Systems and methods for interacting with multiple display devices

Abstract

The method is implemented in a first computer system having a first display generating component in communication with a second computer system having a second display generating component. The method includes, in response to an input, moving a representation of the content across a first display area provided by the first display generating component to a second display area provided by the second display generating component when the display generating components are communicating in the first mode or the second mode. The method includes moving an application window across the first display area to the second display area provided when the display generating components are communicating in the first mode, and preventing moving across the first display area to the second display area when the display generating components are communicating in the second mode.

Description

Technical Field

[0001] (Related Applications) This application is a continuation of U.S. patent application Ser. No. 17 / 831,377, filed Jun. 2, 2022, which claims the benefit of U.S. Provisional Patent Application No. 63 / 197,248, filed Jun. 4, 2021, and U.S. Provisional Patent Application No. 63 / 252,114, filed Oct. 4, 2021. (Technical Field)

[0002] The disclosed embodiments relate to interacting with two or more electronic devices (e.g., a first electronic device and a second electronic device) using one or more input devices associated with only one of the two or more electronic devices. For example, the two or more electronic devices are operated in different modes including a shared input mode (e.g., a first user interface generated by an operating system of the first electronic device and a second user interface generated by an operating system of the second electronic device are controlled by a shared input device) and a companion display mode (e.g., a first user interface generated by an operating system of the first electronic device extends across both the display of the first electronic device and the display of the second electronic device).

Background Art

[0003] Users of computing devices often use multiple computer displays to visually present content simultaneously. By using multiple connected displays, users can open more user interfaces and expand those user interfaces across a larger screen area, often improving user efficiency. However, connecting multiple computer displays together typically requires expertise in the computer's operating system or other display software, requires reconfiguration when the computing device or display is moved, and in other cases can be a difficult, cumbersome, and wasteful process. Also, some computing devices or displays are not even compatible with each other and cannot be connected together to display content simultaneously. Thus, there is a need for systems and methods that enable users to more easily connect and operate multiple computer displays simultaneously.

[0004] Furthermore, the human-machine interfaces for devices operating with multiple displays are typically non-intuitive and cannot provide visual feedback when performing different functions. Thus, there is also a need for a more intuitive human-machine interface, particularly one that enables multiple displays to be used more easily and efficiently. SUMMARY OF THE INVENTION

[0005] The embodiments described herein address the above drawbacks by providing a display device and method that enable a user to intuitively and efficiently connect and operate the displays of multiple display devices (e.g., desktop electronic devices, laptop electronic devices, or tablet electronic devices) that are communicating with each other to share content among the multiple display devices. Such a device and method establish communication between different devices, switch between different display modes (e.g., shared input mode and companion display mode), and share content between different displays, thus requiring little input. Such a display device and method also provide feedback to assist a user in operating different display devices in different display modes. Such a display device and method also provide an improved human-machine interface by, for example, emphasizing information to make it more recognizable on a touch-sensitive display and by requiring less interaction from the user to achieve a desired result from the user. For these reasons and those described below, the devices and methods described herein reduce the power usage of electronic devices and improve battery life.

[0006] According to some embodiments, the method is implemented in a first computer system having a first display generation component. The first computer system communicates with a first input device and a second computer system having a second display generation component. The method includes displaying a first user interface object in a first display area via the first display generation component. While displaying the first user interface in the first display area provided by the first display generation component, a first input including a first movement is detected via the first input device. The input corresponds to a request to drag the first user interface object from the first display area across the first display area to a second display area provided by the second display generation component. In response to detecting the first movement, according to a determination that the first user interface object is a representation of content, when the second display generation component is communicating with the first computer system in a first mode (e.g., extended display mode) or a second mode (e.g., shared input mode), the first user interface object is moved from the first display area to the second display area. According to a determination that the first user interface object is an application window and the first input is detected while the second display generation component is communicating with the first computer system in a first mode (e.g., extended display mode), the first user interface object is moved from the first display area to the second display area. According to a determination that the first user interface object is an application window and the first input is detected while the second display generation component is communicating with the first computer system in a second mode (e.g., shared input mode), the movement of the first user interface object into the second display area provided by the second display generation component is prevented.In some embodiments, in the shared input mode, the first computer system displays a user interface controlled by the first computer system, the second computer system displays a user interface controlled by the second computer system, and the first computer and the second computer system share an input device. In some embodiments, in the extended display mode, both the first display generation component and the second display generation component display a user interface generated by the first computer system, and the first computer and the second computer system receive input via a shared input device.

[0007] According to some embodiments, the method is implemented in a first computer system having a first display generation component. The first computer system communicates with a first input device. The method includes detecting a first event that meets a first criterion while displaying a first user interface within a first display area provided by the first display generation component. In response to detecting a first event that meets the first criterion, an individual visual indication of an individual representative spatial position of a user interface generated by a second display generation component of a second computer system is displayed. Displaying the individual visual indication includes the first computer system sharing the second display generation component of the second computer system and the first input device, and the user interface generated by the second display generation component having a first representative spatial position that can be reached by dragging a user interface object through a first portion of the user interface generated by the first display generation component. Displaying a first visual indication in the first portion of the user interface generated by the first display generation component according to a determination. The first computer system shares the second computer system with which the second display generation component communicates and the first input device, and the user interface generated by the second display generation component has a second representative spatial position that can be reached by dragging a user interface object through a second portion of the user interface generated by the first display generation component. Displaying a second visual indication in the second portion of the user interface generated by the first display generation component according to a determination. While the first computer system is sharing the second computer system and the first input device, a first input via the first input device corresponding to a request to drag a first user interface object across the first display area is detected.In response to detecting a first input, move a first user interface object across a user interface generated by a first display generation component, and when a first event that meets a first criterion is detected while the user interface generated by a second display generation component has an individual representative spatial position represented by an individual visual indication, move the first user interface object across a portion of the first user interface where the individual visual indication was displayed according to a determination that the first input included a movement across the portion of the first user interface where the individual visual indication was displayed, move the first user interface object across the user interface generated by the second display generation component according to the first movement detected via the first input device.

[0008] According to some embodiments, the method is implemented in a first computer system having a first display generation component, the first computer system communicating with a first input device and a second computer system having a second display generation component. The method includes displaying, via the first display generation component, on a first display area, a composited user interface including a first representation of a representative spatial position of a user interface generated by the first display generation component and a second representation of a representative spatial position of a user interface generated by the second display generation component. In accordance with a determination that the first computer system and the second computer system are connected in a first mode (e.g., a shared input mode), the method includes displaying the first representation of the representative spatial position of the user interface generated by the first display generation component and the second representation of the representative spatial position of the user interface generated by the second display generation component using first visual characteristics. In accordance with a determination that the first computer system and the second computer system are connected in a second mode, the first representation of the representative spatial position of the user interface generated by the first display generation component and the second representation of the representative spatial position of the user interface generated by the second display generation component are displayed using second visual characteristics different from the first visual characteristics.

[0009] According to some embodiments, the method is implemented in a first computer system comprising a first display generation component and an input device. The method includes displaying a visual indication of the location of a portal between a first display area associated with the first display generation component and a second display area associated with a second display generation component via the first display generation component. The portal is a portion of the first display area that can move a user interface object between the first display area and the second display area. While displaying the visual indication of the portal, the method includes detecting a first input corresponding to a first movement within the first display area. In response to detecting the first input, the size and / or location of the visual indication of the portal between the first display area associated with the first display generation component and the second display area associated with the second display generation component is changed to indicate a change to the size and / or location of the portal between the first display area associated with the first display generation component and the second display area associated with the second display generation component.

[0010] According to some embodiments, the method is implemented in a first computer system having a display generation component and one or more input devices. The method includes simultaneously displaying, in a display area provided by the display generation component, a dock including a plurality of icons respectively corresponding to a plurality of applications, and a first area and a second area displayed in a split-screen configuration, where the first area displays a first user interface of a first application, and the second area displays a placeholder interface indicating that the second area is available for placement of a user interface of an application other than the first application. The method further includes detecting a first user input corresponding to a selection of an icon in the dock corresponding to a second application while simultaneously displaying the dock, the first user interface of the first application in the first area, and the placeholder interface in the second area, and in response to detecting the first user input, displaying a second user interface of the second application in the second area, where the second user interface of the second application is displayed together with the first user interface of the first application in the split-screen configuration.

[0011] According to some embodiments, a computer system includes a first display generation component, one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above. According to some embodiments, a computer-readable storage medium stores one or more programs, where the one or more programs include instructions for causing a computer system having a first display generation component to perform any of the methods described above when executed by the computer system.

[0012] The systems and methods described herein improve a way of efficiently operating a device having multiple displays simultaneously.

Brief Description of the Drawings

[0013] To better understand the various embodiments described, the following "Modes for Carrying Out the Invention" should be referred to in conjunction with the following drawings, and like reference numerals refer to corresponding parts throughout the following figures.

[0014]

Figure 1A

Figure 1B

[0015]

Figure 2

[0016]

Figure 3A

[0017]

Figure 3B

[0018]

Figure 4A

[0019]

Figure 4B

[0020]

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 5I

Figure 5J

Figure 5K

Figure 5L

Figure 5M

Figure 5N

Figure 5O

Figure 5P

Figure 5Q

Figure 5R

Figure 5S

Figure 5T

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

Figure 6I

Figure 6J

Figure 6K

Figure 6L

Figure 6M

Figure 6N

Figure 6O

Figure 6P

Figure 6Q

Figure 6R

Figure 6S

Figure 6T

Figure 6U

Figure 6V

Figure 6W

Figure 6X

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 7J

Figure 7K

Figure 7L

Figure 7M

Figure 7N

Figure 7O

Figure 7P

Figure 7Q

Figure 7R

Figure 7S

Figure 7T

Figure 7U

Figure 7V

Figure 7W

Figure 7X

Figure 7Y

Figure 7Z

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 8G

Figure 8H

Figure 8I

Figure 8J

Figure 8K

Figure 8L

Figure 8M

Figure 8N

Figure 8O

Figure 8P

Figure 8Q

Figure 8R

Figure 8S

Figure 8T

Figure 8U

Figure 8V

Figure 8W

Figure 8X

Figure 8Y

Figure 8Z

Figure 8AA

Figure 8AB

Figure 8AC

Figure 8AD

Figure 8AE

Figure 8AF

Figure 8AG

Figure 8AH

Figure 8AI

[0021]

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 11A

Figure 11B

Figure 11C

Figure 11D

Figure 11E

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 12F

[0022]

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 13F

Figure 13G

Figure 13H

Figure 13I

Figure 13J

Figure 13K

Figure 13L

Figure 13M

Figure 13N

Figure 13O

Figure 13P

Figure 13Q

Figure 13R

Figure 13S

Figure 13T

Figure 13U

Figure 13V

Figure 13W

Figure 13X

Figure 13Y

Figure 13Z

Figure 13AA

Figure 13AB

Figure 13AC

Figure 13AD

Figure 13AE

Figure 13AF

Figure 13AG

Figure 13AH

Figure 13AI

Figure 13AJ

Figure 13AK

Figure 13AL

Figure 13AM

Figure 13AN

Figure 13AO

[0023]

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 14G

Best Mode for Carrying Out the Invention

[0024] Figures 1A - 4B illustrate exemplary devices in which the methods described herein are implemented and carried out. Figures 5A - 8AI are schematic diagrams of displays used to illustrate exemplary user interfaces for simultaneously launching and interacting with multiple display devices in a shared input mode or a companion display mode (e.g., an extended display mode or a mirroring display mode), and additional explanations regarding the figures of these user interfaces are also provided with reference to methods 9000, 10000, 11000, and 12000 of Figures 9A - 12F. Figures 13A - 13AO are schematic diagrams of display devices used to illustrate exemplary user interfaces for simultaneously displaying and interacting with multiple applications, and additional explanations regarding the figures of these user interfaces are also provided with reference to method 14000 of Figures 14A - 14G. Exemplary Devices and Systems

[0025] Reference is now made in detail to the embodiments shown in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well - known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0026] In this specification, terms such as first, second, etc. are used in some embodiments to describe various elements, but it will also be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first contact can be referred to as the second contact, and similarly, the second contact can be referred to as the first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0027] The terms used in the description of the various embodiments described herein are for the sole purpose of describing a particular embodiment and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a", "an", and "the" are intended to include the plural as well, unless the context clearly indicates otherwise. Also, as used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It should be understood that the terms "includes", "including", "comprises", and / or "comprising", when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0028] As used herein, the term "if (when)" is optionally construed to mean "when", "upon", "in response to determining", or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (a stated condition or event) is detected" are optionally construed to mean "upon determining" or "in response to determining", or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0029] FIG. 1A shows an exemplary system in which a first display device (e.g., the illustrated laptop display device 300) operates with a second display device (e.g., the illustrated tablet display device or desktop computer display device 100). FIG. 1B shows an exemplary system in which a first display device (e.g., the illustrated desktop display device 300) operates in relation to a second display device (e.g., the illustrated tablet display device 100) and a third display device (e.g., the illustrated laptop device 200). Devices 100, 200, and 300 are all display devices that include individual display devices 101, 201, and 301 (also referred to as display generation components). In some embodiments, the display is a touch-sensitive display (e.g., the display 101 of the tablet device 100 is a touch-sensitive display or touch screen). The first display device includes or communicates with one or more input devices (e.g., the mouse input device 202, keyboard input devices 203 and 305, and touch pad 309 shown in FIG. 1B). In some embodiments, the input devices are mounted on the device (e.g., the touch pad 309 and keyboard 305 are part of the laptop device 300). In some embodiments, the input devices communicate wirelessly or wiredly with the device (e.g., the mouse 202 and keyboard 203 communicate wirelessly with the desktop device 200 in FIG. 1B). In some embodiments, the first display device communicates with the second and / or third display devices in a shared input device mode. In the shared input device mode, the first display device shares one or more input devices (e.g., the illustrated mouse input device and / or keyboard input device) with the second display device and / or third display device such that the one or more input devices can be used to operate the second display device or the third display device.In some embodiments, the first electronic device detects an input via one or more input devices with which it is in wireless or wired communication and provides information regarding the detected input to a second computer system and / or a third computer system. In some embodiments, the first computer system as well as the second and / or third computer systems all communicate with the same one or more input devices and detect an input via the one or more input devices. For example, the detected input is processed by the currently active computer system (e.g., the input is directed to the keyboard, mouse, or touchpad of the currently active computer system). In some embodiments, the computer system is currently active if it is displaying a cursor (e.g., in a shared input mode, different computers have a common cursor). Alternatively, the first display device may communicate with the second and / or third display devices in a companion display mode. In the companion display mode, the individual displays of the second display device or the third display device display the content provided by the first display device. For example, the individual displays of the second display device or the third display device operate as a mirror display or an extended display for the display of the first display device. Additional details regarding the shared input mode and the companion display mode are provided below.

[0030] Note that various references are made to the first, second, and third display devices. In certain instances, the first, second, and third display devices can be selected from any type of display device, i.e., an electronic device having a respective display (e.g., a mobile phone, a tablet, a laptop, a wearable, or a desktop display device). Also, the references to tablet, laptop, desktop, wearable, and mobile phone display devices are merely exemplary. The description herein regarding tablet display devices also applies to other portable display devices that execute a mobile operating system (e.g., a smartphone such as an IPHONE of APPLE INC. (Cupertino, CA) that executes the IOS operating system), and the description herein regarding laptop display devices also applies to other desktop-like devices that execute a desktop / laptop operating system.

[0031] Block diagrams showing various components of the first and second electronic devices are shown in FIGS. 2 and 3A - 3B.

[0032] Next, attention is directed to an embodiment of a portable electronic device having a touch-sensitive display. FIG. 2 is a block diagram showing a portable multifunctional device 100 (also referred to interchangeably herein as electronic device 100 or device 100) having a touch-sensitive display 112 according to some embodiments. The touch-sensitive display 112 may be referred to herein as a "touch screen" for convenience, and may also be known or referred to as a touch-sensitive display system. Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a controller 120, one or more processing units (CPUs) 122, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on device 100 (e.g., on a touch-sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating a haptic output on device 100 (e.g., on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or a touch pad of device 100). These components communicate, optionally, via one or more communication buses or signal lines 103.

[0033] Device 100 is merely an example of a portable multifunctional device, and it is to be understood that device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have a different configuration or arrangement of those components. The various components shown in FIG. 1 are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0034] Memory 102 optionally includes a high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, or other random access solid state memory devices), and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 102 optionally includes one or more storage devices that are remotely located from the processor(s) 122. Access to Memory 102 by other components of Device 100, such as CPU 122 and Peripheral Interface 118, is optionally controlled by Controller 120.

[0035] Peripheral Interface 118 can be used to couple the input and output peripheral devices of this device to CPU 122 and Memory 102. One or more processors 122 operate or execute various software programs and / or instruction sets stored in Memory 102 to perform various functions for Device 100 and process data.

[0036] In some embodiments, Peripheral Interface 118, the processor(s) or CPU(s) 122, and Controller 120 are optionally implemented on a single chip such as Chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0037] The RF (radio frequency) circuit 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals or vice versa and communicates with a communication network and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuit 108 optionally communicates wirelessly with a network such as the Internet, also called the World Wide Web (WWW), an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN), and with other devices. The wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies including, but not limited to, Global System for Mobile Communications (GSM) (registered trademark), Enhanced Data GSM Environment (EDGE) (registered trademark), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth (registered trademark), Wireless Fidelity (Wi-Fi) (registered trademark) (e.g., IEEE802.11a, IEEE802.11b, IEEE802.11g, and / or IEEE802.11n).

[0038] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. Also, the audio circuit 110 receives the electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 further includes a headset jack. The headset jack provides an interface between the audio circuit 110 and a detachable audio input / output peripheral device such as an output-only headset or a headset having both an output (e.g., mono or stereo headphones) and an input (e.g., a microphone).

[0039] The I / O subsystem 106 connects the input / output peripheral devices of the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from other input or control devices 116 and transmit electrical signals to them. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to or not coupled to any of pointer devices such as a keyboard, an infrared port, a USB port, and a mouse. The one or more buttons optionally include up / down buttons for volume adjustment of the speaker 111 and / or the microphone 113. The one or more buttons optionally include push buttons.

[0040] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or transmits electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0041] The touch screen 112 has a touch sensing surface, sensor, or set of sensors that receive input from a user based on tactile and / or haptic contact. The touch screen 112 and the display controller 156 (along with any associated modules and / or instruction sets within the memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the area under the user's finger.

[0042] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although in other embodiments, other display technologies are used. The touch screen 112 and the display controller 156 optionally use any of a plurality of touch sensing technologies, now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112, to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology, as seen in, for example, the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) of Apple Inc. (Cupertino, California), is used.

[0043] The touch screen 112 optionally has a video resolution exceeding 400 dpi. In some embodiments, the touch screen 112 has a video resolution of at least 600 dpi. In other embodiments, the touch screen 112 has a video resolution of at least 1000 dpi. The user optionally contacts the touch screen 112 using any suitable object or digit, such as a stylus or a finger. In some embodiments, the user interface is designed to primarily operate with finger-based contact and gestures. In some embodiments, the device converts finger-based input into a command to execute an accurate pointer / cursor position or an action desired by the user.

[0044] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touch pad for activating or deactivating certain functions. In some embodiments, the touch pad, unlike the touch screen, is a touch-sensing area of the device that does not display visual output. The touch pad is optionally a touch-sensing surface separate from the touch screen 112 or an extension of the touch-sensing surface formed by the touch screen.

[0045] The device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indication (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0046] Device 100 also optionally includes one or more optical sensors 164. FIG. 1 shows an optical sensor coupled to an optical sensor controller 158 within I / O subsystem 106. Optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light from the environment projected through one or more lenses and converts that light into data representing an image. Optical sensor 164 optionally captures a still image or video in cooperation with imaging module 143 (also referred to as a camera module). In some embodiments, the optical sensor is disposed on the back of device 100, opposite touch screen 112 on the front of the device, so as to enable the touch-sensitive display to be activated as a viewfinder for obtaining still and / or video images. In some embodiments, another optical sensor is disposed on the front of the device so as to optionally obtain an image of the user while the user is viewing other video conference participants on the touch-sensitive display for a video conference.

[0047] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1 shows a contact intensity sensor coupled to an intensity sensor controller 159 within I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with, or in proximity to, a touch sensing surface (e.g., touch sensing display system 112). In some embodiments, at least one contact intensity sensor is disposed on the back of device 100, on the opposite side of touch screen 112 disposed on the front of device 100.

[0048] Also, device 100 optionally includes one or more proximity sensors 166. FIG. 1 shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 within I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch screen 112 when a multifunction device (e.g., when the user is on a call) is placed near the user's ear.

[0049] Device 100 also optionally includes one or more haptic output generators 167. FIG. 1 shows a haptic output generator coupled to a haptic feedback controller 161 within I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output) into linear movement. The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is disposed on or proximate to a touch sensing surface (e.g., touch sensing display system 112) and optionally generates a haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of device 100) or a lateral direction (e.g., back / forth within the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is disposed on the back of device 100, which is opposite the touch sensing display 112 disposed on the front of device 100.

[0050] Also, device 100 optionally includes one or more accelerometers 168. FIG. 1 shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 within the I / O subsystem 106. In some embodiments, information is displayed on a touch sensing display in a portrait or landscape orientation based on analysis of data received from one or more accelerometers. In addition to the accelerometer(s) 168, device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for obtaining information regarding the location and orientation (e.g., portrait or landscape) of device 100.

[0051] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Further, in some embodiments, the memory 102 stores a device / global internal state 157, as shown in FIG. 1. The device / global internal state 157 includes, if any, an active application state indicating which application is currently active, a display state indicating which application, view, or other information occupies various regions of the touch-sensitive display 112, a sensor state including information obtained from various sensors and input control devices 116 of the device, and one or more of location information regarding the location and / or orientation (i.e., the orientation of the device) of the device.

[0052] The operating system 126 (e.g., an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitate communication between various hardware components and software components.

[0053] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) are adapted to couple to other devices either directly or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is the same as, similar to, and / or compatible with a multi-pin (e.g., 30-pin) connector such as that used in some embodiments of APPLE Inc.'s IPOD devices. In other embodiments, the external port is the same as, similar to, and / or compatible with a multi-pin (e.g., 8-pin) connector such as that used in APPLE Inc.'s Lightning connector.

[0054] The contact / motion module 130 optionally detects contact with the touch screen 112 (in cooperation with the display controller 156) and other touch sensing devices (e.g., a touch pad or a physical click wheel). The contact / motion module 130 performs various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch sensing surface (e.g., detecting one or more events of dragging a finger), and determining whether the contact has ceased (e.g., detecting a finger-up event or an interruption of the contact), and includes various software components for performing these operations. The contact / motion module 130 receives contact data from the touch sensing surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact with one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact with multiple fingers). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0055] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by the user (e.g., to determine whether the user has selected or “clicked” on an affordance). In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds can be adjusted without changing the physical hardware of the device 100, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse “click” threshold of a trackpad or touch-sensitive display can be set to any of a wide range of predetermined thresholds without changing the hardware of the trackpad or touch-sensitive display. Further, in some implementations, the user of the device is provided with a software setting to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once by a system-level click “intensity” parameter).

[0056] The contact / motion module 130 optionally detects gesture inputs from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., the detected movement, timing, and / or intensity of the contact is different). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event followed by a finger up (lift off) event at the same position (or substantially the same position) as the finger down event (e.g., the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger down event followed by one or more finger drag events and, in some embodiments, subsequently detecting a finger up (lift off) event.

[0057] The graphic module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components that change the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphic" includes, without limitation, any object that can be displayed to the user, including text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0058] In some embodiments, the graphic module 132 stores data representing the graphics that will be used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives, from an application or the like, one or more codes that specify the graphics to be displayed, along with coordinate data and other graphic characteristic data as needed, and then generates the image data of the screen to be output to the display controller 156.

[0059] The tactile feedback module 133 includes various software components for generating the instructions used by the tactile output generator(s) 167, and generates tactile output at one or more locations on the device 100 in response to the user's interaction with the device 100.

[0060] The text input module 134 is optionally a component of the graphic module 132 and provides a soft keyboard for entering text in various applications (e.g., the contact module 137, the email client module 140, the IM module 141, the browser module 147, and any other application that requires text input).

[0061] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based phone calls, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0062] The application ("app") 136 optionally includes the following modules (or sets of instructions), or subsets or supersets thereof. ● Contact module 137 (sometimes referred to as an address book or contact list), ● Phone module 138, ● Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ● Fitness module 142, ● Camera module 143 for still images and / or videos, ● Image management module 144, ● Browser module 147, ● Calendar module 148, ● Optionally, a widget module 149 that includes one or more of weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widget 149-6, ● Search module 151, ● Video and music player module 152, optionally composed of a video player module and a music player module, ● Memo module 153, ● Map module 154, and / or ● Online video module 155.

[0063] Examples of other applications 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk authoring applications, spreadsheet applications, JAVA (registered trademark)-compatible applications, encryption, digital rights management, voice recognition, a widget creator module for creating user-created widgets 149-6, and voice replication.

[0064] In cooperation with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the contact module 137 in the memory 102 or the memory 302), which includes adding a name(s) to the address book, deleting a name(s) from the address book, associating a phone number(s), email address(es), address(es), or other information with a name, associating an image with a name, classifying and sorting names, providing a phone number or email address to initiate and / or facilitate communication by the phone module 138, the video conferencing module 139, the email client module 140, or the IM module 141, and so on.

[0065] In cooperation with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the telephone module 138 is optionally used to input a series of characters corresponding to a telephone number, access one or more telephone numbers in the address book 137, modify an input telephone number, dial each telephone number, conduct a conversation, and disconnect the connection or hang up the phone when the conversation is completed. As described above, the wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.

[0066] Together with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact module 130, the graphic module 132, the text input module 134, the contact list 137, and the telephone module 138, the videoconference module 139 includes executable instructions for starting, conducting, and ending a videoconference between the user and one or more other participants according to the user's command.

[0067] In cooperation with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the email client module 140 includes executable instructions for creating, sending, receiving, and managing emails according to the user's command. In cooperation with the image management module 144, the email client module 140 makes it very easy to create and send emails with still or moving images captured by the camera module 143.

[0068] Together with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, and text input module 134, instant messaging module 141 includes executable instructions for inputting a string corresponding to an instant message, modifying an input character, sending each instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), receiving an instant message, and displaying the received instant message. In some embodiments, the instant messages sent and / or received optionally include graphics, photos, audio files, video files, and / or other attached files, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0069] In cooperation with the RF circuit 108, touch screen 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and video and music player module 146, the fitness module 142 includes executable instructions for creating workouts (e.g., with respect to time, distance, and / or calorie consumption goals), communicating with a training sensor (such as a sports device like a wristwatch or pedometer), receiving training sensor data, calibrating sensors used to monitor a workout, selecting and playing music for a workout, and displaying, storing, and transmitting workout data.

[0070] In cooperation with the touch screen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphic module 132, and image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0071] In cooperation with the touch screen 112, display controller 156, contact module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise operating on still images and / or video images, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing them.

[0072] In cooperation with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attached files and other files linked to the web pages.

[0073] In cooperation with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar items, to-do lists, etc.) according to user instructions.

[0074] In cooperation with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, the text input module 134, and the browser module 147, the widget module 149 is a mini-application (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) that is optionally downloaded and used by the user, or a mini-application (e.g., user-created widget 149-6) created by the user. In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0075] In cooperation with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, the text input module 134, and the browser module 147, a widget creator module (not shown) is optionally used by the user to create a widget (e.g., change a user-specified location on a web page to a widget).

[0076] In cooperation with the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, and the text input module 134, the search module 151 includes executable instructions for searching for text, music, sounds, images, videos, and / or other files in the memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to the user's instructions.

[0077] In cooperation with the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, the audio circuit 110, the speaker 111, the RF circuit 108, and the browser module 147, the video and music player module 152 includes executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back videos (e.g., on the touch screen 112 or on an external display connected via the external port 124). In some embodiments, the device 100 optionally has the functionality of an MP3 player such as the IPOD (registered trademark) of APPLE Inc.

[0078] In cooperation with the touch screen 112, the display controller 156, the touch module 130, the graphic module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do lists, etc. according to the user's commands.

[0079] In cooperation with the RF circuit 108, the touch screen 112, the display system controller 156, the touch module 130, the graphic module 132, the text input module 134, the GPS module 135, and the browser module 147, the map module 154 may optionally be used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data) according to the user's commands.

[0080] Together with touch screen 112, display system controller 156, touch module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen or on an external display connected via external port 124), send an email including a link to a particular online video, and otherwise manage one or more file formats of online video such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0081] As shown in FIG. 2, portable multifunctional device 100 also includes companion display module 180 for managing operations associated with companion display mode multitasking on device 100. Companion display module 180 optionally includes the following modules (or set of instructions), or a subset or superset thereof. ● Layout module 182 for determining the layout of displays for laptop and tablet devices adjacent to each other in connection with the companion display mode described herein, ● UI generator module 184 for generating user interfaces and sharing data associated with those user interfaces between different devices along with the companion display and annotation mode, and ● Secure criteria module 186 for monitoring whether a device meets a set of secure connection criteria used to determine when the companion display mode is available for use between different devices (e.g., laptop and tablet devices).

[0082] In cooperation with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the contact intensity sensor(s) 165, the PIP module 186 determines a reduced size of video content and determines an appropriate location on the touch screen 112 for displaying the reduced-size video content (e.g., a location that avoids important content within an active application overlaid by the reduced-size video content), and includes executable instructions for doing so.

[0083] Each of the above-specified modules and applications corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., methods executed by a computer and other information processing methods described herein). Those modules (i.e., the sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of those modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, the memory 102 optionally stores a subset of the modules and data structures identified above. Further, the memory 102 optionally stores additional modules and data structures not described above.

[0084] FIG. 3A is a block diagram of an electronic device 300 according to some embodiments. In some embodiments, the electronic device 300 is a laptop or desktop computer running a desktop operating system different from a mobile operating system.

[0085] The electronic device 300 generally supports various applications such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a video conferencing application, an email application, an instant messaging application, an image management application, a digital camera application, a digital video camera application, a web browser application, and / or a media player application.

[0086] The various applications executed on the electronic device 300 optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed by the electronic device 300, are optionally adjusted and / or changed for each application and / or within an application. Thus, the common physical architecture (such as a touch-sensitive surface) of the electronic device 300 optionally supports various applications having a user interface that is intuitive and transparent to the user.

[0087] The electronic device 300 includes a memory 302 (optionally including one or more computer-readable storage media), a memory controller 322, one or more processing units (CPUs) 320, a peripheral device interface 318, an RF circuit 308, an audio circuit 310, a speaker 311, a microphone 313, an input / output (I / O) subsystem 306, other input or control devices 316, and an external port 324. The electronic device 300 optionally includes a display system 312 which can be a touch-sensitive display (sometimes referred to herein as a "touch screen" or "touch screen display"). The electronic device 300 optionally includes one or more optical sensors 364. The electronic device 300 optionally includes one or more intensity sensors 365 that detect the intensity of contact on a touch-sensitive surface such as a touch-sensitive display or a touch pad. The electronic device 300 optionally includes one or more haptic output generators 367 for generating haptic output on a touch-sensitive surface such as a touch-sensitive display or a touch pad. These components optionally communicate via one or more communication buses or signal lines 303.

[0088] As used herein, the term "intensity" of a contact on a touch-sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensing surface, or a proxy for the force or pressure of the contact on the touch-sensing surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically includes hundreds (e.g., at least 256) of distinct values. The intensity of the contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine the estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensing surface. Alternatively, the size and / or change thereof of the contact area detected on the touch-sensing surface, the capacitance and / or change thereof of the touch-sensing surface proximate to the contact, and / or the resistance and / or change thereof of the touch-sensing surface proximate to the contact are optionally used as a proxy for the force or pressure of the contact on the touch-sensing surface. In some implementations, an alternative measurement of the force or pressure of the contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, an alternative measurement for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).

[0089] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensing surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device that will be detected by the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive surface of the user (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, movement of a touch-sensing surface (e.g., a touch-sensing display or a touch / track pad) may optionally be interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user may feel a tactile sensation such as a "down click" or "up click" even when there is no movement of the physical actuator button associated with the touch-sensing surface that has been physically pressed (e.g., displaced) by the user's action. As another example, movement of the touch-sensing surface may optionally be interpreted or perceived by the user as "roughness" of the touch-sensing surface, even if there is no change in the smoothness of the touch-sensing surface. Such interpretation of touch by the user depends on the user's individual sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise specified, the generated haptic output corresponds to a physical displacement of the device or a component of the device that produces the described sensory perception of a typical (or average) user.

[0090] The electronic device 300 is merely an example, and it should be understood that the electronic device 300 may optionally have more or fewer components than those shown, or may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in FIG. 3A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0091] The memory 302 may optionally include a high-speed random access memory and may also optionally include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to the memory 302 by other components of the electronic device 300, such as the CPU(s) 320 and the peripheral interface 318, may optionally be controlled by a memory controller 322. The peripheral interface 318 may be used to couple input and output peripheral devices to the CPU(s) 320 and the memory 302. One or more processing units 320 perform various functions for the electronic device 300 and operate or execute various software programs and / or instruction sets stored in the memory 302 to process data. In some embodiments, the peripheral interface 318, the CPU(s) 320, and the memory controller 322 may optionally be implemented on a single chip such as the chip 305. In some other embodiments, they may optionally be implemented on separate chips.

[0092] The RF (radio frequency) circuit 308 transmits and receives RF signals, also referred to as electromagnetic signals. The RF circuit 308 converts electrical signals to / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. The RF circuit 308 optionally includes well-known circuits for performing these functions, and these circuits include, but are not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuit 308 optionally communicates wirelessly with a network such as the Internet, also referred to as the World Wide Web, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN), and with other devices. The wireless communication optionally uses any of a plurality of communication standards, communication protocols, and communication technologies, and these communication standards, communication protocols, and communication technologies include the Global System for Mobile Communications (GSM (registered trademark)) for mobile communications, the Enhanced Data GSM Environment (EDGE (registered trademark)), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE (registered trademark)), near field communication (NFC), wideband code division multiple access (WCDMACode division multiple access, W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), WiMAX (registered trademark), protocols for e-mail (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the filing date of the present document, but not limited thereto.

[0093] The audio circuit 310, the speaker 311, and the microphone 313 provide an audio interface between the user and the electronic device 300. The audio circuit 310 receives audio data from the peripheral device interface 318, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 311. The speaker 311 converts the electrical signal into human audible sound waves. Also, the audio circuit 310 receives the electrical signal converted from sound waves by the microphone 313. The audio circuit 310 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 318 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 302 and / or the RF circuit 308 by the peripheral device interface 318. In some embodiments, the audio circuit 310 further includes a headset jack. The headset jack provides an interface between the audio circuit 310 and a detachable audio input / output peripheral device such as an output-only headset or a headset with both output (e.g., mono or stereo headphones) and input (e.g., microphone).

[0094] The I / O subsystem 306 couples input / output peripheral devices of the electronic device 300, such as a display system 312 and other input or output devices 316, to a peripheral device interface 318. The I / O subsystem 306 optionally includes a display controller 356, an optical sensor controller 358, an intensity sensor controller 359, a haptic feedback controller 361, and one or more other input controllers 360 for other input or control devices. The one or more other input controllers 360 receive electrical signals from and transmit electrical signals to other input or control devices 316. The other input or control devices 316 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the other input controller(s) 360 are optionally coupled to (or not coupled to any of) a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more physical buttons optionally include up / down buttons for volume control of the speaker 311 and / or the microphone 313.

[0095] The display system 312 provides an output interface (and optionally an input interface if it is a touch-sensitive display) between the electronic device 300 and the user. The display controller 356 receives electrical signals from and / or transmits electrical signals to the display system 312. The display system 312 displays visual output to the user. This visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects / elements.

[0096] In some embodiments, the display system 312 is a touch-sensitive display having a touch-sensitive surface, sensors, or a set of sensors that receive input from a user based on tactile and / or haptic contact. Accordingly, the display system 312 and the display controller 356 (along with any associated modules and / or instruction sets in the memory 302) detect contact (and any movement or interruption of the contact) on the display system 312 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the display system 312. In an exemplary embodiment, the point of contact between the display system 312 and the user corresponds to the area under the user's finger.

[0097] The display system 312 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, but in other embodiments, other display technologies are used. In some embodiments, when the display system 312 is a touch-sensitive display, the display system 312 and the display controller 356 optionally use any of a plurality of touch-sensing technologies currently known or later developed to detect contact and any movement or interruption of that contact, and those touch-sensing technologies include capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements that determine one or more points of contact with the display system 312, but are not limited thereto. In an exemplary embodiment, projected mutual capacitance sensing technology, as found in the IPHONE (registered trademark), IPOD TOUCH (registered trademark), and IPAD (registered trademark) from Apple Inc. of Cupertino, California, is used.

[0098] The display system 312 optionally has a video resolution greater than 400 dpi (e.g., 500 dpi, 800 dpi, or more). In some embodiments, the display system 312 is a touch-sensitive display that the user can optionally touch using a stylus, finger, etc. In some embodiments, the user interface is designed to primarily operate based on finger-based contacts and gestures. In some embodiments, the electronic device 300 converts rough input by a finger into a command to execute an accurate pointer / cursor position or an action desired by the user.

[0099] In some embodiments, in addition to the display system 312, the electronic device 300 optionally includes a touch pad for activating or deactivating certain functions. In some embodiments, the touch pad is a touch-sensitive area of the electronic device 300 that does not display a visual output, unlike the display system 312. In some embodiments, when the display system 312 is a touch-sensitive display, the touch pad is optionally a touch-sensitive surface separate from the display system 312 or an extension of the touch-sensitive surface formed by the display system 312.

[0100] The electronic device 300 also includes a power system 362 that supplies power to various components. The power system 362 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC), etc.), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indication (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0101] The electronic device 300 also optionally includes one or more optical sensors 364 coupled to an optical sensor controller 358 within the I / O subsystem 306. The optical sensor(s) 364 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor(s) 364 receive light from the environment projected through one or more lenses and convert the light into data representative of an image. In cooperation with the imaging module 343, the optical sensor(s) 364 optionally capture a still image or video. In some embodiments, the optical sensor is disposed on the front of the electronic device 300 so as to optionally obtain an image of the user for a video conference while the user is viewing other video conference participants on the display system 312.

[0102] The electronic device 300 also optionally includes one or more contact intensity sensors 365 coupled to an intensity sensor controller 359 within the I / O subsystem 306. The contact intensity sensor(s) 365 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrokinetic sensors, piezoelectric sensors, optomechanical sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor(s) 365 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to the touch sensing surface.

[0103] The electronic device 300 also optionally includes one or more haptic output generators 367 coupled to a haptic feedback controller 361 within the I / O subsystem 306. The haptic output generator(s) 367 optionally include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output) on the device. The contact intensity sensor(s) 365 receive haptic feedback generation instructions from the haptic feedback module 333 and generate haptic outputs that can be sensed by a user of the electronic device 300. In some embodiments, at least one haptic output generator is juxtaposed with or proximate to the touch sensing surface and optionally generates a haptic output by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of the electronic device 300) or a lateral direction (e.g., back and forth within the same plane as the surface of the electronic device 300).

[0104] The electronic device 300 also optionally includes one or more proximity sensors 366 coupled to the peripheral device interface 318. Alternatively, the proximity sensor(s) 366 are coupled to other input controller(s) 360 within the I / O subsystem 306. The electronic device 300 also optionally includes one or more accelerometers 368 coupled to the peripheral device interface 318. Alternatively, the accelerometer(s) 368 are coupled to other input controller(s) 360 within the I / O subsystem 306.

[0105] In some embodiments, the software components stored in the memory 302 include an operating system 326, a communication module 328 (or instruction set), a touch / motion module 330 (or instruction set), a graphics module 332 (or instruction set), an application 340 (or instruction set), and a touch bar management module 350 (or instruction set). Further, in some embodiments, as shown in FIG. 3A, the memory 302 stores a device / global internal state 357 (or instruction set). The device / global internal state 357 includes, among other things, an active application state indicating which application, if any, is active and / or in focus when there is an active application, a display state indicating which application, view, or other information occupies various regions of the display system 312 and / or the peripheral display system, a sensor state including information obtained from various sensors and input or control devices 316 of the electronic device 300, and location information regarding the location and / or orientation of the electronic device 300.

[0106] The operating system 326 (e.g., an embedded operating system such as DARWIN®, RTXC®, LINUX®, UNIX®, OS X®, WINDOWS®, or VXWorks®) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware components and software components.

[0107] The communication module 328 facilitates communication with other devices via one or more external ports 324 and / or the RF circuit 308, and includes various software components for transmitting / receiving data via the RF circuit 308 and / or the external port 324. The external port 324 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted to couple to other devices either directly or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port 324 is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used on iPod (registered trademark) devices.

[0108] The contact / motion module 330 optionally detects contact with the display system 312 when it is a touch-sensitive display (in cooperation with the display controller 356) and other touch-sensitive devices (e.g., a touch pad or a physical click wheel). The contact / motion module 330 performs various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch-sensitive surface (e.g., detecting one or more events of dragging a finger), and determining whether the contact has ended (e.g., detecting a finger-up event or an interruption of the contact), and includes various software components for this purpose. The contact / motion module 330 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact with one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact with multiple fingers). In some embodiments, the contact / motion module 330 also detects contact on the touch pad.

[0109] In some embodiments, the contact / motion module 330 uses a set of one or more intensity thresholds for determining whether an action has been performed by the user (e.g., for determining whether the user has selected or "clicked" on an affordance). In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the electronic device 300). For example, the mouse "click" threshold of a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of that trackpad or touch screen display. Further, in some implementations, the user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0110] The contact / motion module 330 optionally detects gesture inputs from the user. Different gestures on the touch-sensing surface have different contact patterns (e.g., the detected movement, timing, and / or intensity of the contact is different). Thus, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap contact includes detecting an event of the finger being lowered and subsequently detecting an event of the finger being raised (lift-off) at the same position (or substantially the same position) as the event of the finger being lowered (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensing surface includes detecting an event of the finger being lowered, subsequently detecting one or more events of the finger being dragged, and in some embodiments, also subsequently detecting an event of the finger being raised (lift-off).

[0111] The graphic module 332 includes various known software components for rendering and displaying graphics on the primary display 301 or other displays, including components for changing the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term "graphic" includes, but is not limited to, any object that can be displayed to the user, such as text, web pages, icons (such as user interface objects including soft keys), digital images, videos, and animations. In some embodiments, the graphic module 332 stores data representing the graphics that will be used. Each graphic is optionally assigned a corresponding code. The graphic module 332 receives from an application or the like, as needed, one or more codes specifying the graphic to be displayed, along with coordinate data and other graphic property data, and then generates the screen image data to be output to the display controller 356.

[0112] The tactile feedback module 333 includes various software components that generate instructions for use by the tactile output generator(s) 367 to generate tactile output at one or more locations on the electronic device 300 in response to user interaction with the electronic device 300.

[0113] The application 340 optionally includes the following modules (or sets of instructions) or subsets or supersets thereof. ● An email client module 341 for receiving, sending, configuring, and viewing emails (sometimes referred to herein as an "email app" or "electronic mail app"), ● An imaging module 342 for capturing still images and / or video images, ● An image management module 343 that edits and views still images and / or video images (sometimes referred to as a "photo app" in this specification). ● A media player module 344 that plays audio and / or video (sometimes referred to as a "media player app" in this specification), and ● A web browsing module 345 that connects to and browses the Internet (sometimes referred to as a "web browser" in this specification).

[0114] Examples of other applications 340 that are optionally stored in the memory 302 include messaging and communication applications, word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption applications, digital rights management applications, voice recognition applications, and voice duplication applications.

[0115] In cooperation with one or more of the RF circuit 308, the display system 312, the display controller 356, the touch module 330, and the graphic module 332, the email client module 341 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. In cooperation with the image management module 343, the email client module 341 makes it very easy to create and send an email with a still image or video image captured by the imaging module 342.

[0116] In cooperation with one or more of display system 312, display controller 356, optical sensor(s) 364, optical sensor controller 358, contact module 330, graphic module 332, and image management module 343, imaging module 342 includes executable instructions for capturing a still image or video (including a video stream), storing them in memory 302, changing the characteristics of the still image or video, or deleting the still image or video from memory 302.

[0117] In cooperation with one or more of display system 312, display controller 356, contact module 330, graphic module 332, and imaging module 342, image management module 343 includes executable instructions for arranging, modifying (e.g., editing), or optionally operating on, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0118] In cooperation with one or more of display system 312, display controller 356, contact module 330, graphic module 332, audio circuit 310, speaker 311, RF circuit 308, and web browsing module 345, media player module 344 includes executable instructions enabling a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, and otherwise playing videos.

[0119] In cooperation with one or more of RF circuit 308, display system 312, display controller 356, contact module 330, and graphic module 332, web browsing module 345 includes executable instructions for browsing the Internet based on user commands including searching for, linking to, receiving, and displaying web pages or portions thereof, and attached files and other files linked to the web page.

[0120] As shown in FIG. 3A, device 300 can also include a companion display module 350 for managing operations associated with companion display mode multitasking on device 100. The companion display module 350 can optionally include the following modules (or, sets of instructions), or subsets or supersets thereof. ● An arrangement module 351 for determining the arrangement of displays for laptop and tablet devices adjacent to each other in relation to the companion display mode described herein, ● A UI generator module 352 for generating user interfaces and sharing data associated with those user interfaces between different devices along with the companion display and annotation modes, and ● A security criteria module 353 for monitoring whether a device meets a set of secure connection criteria used to determine when the companion display mode is available for use between different devices (e.g., laptop and tablet devices).

[0121] Each of the above-identified modules and applications corresponds to a set of executable instructions that perform one or more of the functions described above, as well as the methods described in this application (e.g., methods executed by a computer and other information processing methods described herein). Those modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of those modules can optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 302 optionally stores a subset of the modules and data structures identified above. Further, memory 302 optionally stores additional modules and data structures not described above.

[0122] Figure 3B is a block diagram of components for event processing according to some embodiments. In some embodiments, memory 302 (Figure 3A) includes an event sorter 370 (e.g., within operating system 326) and an application 340-1 (e.g., any one of applications 341, 342, 343, 344, or 345 described above).

[0123] The event sorter 370 receives event information and determines an application 340-1 to which the event information is to be delivered and an application view 391 of the application 340-1. The event sorter 370 includes an event monitor 371 and an event dispatcher module 374. In some embodiments, the application 340-1 includes an application internal state 392 that indicates the current application view(s) displayed on the display system 312 when the application is active or running. In some embodiments, the device / global internal state 357 is used by the event sorter 370 to determine which application(s) is currently active or in focus, and the application internal state 392 is used by the event sorter 370 to determine the application view 391 to which the event information is to be delivered.

[0124] In some embodiments, the application internal state 392 includes additional information such as resume information used when the application 340-1 resumes execution, user interface state information indicating information that is being or is ready to be displayed by the application 340-1, a state queue for enabling the user to return to a previous state or view of the application 340-1, and a redo / undo queue of previous actions performed by the user.

[0125] The event monitor 371 receives event information from the peripheral device interface 318. The event information includes information about sub-events (e.g., a user touch on the display system 312 when it is a touch-sensitive display as part of a multi-touch gesture). The peripheral device interface 318 transmits information received from sensors such as the I / O subsystem 306, or proximity sensor(s) 366, accelerometer(s) 368, and / or microphone 313 (via the audio circuit 310). The information that the peripheral device interface 318 receives from the I / O subsystem 306 includes information from the display system 312 when it is a touch-sensitive display or another touch-sensitive surface.

[0126] In some embodiments, the event monitor 371 transmits requests to the peripheral device interface 318 at predetermined intervals. In response, the peripheral device interface 318 transmits event information. In other embodiments, the peripheral device interface 318 transmits event information only when a significant event (e.g., receiving an input that exceeds a predetermined noise threshold and / or is longer than a predetermined duration) exists.

[0127] In some embodiments, the event sorter 370 also includes a hit view determination module 372 and / or an active event recognition unit determination module 373.

[0128] The hit view determination module 372 provides software procedures for determining where views are configured from where a sub-event occurred within one or more views, when the display system 312 displays two or more views, and control and other elements visible to the user on the display.

[0129] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as an application view or a user interface window, in which information is displayed and touch-based gestures occur. The application view (of the application) in which a touch is detected optionally corresponds to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events recognized as appropriate input is optionally determined based at least in part on the hit view of the initial touch that initiates the touch-based gesture.

[0130] The hit view determination module 372 receives information related to sub-events of a touch-based gesture. When the application has a plurality of hierarchically structured views, the hit view determination module 372 identifies the hit view as the lowest-level view within the hierarchy in which the sub-event is to be processed. In most situations, the hit view is the lowest-level view in which the starting sub-event (i.e., the first sub-event in a series of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0131] The active event recognition unit determination module 373 determines which view(s) within the view hierarchy should receive a particular series of sub-events. In some embodiments, the active event recognition unit determination module 373 determines that only the hit view should receive a particular series of sub-events. In other embodiments, the active event recognition unit determination module 373 determines that all views including the physical location of the sub-events are views actively involved, and thus determines that all views actively involved should receive a particular series of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area associated with one particular view, the upper-level views within the hierarchy remain views actively involved.

[0132] The event dispatcher module 374 dispatches event information to the event recognition unit (e.g., event recognition unit 380). In embodiments including the active event recognition unit determination module 373, the event dispatcher module 374 distributes event information to the event recognition unit determined by the active event recognition unit determination module 373. In some embodiments, the event dispatcher module 374 stores the event information retrieved by the individual event receiver 382 in the event queue.

[0133] In some embodiments, the operating system 326 includes an event sorter 370. Alternatively, the application 340-1 includes the event sorter 370. In still other embodiments, the event sorter 370 is a stand-alone module or part of another module stored in the memory 302 such as the touch / motion module 330.

[0134] In some embodiments, application 340-1 includes a plurality of event processing units 390 and one or more application views 391, each including instructions for processing touch events that occur within an individual view of the user interface of the application. Each application view 391 of application 340-1 includes one or more event recognition units 380. Generally, an application view 391 includes a plurality of event recognition units 380. In other embodiments, one or more of the event recognition units 380 are part of a separate module, such as a user interface kit or a higher-level object from which application 340-1 inherits methods and other characteristics. In some embodiments, an individual event processing unit 390 includes one or more of event data 379 received from data update unit 376, object update unit 377, GUI update unit 378, and / or event sorter 370. The event processing unit 390 optionally utilizes or calls data update unit 376, object update unit 377, or GUI update unit 378 to update the internal state 392 of the application. Alternatively, one or more of the application views 391 include one or more individual event processing units 390. Also, in some embodiments, one or more of data update unit 376, object update unit 377, and GUI update unit 378 are included in the application view 391.

[0135] An individual event recognition unit 380 receives event information (e.g., event data 379) from event sorter 370 and identifies an event from the event information. The event recognition unit 380 includes an event receiving unit 382 and an event comparing unit 384. In some embodiments, the event recognition unit 380 also includes at least one subset of metadata 383 and event delivery instructions 388 (optionally including sub-event delivery instructions).

[0136] The event receiving unit 382 receives event information from the event sorter 370. The event information includes sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event relates to the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0137] The event comparison unit 384 compares event information with a predefined event or sub-event definition, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 384 includes an event definition 386. The event definition 386 includes definitions of events (e.g., a predefined series of sub-events) such as, for example, event 1 (387-1) and event 2 (387-2). In some embodiments, the sub-events in event 387 include, for example, the start of a touch, the end of a touch, the movement of a touch, the cancellation of a touch, and multiple touches. In one example, the definition of event 1 (387-1) is a double-tap on a displayed object. A double-tap includes, for example, a first touch (start of touch) at a predetermined stage on the displayed object, a first lift-off (end of touch) at a predetermined stage, a second touch (start of touch) at a predetermined stage on the displayed object, and a second lift-off (end of touch) at a predetermined stage. In another example, the definition of event 2 (387-2) is a drag on a displayed object. A drag includes, for example, a touch (or contact) at a predetermined stage on the displayed object, movement of the touch across the display system 312 when it is a touch-sensitive display, and lift-off of the touch (end of touch). In some embodiments, an event also includes information regarding one or more associated event processing units 390.

[0138] In some embodiments, the event definition 387 includes the definition of events for individual user interface objects. In some embodiments, the event comparison unit 384 performs a hit test to determine which user interface object is associated with the sub - event. For example, in an application view where three user interface objects are displayed on the display system 312, when a touch is detected on the display system 312 when it is a touch - sensitive display, the event comparison unit 384 performs a hit test to determine which of the three user interface objects is associated with the touch (sub - event). If each display object is associated with an individual event processing unit 390, the event comparison unit uses the result of the hit test to determine which event processing unit 390 should be activated. For example, the event comparison unit 384 selects the sub - event that triggers the hit test and the event processing unit associated with the object.

[0139] In some embodiments, the definition of an individual event 387 also includes a delay action that delays the delivery of event information until it is determined whether a series of sub - events correspond to the event type of the event recognition unit.

[0140] When the individual event recognition unit 380 determines that a series of sub - events do not match any of the events in the event definition 386, the individual event recognition unit 380 enters a state of event impossible, event failure, or event end, and then ignores subsequent sub - events of the touch - based gesture. In this situation, if there is another event recognition unit that remains active for the hit view, that event recognition unit continues to track and process the sub - events of the ongoing touch - based gesture.

[0141] In some embodiments, the individual event recognition unit 380 includes metadata 383 having configurable properties, flags, and / or lists indicating how the event distribution system should actively participate in implementing sub - event distribution. In some embodiments, the metadata 383 includes configurable properties, flags, and / or lists indicating how the event recognition units interact with each other or how they can interact with each other. In some embodiments, the metadata 383 includes configurable properties, flags, and / or lists indicating whether sub - events are distributed to various levels in the view hierarchy or program hierarchy.

[0142] In some embodiments, the individual event recognition unit 380 activates the event processing unit 390 associated with the event when one or more specific sub - events of the event are recognized. In some embodiments, the individual event recognition unit 380 distributes event information associated with the event to the event processing unit 390. Activating the event processing unit 390 is different from sending (and deferring sending) sub - events to individual hit views. In some embodiments, the event recognition unit 380 sets a flag associated with the recognized event, and the event processing unit 390 associated with that flag catches the flag and performs a predefined process.

[0143] In some embodiments, the event distribution command 388 includes a sub - event distribution command that distributes event information about sub - events without activating the event processing unit. Instead, the sub - event distribution command distributes event information to an event processing unit associated with a series of sub - events or to a view that is actively involved. The event processing unit associated with a series of sub - events or an actively involved view receives the event information and executes a predetermined process.

[0144] In some embodiments, the data update unit 376 creates and updates the data used by application 340-1. For example, the data update unit 376 stores the video files used by the media player module 344. In some embodiments, the object update unit 377 creates and updates the objects used by application 340-1. For example, the object update unit 376 creates a new user interface object or updates the position of a user interface object. The GUI update unit 378 updates the GUI. For example, the GUI update unit 378 prepares the display information and sends it to the graphic module 332 for display on the display system 312.

[0145] In some embodiments, the event processing unit(s) 390 includes or has access to the data update unit 376, the object update unit 377, and the GUI update unit 378. In some embodiments, the data update unit 376, the object update unit 377, and the GUI update unit 378 are included in a single module of application 340-1 or application view 391. In other embodiments, they are included in two or more software modules.

[0146] The foregoing considerations regarding event processing of user touches on a touch-sensitive display also apply to other forms of user input for operating the electronic device 300 using an input device, it being understood that not all of them are initiated on the touch screen. For example, the movement of a mouse and the pressing of a mouse button, contact movements such as taps, drags, and scrolls on a touch pad, pen stylus input, movement of the device, verbal commands, detected eye movements, biometric input, and / or any combination thereof, optionally in association with a single or multiple presses or holds of a keyboard, are used as input corresponding to sub-events that define events to be optionally recognized.

[0147] As used herein, the term "focus selector" refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other location marker, the cursor functions as a "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3, or touch-sensitive surface 451 in FIG. 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display that enables direct interaction with user interface elements on the touch screen display, a contact detected on the touch screen operates as a "focus selector" such that when an input (e.g., a press input by the contact) is detected on the touch screen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one region of the user interface to another region of the user interface (e.g., by moving the focus from one button to another button using a tab key or arrow keys) without movement of the corresponding cursor or movement of the contact on the touch screen display. In these implementations, the focus selector moves in accordance with the movement of the focus between various regions of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on a touch screen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the user interface element with which the user intends to interact).For example, the position of a focus selector (e.g., a cursor, a contact, or a selection box) over an individual button while a press input is detected on a touch-sensing surface (e.g., a touch pad or a touch screen) indicates that the user intends to activate that individual button (as opposed to other user interface elements shown on the device's display).

[0148] As used in this specification and the claims, the term "intensity" of a contact on a touch-sensing surface refers to the force, or pressure (force per unit area), of a contact (e.g., finger contact or stylus contact) on the touch-sensing surface, or a proxy for the force or pressure of the contact on the touch-sensing surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically includes hundreds (e.g., at least 256) of distinct values. The intensity of the contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or sum) to identify the estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensing surface. Alternatively, the size and / or change thereof of the contact area detected on the touch-sensing surface, the capacitance and / or change thereof of the touch-sensing surface in proximity to the contact, and / or the resistance and / or change thereof of the touch-sensing surface in proximity to the contact are optionally used as a proxy for the force or pressure of the contact on the touch-sensing surface. In some implementations, an alternative measurement of the force or pressure of the contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, an alternative measurement for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds the intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).Using the intensity of contact as an attribute of user input enables the user to access additional device functions that may not otherwise be easily accessible to the user on a device of limited or reduced size where assets for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons) are limited.

[0149] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the portable computing device 100). For example, the mouse "click" threshold for a trackpad or touch screen display can be set to any of a wide range of default thresholds without changing the hardware of the trackpad or touch screen display. Further, in some implementations, a software setting is provided for the user of the device to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0150] As used in this specification and the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected during a predetermined time (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined number of strength samples, i.e., a predetermined event (e.g., after detecting the contact, before detecting the lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of the maximum value of the strength of the contact, the median value of the strength of the contact, the average value of the strength of the contact, the top 10 percent value of the strength of the contact, a value that is half of the maximum value of the strength of the contact, a value that is 90 percent of the maximum value of the strength of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic strength (e.g., when the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an action has been performed by a user. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact having a characteristic strength that does not exceed the first threshold results in a first operation, a contact having a characteristic strength that exceeds the first threshold and does not exceed the second threshold results in a second operation, and a contact having a characteristic strength that exceeds the second threshold results in a third operation. In some embodiments, the comparison between the characteristic strength and the one or more strength thresholds is not used to determine whether to perform a first operation or a second operation, but rather is used to determine whether to perform one or more operations (e.g., whether to perform each option or refrain from performing each operation).

[0151] In some embodiments, for the purpose of determining characteristic strength, a portion of the gesture is identified. For example, the touch sensing surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a starting position and reaches an ending location, and at that point the intensity of the contact increases. In this example, the characteristic strength of the contact at the ending location may be based on only a portion of the continuous swipe contact (e.g., only a portion of the swipe contact at the ending location) rather than the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before identifying the characteristic strength of the contact. For example, the smoothing algorithm may optionally include one or more of a non-weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate narrow spikes or dips in the width of the swipe contact intensity for the purpose of determining characteristic strength.

[0152] In some embodiments, one or more predetermined intensity thresholds are used to determine whether a particular input meets an intensity-based criterion. For example, the one or more predetermined intensity thresholds may include (i) a contact detection intensity threshold IT0, (ii) a light press intensity threshold IT L , (iii) a deep press intensity threshold IT L (e.g., at least initially higher than I D ), and / or (iv) one or more other intensity thresholds (e.g., an intensity threshold I L lower than I H) can be mentioned. In some embodiments, the light pressing intensity threshold generally corresponds to the intensity at which the device performs an operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep pressing intensity threshold generally corresponds to the intensity at which the device performs an operation different from the operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact having a characteristic intensity below the light pressing intensity threshold (for example, exceeding a nominal contact detection intensity threshold IT0 below which contact is no longer detected) is detected, the device moves the focus selector according to the movement of the contact on the touch sensing surface without performing an operation associated with the light pressing intensity threshold or the deep pressing intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent among various sets of user interface values.

[0153] In some embodiments, the device's response to an input detected by the device depends on a criterion based on the contact intensity during the input. For example, in the case of some "light press" inputs, a contact intensity exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on a criterion that includes both the contact intensity and a time-based criterion during the input. For example, in the case of some "deep press" inputs, a contact intensity exceeding a second intensity threshold during the input that is greater than the first intensity threshold for light presses triggers a second response only if a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold. This delay time is typically less than 200 milliseconds in duration (e.g., 40 milliseconds, 100 milliseconds, or 120 milliseconds depending on the magnitude of the second intensity threshold, and the delay time increases as the second intensity threshold increases). This delay time helps to avoid accidental deep press inputs. As another example, in the case of some "deep press" inputs, there is a period of time with reduced sensitivity that occurs after the time when the first intensity threshold is satisfied. During the period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. In the case of other deep press inputs, the response to the detection of the deep press input does not depend on a time-based criterion.

[0154] In some embodiments, one or more of the input intensity thresholds and / or the corresponding outputs vary based on one or more factors such as user settings, movement of the contact, input timing, the application being run, the rate at which the intensity is applied, the number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), the position of the focus selector, etc. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are hereby incorporated by reference in their entirety.

[0155] For ease of explanation, descriptions of operations performed in response to a pressing input associated with a pressing input strength threshold, or in response to a gesture including the pressing input, are optionally based on an increase in the intensity of contact exceeding the pressing input strength threshold, an increase in the intensity of contact from an intensity below a hysteresis strength threshold to an intensity exceeding the pressing input strength threshold, a decrease in the intensity of contact below the pressing input strength threshold, or a decrease in the intensity of contact below a hysteresis strength threshold corresponding to the pressing input strength threshold. Further, in examples where an operation is described as being performed in response to detecting a decrease in the intensity of contact below the pressing input strength threshold, the operation is optionally performed in response to detecting a decrease in the intensity of contact corresponding to and below a hysteresis strength threshold lower than the pressing input strength threshold. As described above, in some embodiments, the triggering of these responses also depends on time-based criteria being met (e.g., the delay time elapsing between the first intensity threshold being met and the second intensity threshold being met).

[0156] FIG. 4A shows an exemplary user interface 400 for a menu of an application on a portable multifunctional device 100 according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or a subset or superset thereof.

[0157] Signal strength indication(s) for wireless communication(s) such as cellular signal and Wi-Fi signal

[0158] Time

[0159] Bluetooth indication

[0160] Battery status indication

[0161] A tray 408 including icons for frequently used applications such as the following

[0162] An icon 416 for a telephone module 138 labeled "Telephone", optionally including an indication 414 of the number of incoming calls or voicemail messages.

[0163] An icon 418 for an email client module 140 labeled "Mail", optionally including an indication 410 of the number of unread emails.

[0164] An icon 420 for a browser module 147 labeled "Browser", and

[0165] An icon 422 for a video and music player module 152 labeled "Music", and

[0166] Icons for other applications, such as the following.

[0167] An icon 424 for an IM module 141 labeled "Message".

[0168] An icon 426 for a calendar module 148 labeled "Calendar".

[0169] An icon 428 for an image management module 144 labeled "Photo".

[0170] An icon 430 for a camera module 143 labeled "Camera".

[0171] An icon 432 for an online video module 155 labeled "Online Video".

[0172] An icon 434 for a stock price widget 149-2 labeled "Stock Price".

[0173] An icon 436 for a map module 154 labeled "Map".

[0174] An icon 438 for the weather widget 149-1 labeled "Weather",

[0175] An icon 440 for the alarm clock widget 149-4 labeled "Clock",

[0176] An icon 442 for the training support module 142 labeled "Training Support",

[0177] An icon 444 for the memo module 153 labeled "Memo", and

[0178] An icon 446 for a settings application or module that provides access to settings related to the device 100 and its various applications 136.

[0179] Note that the labels of the icons shown in FIG. 4A are merely examples. For example, other labels may optionally be used for the various application icons. In some embodiments, the label for an individual application icon includes the name of the application corresponding to the individual application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0180] FIG. 4B shows an exemplary user interface on a device (e.g., device 300, FIG. 3) having a touch sensing surface 451 (e.g., a tablet or touch pad 355, FIG. 3) separated from a display 450. Many of the following examples are given with reference to inputs on a touch screen display 112 (when the touch sensing surface and the display are combined), but in some embodiments, the device detects inputs on a touch sensing surface separated from the display, as shown in FIG. 4B. In some embodiments, the touch sensing surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) corresponding to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to those embodiments, the device detects contacts (e.g., 460 and 462 in FIG. 4B) with the touch sensing surface 451 at locations (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470) corresponding to respective locations on the display. Thus, when the touch sensing surface is separated from the display, user inputs (e.g., contacts 460 and 462, and their movements) detected by the device on the touch sensing surface (e.g., 451 in FIG. 4B) are used by the device to operate the user interface on the display (e.g., 450 in FIG. 4B) of the multifunctional device. It should be understood that a similar method is optionally used for other user interfaces described herein.

[0181] In addition, the following examples are given mainly with reference to finger inputs (e.g., finger contact, finger tap gesture, finger swipe gesture, etc.), but it should be understood that in some embodiments, one or more of those finger inputs are replaced with inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by a mouse click with movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click while the cursor is located on the location of the tap gesture (e.g., instead of detecting contact and subsequently ceasing to detect contact). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0182] As used herein, the term "focus selector" refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor functions as a "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display (e.g., touch-sensitive display system 112 in FIG. 1A or touch screen in FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, a contact detected on the touch screen functions as a "focus selector" such that when an input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one region of the user interface to another region of the user interface without moving the corresponding cursor or the contact on the touch screen display (e.g., by using the tab key or arrow keys to move the focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of the focus between various regions of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the user interface element through which the user intends to interact).For example, the position of a focus selector (e.g., a cursor, a contact, or a selection box) over an individual button while a press input is detected on a touch-sensing surface (e.g., a touch pad or a touch screen) indicates that the user intends to activate that individual button (as opposed to other user interface elements shown on the device's display).

[0183] As used in this specification and the claims, the term "intensity" of a contact on a touch sensing surface refers to the force, or pressure (force per unit area), of a contact (e.g., a finger contact or a stylus contact) on the touch sensing surface, or an alternative (proxy) for the force or pressure of the contact on the touch sensing surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically includes hundreds (e.g., at least 256) of distinct values. The intensity of the contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch sensing surface are optionally used to measure the force at various points on the touch sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or sum) to identify the estimated contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch sensing surface. Alternatively, the size and / or change thereof of the contact area detected on the touch sensing surface, the capacitance and / or change thereof of the touch sensing surface proximate to the contact, and / or the resistance and / or change thereof of the touch sensing surface proximate to the contact are optionally used as an alternative for the force or pressure of the contact on the touch sensing surface. In some implementations, an alternative measurement of the force or pressure of the contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, the alternative measurement for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds the intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure).Using the intensity of contact as an attribute of user input enables the user to access additional device functions that may not otherwise be easily accessible to the user on a device of limited or reduced size where assets for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons) are restricted.

[0184] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds may not be determined by the activation threshold of a particular physical actuator and may be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to any of a wide range of default thresholds without changing the hardware of that trackpad or touchscreen display. Further, in some implementations, a software setting is provided for the user of the device to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0185] As used in this specification and the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected over a predetermined time (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined number of strength samples, i.e., a predetermined event (e.g., after detecting the contact, before detecting the lift-off of the contact, before or after detecting the start of the movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of the contact is optionally one or more of the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the top 10% value of the contact strength, a value that is half of the maximum value of the contact strength, a value that is 90% of the maximum value of the contact strength, a value generated by low-pass filtering the strength of the contact over a predefined period, or at a predefined time. In some embodiments, the duration of the contact is used when determining the characteristic strength (e.g., when the characteristic strength is the average of the contact strength over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by the user. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a first operation is performed as a result of a contact having a characteristic strength that does not exceed the first threshold, a second operation is performed as a result of a contact having a characteristic strength that exceeds the first threshold and does not exceed the second threshold, and a third operation is performed as a result of a contact having a characteristic strength that exceeds the second threshold. In some embodiments, the comparison between the characteristic strength and the one or more strength thresholds is not used to determine whether to perform a first operation or a second operation, but rather is used to determine whether to perform one or more operations (e.g., whether to perform each option or refrain from performing each operation).

[0186] In some embodiments, for the purpose of determining characteristic intensity, a portion of the gesture is identified. For example, the touch sensing surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a start position and reaches an end location, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end location may be based on only a portion of the continuous swipe contact (e.g., only a portion of the swipe contact at the end location) rather than the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before identifying the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of a non - weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate narrow spikes or drops in the width of the swipe contact intensity for the purpose of determining characteristic intensity.

[0187] The figures of the user interfaces described herein optionally include one or more intensity thresholds (e.g., a contact detection intensity threshold IT0, a light press intensity threshold IT L , a deep press intensity threshold IT D (e.g., at least initially higher than IT L ), and / or one or more other intensity thresholds (e.g., an intensity threshold IT L lower than IT H) includes charts of various intensities indicating the current intensity of contact on the touch sensing surface with respect to. This intensity diagram is typically not part of the displayed user interface but is provided to assist in the interpretation of the diagram. In some embodiments, the light press intensity threshold typically corresponds to the intensity at which the device performs an operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep press intensity threshold typically corresponds to the intensity at which the device performs an operation different from the operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact having a characteristic intensity below the light press intensity threshold (e.g., above a nominal contact detection intensity threshold IT0 below which the contact is no longer detected) is detected, the device moves the focus selector in accordance with the movement of the contact on the touch sensing surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent among various sets of values of the user interface.

[0188] In some embodiments, the device's response to an input detected by the device depends on a criterion based on the contact intensity during the input. For example, in the case of some "light press" inputs, a contact intensity exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on a criterion that includes both the contact intensity and a time-based criterion during the input. For example, in the case of some "deep press" inputs, a contact intensity exceeding a second intensity threshold during the input that is greater than the first intensity threshold for light presses triggers a second response only if a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold. This delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms depending on the magnitude of the second intensity threshold, and the delay time increases as the second intensity threshold increases). This delay time helps to avoid accidental recognition of deep press inputs. As another example, in the case of some "deep press" inputs, there is a time period of reduced sensitivity that occurs after the time when the first intensity threshold is satisfied. During the time period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. In the case of other deep press inputs, the response to the detection of a deep press input does not depend on a time-based criterion.

[0189] In some embodiments, one or more of the input intensity thresholds and / or the corresponding outputs vary based on one or more factors such as user settings, movement of the contact, input timing, the application being executed, the rate at which the intensity is applied, the number of simultaneous inputs, the user history, environmental factors (e.g., ambient noise), the position of the focus selector, etc. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are hereby incorporated by reference in their entirety. User Interface and Related Processes

[0190] Next, attention is directed to embodiments of a user interface ("UI") and related processes that may be implemented on a system including a laptop device 300 (FIG. 1A), a tablet device 100 (FIGS. 2A-2B), and / or a desktop device 200 (e.g., FIG. 1B). The system can operate in different modes including a shared input mode and a companion display mode. In the shared input mode, the user interface generated by each device (e.g., the laptop device 300, the tablet device 100, or the desktop device 200) is presented on the respective display of the device (e.g., the displays 301, 101, and 201 of the laptop device 300, the tablet device 100, or the desktop device 200, respectively) such that the devices share the same input device (e.g., the mouse 202 and the keyboard 203, or the keyboard 305 and / or the touchpad 309). In the companion display mode, the user interface generated by one device (e.g., the laptop device 300 of FIG. 1A) is presented on another device (e.g., the tablet device 100 of FIG. 1A). The devices described herein (e.g., desktops, laptops, tablets, mobile phones) are used as exemplary examples in the following description, and the techniques described herein are equally applicable to any device running a desktop / laptop / tablet operating system, or in some cases, operations described as being implemented on a laptop may also be implemented by a tablet device or a desktop, and vice versa, which will be readily understood by those skilled in the art. The following examples illustrate one or more embodiments.

[0191] Figure 1A shows that the laptop device 300 has a connection 194 (e.g., a wired or wireless connection), is associated with (e.g., logged in to) the same user account as the tablet device 100, and has established a trusted connection with the tablet device (e.g., a trust prompt as described below has been accepted by the user of the device). The laptop includes a display 301 that can be a touch-sensitive display. Additionally, in some embodiments, the laptop can also include a dynamic function row 304 for displaying additional information (additional details regarding such dynamic function row 304 are provided in U.S. Patent Application No. 15 / 655,707, which application is incorporated herein by reference in its entirety). Further, the laptop also includes a keyboard 305 and a touchpad 309. Regarding the tablet device 100, the tablet device 100 includes a touch-sensitive display 101 that can be capacitive sensing, and the device 100 can also receive input from an input device such as a stylus or the user's finger. Figure 1A also shows performing a selection operation using a cursor on the maximize button 196 of the photo application window 189 (e.g., a button that can be displayed between two other buttons at the corner of the user interface window and the maximize button can also be presented in green) (e.g., by hovering or performing a right click).

[0192] In some embodiments, when in the shared input mode, both devices 100 and 300 execute their respective operating systems while sharing input devices (e.g., keyboard 305 and touchpad 309) implemented on device 300. In some embodiments, when in the companion display mode, device 100 continues to execute its operating system and then receives from device 300 information that enables device 100 to display a user interface generated by device 300 (in some cases, device 100 also stops displaying any user interface elements associated with its operating system when the companion display mode is initiated). The companion display mode includes an extended display mode and a mirroring display mode. In the extended display mode, the displays of devices 100 and 300 display a continuous view of the content generated by device 300 (e.g., the display of device 100 extends the display of device 300). In the mirroring display mode, the display of device 100 displays a mirror image of the display of device 300, and the content on the display is generated by device 300. In some embodiments, two or more devices (e.g., two tablet devices running a mobile operating system, or two laptop devices running a desktop operating system) can execute the same operating system. For example, in FIG. 1B, device 200 is in a trusted (195) connection 194 with devices 300 and 100 and shares the same user account 193. Device 200 can be in the shared input mode or the companion display mode with either or both of devices 300 and 100.

[0193] FIG. 1A shows two devices, namely a laptop device 300 and a tablet device 100, both signed in to the same user account 193 (e.g., the same iCloud account of APPLE INC. (Cupertino, CA) on the displays of both two devices) and having an established connection 194 (e.g., a wired connection or a wireless connection). When the two devices are logged in to the same user account and have an established connection, the companion display mode or the shared input mode may not yet be available until the devices have a trusted connection (e.g., 195). The laptop device 300 and the tablet device 100 are both connected to the same Wi-Fi wireless network, indicating that the devices have an established connection 194. In some embodiments, the user may not need to be on the same Wi-Fi network, and other forms of connection between the two devices, such as near field communication (NFC), Bluetooth, or other short-range communication protocols, may be possible.

[0194] FIGS. 5A-8AI are schematic diagrams of a laptop display 301, a desktop display 201, and a touch-sensitive display 101 of a tablet device, used to illustrate exemplary user interfaces according to some embodiments. The user interfaces in these figures are used to illustrate the methods and / or processes described below. Those skilled in the art will understand that the following user interfaces are merely examples, and the user interfaces shown in each figure can be invoked in any particular order. Further, those skilled in the art will understand that different layouts with additional or fewer affordances, user interface elements, or graphics can be used in various situations. It should also be understood that any one of the following exemplary user interfaces can correspond to a separate embodiment and does not need to follow any particular order. The user interfaces of these figures are used to illustrate the processes described below, including the processes of FIGS. 9A-12F.

[0195] Figures 5A - 5E show user input for moving an object (e.g., a user interface object) from a first display of a first device to a second display of a second device when the first device and the second device are in an extended display mode, according to some embodiments. In Figure 5A, device 200 (e.g., a desktop device) displays a first user interface 5010 (e.g., a home screen user interface), a menu bar (e.g., menu bar 5008), and a dock (e.g., dock 5006). As referred to herein, the menu bar includes a plurality of affordances for invoking commands, performing system - level or application - level tasks, and / or status indications (e.g., time, signal strength). As referred to herein, the dock includes a plurality of application icons (e.g., icons for engaging or launching an application). In some embodiments, at least some of the application icons in the dock correspond to recently viewed applications, frequently used applications, and / or applications based on user preferences or selections. Device 200 further displays a user interface object (e.g., photo object 5004) and a cursor (e.g., cursor 5002) positioned on the object. Movement of the cursor is controlled by an input device (e.g., mouse 202 and / or keyboard 203 that communicate with device 200). In Figures 5A - 5E, the desktop device 200 is communicating with the tablet device 100 in an extended display mode. In some embodiments, as described with respect to Figure 1A, the desktop device 200 and the tablet device 100 are both signed in to the same user account 193 and have established a trusted (e.g., 195) connection 194 (e.g., a wired or wireless connection), the desktop device 200 is communicating with the tablet device 100.In some embodiments, when the desktop device 200 is communicating with the tablet device 100, both the desktop device 200 and the tablet device 100 are connected to the same Wi-Fi wireless network or other form of connection (e.g., Near Field Communication (NFC), Bluetooth, or other short-range communication protocol). Specifically, the device 200 is positioned next to the device 100 such that a first edge (e.g., edge 201-1) of the display 201 of the device 200 is substantially parallel to and proximate (e.g., within 1 foot or 2 feet) to a first edge (e.g., edge 101-1) of the display 101 of the device 100. In some embodiments, an edge of a display refers to a location on the device (e.g., the touch-sensitive display or touch screen 112 or display system 312 described above) where the display area ends and the edge of the frame of the display device begins (or in the case of an edge-less display, the side of the device). In the extended display mode, the displays of the devices 100 and 200 display a continuous view of the display content generated by the device 200, as shown with corresponding horizontal desktop wallpapers (e.g., the background image of the user interface 5010) displayed on the displays 201 and 101 (e.g., the desktop on the display of the device 100 extends across the display of the device 300). In some embodiments, as shown, the device 100 also displays a menu bar 5008 corresponding to the menu bar displayed on the device 200. Note that some aspects of user interface features such as menu bars may appear different, or include different selections of indicia and / or affordances, for example, based on the size and orientation of different displays. The edges 201-1 and 101-1 of each device include a portal for moving a particular object (e.g., a cursor or user interface object) from the display 201 across the edge 201-1 onto the display 101.For example, the portal corresponds to a first portion of an individual edge that enables moving a particular object from display 201 to display 101. A portion of the individual edge that is outside the first portion does not permit movement of the particular object. In some embodiments, the portal extends along a portion of the individual edges of edges 201-1 and 101-1 (e.g., a portion corresponding to about 25%, about 50%, or about 75%), or along the entire length of the individual edges of edges 201-1 and 101-1. For example, the portal extends along about 50% of the width of an individual display.

[0196] In FIG. 5A, while the cursor 5002 is displayed on the photo object 5004, the device 200 detects an input via the mouse 202 (e.g., an input including a press 5011-A (e.g., a gray dot) on the mouse 202 and a movement 5012-A of the mouse 202 in the direction indicated by the arrow in FIG. 5A). In some embodiments, the input corresponds to a request to select and move the photo object 5004. In some embodiments, the input corresponds to a request to select and drag the photo object 5004 according to the movement of the mouse 202 (e.g., movement 5012-A). For example, the user input includes a press 5011-A (e.g., indicated by the gray dot on the mouse 202) and a drag input that are performed via the mouse 202 to select the photo object 5004 and move the photo object according to the drag input. In FIG. 5B, the device 200 displays the moving photo object 5004 according to the movement 5012-A of the mouse 202 toward the edge 201-1 of the display 201. In some embodiments, the photo object 5004 is moved in the direction corresponding to the movement 5012-A at a speed and / or acceleration corresponding to the movement 5012-A. For example, the movement of the photo object 5004 may increase, decrease, or halt according to the movement 5012-A. As long as the photo object 5004 remains selected, the object follows the movement of the mouse 202. In some embodiments, in response to detecting that the photo object 5004 is being moved toward the edge 201-1 including a portal for moving the object to the display 201 of the device 100, the device 200 displays an indication (e.g., indication 5014-A) indicating the position of the portal that enables the object to be moved from the display 101 to the display 101. In some embodiments, the length of the indication 5014-A corresponds to the size of the portal. In some embodiments, the length of the indication 5014-A corresponds to the length of the edge 101-1 of the device 100 (e.g., an extended display).In some embodiments, while the display 101 displays the indication 5014-A, the display 201 simultaneously displays the corresponding indication 5014-B. The indication 5014-B extends along the edge 101-1 of the display 101 facing the edge 201-1 of the display 201. In some embodiments, the indication 5014-A and / or the indication 5014-B are displayed according to a determination that an object (or a set of one or more objects) being moved towards the edge 201-1 is a representation of content that can be moved from the display 201 to the display 101. In some embodiments, the content that can be moved from the display 101 to the display 201 corresponds to text content (e.g., one or more text files or a selection of text), photo objects (e.g., one or more photos), video content, audio content, presentation content. In FIG. 5C, according to a determination that the photo object 5004 can be moved from the display 201 to the display 101, when the devices 200 and 100 are in the extended display mode, the photo object 5004 moves through the portal across the edges 201-1 and 101-1. In FIG. 5C, the user input further includes a movement 5012-B to move the photo object 5004 to a desired position on the display 101. In some embodiments, the movements 5012-A and 5012-B are part of a continuous drag input. In some embodiments, the appearance (e.g., size, shape, color, pattern) of the cursor 5002 changes according to the device on which it is displayed. As shown, the appearance of the cursor 5002 changes as the cursor moves from the display 201 (e.g., a desktop display) to the display 101 (e.g., a tablet touch screen). In some embodiments, the appearance of the moved object is also changed. For example, the device 200 with a smaller display size can display an object with a reduced size compared to the device 100 with a larger display size.In FIG. 5D, the photo object 5004 is moved to the central portion of the display 201. In FIG. 5D, the device 200 further detects the end of a gesture for moving the photo object (e.g., the press gesture on the mouse 202 is released). In some embodiments, the release of the input corresponds to a request to drop the photo object 5004 at the position of the cursor 5002. As shown in FIG. 5E, the photo object 5004 is positioned (e.g., dropped) at a position corresponding to the position of the cursor 5002 when the press on the mouse 202 is released. As shown, the photo object 5004 remains in the same position while the cursor 5002 is moved away from the object. As described with respect to FIGS. 5A-5F, using the input detected on the mouse to select, move, and release an object is applicable to other user interfaces described below. It is also understood that moving a user interface object (e.g., the photo object 5004) from the display 201 to the display 101 is exemplary. It is understood that the user interface object can also be moved from the display 101 to the display 101 with a similar operation.

[0197] Figures 5F - 5J show user input for moving an object from the first display of the first device to the second display of the second device when the first device and the second device are in a shared input mode, according to some embodiments. In FIG. 5A, device 200 displays an application user interface (e.g., photo application 5016) on top of the first user interface 5010. The photo application 5016 includes a photo object 5004 displayed on the user interface 5010 of FIG. 5A. Device 200 is positioned adjacent to device 100 such that a first edge (e.g., edge 201 - 1) of the display 201 of device 200 is substantially parallel and adjacent to a first edge (e.g., edge 101 - 1) of the display 101 of device 100. In FIGS. 5F - 5J, the desktop device 200 and the tablet device 100 are in a shared input mode. As described above, in the shared input mode, devices 100 and 200 share one or more input devices (e.g., keyboard 203 and mouse 202). However, devices 100 and 200 operate their respective operating systems while receiving input via the same input devices (e.g., keyboard 203 and mouse 202 that communicate with device 200). In some embodiments, device 200 receives input and communicates the input to device 100 according to a determination that the cursor is displayed on the display 101 of device 100. Thereby, device 100 receives information regarding the input from device 200 and performs an operation accordingly. As shown in FIG. 5F, the display 101 of device 100 displays a different user interface (e.g., email user interface 5018) than the user interface (e.g., user interface 5010 and application user interface 5016) displayed on the display 201 of device 200.

[0198] In FIG. 5F, the device 200 detects an input via the mouse 202 (e.g., a press on the mouse 202 followed by a drag movement in the direction indicated by the illustrated arrow) corresponding to a request to select and move the photo object 5004. As described above, the input corresponds to a request to drag the photo object 5004 as the mouse 202 moves. In FIG. 5G, the device 200 displays the moving photo object 5004 in accordance with the movement of the mouse 202 toward the edge 201-1 of the display 201 (e.g., in accordance with the direction, speed, acceleration, etc. of the movement of the mouse 202). In some embodiments, in response to detecting that the photo object 5004 is being moved toward the edge 201-1 that includes a portal for moving the object to the display 201 of the device 100, the device 200 displays an indication (e.g., indication 5014-A) indicating the position of the portal that enables the object to be moved from the display 101 to the display 101. In FIG. 5H, in accordance with the determination that the photo object 5004 can be moved from the display 201 to the display 101, when the devices 200 and 100 are in a shared input mode, the photo object 5004 is moved from the display 201 to the display 101 across the edges 201-1 and 101-1 through the portal. In FIG. 5I, the photo object 5004 is moved to the central portion of the display 201 and positioned on the application user interface 5018. In FIG. 5I, the device 200 further detects the end of the gesture for moving the photo object (e.g., the press gesture on the mouse 202 is released). In some embodiments, the release of the input corresponds to a request to drop the photo object 5004 at the position of the cursor 5002. As shown in FIG. 5J, the photo object 5004 is positioned (e.g., dropped) at a position corresponding to the position of the cursor 5002 when the press on the mouse 202 is released. As illustrated, the photo object 5004 remains in the same position while the cursor 5002 is moved away from the object.As shown in FIGS. 5F to 5J, in addition to moving user interface objects between system-level user interfaces (e.g., user interface 5010 such as the home screen user interface in FIGS. 5A to 5F), the user interface object can be moved from a first application window (e.g., photo application user interface 5016 on display 201 of device 200) displayed on a first display of a first device to a second application window (e.g., email application 5018 on display 101 of device 100) displayed on a second display of a second device. The first application and the second application can be user interfaces of the same application or different applications.

[0199] Figures 5K - 5N show user input for moving an application window from the first display of the first device to the second display of the second device when the first device and the second device are in an extended display mode, according to some embodiments. In Figures 5K - 5N, devices 200 and 100 are in an extended display mode as described above with respect to Figure 5A. In Figure 5K, device 200 displays an application user interface (also referred to as an application window) (e.g., photo application 5016). In Figure 5K, device 200 detects input (e.g., a press followed by a movement in the direction indicated by an arrow) via mouse 202. In Figure 5K, cursor 5001 is positioned in a default area of the application window associated with the selection of the user interface instead of a user interface object (e.g., an area corresponding to the menu bar of the application user interface). In some embodiments, a press input received on mouse 202 while cursor 5002 is positioned over the default area of application user interface 5016 selects application user interface 5016 and corresponds to a request to move application user interface 5016 in accordance with the movement of the mouse.

[0200] In FIG. 5L, according to the determination that the application user interface 5016 can be moved from the display 201 to the display 101, when the devices 200 and 100 are in the extended display mode, the device 200 displays the application user interface 5016 that moves to the display 101 across the edge 201-1 of the display 201 in accordance with the movement of the mouse 202. As shown in the figure, in some embodiments, the indication 5014-A and / or the indication 5014-B are displayed according to the determination that the application user interface 5016 has been moved toward the edge 201-1. In FIG. 5M, the application user interface 5016 is moved to the central portion of the display 201. In FIG. 5M, the device 200 further detects the end of the gesture for moving the photo object (for example, the press gesture on the mouse 202 is released). In some embodiments, the release of the input corresponds to a request to drop the application user interface 5016 at the position of the cursor 5002. As shown in FIG. 5N, the application user interface 5016 is positioned (for example, dropped) at a position corresponding to the position of the cursor 5002 when the press on the mouse 202 is released. As shown in the figure, the application user interface 5016 remains in the same position while the cursor 5002 is moved away from the application user interface (for example, via the movement of the mouse 202).

[0201] Figures 5O - 5P show user input for attempting to move an application window from the first display of the first device to the second display of the second device when, according to some embodiments, the first device and the second device are in an extended display mode and the first display provides an indication that moving the application window to the second display is blocked. In FIGS. 5O - 5P, devices 200 and 100 are in a shared input mode as described above with respect to FIG. 5F. In FIG. 5P, device 100 displays a system - level user interface (e.g., home screen 5020) different from the system - level user interface (e.g., home screen user interface 5010) displayed by device 200 (as shown, for example, by the diagonal stripe wallpaper of device 100 being different from the horizontal stripe wallpaper of device 200). Further, device 200 displays a status bar 5009 (including, for example, indications of signal strength, battery, and Wi - Fi signal) different from the menu bar 5008 displayed by device 200. In FIG. 5O, device 200 displays an application user interface 5016. In FIG. 5K, device 200 detects input via mouse 202 (e.g., a press followed by a movement in the direction indicated by an arrow) while cursor 5001 is positioned within a default region of the application window associated with selection of the user interface (e.g., a region corresponding to the menu bar of the application user interface) instead of a user interface object. In some embodiments, a press input received on mouse 202 while cursor 5002 is positioned over the default region of application user interface 5016 selects application user interface 5016 and corresponds to a request to move application user interface 5016 in accordance with the movement of the mouse.

[0202] In FIG. 5P, the application user interface 5016 is moved to the edge 201-1 of the display 201 (e.g., the application user interface 5016 has reached the edge 201-1 of the display 201 or has reached a distance within a threshold distance from the edge 201-1). When the devices 200 and 100 are in the shared display mode, in accordance with the determination that the moved object is an application user interface that cannot move from the display 201 to the display 101, the device 200 halts the movement of the application user interface 5016 even if the movement of the mouse continues in the projection direction. In some embodiments, in accordance with the determination that the moved object is an application user interface and cannot move between displays, the device 200 displays an indication (e.g., indication 5014-C) indicating that moving objects between displays is not permitted. In some embodiments, the indication corresponds to a bar extending along the edge 201-1 (e.g., similar to the indication 5014-A described with respect to FIG. 5B), but the indication 5014-C has an appearance different from the appearance of the indication 5014-A indicating that it is possible to move an object between displays. For example, the indication 5014-C has a color, size, or pattern different from the color, size, or pattern of the indication 5014-A. In some embodiments, the device 200 provides other visual indications that the movement of the application user interface 5016 from the display 201 to the display 101 is not permitted. In some embodiments, the device 200 displays an animation indicating that the application user interface 5016 appears to bounce back from the edge 201-1. In some embodiments, the device 200 displays an animation indicating that the application user interface 5016 appears to bounce back to its original position on the display 201 (e.g., the position when the selection and drag user input was initiated).In some embodiments, device 200 displays an animation indicating that edge 201-1 resists the movement of application user interface 5016. For example, cursor 5002 continues to move toward edge 201-1 in accordance with the movement of mouse 202, but application user interface 5016 stops moving when it reaches edge 201-1.

[0203] Figures 5Q-5T show user input to move selected text from the first display of the first device to the third display of the third device when, according to some embodiments, the first device and the third device are in an extended display mode and the first device is also communicating with the second device. In Figure 5Q, desktop device 200 communicates with tablet device 100 and laptop device 300. Specifically, device 300 is positioned next to device 200 such that a first edge (e.g., edge 301-1) of display 201 of device 200 is substantially parallel and proximate to a second edge (e.g., edge 201-2) of display 301 of device 200. Device 200 is communicating with device 300 in a shared input mode (e.g., device 300 is operating its own operating system while receiving commands from an input device that is communicating with device 200). As shown, device 300 displays a system-level user interface (e.g., home screen user interface 5024) that includes a menu bar (e.g., menu bar 5028). Home screen 5025 and menu bar 5028 are different from home screen user interface 5010 and menu bar 5008, respectively, displayed by device 200. Device 200 can communicate with device 100 in a shared input mode, extended display mode, or mirroring display mode. In Figure 5Q, device 200 displays an application user interface (e.g., memo application 5030) that includes text arranged in two lines. As shown, text line 5032 is selected as indicated by the highlighting. For example, device 200 has detected a user input corresponding to a request to select the text. In some embodiments, the user input corresponding to the request to select text line 5032 includes pressing and holding while cursor 5002 is positioned over the text to be selected and then dragging cursor 5002 to expand the selection to the desired size.In FIG. 5Q, while the cursor 5001 is on the selected text line 5032, the device 200 detects user input (e.g., after pressing, movement in the direction indicated by the arrow) via the mouse 202. In some embodiments, the text line 5032 is moved according to the movement of the mouse. In FIG. 5R, the text line 5053 is moved to the edge 201-2 of the display 201. In some embodiments, according to the determination that the movement of the text line 5034 is towards the edge 201-2, the indication 5014-C is displayed along the edge 201-2, indicating the position of the portal that enables moving an object from the display 201 to the display 301. In some embodiments, according to the determination that the movement of the text line 5034 is towards the edge 201-2 (not towards the edge 201-1), the device 200 stops displaying an indication (e.g., the indication 5014-A in FIG. 5B) along the edge 201-1. When the devices 200 and 100 are in the shared input mode, according to the determination that the text line where the moved object can move from the display 201 to the display 301, the device 200 displays the text line 5034 that moves across the edge 201-2 of the display 201 to the display 101 according to the movement of the mouse 202. In FIG. 5S, the text line 5032 is moved to the email application 5026. In FIG. 5S, the device 200 further detects the end of the gesture for the text line 5032 (e.g., the press input on the mouse 202 is released). In some embodiments, the release of the input corresponds to a request to drop the text line 5034 to the position of the cursor 5002 on the email application 5026. As shown in FIG. 5T, the text line 5034 is positioned (e.g., dropped) at a position corresponding to the position of the cursor 5002 when the press of the mouse 202 is released. As shown, the text line 5034 remains in the same position while the cursor 5002 is moved away from the text line 5034 (e.g., via the movement of the mouse 202).Based on a request to copy or move an object such as text line 5032, the object can be copied or moved to a new position on a different display. As shown in FIG. 5T, text line 5032 is copied from the memo application 5030 on display 201 to the email application so that text line 5034 continues to be displayed on the memo application 5030. Alternatively, text line 5034 is moved from the memo application 5030 on display 201 to the email application, and as a result, text line 5034 is no longer displayed on the memo application 5030.

[0204] Figures 6A - 6D illustrate that, in some embodiments, when the second display of the second display device is available for moving content from the first display to the second display, an indication is displayed on the first display of the first device. In Figure 6A, the tablet device 100 is positioned near the desktop device 200. The display device 100 is in the sleep mode 6006 (e.g., power saving mode or low power mode). In some embodiments, when in the sleep mode, the device turns off the touch screen display 101, thereby reducing the power consumption of the device 100. For example, the device turns off any display by a display generation component (e.g., the display controller 156 associated with the display 101). In some embodiments, the device turns on the sleep mode according to the determination that the user interaction device 100 has not been present for a predetermined period. In some embodiments, the device 100 turns on the sleep mode in response to detecting a user input for turning on the sleep mode. In some embodiments, the device 100 turns off the sleep mode (e.g., wakes up the display 101) in response to detecting a user input. In some embodiments, the user input is a gesture (e.g., a tap or a swipe gesture by a contact 6002 - 2 on the touch - sensitive display 101) or a button press (e.g., a press 6002 - 1 on the button 6004 (e.g., the start button)). In some embodiments, the device turns off the low - power mode in response to detecting a change in the orientation of the device 100. For example, the orientation of the device 100 changes from a horizontal orientation (e.g., lying horizontally) to a vertical position. When the device exits the sleep mode, the display 101 displays a wake - screen user interface (e.g., the wake - screen user interface 6008) as shown in Figure 6B.In some embodiments, the wake screen user interface is initially displayed in a locked state and then transitions to an unlocked state after authentication information is obtained (e.g., through password entry or biometric verification). In some embodiments, the wake screen user interface and the lock screen user interface have a similar appearance. In some embodiments, the wake screen user interface includes a time element (e.g., time element 6010) that displays the current time and optionally the date. In some embodiments, the wake screen usage interface includes a status bar 5009. When device 100 is in sleep mode 6006 or when the wake screen user interface 6008 is being displayed, the display 101 is not available to receive objects that have been moved from the display 201 of device 200 to the display 101 of device 200 (e.g., as described with respect to FIGS. 5A - 5T). In some embodiments, communication between devices 200 and 100 is not established (e.g., the devices operate independently without any communication between each other, or the devices do not share a user account 193 and / or do not have a trusted (195) connection 194). In some embodiments, device 100 ends the wake screen and / or the lock screen in response to detecting a user input and / or in response to receiving authentication information. For example, device 100 detects a user input (e.g., a tap or swipe gesture by contact 6002 - 3 on the touch - sensitive display 101 shown in FIG. 6B, or a user input on a button).

[0205] In FIG. 6C, after the device 100 wakes the screen and / or locks the screen in response to detection of a user input (e.g., by contact 6002-3), the system-level user interface (e.g., the home screen 5020) is displayed. The home screen 5020 includes a plurality of application icons 6012 (e.g., application launch icons). In some embodiments, the application icons 6012 correspond to the application icons described with respect to FIG. 4A. When the device 100 begins to display the home screen 5020 or an application user interface (e.g., the email user interface 5018 of FIG. 5F), after the sleep mode and / or the wake / lock screen user interface 6008, the device 100 may be available to receive one or more user interface objects that are moved (e.g., dragged) from the display 201 of the device to the display 101 of the device 100. In response to detecting an event (e.g., starting to display the home screen 5020 and / or the application user interface 5018 after the sleep mode and / or the wake / lock screen user interface 6008), the device 200 determines whether the display 101 meets a first set of criteria for being available to receive one or more user interface objects that are moved between the display 101 and the display 201. In some embodiments, the first set of criteria includes the device sharing a user account 193 and / or having a trusted (195) connection 194, or being able to establish such a trusted connection. In some embodiments, the first set of criteria includes the device having established communication in a shared display mode, as described above. In some embodiments, the first set of criteria includes the device having established communication in an extended display mode.In some embodiments, the first set of criteria includes that the distance between device 200 and device 100 is below a threshold distance (such as determined based on Bluetooth range or based on ultra-wideband (UWB) positioning technology). In some embodiments, the first set of criteria includes that both the display 101 of device 100 and the display 201 of device 200 display a user interface that enables the display of movable user interface objects (e.g., devices 100 and 200 are not in sleep mode, or display a wake screen user interface or a lock screen user interface). In response to a determination that the first set of criteria for being able to move user interface objects between devices is met, device 200 displays an indication (e.g., indication 6014-A) on a first area of display 201. In some embodiments, indication 6014-A is displayed adjacent to edge 201-1 of display 201. In some embodiments, indication 6014-A has an appearance similar to indication 5014-A described with respect to FIG. 5B. In some embodiments, indication 6014-A corresponds to indication 5014-A. For example, indication 6014-A indicates the position of a portal that enables an object to be moved from display 201 to display 101. In response to a determination that the first set of criteria for being able to move user interface objects between devices is met, display 101 displays indication 6014-B in a first area of display 101. In some embodiments, indication 6014-B is displayed adjacent to edge 101-1 of display 101 (e.g., edge 101-1 opposite edge 201-1 of display 201). In some embodiments, indication 6014-B has an appearance similar to indication 5014-B described with respect to FIG. 5B. In some embodiments, indication 6014-B corresponds to indication 5014-B.For example, indication 6014-B indicates the position of a portal that enables an object to move from display 201 to display 101. As shown in FIG. 6C, indication 6014 is displayed at a position corresponding to each part of displays 201 and 101 corresponding to a portal that enables a user interface object to move between displays.

[0206] In FIG. 6D, device 100 has been moved to a different side of device 200 (e.g., device 100 faces edge 201-2 of display 201 that is opposite edge 201-1). In some embodiments, the relative positions of devices 100 and 200 are determined based on Bluetooth, Wi-Fi, or UWB signals. In some embodiments, the relative positions of devices 100 and 200 are determined based on user input (e.g., while displaying a display preference user interface as described with respect to FIGS. 7A-7G). As shown in FIG. 6D, device 200 displays indication 6014-D on a second part of display 201 (e.g., along edge 201-2 facing device 100). In some embodiments, device 100 displays indication 6014-C on a second part of display 101 (e.g., along edge 101-2 facing device 200). As shown in FIGS. 6C-6D, device 200 and optionally device 100 display an indication to indicate that it is possible to move a user interface object between devices in the respective display areas corresponding to the portal (e.g., the respective edge portions that can be used to drag a user interface object between displays).

[0207] Figures 6E - 6F show switching of the display modes of a first device and a second device communicating with each other in the display settings user interface according to some embodiments. In Figure 6E, device 200 displays the display settings user interface 6016. In some embodiments, the display settings user interface 6016 is displayed in response to user input for an affordance or combination of keystrokes corresponding to a request to the display settings user interface 6016. The display settings user interface 6016 includes a representation of device 200 (e.g., representation 6018 - A) and a representation of device 100 (e.g., representation 6018 - B). In some embodiments, the representation includes a text identifier (e.g., "Joe's tablet and Joe's desktop computer"). In some embodiments, the representation displays a pattern and / or figure corresponding to the wallpaper of each device. In Figure 6E, device 200 and device 100 are in a shared input mode. In some embodiments, the mode is indicated by a text indication (e.g., "Shared input mode") on the display settings user interface 6016. Also, as shown in Figure 6E, representation 6018 - A has a horizontal stripe pattern corresponding to the wallpaper displayed on the home screen user interface 5010 of device 200, and representation 6018 - B has a diagonal stripe pattern corresponding to the wallpaper displayed on the home screen user interface 5020 of device 100. In Figure 6E, representations 6018 - A and 6018 - B are displayed separately from each other. In some embodiments, representation 6018 - B is separated from representation 6018 - A by a distance greater than a predetermined distance. In Figure 6E, device 200 detects user input (e.g., pressing mouse 202 and subsequently moving mouse 202 in the direction of the arrow) while cursor 5002 is displayed on representation 6018 - B. In some embodiments, the user input in Figure 6E corresponds to a request to select representation 6018 - B and move representation 6018 - B according to the movement of mouse 202. In Figure 6F, representation 6018 - B is moved to be adjacent to (e.g., in direct contact with) representation 6018 - A.In some embodiments, when representation 6018-B reaches a predetermined distance from representation 6018-A, the device 200 displays an animation that causes the representations to "snap" into contact with each other (e.g., the representations slide together). In some embodiments, user input for moving the representations adjacent to each other corresponds to a request to change the mode from the shared input mode to the expanded display mode, as shown in FIG. 6F. In FIG. 6F, in response to changing the mode to the expanded display mode, the display 101 displays a home screen user interface 5010 and a menu bar 5008 corresponding to the home screen user interface and menu bar displayed on the display 201. At the same time, the appearance of representation 6018-B changes to reflect the appearance of the display 101 (e.g., representation 6018-B has a horizontal stripe pattern corresponding to the wallpaper of the home screen user interface 5010).

[0208] Figures 6G-6H illustrate that while a first device is also communicating with a second device, an indication is displayed on a first display of the first device when a third display of a third display is available to move content from the first display to the third display. In Figure 6G, the laptop device 300 is positioned at a distance greater than a threshold distance (e.g., distance D) for establishing communication (e.g., in a shared input mode or an extended display mode) between devices 200 and 300 from the desktop device 200. Thus, in Figure 6G, device 300 is not communicating with device 200 in a shared input mode or an extended display mode. Instead, device 300 operates its own operating system. As shown, device 300 displays a system-level user interface (e.g., home screen user interface 5024), a dock (e.g., dock 5022), and a menu bar (e.g., menu bar 5028) that are different from the home screen user interface, dock, and menu bar displayed by the display 201 of device 200. In Figure 6H, device 300 is moved closer to device 100 such that the distance between the devices is less than the threshold distance D for establishing communication (e.g., in a shared input mode or an extended display mode) between devices 200 and 300. In accordance with the determination that device 300 is within the threshold distance D for establishing communication, and in accordance with the determination that any other criteria of a first set of criteria for moving user interface objects between device 200 and device 300 are satisfied, device 200 displays an indication 6014-D within a second region of display 201 (e.g., along edge 201-1 of display 201). In some embodiments, display 201 of device 200 also displays an indication (e.g., indication 6014-E along edge 301-1 facing device 200) within a first region of display 301.

[0209] Figures 6I - 6K show user input for moving an object (e.g., a user interface object) from the first display of a first device to the second display of a second device when the first device and the second device are in a shared input mode. In Figure 6I, the display 201 of device 200 shows a photo application user interface 5016 that includes photo objects 5004 - A and 5004 - B. The display 201 of device 100 shows a memo application user interface 6020. Devices 200 and 100 are in a shared input mode. In Figure 6I, as shown by the gray highlighting, photo objects 5004 - A and 5004 - B are selected, and cursor 5002 is positioned on the selected photo object 5004 - B. For example, multiple user interface objects may be selected by user input corresponding to a drag (e.g., moving mouse 202 while holding down a press on mouse 202) over an area that includes the multiple user interface objects. In Figure 6I, while cursor 5002 is shown over the selected photo objects 5004 - A and 5004 - B, device 200 detects user input (e.g., a press on mouse 202 followed by movement in the direction indicated by the arrow). In some embodiments, the user input of Figure 6I corresponds to a request to move the selected photo objects 5004 - A and 5004 - B according to the movement of mouse 202. In some embodiments, indication 6014 - A is displayed along the edge 201 - 1 of display 201 to indicate that the user interface object can be moved from display 201 across the portal indicated by indication 6014 - A to the display 101 of device 100. In Figure 6J, the selected photo objects 5004 - A and 5004 - B are being moved towards the edge 201 - 1 of display 201 according to the movement of mouse 202.In FIG. 6K, the selected photo objects 5004-A and 5004-B have been moved to the central portion of the display 101 on the memo application user interface 602. In FIG. 6K, the device further detects the release of a user input (e.g., the lift-off of a press gesture on the mouse 202). In response to this release, the photo objects 5004-A and 5004-B are positioned at a position corresponding to the position of the cursor 5002 when the user input was released.

[0210] Figures 6L - 6M show user input for attempting to move an object from a first display of a first device to a second display of a second device across a location outside a portal indicated by an indication, according to some embodiments. As described above, in some embodiments, an indication such as 6014 - A described with respect to FIG. 6C indicates the location of an individual edge portion that enables dragging of a user interface object between displays of different devices. As a result, in some embodiments, a user interface object cannot be dragged between displays of different devices at a location outside these indications. In FIG. 6L, the display 201 of device 200 displays a photo application user interface 5016 including photo objects 5004 - A and 5004 - B, and the display 201 of device 100 displays a memo application user interface 6020. Devices 200 and 100 are in a shared input mode. In FIG. 6I, in FIG. 6L, while the cursor 5002 is displayed over the photo object 5004 - B, device 200 detects a user input (e.g., pressing of the mouse 202 and subsequent movement in the direction indicated by the arrow). In some embodiments, the user input in FIG. 6L corresponds to a request to select and move the photo object 5004 - B in accordance with the movement of the mouse 202. In some embodiments, the indication 6014 - A is displayed along the edge 201 - 1 of the display 201 to indicate that a user interface object can be moved from the display 201 across the inter - display portal indicated by the indication 6014 - A to the display 101 of device 100. However, in FIG. 6L, the movement of the mouse 202 in the direction indicated by the arrow has a projection direction towards an edge region of the display 201 outside the indication 6014 - A (e.g., the upper region of the edge 201 - 1). In FIG. 6M, the photo object 5004 - B is being moved towards the upper region of the edge 201 - 1 of the display 201 in accordance with the movement of the mouse 202.As shown in FIG. 6M, since the photo object 5004-B has reached the outer edge 201-1 of the inter-display portal indicated by the indication 6014-A, the photo object 5004-B cannot be dragged across the edge 201-1 to the display 101 of the device 100.

[0211] Figures 6N - 6O show user input on the touchpad of a third device to move an object from the first display of a first device to the second display of a second device when the first device is in a shared input mode with the second and third devices. As described above, two or more display devices can operate in communication with each other (e.g., in a shared input mode or a companion display mode) while sharing one or more input devices. In some embodiments, one or more input devices communicate with a single display device. For example, a keyboard 203 and a mouse 202 that communicate with a desktop device 200 are shared among the desktop device 200, a tablet device 100, and a laptop device 300. In some embodiments, one or more input devices communicate with two or more display devices. For example, in addition to the keyboard 203 and the mouse 202, the desktop device 200, the tablet device 100, and the laptop device 300 may also be operated by the touchpad 309 and the keyboard 305 of the laptop device 300, as shown in Figure 6N. In Figure 6N, device 200 is communicating with devices 100 and 300 in a shared input mode. In some embodiments, device 200 is communicating with devices 100 and 300 in an extended display mode (e.g., displays 201, 101, and 301 all display an extended view of the user interface provided by device 200). In any of these embodiments, devices 200, 300, and 100 can all be input devices (e.g., mouse 202, keyboard 203, keyboard 305, and touchpad 309). In some other embodiments, device 200 communicates with device 100 in a shared input mode and with device 300 in an extended display mode, or vice versa.

[0212] In FIG. 6N, device 200 displays a memo application user interface 6022 that includes text arranged in two lines, and device 100 displays a memo application user interface 6020. As shown, text line 5032 is selected as indicated by the highlighting. In FIG. 6N, while cursor 5002 is displayed on selected text line 5032, device 300 detects a user input (e.g., contact 6026) on touchpad 309. For example, the user input corresponds to a press of contact 6026 followed by a movement in the direction indicated by the arrow. In some embodiments, the user input corresponds to a request to move (e.g., drag) the selected text line 5032 in accordance with the movement of contact 6026 on touchpad 309 (e.g., in the direction indicated by the arrow). In some embodiments, indication 6014-A is displayed along edge 6014-1 of display 201 to indicate that a user interface object can be moved from display 201 of device 100 to display 101 across the portal indicated by indication 201-A. In some embodiments, device 200 displays indication 6014-A along edge 201-1 in response to device 300 detecting that the movement of contact 6026 for dragging text line 5032 is towards edge 201-1. In some embodiments, simultaneously with displaying indication 6014-A, device 100 displays indication 6014-B along edge 101-1. In FIG. 6O, text line 5032 is moved to the central portion of display 101 on memo application 6020 in accordance with the movement of contact 6026. As described above, in response to device 300 detecting the end of the user input (e.g., lift-off of contact 6026), text line 5032 is dropped at the location of cursor 5002 at the time the end of the user input is detected.

[0213] FIG. 6P shows user input in a predetermined manner on the touchpad of a third device that highlights the second display of the second display when the cursor is displayed on the second display according to some embodiments. In some cases, when operating multiple display devices simultaneously, the user may find it difficult to recognize the display on which the cursor 5002 is currently displayed, and a way to provide an indication of such a display to the user is useful. In FIG. 6P, the device 300 detects user input (e.g., contact 6028) on the touchpad 309. The user input includes moving the contact 6028 on the touchpad 309 in a predetermined movement. For example, the movement includes circular movement that moves the cursor 5002 in a circular movement (e.g., the cursor 5002 is hovering over an area on the display 101). In response to identifying that the user input includes movement in a predetermined movement and that the cursor 5002 is displayed on the display 101 of the device 100, the display 101 displays an indication indicating that the cursor 5002 is located on that display. In some embodiments, the indication includes increasing the brightness of the display 101 or a portion of the display 101. In FIG. 6P, the indication includes a brightened rim 6032 of the display 101.

[0214] Figures 6Q - 6T illustrate user input on the touchpad of a third device that causes operations on a second display and a first display based on where a cursor is displayed when user input is received, according to some embodiments. In Figures 6Q - 6T, device 300 is communicating with devices 100 and 200 in a shared input mode. In Figure 6Q, device 200 is displaying a home screen user interface 5010 and device 100 is displaying a home screen user interface 5020. In Figure 6Q, device 300 detects user input (e.g., contact 6030) on touchpad 309 while cursor 5002 is displayed on application icon 6012 - 1 (e.g., a photo application icon) of application icon 6012 on home screen 5020 on display 101. For example, the user input corresponds to a press of contact 6030 followed by a movement in the direction indicated by the arrow. In some embodiments, the user input corresponds to a request to select and move (e.g., drag) application icon 6012 - 1 in accordance with the movement of contact 6030 on touchpad 309 (e.g., in the direction indicated by the arrow). In Figure 6R, device 100 is moving application icon 6012 - 1 to the central portion of display 101 in accordance with the movement of contact 6030. As described above, in response to detection of the end of user input by device 300 (e.g., lift - off of contact 6030), application icon 6012 - 1 is dropped at the location of cursor 5002 at the time when the end of user input is detected.

[0215] In FIG. 6S, device 200 displays photo application user interface 5016 on top of home screen user interface 5010. The photo application user interface 5016 includes a plurality of representations corresponding to photos (e.g., photo object 5004). In FIG. 6S, while cursor 5002 is displayed on photo object 5004 on photo application user interface 5016, device 300 detects a user input (e.g., contact 6031) on touchpad 309. For example, the user input corresponds to a tap of contact 6031 (e.g., a quick touch and lift-off at the location corresponding to contact 6031). In some embodiments, the tap user input corresponds to a request to select and display (e.g., open) photo object 5004. In FIG. 6T, in response to detecting the tap user input, device 200 displays photo 5004 on display 201.

[0216] Figures 6U - 6V show user input for displaying a control panel on a first display of a first device according to some embodiments, the control panel including a plurality of affordances for controlling display settings. In Figure 6U, device 200 displays a home screen user interface 5010 including a menu bar 5008. The menu bar 5008 includes a plurality of indications (e.g., status indications) and a plurality of affordances including an affordance 5008 - 1 for a control panel user interface. As used herein, a control panel user interface (also referred to as a control center user interface or a control UI) is used to control a plurality of system - level operations. The control panel user interface includes a plurality of controls (e.g., affordances) corresponding to a plurality of system functions of the device. In Figure 6U, device 200 detects user input (e.g., a press 6034 on mouse 202) via mouse 202 while cursor 5002 is displayed over affordance 5008 - 1. In some embodiments, the user input corresponds to a request to display a control panel user interface (e.g., control panel user interface 5036 of Figure 6V). In response to detecting user input via mouse 202, device 200 displays control panel user interface 5036 as shown in Figure 6V. The control panel user interface includes an affordance 5036 - 1 for controlling a wireless network, an affordance 5036 - 2 for controlling silencing of calls and notifications, an affordance 5036 - 3 for controlling the brightness of a keyboard, an affordance 5036 - 4 for display preferences, and an affordance 5036 - 5 for controlling the brightness of a display. These affordances are exemplary, and it should be understood that control panel 5036 may also include different sets of affordances.The affordance 5036-4 for display preferences includes a selectable affordance 5038 for opening a display preference user interface (as described with respect to, for example, FIG. 7A). The affordance 5036-4 further includes a plurality of slide affordances (e.g., affordance 5039) for toggling different display modes on and off. For example, in FIG. 6V, the affordance associated with the shared input mode (e.g., "share mouse and keyboard") is turned on, and the affordances associated with the extended display mode and the mirroring display mode (e.g., "display as extended display" and "display as mirror display", respectively) are turned off.

[0217] Figures 6W - 6X illustrate updating a touch screen bar of a third device in response to user input on a touch pad of the third device based on where a cursor is displayed when user input is received, according to some embodiments. In some embodiments, the tablet device 300 includes a touch screen bar (also referred to as a touch bar) (e.g., touch screen bar 6040 positioned adjacent to the keyboard 305). As referred to herein, the touch screen bar is an input device having a function of displaying a plurality of changing affordances for causing operations on the device 300 and / or the devices 100 and 200 communicating with the device 300. User input on the plurality of changing affordances can be touch gestures including, for example, tap, swipe, flick, drag gestures. In some embodiments, the plurality of affordances (e.g., selectable icons) are displayed according to the currently active user interface. For example, when the currently active user interface is an application user interface, the touch screen bar displays a first plurality of affordances for operating the application user interface, and when the currently active user interface is a home screen user interface, the touch screen bar displays a second plurality of affordances different from the first set of affordances for operating the home screen user interface. In some embodiments, the user interface is determined to be active when a cursor is displayed on the user interface.

[0218] In FIG. 6W, cursor 5002 is displayed on home screen user interface 5024 of device 300, and touch screen bar 6040 displays a first set of one or more affordances (e.g., affordance 6040-1 for performing an operation on home screen user interface 5024 of device 300). Device 300 detects a user input (e.g., contact 6042) on touch pad 309. For example, the user input corresponds to a press by contact 6042 and subsequent movement in the direction indicated by the arrow. In some embodiments, the user input corresponds to a request to move cursor 5002 in accordance with the movement of contact 6042. In FIG. 6X, cursor 5002 is moved to home screen user interface 5020 of display 101. In response to moving cursor 5002 to a different user interface, touch screen bar 6040 is updated to display a second set of one or more affordances (e.g., affordance 6040-2 for performing an operation on home screen 5020 of device 100).

[0219] Figures 7A-7G illustrate changing the display modes of a first device and a second device communicating with each other by changing the positions of the respective representations of the first device and the second device within a display preference user interface, according to some embodiments. In Figure 7A, the display 201 of the desktop device 200 displays a display preference user interface (e.g., display preference user interface 7002). In some embodiments, the display preference user interface 7002 is displayed in response to user input on an affordance within a control panel user interface (e.g., affordance 5038 within the control panel user interface 5036 of Figure 6V). The display preference user interface 7002 provides affordances for adjusting properties and settings related to the display. In some embodiments, the display preference user interface 7002 includes a plurality of tabs for adjusting different aspects of the display. For example, tab 7002-1 is for adjusting different aspects of an individual display, tab 7002-2 is for adjusting the arrangement of displays when an individual display is communicating with one or more displays, and tab 7002-3 is for adjusting the color settings of an individual display.

[0220] FIG. 7A shows the content of tab 7002-2 for adjusting the arrangement of a plurality of displays. Tab 7002-2 includes a representation 7004-A corresponding to a desktop device 200 (e.g., named "Joe's desktop computer") and a representation 7004-B corresponding to a tablet device 100 (e.g., named "Joe's tablet"). In FIG. 7A, device 200 is communicating with device 100 in a shared input mode. Representation 7004-A has an emphasized rim, indicating that representation 7004-A corresponds to display 201 on which the display preference user interface 7002 is currently being displayed (e.g., display 201 is the currently active display). Device 200 displays the home screen user interface 5010, and device 100 displays the home screen user interface 5020. In the display preference user interface 7002, to indicate that devices 200 and 100 are in the shared input mode, representation 7004-A has a first appearance (e.g., size, shape, photo, or pattern), and representation 7004-B has a second appearance different from the first appearance. In some embodiments, the sizes of representations 7004-A and 7004-B correspond to the relative sizes of displays 200 and 100, respectively. For example, representation 7004-A has a larger size than representation 7004-B because display 201 has a larger size than display 101. In some embodiments, the shapes of representations 7004-A and 7004-B correspond to the respective shapes of displays 200 and 100. For example, representation 7004-B has a vertically elongated rectangular shape corresponding to the orientation of display 101, and representation 7004-A has a horizontally elongated rectangular shape corresponding to the orientation of display 201. In some embodiments, the photos or wallpapers of representations 7004-A and 7004-B correspond to the respective photos or wallpapers of displays 200 and 100.For example, representation 7004-A has a horizontal stripe wallpaper corresponding to the horizontal stripe wallpaper of the home screen user interface 5010 of device 200, and representation 7004-B has a diagonal stripe wallpaper corresponding to the diagonal stripe wallpaper of the home screen user interface 5020 of device 100. When in the shared input mode, representation 7004-B is separated from representation 7004-B by a certain distance (e.g., a distance greater than a threshold distance for changing between the shared input mode and the extended display mode).

[0221] In FIG. 7B, device 200 is communicating with device 100 in the extended display mode. In contrast to the shared input mode of FIG. 7A, both displays 101 and 201 display a home screen user interface 5010 that includes a menu bar 5008 provided by device 200. As a result, representation 7004-B changes its appearance to correspond to the wallpaper of the home screen user interface 5010. As shown, representation 7004-B has the same horizontal stripe wallpaper as representation 7004-A. Further, in FIG. 7B, representation 7004-B is adjacent to (e.g., in direct contact with) representation 7004-A (e.g., no gap between the representations is shown).

[0222] In FIG. 7C, device 200 is communicating with device 100 in mirror display mode. As shown, display 101 of device 100 displays a mirror image or replica of what is displayed on display 201 of device 200. Since display 101 has a smaller size than display 201, it is understood that some affordances and / or features are displayed at a smaller scale or excluded. As a result, when in mirror display mode, representations 7004-A and 7004-B are displayed overlapping (e.g., as a stack) in display preference user interface 7002 (e.g., representation 7004-B is displayed behind representation 7004-A). Representations 7004-A and 7004-B have corresponding appearances to indicate that representation 7004-B is actually a mirror image of representation 7004-A. For example, representations 7004-A and 7004-B have the same size, the same shape, and include the same horizontal striped wallpaper. FIGS. 7D-7E show changing the display mode from shared input mode to extended display mode by moving (e.g., dragging) representation 7004-A adjacent to representation 7004-B. In FIG. 7D, device 200 is communicating with device 100 in shared input mode, and display preference user interface 7002 is as described above with respect to FIG. 7A. In FIG. 7D, device 200 detects user input on mouse 202 while cursor 5002 is displayed over representation 7004-B. In some embodiments, the user input corresponds to a movement of mouse 202 in the direction indicated by arrow 7005-A following a press on mouse 202 (e.g., as indicated by the gray dot on mouse 202). In some embodiments, the user input corresponds to a request to select and move representation 7004-B following the movement of the mouse. In FIG. 7E, device 200 is moving representation 7004-B so that it is adjacent to (e.g., in direct contact with) representation 7004-A. As a result, device 200 is changing the display mode from shared input mode to extended display mode (e.g., as described with respect to FIG. 7B).In addition, the representation 7004-B and the display 101 change their wallpapers to correspond to the horizontal-striped wallpaper of the home screen user interface 5010 of the device 200. In some embodiments, the device 200 displays an animation that "snaps" representations that are in contact with each other when the representation 7004-B has moved a predetermined distance from the representation 7004-A (e.g., the representation 7004-B slides and contacts the representation 7004-A). It is understood that the display mode can be changed from the extended display mode to the shared input mode in accordance with a user input (e.g., a user input corresponding to a request to drag the representation 7004-B away from the representation 7004-A) that requires the representations 7004-A and 7004-B to be displayed separately from each other. In FIGS. 7D-7E, changing the display mode is shown by moving the representation 7004-B relative to the representation 7004-A while the representation 7004-B remains in its original position. However, it should be understood that the display mode can be changed by moving either or both of the representations 7004-A and 7004-B.

[0223] In FIG. 7E, device 200 further detects user input on mouse 202 while cursor 5002 is displayed over representation 7004-B. In some embodiments, the user input corresponds to a movement of mouse 202 in the direction indicated by arrow 7005-B following a press on mouse 202 (e.g., as indicated by the gray dot on mouse 202). In some embodiments, the user input corresponds to a request to select and move representation 7004-B in accordance with the movement of the mouse. In FIG. 7F, device 200 continues to detect user input. In FIG. 7F, the user input includes a movement of mouse 202 in the direction indicated by arrow 7005-B. While representation 7004-B is being moved and not in contact with representation 7004-A, devices 200 and 100 are in a shared input mode (e.g., each displaying their respective home screen user interface). In FIG. 7G, device 200 moves representation 7004-B to the opposite side of representation 7004-A in accordance with the movement of mouse 202. Additionally, the user positions device 100 on the opposite side of device 200. As indicated by the arrangement and appearance of representations 7004-A and 7004-B, device 200 is currently communicating with device 100 in a shared input mode.

[0224] Figures 7H - 7I illustrate changing the display modes of a first device and a second device that are communicating with each other using affordances displayed on a representation of the second device within a display preference user interface, according to some embodiments. Figures 7H - 7I illustrate yet another way to change the display modes of devices 200 and 100 that are communicating with each other. In Figure 7H, device 200 displays an affordance 7006 for changing the display mode. In Figure 7H, an affordance 7006 for adjusting the display preference corresponding to display 101 of device 100 is displayed on representation 7004 - B. Similarly, affordance 7006 may alternatively or additionally be displayed on representation 7004 - A to adjust the display preference corresponding to display 201 of device 100. It is understood that the display mode may be changed by adjusting any of the display preferences of the displays. In Figure 7H, device 200 detects user input (e.g., a click) on mouse 202 while cursor 5002 is over affordance 7006. In some embodiments, the user input corresponds to a request to display a dropdown menu that includes a set of selectable affordances for selecting a display mode (e.g., "Share mouse and keyboard", "Use as external display", and "Use as mirror display"). For example, user input on "Share mouse and keyboard" (e.g., a click on mouse 202 while cursor 5002 is over the desired option) changes the display modes of devices 200 and 100 to the shared input mode. User input regarding "Use as external display" changes the display modes of devices 200 and 100 to the extended display mode such that display 101 operates as an extended display for displaying the user interface provided by device 200. User input on "Use as mirror display" changes the display modes of devices 200 and 100 to the mirror display mode such that display 101 displays a mirror image of display 201.

[0225] Figures 7J - 7K show user input on the representation of devices within a display preference user interface that provides an indication on the display of individual devices, according to some embodiments. In some cases, when operating multiple display devices simultaneously, the user may find it difficult to recognize the display on which the cursor 5002 is currently being displayed, and a method for providing an indication of such a display to the user is useful. In Figure 7J, device 200 is communicating with devices 300 and 200 in a shared input mode. The display preference user interface 7002 includes representations 7004 - A, 7004 - B, and 7004 - C corresponding to devices 200, 100, and 300, respectively. In Figure 7J, device 200 detects user input via mouse 202 while the cursor 5002 is displayed over representation 7004 - B. The user input includes moving the mouse 202 in a predetermined movement. For example, the movement includes a circular movement that moves the cursor 5002 in a circular motion (e.g., the cursor 5002 is hovering over representation 7004 - B). In response to identifying that the user input includes movement in a predetermined movement and that the cursor 5002 is displayed over representation 7004 - B, display 101 displays an indication that the cursor 5002 is located on display 101. In Figure 7K, the indication includes a highlighted bezel region 7008 of display 101 (e.g., the region of the display adjacent to the bezel of device 100). In some embodiments, the indication includes increasing the luminance of display 101, a portion of display 101, or the rim of display 101. In some embodiments, the indication is a text notification (e.g., a pop - up window).

[0226] Figure 7L shows adjusting the display of different features in a display preference user interface according to some embodiments. In Figure 7L, device 200 displays tab 7002-1 of display preference user interface 7002 for adjusting different modes of display 201 of device 200. Tab 7002-1 includes affordance 7010-1 for displaying a control strip (such as shown in Figure 7R) for accessing modifier keys frequently used on a touch screen display, affordance 7010-2 for enabling and disabling a touch screen bar (such as touch screen bar 6040 shown in Figures 6W - 6X), affordance 7010-3 for enabling and disabling using a wireless stylus pen work as an input device (such as shown in Figure 7R), and affordance 7010-4 for enabling and disabling content casting (such as enabling and disabling a nearby device that plays the content currently being played on the device), etc., including a plurality of affordances for enabling and disabling specific operations or features available on device 200.

[0227] Figures 7M through 7Q illustrate the streaming of content (e.g., display of video content) from the display of a second device to the display of a first device when the device is in a shared input mode, according to some embodiments. In Figure 7M, device 200 is communicating with device 200 in a shared input mode. Device 100 is playing video content in video user interface 7012. Video user interface 7012 includes a set of controls such as affordance 7012-1 for turning off the video content, affordance 7014-2 for playing the video content, affordance 7014-3 for pausing the video content, and affordance 7014-4 for casting the currently playing video content to a different device that is communicating with device 100. In Figure 7M, device 100 detects a user input (e.g., a tap gesture) on affordance 7014-4 to enable casting the video content to a different device. In response to the tap gesture on affordance 7014-4, device 100 displays a user interface that includes affordance 7016 for selecting the device to which the content will be cast. In Figure 7N, device 100 detects that device 200, identified as "Joe's desktop computer", has been selected. In response to the selection, device 200 streams the video content displayed by device 100 (e.g., displays 101 and 201 play the same video content simultaneously). Such streaming enables the user to enjoy watching the video content from device 100 on the larger-sized display of device 200. While device 200 is streaming the video content, other operations of device 200 are set to a lock mode. For example, when in the lock mode, it is necessary to obtain authentication information from the user (e.g., through password input or biometric verification) to perform any operation on device 200 other than streaming the video content.In FIG. 7O, the device 200 detects a user input (e.g., a keystroke by contact 7018) corresponding to a request to unlock the device 200. In response to detecting the user input, the device 200 stops displaying video content on the video user interface 7012. Instead, the device 200 displays a lock screen user interface (e.g., lock screen user interface 7020) that includes an affordance for entering a passcode (e.g., passcode input affordance 7022). In some embodiments, the device obtains biometric information (e.g., fingerprint or face recognition) from the user to unlock the device 100. In response to obtaining the passcode (e.g., by entering the passcode via user input on the keyboard 203), the device 200 ends the lock screen user interface 7020 and displays the home screen user interface 5010 as shown in FIG. 7Q.

[0228] Figure 7R shows a second display that presents a control strip and receives input via a stylus input device, according to some embodiments. In some embodiments, the control strip and stylus input device are enabled using affordances on a display preference user interface (e.g., as described above with respect to Figure 7L). In Figure 7R, device 100 presents control strip 7028 on display 101. In Figure 7R, control strip 7028 is a bar positioned along edge 101-1 of display 101. In some embodiments, control strip 7028 can be positioned on any other edge of display 101. Control strip 7028 includes a plurality of icons corresponding to frequently used modifier controls (e.g., commands, options, controls, shift, undo, redo). By presenting these frequently used modifier controls on control strip 7029, a user can access such controls without having to navigate through embedded menus to access these controls when operating multiple displays. In Figure 7R, device 100 also presents photo application user interface 7024 and detects user input via stylus 7026. In some embodiments, stylus 7026 operates as an input device (e.g., as a wireless pen). When the stylus is in physical contact with display 101 and dragged along display 101 or moved a predefined distance over the display (e.g., a hover movement where the cursor is over the display but not in contact with the display), cursor 5002 follows the movement of stylus 7026. In some embodiments, stylus 7026 is used to control the movement of cursor 5002 (e.g., instead of a mouse or touch contact by a user's finger that controls the movement of cursor 5002).

[0229] Figures 7S - 7T illustrate user input on a second display (e.g., a touch screen) of a second device that causes an operation on a first display of a first device according to some embodiments. In some embodiments, when a mobile device is communicating with a desktop or laptop device in a shared input mode, operations on the desktop or laptop device can be performed by touch gesture inputs received on a touch - sensitive display of the mobile device (e.g., a tablet device or a mobile phone device). In FIG. 7S, device 200 is communicating with device 100 in a shared input mode. In FIG. 7S, cursor 5002 is displayed over an application icon (e.g., photo application icon 5006 - 1) on dock 5006. While cursor 5002 is displayed over photo application icon 5006 - 1, device 100 detects a touch input (e.g., contact 7030) on display 101 (e.g., a touch - sensitive display). For example, the touch input of contact 7030 corresponds to a tap gesture. In some embodiments, the touch input of contact 7030 corresponds to a request to select and display the application corresponding to application icon 5006 - 1 over which cursor 5002 is displayed. In FIG. 7T, in response to detecting a touch input on display 101 of device 100, device 200 displays photo application 5016. It is understood that other operations of device 200 can be performed in response to gesture inputs received on display 101 of device 100, similar to displaying an application user interface in response to a tap gesture while cursor 5002 is over an individual application icon. For example, gesture inputs received on display 101 of device 100 can be used to provide a selection on an affordance or to move a user interface object. The operations performed are selected based on the gesture input and the location of the cursor when the gesture input is received.

[0230] Figures 7U through 7Z illustrate user input on a keyboard for causing a search user interface to be displayed on a first display of a first device or a second display of a second device based on a cursor position when the first device and the second device are in a shared input mode, according to some embodiments. Generally, a user can conveniently display frequently used user interfaces by providing user input that includes a combination of keystrokes. When operating sharing in a multi-device input mode, the same user input can be used to display the same frequently used user interface on the device that is displaying the cursor when receiving the user input. In some embodiments, a combination of keystrokes (e.g., command + space) causes the device to display a search user interface (e.g., for searching for content from the device and / or network). In FIG. 7U, device 200 is communicating with device 100 in a shared input mode. In FIG. 7U, device 200 detects a user input corresponding to a combination of keystrokes (e.g., contact 7032 corresponding to simultaneously pressing the command and space keystrokes). The user input is detected while cursor 5002 is displayed on display 101 of device 100. In some embodiments, a user input that includes contact 7032 for simultaneously pressing the command and space keystrokes corresponds to a request to display a search user interface. In response to detecting the user input, device 100 displays corresponding text.

[0231] In response to receiving user input via FIG. 7U, device 100 displays a search user interface 7034 (e.g., a search input area or search bar) on the home screen user interface 5020. While cursor 5002 is displayed on search user interface 7034, device 200 further detects a user input (e.g., a text user input) corresponding to a text string (e.g., "APPL"). For example, the user input includes a plurality of key strokes (e.g., pressing of the key strokes by contact 7036) corresponding to the search term "APPL", as shown in FIG. 7V. In response to the detection of the user input, the corresponding text string is displayed on search user interface 7034.

[0232] In response to receiving text input at the search user interface 7034, the device 100 (e.g., the search module 151 of the device 100) uses the text input as a search criterion (e.g., optionally, together with other context information (e.g., time, location, past searches, past user interactions, etc.) as supplementary search criteria and / or search filters) to perform a search to identify relevant content corresponding to the search criteria. In some embodiments, the search is performed in a search corpus corresponding to different sources of content, including content associated with applications installed on the device (e.g., content and / or data within the application (e.g., files, messages generated or stored within the application), metadata associated with the application (e.g., application name, application icon, etc.)), content from external sources (e.g., the Internet, other related devices connected to the device, etc.), files stored on the device and / or stored on a user account associated with the device. In some embodiments, the search is performed in a search corpus corresponding to different categories or content types for search results, including images, photos, videos, media files, contacts having contact information (e.g., name, address, username, alias, web address, social media handle, etc.), applications, actions or operations that can be performed on the device. In some embodiments, the search is updated as the user types the input (e.g., without the user having to select "search" or "return"). In response to detecting a search input (e.g., partial or complete), the search user interface 7034 updates (e.g., refreshes or replaces) with search results (e.g., search results 7038) corresponding to the detected search input, as shown in FIG. 7W. The search results can include content from various applications on the device 100 identified as being related to the received search input. In FIG. 7W, the search results include a photo object (e.g., search results 7038 including the photo object "apple.jpeg"). Figures 7X - 7Z show the same search operations as described with respect to Figures 7U - 7W. In Figure 7X, while the cursor 5002 is displayed on the display 201 of the device 200, a user input received via the keyboard 203 for displaying the search user interface (e.g., contact 7032 corresponding to pressing the command and space key strokes simultaneously) is detected. In response to the user input detected in Figure 7X, the device 200 displays a search user interface 7043 (e.g., a search input area) on the display 201, as shown in Figure 7Y. In some embodiments, the search user interface 7034 is displayed on top of any concurrently displayed user interface (e.g., the photo application user interface 5016 in Figure 7Y). In Figure 7Y, the device 200 detects the same text user input (e.g., the text string "APPL") as in Figure 7V. In response to receiving the text input in the search user interface 7034, the device 200 (e.g., the search module 151 of the device 200) performs a search using the text input as a search criterion, as described above with respect to the search performed by the device 100. In response to detecting the search input, the search user interface 7034 is updated with search results (e.g., search results 7040) corresponding to the detected search input, as shown in Figure 7Z. In Figure 7Z, the search results include content from various applications on the device 200, identified as being related to the received search input. In Figure 7Z, the search results include a presentation ("Apple Presentation.key") and a text file ("Apples.txt").

[0233] Figures 8A - 8C show a user input for displaying the movement of a cursor from the middle part of the first display area to the edge area of the first display area of a first computer system according to some embodiments, and for displaying an indication showing the location of a portal for moving a user interface object between the first display area and the second display area of a second computer system. In FIG. 8E, the desktop device 200 communicates with the tablet device 100 and the tablet device 100 - 1. The tablet devices 100 and 100 - 1 are positioned on the opposite side of the display 201 (for example, the device 100 faces the edge 201 - 1 of the display 201, and the device 100 - 1 faces the edge 201 - 2 of the display 201). As shown, the device 200 is displaying the home screen user interface 5010. The device 200 is in the sleep mode 6006 (for example, a power saving mode or a low power mode). In some embodiments, when in the sleep mode, the device turns off the display (for example, does not light up), thereby reducing the power consumption of the device. In FIG. 8A, the device 200 detects a user input including a movement in the direction indicated by the arrow shown (for example, pressing and dragging on the mouse 202). In some embodiments, the user input corresponds to a request to move the cursor 5002 in accordance with the movement of the mouse 202. In FIG. 8B, the device 200 is moving the cursor 5002 in accordance with the movement of the mouse 202 towards the edge 201 - 1 (for example, the edge area corresponding to the area near the edge 201 - 1). In response to the movement towards the edge 201 - 1, the device 200 facilitates waking up one of the devices that the device 200 is communicating with from the sleep mode 6006, and starts displaying a lock screen user interface (for example, the lock screen user interface 7020 on the display 101) according to a determination of which device is the most recently opened file device.In some embodiments, in accordance with the determination of the most recently opened file device, device 200 wakes up the most recently opened file device from sleep mode 6006 and provides the device with information to display the lock screen user interface 7020. In some embodiments, the most recently opened file device includes the device that most recently detected input from the user or the device that most recently had a cursor displayed thereon. In some embodiments, the most recently opened file device is not determined based on receiving a notification or generating a notification (e.g., related to an application). In FIG. 8B, device 100 is determined to be the most recently active device of devices 100 and 100-1, although it is understood that the device could have been device 100-1 (e.g., the relative position of the devices is not a determining factor for waking up the most recently opened file device). In some embodiments, device 100 ends the lock screen user interface 7020 in response to detecting user input and / or in response to receiving authentication information (e.g., a password or biometric information on the password input affordance 7022).

[0234] In FIG. 8C, device 100 exits the speed mode user interface 6006 and displays the home screen user interface 5020. As described above, when device 100 is displaying the home screen user interface 5020 and devices 100 and 200 are communicating directly or indirectly with each other (e.g., in a shared input mode or an extended display mode), user interface objects can be moved (e.g., dragged) between displays 101 and 201 via portals on the edges of displays 101 and 201. A portal refers to a portion of each edge of displays 101 and 201 that enables the movement of user interface objects between displays 101 and 201 (e.g., any of the operations described above with respect to FIGS. 5A-5T). In accordance with the determination that user interface objects can be moved between displays 101 and 201 via the portals, device 200 displays an indication (e.g., an indication user interface) indicating the location of such portals on display 201 and / or display 101. In some embodiments, the indication of the portal includes indication 8002-A (e.g., the first portion of the indication of the portal), which is displayed along the edge of display 201 (e.g., edge 201-1 in FIG. 8C) that can be crossed to move the user interface object from display 201 to display 101. In some embodiments, indications 8002-A and 8002-B are displayed while device 100 is in the lock mode. For example, indication 8002-B is displayed on the lock screen user interface 7020 of device 100 in FIG. 8B. In some embodiments, indication 8002-A corresponds to indication 5014-A described above with respect to FIGS. 5A-5C.In some embodiments, the portal indication includes an indication 8002-B (e.g., a second portion of the portal indication) that can be traversed to move a user interface object from display 101 to display 101 and is displayed along an edge (e.g., edge 101-1) of display 101. In some embodiments, indications 8002-A and 8002-B optionally include representations (e.g., representation 8004-A within indication 8002-A corresponding to a photograph representing device 100, and representation 8004-B within indication 8002-B corresponding to a photograph representing device 200) of devices for which individual portals can be used to drag a user interface object. In some embodiments, representation 8004-A is displayed in indication 8002-A, but representation 8004-B is not displayed. In some embodiments, representation 8004-B is displayed within indication 8002-B, but representation 8004-A is not displayed. In some embodiments, indications 8002-A and 8002-B include the names of devices for which a user interface object can be dragged using individual portals. In some embodiments, indications 8002-A and 8002-B have a length along the edge corresponding to the length of edge 101-1 of display 101, which is the smaller of the two displays. In some embodiments, indication 8002-A has a first length and indication 8000-B has a second length different from the first length. For example, indications 8002-A and 8002-B have a pre-set length when communication between the devices is established. In some embodiments, indications 8002-A and 8002-B have a pre-set location along their respective edges at the time when communication between the devices is established. However, as seen in FIG. 8C, in some cases, the locations and / or sizes of indications 8002-A and 8002-B do not match (e.g., are not aligned).In such a case, a cursor or user interface object being dragged between different displays does not move smoothly as projected by the movement of the mouse 202. For example, when the cursor moves from a first location on the display 201 to a first location on the display 201 following the linear continuous movement of the mouse 202, instead of linearly transitioning from the first location on the display 201 to the first location on the display 101, the cursor jumps between the displays. Such behavior may reduce the user experience of operating two displays simultaneously. The operations described with respect to FIGS. 8E through 8AJ show means for adjusting the position of the portal to display a continuous movement of an object between different displays.

[0235] FIG. 8D shows an edge region to which a cursor can reach to start displaying an indication of the location of a portal according to some embodiments. In some embodiments, displaying indication 7002-A and / or indication 7002-B is initiated according to a determination that cursor 5002 has been moved to the edge region of the display as the input moves. In FIG. 8D, device 200 detects an input on mouse 202 corresponding to a request to move cursor 5002 according to the movement of the mouse. In FIG. 8D, cursor 5002 is being moved from the central portion of display 201 towards edge 201-1 of display 201. In some embodiments, device 200 starts displaying indication 7002-A according to a determination that cursor 5002 has reached edge 201-1 (e.g., cursor 5002 is displayed adjacent to edge 201-1 or appears to touch edge 201-1). In some embodiments, device 200 starts displaying indication 7002-A according to a determination that cursor 5002 has reached a threshold distance (e.g., threshold distance T in FIG. 8D) (e.g., cursor 5002 is at least partially displayed within an edge region that extends a distance T from edge 201-1 towards the center of display 201). In some embodiments, device 200 starts displaying indication 7002-A according to a determination that cursor 5002 has reached a location on edge 201-1 (or an edge region extending from edge 201-1) and has remained at that location for a duration longer than a threshold duration for starting to display indication 7002-A (e.g., the movement of cursor 5002 has been paused). In some embodiments, indication 7002-B is displayed simultaneously with indication 7002-A (e.g., the display of indication 7002-B starts at the same time as the start of the display of indication 7002-A). In some embodiments, device 200 generates indications 7002-A and 7002-B.In some embodiments, device 200 provides device 100 with information to cause device 100 to display indication 8002-B at the same time that display 201 displays indication 8002-A.

[0236] Figures 8E - 8L show user input for changing the size and / or location of portal indications to move user interface objects between different displays, according to some embodiments. In Figure 8E, device 200 displays indication 8002 - A along edge 201 - 1 of display 201 and indication 8002 - B along edge 101 - 1 of display 101. In Figure 8E, while cursor 5002 is displayed on indication 8002 - A, device 200 detects input on mouse 202. In some embodiments, the input includes movement 8006 - A in the direction indicated by the arrow. In some embodiments, the input corresponds to a request to move indication 5002 - A and / or change its size according to the movement of mouse 202. In some embodiments, the input corresponds to a request to move indication 5002 - B and / or change its size according to the movement of mouse 202. In some embodiments, the input corresponds to a request to move indications 5002 - A and 5002 - B simultaneously and / or change their sizes. In some embodiments, the input corresponds to a request to move and / or change the size of an indication displayed on a larger - sized display 201 or display 101 (e.g., display 201 of device 200 has a larger size than display 101 of device 100 in Figure 8E). In Figure 8E, indication 8002 - A has size L1 (e.g., size L1 refers to the length of indication 8002 - A along edge 201 - 1), and indication 8002 - B has size L2 (e.g., size L2 refers to the length of indication 8002 - B along edge 101 - 1). In some embodiments, size L1 corresponds to size L2 (e.g., L2 corresponds to the full length of edge 101 - 1 of device 100). In some embodiments, size L1 is different from size L2 (e.g., L1 and L2 are based on pre - set lengths).

[0237] In FIG. 8F, device 200 detects movement 8006-A of mouse 202. In accordance with this movement, cursor 5002 and indication 8002-A are moved downward along edge 201-1 (e.g., the relative position of cursor 5002 on indication 8002-A is maintained). In FIG. 8G, device 200 detects further movement 8006-A of the mouse. In accordance with this movement, the cursor and indication 8002-A are below edge 201-1. As shown, indication 8002-A has reached the lower edge of display 201-1. In response to reaching the lower edge, device 200 reduces the size (e.g., the length along edge 201-1) of indication 8002-A such that the size of indication 8002-A in FIG. 8G is smaller than size L1 shown in FIG. 8E. In FIG. 8G, the upper edge of indication 8002-A has reached a location that substantially corresponds to the location of the upper edge of indication 8002-B in the vertical direction. In some embodiments, in response to determining that the upper edge of indication 8002-A has vertically reached the location of the upper edge of indication 8002-B, device 200 holds the location of the upper edge of indication 8002-A and the location of the upper edge of indication 8002-A (e.g., the location of the upper edge is “locked”) unless device 200 detects a user input that requests not to hold the location. For example, device 200 holds the location of the upper edge and continues to move the lower edge of each indication if device 200 continues to detect movement 8006-A of mouse 202. However, in response to detecting movement of mouse 202 in the opposite direction of movement 8006-A (e.g., the user wants to move cursor 5002 upward), device 200 releases the hold on the location of the upper edges of indications 8002-A and 8002-B.

[0238] Even after indication 8002-A reaches the lower edge of display 201, device 200 detects movement 8006-A (e.g., the user input continues in the direction indicated by the arrow). In response, while continuing to reduce the size of indication 8002-A, device 200 provides device 100 with information to reduce the size of indication 8002-B in device 100 (e.g., in FIG. 8G, the lower edge of indication 8002-B is moved upward). In some embodiments, device 200 further detects movement of mouse 202 in a direction opposite to the direction of movement 8006-A. In accordance with movement of mouse 202 in a direction opposite to movement 8006-A, device 200 facilitates resizing and / or changing the location of one or both of indications 8002-A and 8002-B. For example, in accordance with movement of mouse 202 in a direction opposite to movement 8006-A, device 200 facilitates increasing the size of indication 8002-A and / or moving indication 8002-A upward along edge 201-1. In some embodiments, the device continues to change the size and / or location of one or both of indications 8002-A and 5002-B in accordance with user input including movement in a direction corresponding to movement of cursor 5002 moving up or down within indication 8002-A. In some embodiments, after an individual indication reaches the upper or lower edge of the display, in response to device 200 detecting continued movement of mouse 202 in the direction of or beyond the reached edge, device 200 increases the size of the individual indication. For example, as shown in FIG. 8G, when indication 8002-A reaches the lower edge of display 201 and device 200 detects continuation of movement 8006-A (e.g., the user is attempting to drag indication 8002-A beyond the lower edge of display 201), device 200 expands the size of indication 8002-A.In some embodiments, displays 201 and 101 display the movement of indications 8002-A and 8002-B as an animation, as described with respect to FIGS. 8D-8G. For example, changes in the size and / or location of indications 8002-A and 8002-B are displayed as a continuous animation.

[0239] In FIG. 8H, indications 8002-A and 8002-B have reached their corresponding sizes and locations. For example, indication 8002-A has substantially the same size and substantially the same vertical location (e.g., a vertical location along parallel edges 201-1 and 101-1) as indication 8002-B. In some embodiments, the location and / or size are held by device 200 unless a user interface is detected that does not hold the location and / or size requirements. In FIG. 8H, device 200 detects a user input that includes a movement 8006-B having a direction that is substantially perpendicular to the direction of movement 8006-A. In some embodiments, movement 8006-B corresponds to a request to move cursor 5002 from indication 8002-A to indication 8002-B (e.g., across a portal between displays 201 and 101). In accordance with movement 8006-B, device 200 facilitates the display of movement cursor 5002 from indication 8002-A to indication 8002-B, as shown in FIG. 8I. In some embodiments, cursor 5002 has a first appearance on display 201 (e.g., cursor 5002 is the arrow in FIG. 8H), and cursor 5002 has a second appearance on display 101 (e.g., cursor 5002 is the dot in FIG. 8I). In FIG. 8J, device 100 (or alternatively, device 200) detects a user input that includes a movement 8006-C. In some embodiments, both device 200 and device 100 communicate with input devices (e.g., mouse 202 and keyboard 203). In some embodiments, an input received via an input device is detected by the individual device that is currently active (e.g., currently displaying cursor 5002). In some embodiments, an input received via an input device is detected by device 200, and device 200 provides information regarding the detected input to device 100 and causes device 100 to perform an operation on display 101 in accordance with the detected input when cursor 5002 is displayed on display 101.In some embodiments, a user input involving movement 8006-C corresponds to a request to move cursor 5002 in the direction of movement 8006-C (e.g., upward along indication 8002-C). In FIG. 8K, device 100 detects a user input including movement 8006-D. In some embodiments, a user input involving movement 8006-D corresponds to a request to move cursor 5002 in the direction of movement 8006-D (e.g., downward along indication 8002-C). In some embodiments, moving cursor 5002 up and down within indication 8002-A or indication 8002-B has a first movement resistance (e.g., interference or deceleration of movement), and moving the cursor from left to right or right to left such that cursor 5002 exits an individual indication has a second movement resistance. In some embodiments, the second movement resistance is greater than the first movement resistance. In some embodiments, movement resistance refers to decelerating or interfering with a proportional movement of the cursor in response to movement of the input. For example, the cursor moves proportionally less for a given amount of input movement (e.g., movement of contact on a touchpad or movement of a mouse) when the resistance is high than when the resistance is low. In some embodiments, due to normal resistance when mouse 202 moves a distance corresponding to distance X, cursor 5002 moves a distance Y on the display. At a higher resistance, when mouse 202 moves distance X, cursor 5002 moves distance Z, and distance Z is less than distance Y.

[0240] FIG. 8L shows providing an indication of the location of a cursor while it is displayed within an individual indication for a portal, according to some embodiments. In FIG. 8L, device 100 detects a pause in the movement of the mouse for a duration longer than a threshold duration for display of an indication of an indicator (e.g., indicator 8008) for indicating the location of cursor 5002. In response to detecting the pause, display 101 displays indicator 8008.

[0241] Figures 8M - 8Q show user input for causing an animated end of a cursor from an individual indication to a portal in accordance with some embodiments. In Figure 8M, device 100 detects user input including movement of mouse 202 in the direction indicated by the arrow (e.g., movement 8006 - E). In some embodiments, the user input corresponds to a request to move cursor 5002 away from edge 101 - 1 of display 101 (e.g., towards the central portion of display 101). In accordance with movement 8006 - E of mouse 202, display 101 displays cursor 5002 moving towards the central portion of display 101 such that cursor 5002 exits indication 8002 - B, as shown in Figures 8N - 8Q. In some embodiments, display 101 displays an animation including expanding a portion (e.g., portion 8010) when it displays cursor 5002 exiting indication 8002 - B. As shown in Figures 8N and 8O, portion 8010 of indication 8002 - B corresponding to the location of cursor 5002 stretches or expands as cursor 5002 moves towards the central portion of display 101. In Figure 8P, display 101 displays an instance where cursor 5002 has exited indication 8002 - B while portion 8010 of indication 8002 - B is still stretched. In Figure 8Q, portion 8010 of indication 8002 - B does not stretch (e.g., begins to return to its original size) while cursor 5002 is displayed outside of indication 8002 - B on home screen user interface 5020 of device 100.

[0242] As the cursor 5002 exits the indication 5002-B, the devices 100 and 200 maintain the size and location of the portals corresponding to the indications 8002-A and 8002-B. In some embodiments, in accordance with the determination that the cursor 5002 is no longer displayed on the indications 8002-A and 8002-B, the displays 201 and 101 stop displaying the indications 8002-A and 8002-B while maintaining the size and location of the corresponding portals. As shown in FIG. 8R, while the device 100 is detecting user input via the mouse 202 corresponding to the movement of the location of the application icon on the home screen user interface 5020 (e.g., selection by pressing and dragging in the direction of movement 8006-E indicated by the arrow), the displays 201 and 101 may stop displaying the indications 8002-A and 8002-B (as indicated by the dashed lines).

[0243] Figures 8S - 8T show moving a user interface object according to the movement of an input from a first display area to a second display area after the display areas are aligned, according to some embodiments. In Figure 8S, while devices 100 and 200 maintain the size and location of the portals corresponding to indications 8002 - A and 8002 - B, and cursor 5002 is displayed over photo object 5004, device 200 detects user input via mouse 202 including movement 8006 - F (e.g., after a press, followed by movement 8006 - F of mouse 202 in t...

Claims

Claim 1 A method, comprising: In a first computer system having a first display generation component, the first computer system communicating with a second computer system having a first input device and a second display generation component different from the first display generation component; Displaying, via the first display generation component, a first user interface object at a first location within a first display area provided by the first display generation component, the first user interface object being within the first display area; While displaying the first user interface within the first display area provided by the first display generation component, detecting, via the first input device, a first input corresponding to a first movement, the first movement including a first portion and a second portion, the first input including a request to drag the first user interface object across the first display area provided by the first display generation component according to the first portion of the first movement, and subsequently a request to drag the first user interface object from the first display area provided by the first display generation component to a second display area provided by the second display generation component according to the second portion of the first movement; In response to detecting the first movement; Based on a determination that the first user interface object is a representation of content and that the first input was detected while the second display generation component was communicating with the first computer system in a first mode; Moving the first user interface object from the first location to a second location within the first display area across the first display area provided by the first display generation component according to the first portion of the first movement; Moving the first user interface object across the second display area provided by the second display generation component according to the second portion of the first movement; in accordance with a determination that the first input is detected while the first user interface object is a representation of content and the second display generation component is communicating with the first computer system in a second mode different from the first mode, moving the first user interface object from the first location to the second location within the first display area provided by the first display generation component across the first display area according to the first part of the first movement; moving the first user interface object across the second display area provided by the second display generation component according to the second part of the first movement; in accordance with a determination that the first input is detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the first mode, moving the first user interface object from the first location to a second location within the first display area provided by the first display generation component across the first display area according to the first part of the first movement; moving the first user interface object across the second display area provided by the second display generation component according to the second part of the first movement; preventing the first user interface object from moving into the second display area provided by the second display generation component in accordance with a determination that the first input is detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the second mode. A method comprising: **Claim 2** Moving the first user interface object across the second display area includes moving the first user interface object from a first location on the second display area to a second location on the second display area, The method further includes detecting, via the first input device, a second input, wherein the second input is a second location on the second display area, and the second location on the second display area is determined according to a determination of a location for receiving the second input, and the second input corresponds to a request for positioning the first user interface object at the second location. The method according to claim 1.

3. When the first user interface object is a representation of content, maintaining the display of the first user interface object while the first user interface object is being moved across the first display area and the second display area. The method according to claim 1, further including.

4. When the first user interface object is an application window, maintaining the display of the application window as an open application window while the first user interface object is being moved. The method according to claim 1, further including.

5. Preventing the first user interface object from moving into the second display area includes automatically moving the first user interface object back from the second location to the first location across the first display area provided by the first display generation component. The method according to claim 1.

6. Preventing the first user interface object from moving into the second display area includes temporarily stopping the movement of the first user interface object at the second location. The method according to claim 1.

7. The method according to claim 1, wherein the first computer system communicates with the second computer system via a wired connection.

8. The method according to claim 1, wherein the first computer system communicates with the second computer system via a wireless connection.

9. The method according to claim 1, wherein the first computer system having the first display generation component further communicates with a third computer system having a third display generation component different from the first display generation component and the second display generation component.

10. Displaying a first visual indication in accordance with a determination that the first user interface object can be moved from the first display area onto the second display area while moving the first user interface object across the first display area provided by the first display generation component. The method according to claim 1, further comprising.

11. Displaying a second visual indication different from the first visual indication in accordance with a determination that the first user interface object cannot be moved from the first display area onto the second display area. The method according to claim 10, further comprising.

12. While moving the first user interface object across the first display area according to the first part of the first movement, in accordance with a determination that a first criterion for moving the first user interface object onto the second display area provided by the second display generation component or onto a third display area provided by a third display generation component is satisfied, displaying at a first position of the first display area a third visual indication indicating that it is possible to move the first user interface object onto the second display area, or at a second position of the first display area a fourth visual indication indicating that it is possible to move the first user interface object onto the third display area. The method according to claim 1, further comprising.

13. In accordance with the determination that the first user interface object has a location at an individual time between the first portions of the first movement that is spatially closer to the second display area provided by the second display generation component than the third display area provided by the third display generation component, display the third visual indication indicating that it is possible to move the first user interface object onto the second display area at the first position of the first display area, and stop displaying the fourth visual indication indicating that it is possible to move the first user interface object onto the third display area at the second position of the first display area, The method according to claim 12, further comprising.

14. In accordance with the determination that the first input is detected while the second display generation component is communicating with the first computer system in the second mode, moving the first user interface object across the second display area provided by the second display generation component to move the first user interface object to a location on the native user interface of the second computer system, The method according to claim 1, comprising.

15. A first computer system, A first display generation component, One or more processors, A memory, One or more programs, and the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are, The first computer system having the first display generation component, wherein the first computer system communicates with a first input device and a second computer system having a second display generation component different from the first display generation component, in the first computer system, Displaying a first user interface within the first display area provided by the first display generation component, including displaying a first user interface object at a first location within the first display area via the first display generation component, While displaying the first user interface within the first display area provided by the first display generation component, a first input, the first input including a first movement, according to a first portion of the first movement, a request to drag the first user interface object across the first display area provided by the first display generation component, and subsequently, according to a second portion of the first movement, a request to drag the first user interface object from the first display area provided by the first display generation component towards a second display area provided by the second display generation component beyond the first display area, detect a first input corresponding thereto via the first input device, In response to detection of the first movement, According to a determination that the first user interface object is a representation of content and the first input is detected while the second display generation component is communicating with the first computer system in a first mode, According to the first portion of the first movement, move the first user interface object from the first location within the first display area to a second location within the first display area across the first display area provided by the first display generation component, According to the second portion of the first movement, move the first user interface object across the second display area provided by the second display generation component, According to a determination that the first user interface object is a representation of content and the first input is detected while the second display generation component is communicating with the first computer system in a second mode different from the first mode, According to the first portion of the first movement, move the first user interface object from the first location within the first display area to a second location within the first display area across the first display area provided by the first display generation component, According to the second portion of the first movement, move the first user interface object across the second display area provided by the second display generation component, in accordance with a determination that the first input is detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the first mode, move the first user interface object from the first location to a second location within the first display area across the first display area provided by the first display generation component according to the first portion of the first movement, move the first user interface object across the second display area provided by the second display generation component according to the second portion of the first movement, a first computer system including instructions to prevent movement of the first user interface object into the second display area provided by the second display generation component in accordance with a determination that the first input is detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the second mode.

16. A first computer system, a first display generation component, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 2 to 14.

17. A computer program comprising instructions that, when executed by a first computer system having a first display generation component, cause the first computer system to in the first computer system having the first display generation component, the first computer system communicating with a first input device and a second computer system having a second display generation component different from the first display generation component Displaying a first user interface object at a first location within a first display area, including causing the first display generation component to display the first user interface within the first display area provided by the first display generation component, While displaying the first user interface within the first display area provided by the first display generation component, detecting, via the first input device, a first input that includes a first movement and, in accordance with a first portion of the first movement, a request to drag the first user interface object across the first display area provided by the first display generation component, and subsequently, in accordance with a second portion of the first movement, a request to drag the first user interface object from the first display area provided by the first display generation component towards a second display area provided by the second display generation component beyond the first display area, In response to detecting the first movement, Based on the determination that the first user interface object is a representation of content and the first input was detected while the second display generation component was communicating with the first computer system in a first mode, Moving the first user interface object from the first location within the first display area to a second location within the first display area across the first display area provided by the first display generation component in accordance with the first portion of the first movement, Moving the first user interface object across the second display area provided by the second display generation component in accordance with the second portion of the first movement, Based on the determination that the first user interface object is a representation of content and the first input was detected while the second display generation component was communicating with the first computer system in a second mode different from the first mode, Moving the first user interface object from the first location within the first display area to a second location within the first display area across the first display area provided by the first display generation component in accordance with the first portion of the first movement, Move the first user interface object across the second display area provided by the second display generation component according to the second part of the first movement. According to the determination that the first input was detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the first mode. Move the first user interface object from the first location to a second location within the first display area across the first display area provided by the first display generation component according to the first part of the first movement. Move the first user interface object across the second display area provided by the second display generation component according to the second part of the first movement. A computer program that prevents the first user interface object from moving into the second display area provided by the second display generation component according to the determination that the first input was detected while the first user interface object is an application window and the second display generation component is communicating with the first computer system in the second mode. **Claim 18** A computer program including instructions that cause the first computer system to perform the method according to any one of claims 2 to 14 when executed by the first computer system having a first display generation component.

Images