System and method for interacting with a companion display mode of an electronic device having a touch-sensitive display

Abstract

Disclosed are systems and methods for interacting with a companion display mode of an electronic device having a touch-sensitive display, that is, systems and methods that allow activation of a companion display mode of an electronic device and intuitive interaction with the companion display mode. An exemplary method includes: receiving an instruction to operate a first electronic device in a companion display mode in which a user interface generated by a second electronic device is displayed at the first electronic device, wherein the second electronic device is separate from the first electronic device; in response to receiving the instruction to operate in the companion display mode: simultaneously displaying on the touch-sensitive display of the first electronic device: a user interface generated by the second electronic device; and a plurality of user interface objects, including a first user interface object associated with a first function among a plurality of functions and a second user interface object associated with a second function among a plurality of functions, the plurality of functions being used to control the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode.

Description

[0001] This application is a divisional application of an invention patent application with an application date of April 2, 2020, application number 202080029421.0, and invention name “System and method for interacting with a companion display mode of an electronic device with a touch-sensitive display”. Technical Field

[0002] Embodiments disclosed herein relate to initiating and interacting with a companion display mode of an electronic device having a touch-sensitive display, and more particularly, to extending a user interface generated by a desktop operating system to a touch-sensitive display controlled by a standalone operating system of a portable device, including techniques for determining whether input is processed as touch input or desktop input. Background Art

[0003] Utilizing an auxiliary display allows users to separate the various projects they are working on and take advantage of the exclusivity of different types of displays for different types of projects. However, in some cases, some users cannot easily utilize the auxiliary display because these users cannot invoke the menus and input sequences required to use the device as an auxiliary display, and even if the user is able to invoke such menus and input sequences, those users may have to waste excessive time navigating the required menus and input sequences, which negatively affects their productivity and satisfaction with their device. Therefore, there is a need for features that allow for quick operation of a device as an auxiliary display.

[0004] Furthermore, human-machine interfaces for devices operating as secondary displays are often not intuitive and do not allow a user to use different types of input devices (e.g., finger, stylus, etc.) to perform different functions. Thus, there is a need for more intuitive human-machine interfaces, and in particular, a need for human-machine interfaces that allow the use of different types of input devices when a device is operating as a secondary display. Summary of the invention

[0005] The embodiments described herein solve the above-mentioned shortcomings by providing a device and method that allows a user to easily and quickly operate a first device (e.g., a tablet electronic device) in a companion display mode in which a user interface generated by another device (e.g., a laptop electronic device) is displayed. Such devices and methods also require minimal input to locate for activating and using the companion display mode. Such devices and methods also enable more relevant information to be provided on a limited screen (e.g., a touch-sensitive display of a tablet electronic device is used to display relevant information from both a desktop operating system and a mobile operating system using limited screen space). Such devices and methods also provide an improved human-computer interface, for example, by providing an emphasis effect to make information more discernible on a touch-sensitive display (which can be generated by different operating systems), by providing continuous interaction so that continuous input from a user pointing to a desktop operating system or a mobile operating system causes the device (which operates in a companion display mode) to provide outputs, which are then used to promote additional inputs from the user, and by reducing the user's interaction to achieve the desired result. For these reasons and those described below, the devices and methods described herein reduce power usage and extend the battery life of electronic devices.

[0006] According to some embodiments, a method (e.g., for sharing a user interface between different electronic devices) is performed at a first electronic device (e.g., a tablet electronic device). The method includes receiving an instruction to operate the first electronic device in a companion display mode in which a user interface generated by a second electronic device (e.g., a laptop electronic device) is displayed at the first electronic device, and the second electronic device is separated from the first electronic device. In response to receiving an instruction to operate in the companion display mode, the method includes: simultaneously displaying on the touch-sensitive display of the first electronic device: (i) a user interface generated by the second electronic device; and (ii) multiple user interface objects, including (i) a first user interface object associated with a first function among multiple functions, and (ii) a second user interface object associated with a second function among the multiple functions, the multiple functions being used to (only) control the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode.

[0007] When users interact with the auxiliary display, they must often navigate through complex menu sequences to adjust the display according to their needs at different points in time. Fig. 7AThe control strip 197 shown near the left edge of the illustrated tablet device in the user interface of , and other figures showing such control strips) is displayed immediately (and without any other human intervention), allowing the user to avoid additional inputs to access menus for controlling the touch-sensitive display of the first electronic device. Reducing the number of inputs required to control the touch-sensitive display enhances the operability of the device and makes the human-machine interface more efficient (e.g., by helping the user reduce the number of inputs the user needs to make), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0008] According to some embodiments, a method is performed at a first electronic device (e.g., a tablet electronic device). The method includes: operating the first electronic device in a companion display mode in which a user interface generated by a second electronic device (e.g., a laptop computer or a desktop electronic device) is displayed at the first electronic device, and the second electronic device is separated from the first electronic device. When operating in the companion display mode, the method includes: displaying the user interface generated by the second electronic device on a touch-sensitive display of the first electronic device; and detecting a gesture using an input object at the first electronic device. In response to detecting a gesture, the method includes: performing a first operation on the touch-sensitive display based on a determination that the input object is one or more fingers; and performing a second operation different from the first operation on the touch-sensitive display based on a gesture based on a determination that the input object is a stylus.

[0009] When interacting with a touch-sensitive display, users are typically limited to a specific set of predefined inputs based on the dexterity of the human hand, which limits the number of operations that can be performed. Allowing a single gesture to have multiple purposes based on an input device (e.g., a finger or a stylus) allows the user to perform more operations than is typically possible and enables efficient interaction with the accompanying display mode. Increasing the number of operations that can be performed based on a set number of gestures enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user reduce the number of gestures required for the user to perform an operation), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0010] According to some embodiments, a method is performed at a first electronic device (e.g., a laptop computing device) including a display. The method includes: displaying a user interface associated with an application on a display, the user interface being displayed together with a control user interface element (e.g., a green button control element mentioned herein) for changing (only) the display properties of the user interface (examples of display properties include the size, display position, etc. of the user interface). For example, clicking the green button on the upper left corner of the window maximizes the window. Then detecting input pointing to (e.g., hovering or right clicking) the control user interface element. The method also includes: in response to detecting the input, and while continuing to display the user interface: simultaneously displaying on the display: (i) a first selectable option for changing the display properties of the user interface on the display of the first electronic device; and (ii) a second selectable option for requesting display of a user interface including content from the user interface at a second electronic device different from the first electronic device (e.g., at a tablet electronic device). In other embodiments, the control user element has a single function, namely requesting display of a user interface including content from the user interface at a second electronic device different from the first electronic device (e.g., at a tablet electronic device).

[0011] In addition, changing the arrangement of a user interface running on multiple displays may sometimes require repeated dragging operations or the use of multiple keyboard commands to achieve the desired orientation. Allowing a user to select a single control user interface element that populates a list of multiple selectable options for changing display properties (e.g., maximizing a window, sending to another display, etc.) ensures that such display properties are changed with a minimum number of inputs. Reducing the number of inputs to change these display properties enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user reduce the number of gestures required by the user to perform an operation), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0012] According to some embodiments, a method is performed at a first electronic device including a display device. The method includes: receiving a request to annotate content on the first electronic device. The method also includes: in response to receiving the request: based on determining that a second electronic device different from the first electronic device can be used to display content in annotation mode, and using the second electronic device to display content in annotation mode has not been previously approved, displaying a selectable option via the display device, which when selected causes the first electronic device to send an instruction to display content in annotation mode at the second electronic device; and based on determining that the second electronic device can be used to display content in annotation mode and using the second electronic device to display content in annotation mode has been previously approved, sending an instruction to the second electronic device to automatically display content in annotation mode without further user intervention. In some embodiments, the annotation mode is a mode in which inputs are received at certain locations above the content and those inputs are then used to annotate the content (including drawing lines, circles, handwriting, shapes, etc.).

[0013] If the user has indicated that the second device is approved to display content in annotation mode, it will waste time and require redundant input to continuously ask the user to re-approve the second device. Therefore, by determining whether the second device is available to display content in annotation mode and whether the second device has been previously approved to respond to a request to annotate content, it is ensured that the user does not have to waste time providing additional input to re-approve the second device. In this way, the human-computer interface is improved and continuous interaction with two different devices is possible.

[0014] The descriptions of the first electronic device and the second electronic device herein are interchangeable. In other words, the description of the operations at the first electronic device also applies to the operations that can be performed at the second electronic device, and vice versa.

[0015] According to some embodiments, a first electronic device (e.g., a device running a desktop operating system or a mobile operating system, such as a laptop running a desktop operating system or a tablet device running a mobile operating system) includes a display (which may be a touch-sensitive display) and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors and the one or more programs including instructions for performing or causing the operation of any of the methods described herein. According to some embodiments, the first electronic device has stored therein instructions that, when executed by the first electronic device, cause the device to perform or cause the operation of any of the methods described herein. According to some embodiments, a graphical user interface on a display of a first electronic device is provided, and the graphical user interface includes one or more elements of the elements displayed in any of the methods described herein, which are updated in response to input, as described in any of the methods described herein. According to some embodiments, the first electronic device includes a device for performing or causing the operation of any of the methods described herein. According to some embodiments, an information processing device for use in a first electronic device includes a device for performing or causing the operation of any of the methods described herein.

[0016] The systems and methods described herein improve the operability of electronic devices by, for example, enabling interactions that require fewer inputs without wasting time searching for affordances that may be difficult to locate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0018] Figure 1A to Figure 1B An exemplary system according to some embodiments is shown in which a first electronic device (e.g., a tablet electronic device) operates in a companion display mode for a second electronic device (e.g., a laptop computer).

[0019] Figure 2 is a block diagram of an electronic device (e.g., a device running a mobile operating system that is different from a desktop operating system) according to some embodiments.

[0020] Figure 3A is a block diagram of an electronic device (e.g., a device running a desktop operating system) according to some embodiments.

[0021] Figure 3B According to some embodiments, Figure 3A A block diagram of the components of event processing.

[0022] Figures 4A to 4QQ is a schematic diagram of a display for illustrating an exemplary user interface for initiating and interacting with a companion display mode according to some embodiments. Additional details regarding these figures are also provided below with reference to the description of methods 800, 900, 1000, and 1100.

[0023] Figures 5A-1 to 5F-2 is a schematic diagram of a display showing different exemplary user interface responses based on different input types when operating a tablet device in a companion display mode. Additional details regarding these figures are also provided below with reference to the description of methods 800 , 900 , 1000 , and 1100 .

[0024] FIG. 6A to FIG. 6L is a schematic diagram of a display for illustrating an exemplary user interface for initiating and interacting with an annotation mode according to some embodiments. Additional details regarding these figures are also provided below with reference to the description of methods 800, 900, 1000, and 1100.

[0025] 7A to 7W is a schematic diagram of a display for illustrating an exemplary user interface for interacting with a companion display mode according to some embodiments. Additional details regarding these figures are also provided below with reference to the description of methods 800, 900, 1000, and 1100.

[0026] FIG. 8A to FIG. 8G is a flowchart of a method for initiating and interacting with a companion display mode according to some embodiments.

[0027] 9A to 9E is a flowchart of a method for performing different operations depending on which type of input object is used in conjunction with a companion display mode according to some embodiments.

[0028] FIG. 10A to FIG. 10C is a flowchart of a method for interacting with a control user interface element to invoke a companion display mode according to some embodiments.

[0029] FIG. 11A to FIG. 11D is a flowchart of a method for sending content from one device to another device for use with an annotation mode according to some embodiments. DETAILED DESCRIPTION

[0030] 1 to 3 illustrate exemplary devices on which the methods described herein are implemented and executed. Figures 4A to 7W is a schematic diagram of a display illustrating an exemplary user interface for initiating and interacting with a companion display mode, and additional descriptions of these user interface figures are provided with reference to methods 800, 900, 1000, and 1100 below.

[0031] Exemplary Devices and Systems

[0032] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0033] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish an element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

[0034] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "a", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0035] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0036] Figure 1A to Figure 1BAn exemplary system is shown in which a first electronic device (e.g., a tablet electronic device shown) is operating in a companion display mode for a second electronic device (e.g., a laptop electronic device shown). Additional details about the companion display mode are provided below. It is also noted that various references are made to the first electronic device and the second electronic device, and in some cases, the second device can be a tablet electronic device and the first device can be a laptop electronic device. In addition, references to tablet computers and laptop electronic devices are merely illustrative examples. The description herein of tablet electronic devices also applies to other portable electronic devices running mobile operating systems (e.g., smart phones, such as an IPHONE running an IOS operating system from APPLE INC. (Cupertino, CA) of Cupertino, California), and the description herein of laptop electronic devices also applies to other desktop devices running desktop computer / laptop computer operating systems.

[0037] A block diagram showing various components of a first electronic device and a second electronic device is shown in Figure 2 and FIG. 3A to FIG. 3B Shown in.

[0038] Attention is now directed to embodiments of portable electronic devices having touch-sensitive displays. Figure 2 1 is a block diagram illustrating a portable multifunction device 100 (also interchangeably referred to herein as electronic device 100 or device 100) with a touch-sensitive display 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a touch-sensitive display system. Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), controller 120, one or more processing units (CPUs) 122, peripheral device interface 118, RF circuit 108, audio circuit 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface, such as touch-sensitive display system 112 of device 100 or a touch pad of device 100). These components optionally communicate via one or more communication buses or signal lines 103.

[0039] It should be understood that device 100 is only one example of a portable multifunction device, and device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of these components. The various components shown in FIG. 1 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0040] The memory 102 optionally includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, or other random access solid-state memory devices), and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The memory 102 optionally includes one or more storage devices located remotely from the one or more processors 122. Access to the memory 102 by other components of the device 100 (such as the CPU 122 and the peripheral device interface 118) is optionally controlled by the controller 120.

[0041] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 122 and memory 102. One or more processors 122 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0042] In some embodiments, peripherals interface 118, CPU 122, and controller 120 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0043] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). Wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n).

[0044] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into a sound wave audible to humans. The audio circuit 110 also receives an electrical signal converted from a sound wave by the microphone 113. The audio circuit 110 converts the electrical signal into audio data, and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack. The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0045] The I / O subsystem 106 connects input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any one of the following (or not to any one of the following): a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons may optionally include an increase / decrease button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button.

[0046] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.

[0047] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the area under the user's finger.

[0048] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in Apple Inc. (Cupertino, California) and Technology in.

[0049] The touch screen 112 optionally has a video resolution of more than 400 dpi. In some embodiments, the touch screen 112 has a video resolution of at least 600 dpi. In other embodiments, the touch screen 112 has a video resolution of at least 1000 dpi. The user optionally uses any suitable object or finger such as a stylus or finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures. In some embodiments, the device converts finger-based input into precise pointer / cursor positions or commands for performing actions desired by the user.

[0050] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0051] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0052] The device 100 optionally also includes one or more optical sensors 164. FIG. 1 shows an optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen 112 on the front of the device, so that the touch-sensitive display can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device so that the user can optionally obtain the image of the user for video conferencing while watching other video conference participants on the touch-sensitive display.

[0053] The device 100 optionally also includes one or more contact force sensors 165. FIG. 1 shows a contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106. The contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact force sensor 165 receives contact force information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch screen 112 located on the front of the device 100.

[0054] Device 100 optionally also includes one or more proximity sensors 166. FIG. 1 shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call), the proximity sensor turns off and disables touch screen 112.

[0055] The device 100 optionally also includes one or more tactile output generators 167. FIG. 1 shows a tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The contact intensity sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112), and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of device 100) or laterally (e.g., back and forward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100 opposite to touch-sensitive display 112 located on the front of device 100.

[0056] The device 100 optionally also includes one or more accelerometers 168. FIG. 1 shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch-sensitive display based on an analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0057] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, the memory 102 stores a device / global internal state 157, as shown in Figure 1. The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display 112; a sensor state, including information obtained from various sensors and input control devices 116 of the device; and location information about the location and / or posture of the device (e.g., the orientation of the device).

[0058] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0059] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., a universal serial bus (USB), FireWire, etc.) is suitable for direct coupling to other devices, or indirectly coupling through a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to, and / or compatible with the 30-pin connector used in some embodiments of the IPOD device from APPLE Inc. In other embodiments, the external port is a multi-pin (e.g., 8-pin) connector that is the same as, or similar to, and / or compatible with the 8-pin connector used in the LIGHTNING connector from APPLE Inc.

[0060] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single contact (e.g., one finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on a touch pad.

[0061] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touch-sensitive display hardware, the mouse "click" threshold of a touchpad or touch-sensitive display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0062] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event and then detecting one or more finger drag events, and in some embodiments, also subsequently detecting a finger lift (lift-off) event.

[0063] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0064] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0065] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0066] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts module 137, email client module 140, IM module 141, browser module 147, and any other application requiring text input).

[0067] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0068] Application program ("app") 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0069] Contacts module 137 (sometimes called address book or contact list);

[0070] Telephone module 138;

[0071] Video conferencing module 139;

[0072] Email client module 140;

[0073] Instant messaging (IM) module 141;

[0074] Fitness module 142;

[0075] A camera module 143 for still images and / or video images;

[0076] Image management module 144;

[0077] Browser module 147;

[0078] Calendar module 148;

[0079] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0080] Search module 151;

[0081] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0082] Memo module 153;

[0083] Map module 154; and / or

[0084] Online video module 155.

[0085] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk editing applications, spreadsheet applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, a widget creator module 149-6 for making user-created widgets, and voice replication.

[0086] In combination with the touch screen 112, display controller 156, contact module 130, graphics module 132 and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (for example, stored in the contact module 137 in the memory 102 or the memory 370), including: adding one or more names to the address book; deleting a name from the address book; associating a phone number, email address, physical address or other information with a name; associating an image with a name; categorizing and classifying names; providing a phone number or email address to initiate and / or facilitate communications via the telephone module 138, video conferencing module 139, email client module 140 or IM module 141; and the like.

[0087] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a sequence of characters corresponding to a phone number, access one or more phone numbers in address book 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0088] In combination with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137 and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0089] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0090] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for inputting a sequence of characters corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0091] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 146, the health module 142 includes executable instructions for the following operations: creating a fitness (e.g., with a time, distance and / or calorie burn goal); communicating with a fitness sensor (a sports device such as a watch or a pedometer); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting fitness music and playing it; and displaying, storing and transmitting fitness data.

[0092] In conjunction with touch screen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or videos (including video streams) and store them in memory 102; modify characteristics of still images or videos; or delete still images or videos from memory 102.

[0093] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0094] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0095] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0096] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0097] In combination with the RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134 and browser module 147, a desktop widget creator module (not shown) is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page to a desktop widget).

[0098] In combination with the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0099] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions to display, present, or otherwise play back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes functionality of an MP3 player, such as the IPOD from APPLE Inc.

[0100] In conjunction with the touch screen 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0101] In combination with the RF circuit 108, the touch screen 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store executable instructions for maps and data associated with the maps (e.g., driving directions; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0102] In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0103] like Figure 2 As shown, portable multifunction device 100 also includes a companion display module 180 for managing operations associated with companion display mode multitasking on device 100. Companion display module 180 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0104] An arrangement module 182 for determining an arrangement of displays of a laptop computer and a tablet device adjacent to each other in conjunction with the companion display mode described herein;

[0105] A UI generator module 184 for generating user interfaces in conjunction with the companion display and annotation modes and sharing data related to those user interfaces between different devices; and

[0106] • A security criteria module 186 for monitoring whether the devices have satisfied a set of security connection criteria used to determine when a companion display mode can be used between different devices (eg, a laptop and a tablet device).

[0107] In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and contact intensity sensor 165, PIP module 186 includes executable instructions for determining a reduced size of the video content and determining an appropriate location on touch screen 112 for displaying the reduced size video content (e.g., a location that avoids important content in an active application being covered by the reduced size video content).

[0108] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0109] Figure 3A is a block diagram of an electronic device 300 according to some embodiments. In some embodiments, the electronic device 300 is a laptop or desktop computer running a desktop operating system different from a mobile operating system.

[0110] The electronic device 300 typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a video conferencing application, an email application, an instant messaging application, an image management application, a digital camera application, a digital video camera application, a web browser application, and / or a media player application.

[0111] Various applications executed on the electronic device 300 optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed by the electronic device 300 are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within an application. In this way, the common physical architecture of the electronic device 300 (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0112] The electronic device 300 includes a memory 302 (which optionally includes one or more computer-readable storage media), a memory controller 322, one or more processing units (CPUs) 320, a peripheral device interface 318, an RF circuit 308, an audio circuit 310, a speaker 311, a microphone 313, an input / output (I / O) subsystem 306, other input or control devices 316, and an external port 324. The electronic device 300 optionally includes a display system 312, which can be a touch-sensitive display (sometimes also referred to herein as a "touch screen" or "touch screen display"). The electronic device 300 optionally includes one or more optical sensors 364. The electronic device 300 optionally includes one or more intensity sensors 365 for detecting contact intensity on a touch-sensitive surface such as a touch-sensitive display or a touch pad. The electronic device 300 optionally includes one or more tactile output generators 367 for generating tactile output on a touch-sensitive surface such as a touch-sensitive display or a touch pad. These components optionally communicate via one or more communication buses or signal lines 303.

[0113] As used in this specification, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or changes thereto, the capacitance of the touch-sensitive surface near the contact and / or changes thereto, and / or the resistance of the touch-sensitive surface near the contact and / or changes thereto are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurement of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units).

[0114] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or a touchpad / trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0115] It should be understood that the electronic device 300 is only one example, and the electronic device 300 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 3A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0116] Memory 302 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state memory devices. The access of other components of electronic device 300 (such as CPU 320 and peripheral device interface 318) to memory 302 is optionally controlled by memory controller 322. Peripheral device interface 318 can be used to couple input and output peripheral devices to CPU 320 and memory 302. One or more processing units 320 run or execute various software programs and / or instruction sets stored in memory 302 to perform various functions of electronic device 300 and process data. In some embodiments, peripheral device interface 318, CPU 320 and memory controller 322 are optionally implemented on a single chip such as chip 304. In some other embodiments, they are optionally implemented on independent chips.

[0117] RF (radio frequency) circuit 308 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuit 308 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 308 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, etc. RF circuit 308 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoiP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilization Extensions (SIMPLE), Instant Messaging and Presence Services (IMPS)) and / or Short Message Service (SMS), or any other suitable communications protocol, including communications protocols that have not yet been developed on the date of filing of this document.

[0118] The audio circuit 310, the speaker 311 and the microphone 313 provide an audio interface between the user and the electronic device 300. The audio circuit 310 receives audio data from the peripheral device interface 318, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 311. The speaker 311 converts the electrical signal into a sound wave audible to humans. The audio circuit 310 also receives an electrical signal converted by the microphone 313 according to the sound wave. The audio circuit 310 converts the electrical signal into audio data and transmits the audio data to the peripheral device interface 318 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 302 and / or the RF circuit 308 by the peripheral device interface 318. In some embodiments, the audio circuit 310 also includes an earphone jack. The headphone interface provides an interface between the audio circuit 310 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0119] The I / O subsystem 306 couples the input / output peripherals of the electronic device 300, such as the display system 312 and other input or control devices 316, to the peripheral device interface 318. The I / O subsystem 306 optionally includes a display controller 356, an optical sensor controller 358, an intensity sensor controller 359, a tactile feedback controller 361, and one or more other input controllers 360 for other input or control devices. One or more other input controllers 360 receive electrical signals from / send electrical signals to the other input or control devices 316. Other input control devices 316 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, other input controllers 360 are optionally coupled to any one (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more physical buttons may optionally include an increase / decrease button for volume control of a speaker 311 and / or a microphone 313.

[0120] The display system 312 provides an output interface (and optionally an input interface when it is a touch-sensitive display) between the electronic device 300 and the user. The display controller 356 receives and / or sends electrical signals to the display system 312. The display system 312 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects / elements.

[0121] In some embodiments, the display system 312 is a touch-sensitive display having a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. Thus, the display system 312 and display controller 356 (together with any associated modules and / or instruction sets in memory 302) detect contact (and any movement or interruption of that contact) on the display system 312 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the display system 312. In an exemplary embodiment, the point of contact between the display system 312 and the user corresponds to the area under the user's finger.

[0122] The display system 312 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, LED (light emitting diode) technology, or OLED (organic light emitting diode) technology, although other display technologies are used in other embodiments. In some embodiments, when the display system 312 is a touch-sensitive display, the display system 312 and the display controller 356 optionally use any of a variety of touch sensing technologies now known or later developed and other proximity sensor arrays or other elements for determining one or more points of contact with the display system 312 to detect contact and any movement or interruption thereof, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in Apple Inc. (Cupertino, California) and Technology in.

[0123] The display system 312 optionally has a video resolution of more than 400 dpi (e.g., 500 dpi, 800 dpi or greater). In some embodiments, the display system 312 is a touch-sensitive display, and the user optionally contacts the touch-sensitive display using a stylus, finger, etc. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures. In some embodiments, the electronic device 300 converts rough finger-based input into precise pointer / cursor positions or commands for performing actions desired by the user.

[0124] In some embodiments, in addition to the display system 312, the electronic device 300 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the electronic device 300 that, unlike the display system 312, does not display visual output. In some embodiments, when the display system 312 is a touch-sensitive display, the touchpad is optionally a touch-sensitive surface separate from the display system 312, or an extension of the touch-sensitive surface formed by the display system 312.

[0125] The electronic device 300 also includes a power system 362 for powering the various components. The power system 362 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), etc., a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0126] The electronic device 300 optionally also includes one or more optical sensors 364 coupled to the optical sensor controller 358 in the I / O subsystem 306. The optical sensor 364 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 364 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 343, the optical sensor 364 optionally captures a still image or video. In some embodiments, the optical sensor is located on the front of the electronic device 300 so that the user can optionally obtain an image of the user for video conferencing while viewing other video conference participants on the display system 312.

[0127] The electronic device 300 optionally also includes one or more contact force sensors 365 coupled to the force sensor controller 359 in the I / O subsystem 306. The contact force sensor 365 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact force sensor 365 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to the touch-sensitive surface.

[0128] The electronic device 300 optionally also includes one or more tactile output generators 367 coupled to the tactile feedback controller 361 in the I / O subsystem 306. The tactile output generator 367 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device) for converting energy into linear motion. The contact force sensor 365 receives tactile feedback generation instructions from the tactile feedback module 333 and generates tactile outputs that can be sensed by a user of the device 300. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to the touch-sensitive surface and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the electronic device 300) or laterally (e.g., backward and forward in the same plane as the surface of the electronic device 300).

[0129] The electronic device 300 optionally further includes one or more proximity sensors 366 coupled to the peripheral device interface 318. Alternatively, the proximity sensor 366 is coupled to other input controllers 360 in the I / O subsystem 306. The electronic device 300 optionally further includes one or more accelerometers 368 coupled to the peripheral device interface 318. Alternatively, the accelerometer 368 is coupled to other input controllers 360 in the I / O subsystem 306.

[0130] In some embodiments, the software components stored in memory 302 include an operating system 326, a communication module 328 (or an instruction set), a contact / motion module 330 (or an instruction set), a graphics module 332 (or an instruction set), an application 340 (or an instruction set), and a touch bar management module 350 (or an instruction set). In addition, in some embodiments, memory 302 stores device / global internal state 357 (or an instruction set), such as Figure 3A Device / global internal state 357 includes one or more of the following: active application state, which indicates which applications (if any) are currently active and / or in focus; display state, which indicates what applications, views, or other information occupy various areas of display system 312 and / or peripheral display systems; sensor state, including information obtained from various sensors and input or control devices 316 of electronic device 300; and position information about the position and / or posture of electronic device 300.

[0131] The operating system 326 (e.g., DARWIN, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VXWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware components and software components.

[0132] The communication module 328 facilitates communication with other devices through one or more external ports 324 and / or RF circuitry 308, and also includes various software components for sending / receiving data via RF circuitry 308 and / or external ports 324. External ports 324 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for connecting directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, external ports 324 are connected to A multi-pin (eg, 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used on the device.

[0133] When the display system 312 is a touch-sensitive display (in conjunction with the display controller 356) and other touch-sensitive devices (e.g., a touchpad or physical click wheel), the contact / motion module 330 optionally detects contact with the display system. The contact / motion module 330 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 330 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contact). In some embodiments, the contact / motion module 330 also detects contact on a touchpad.

[0134] In some embodiments, the contact / motion module 330 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the electronic device 300). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, the user of the device is provided with a software setting for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0135] The contact / motion module 330 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap contact includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event and then detecting one or more finger drag events, and in some embodiments, also subsequently detecting a finger lift (lift-off) event.

[0136] The graphics module 332 includes various known software components for presenting and causing the display of graphics on the main display 102 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast or other visual features) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to the user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc. In some embodiments, the graphics module 332 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 332 receives one or more codes for specifying the graphics to be displayed from an application, etc., and also receives coordinate data and other graphics attribute data together when necessary, and then generates screen image data to be output to the display controller 356.

[0137] The tactile feedback module 333 includes various software components for generating instructions that are used by the tactile output generator 367 to produce tactile output at one or more locations on the electronic device 300 in response to user interaction with the electronic device 300 .

[0138] The application 340 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0139] An email client module 341 (also sometimes referred to herein as a "mail application" or "email application") for receiving, sending, composing, and viewing emails;

[0140] An imaging module 342 for capturing still images and / or video images;

[0141] An image management module 343 (also sometimes referred to herein as a "photo application") for editing and viewing still images and / or video images;

[0142] A media player module 344 (also sometimes referred to herein as a "media player application") for playing back audio and / or video; and

[0143] Web browsing module 345 (sometimes also referred to herein as a "web browser") for connecting to and browsing the Internet.

[0144] Examples of other applications 340 that are optionally stored in memory 302 include messaging and communication applications, word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption applications, digital rights management applications, voice recognition applications, and voice replication applications.

[0145] In conjunction with one or more of the RF circuit 308, the display system 312, the display controller 356, and the contact module 330, the graphics module 332, the email client module 341 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 343, the email client module 341 makes it very easy to create and send emails with still images or video images captured by the imaging module 342.

[0146] In conjunction with one or more of the display system 312, display controller 356, optical sensor 364, optical sensor controller 358, contact module 330, graphics module 332, and image management module 343, imaging module 342 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 302, modifying characteristics of still images or videos, or deleting still images or videos from memory 302.

[0147] In conjunction with one or more of the display system 312, display controller 356, contact module 330, graphics module 332, and imaging module 342, image management module 343 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0148] In conjunction with the display system 312, display controller 356, contact module 330, graphics module 332, audio circuit system 310, speaker 311, RF circuit 308 and web browsing module 345, the multimedia player module 344 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting or otherwise playing back videos.

[0149] In combination with RF circuit 308, display system 312, display controller 356, contact module 330 and graphics module 332, web browsing module 345 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0150] like Figure 3A As shown, the device 300 may further include a companion display module 350 for managing operations associated with companion display mode multitasking on the device 100. The companion display module 350 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0151] An arrangement module 351 for determining an arrangement of displays of a laptop computer and a tablet device adjacent to each other in conjunction with the companion display mode described herein;

[0152] A UI generator module 352 for generating user interfaces in conjunction with the companion display and annotation modes and sharing data related to those user interfaces between different devices; and

[0153] • A security criteria module 353 for monitoring whether the devices have satisfied a set of security connection criteria used to determine when a companion display mode can be used between different devices (eg, a laptop and a tablet device).

[0154] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 302 optionally stores a subset of the above modules and data structures. In addition, memory 302 optionally stores additional modules and data structures not described above.

[0155] Figure 3B According to some embodiments, Figure 3A In some embodiments, the memory 302 ( Figure 3A ) includes an event classifier 370 (e.g., in the operating system 326) and an application 340-1 (e.g., any of the aforementioned applications 341, 342, 343, 344 or 345).

[0156] The event classifier 370 receives event information and determines the application 340-1 and the application view 391 of the application 340-1 to which the event information is to be delivered. The event classifier 370 includes an event monitor 371 and an event distributor module 374. In some embodiments, the application 340-1 includes an application internal state 392 that indicates the current application view displayed on the display system 312 when the application is active or executing. In some embodiments, the device / global internal state 357 is used by the event classifier 370 to determine which application(s) is currently active or in focus, and the application internal state 392 is used by the event classifier 370 to determine the application view 391 to which the event information is to be delivered.

[0157] In some embodiments, the application internal state 392 includes additional information, such as one or more of the following: resumption information to be used when the application 340-1 resumes execution, user interface state information indicating information being displayed by the application 340-1 or information ready to be displayed by the application, a state queue for enabling a user to return to a previous state or view of the application 340-1, and a repeat / undo queue of previous actions taken by the user.

[0158] Event monitor 371 receives event information from peripheral device interface 318. Event information includes information about sub-events (e.g., user touches on display system 312 as part of a multi-touch gesture when display system 312 is a touch-sensitive display). Peripheral device interface 318 transmits information it receives from I / O subsystem 306 or sensors such as proximity sensor 366, accelerometer 368, and / or microphone 313 (through audio circuit 310). The information that peripheral device interface 318 receives from I / O subsystem 306 includes information from display system 312 when it is a touch-sensitive display or another touch-sensitive surface.

[0159] In some embodiments, event monitor 371 sends requests to peripheral device interface 318 at predetermined intervals. In response, peripheral device interface 318 transmits event information. In other embodiments, peripheral device interface 318 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0160] In some embodiments, the event classifier 370 also includes a hit view determination module 372 and / or an active event identifier determination module 373.

[0161] The hit view determination module 372 provides a software process for determining where a sub-event has occurred within one or more views when the display system 312 displays more than one view, where a view consists of controls and other elements that a user can see on the display.

[0162] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0163] The hit view determination module 372 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, the hit view determination module 372 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0164] Active event recognizer determination module 373 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 373 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 373 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0165] Event distributor module 374 distributes event information to event identifiers (e.g., event identifier 380). In embodiments including active event identifier determination module 373, event distributor module 374 delivers the event information to the event identifier determined by active event identifier determination module 373. In some embodiments, event distributor module 374 stores the event information in an event queue, which is retrieved by corresponding event receiver 382.

[0166] In some embodiments, operating system 326 includes event classifier 370. Alternatively, application 340-1 includes event classifier 370. In yet another embodiment, event classifier 370 is a standalone module, or is part of another module stored in memory 302, such as contact / motion module 330.

[0167] In some embodiments, application 340-1 includes multiple event handlers 390 and one or more application views 391, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 391 in application 340-1 includes one or more event recognizers 380. Typically, application view 391 includes multiple event recognizers 380. In other embodiments, one or more event recognizers in event recognizers 380 are part of an independent module (such as a user interface toolkit (not shown) or a higher-level object from which application 340-1 inherits methods and other properties). In some embodiments, the corresponding event handler 390 includes one or more of the following: data updater 376, object updater 377, GUI updater 378 and / or event data 379 received from event classifier 370. Event handler 390 optionally utilizes or calls data updater 376, object updater 377 or GUI updater 378 to update application internal state 392. Alternatively, one or more of the application views in the application view 391 include one or more corresponding event handlers 390. In addition, in some embodiments, one or more of the data updater 376, the object updater 377, and the GUI updater 378 are included in the application view 391.

[0168] The corresponding event identifier 380 receives event information (e.g., event data 379) from the event classifier 370 and identifies the event from the event information. The event identifier 380 includes an event receiver 382 and an event comparator 384. In some embodiments, the event identifier 380 also includes metadata 383 and at least a subset of event delivery instructions 388 (which optionally include sub-event delivery instructions).

[0169] The event receiver 382 receives event information from the event classifier 370. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0170] Event comparator 384 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 384 includes event definition 386. Event definition 386 includes the definition of an event (e.g., a predefined sub-event sequence), such as event 1 (387-1), event 2 (387-2), and other events. In some embodiments, the sub-events in event 387 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (387-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (387-2) is the dragging of a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the display system when the display system 312 is a touch-sensitive display, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 390.

[0171] In some embodiments, event definition 387 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 384 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view in which three user interface objects are displayed on display system 312, when a touch is detected on display system 312 (when the display system is a touch-sensitive display), event comparator 384 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 390, the event comparator uses the result of the hit test to determine which event handler 390 should be activated. For example, event comparator 384 selects an event handler associated with a sub-event and an object that triggers the hit test.

[0172] In some embodiments, the definition of the corresponding event 387 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0173] When the corresponding event recognizer 380 determines that the sub-event sequence does not match any event in the event definition 386, the corresponding event recognizer 380 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0174] In some embodiments, the corresponding event identifiers 380 include metadata 383 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event identifiers. In some embodiments, metadata 383 includes configurable properties, flags, and / or lists that indicate how event identifiers interact or can interact with each other. In some embodiments, metadata 383 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0175] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 380 activates an event handler 390 associated with the event. In some embodiments, the corresponding event recognizer 380 delivers event information associated with the event to the event handler 390. Activating the event handler 390 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 380 throws a tag associated with the identified event, and the event handler 390 associated with the tag obtains the tag and performs a predefined process.

[0176] In some embodiments, the event delivery instructions 388 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0177] In some embodiments, data updater 376 creates and updates data used in application 340-1. For example, data updater 376 stores video files used by media player module 344. In some embodiments, object updater 377 creates and updates objects used by application 340-1. For example, object updater 376 creates a new user interface object or updates the position of a user interface object. GUI updater 378 updates the GUI. For example, GUI updater 378 prepares display information and sends the display information to graphics module 332 for display on display system 312.

[0178] In some embodiments, event handler 390 includes or has access to data updater 376, object updater 377, and GUI updater 378. In some embodiments, data updater 376, object updater 377, and GUI updater 378 are included in a single module of application 340-1 or are included in application view 391. In other embodiments, they are included in two or more software modules.

[0179] It should be understood that the above discussion on event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the electronic device 300, and not all user inputs are initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0180] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., trackpad 355 or 356 in FIG. 3 ). Figure 4BIn some implementations of a touch screen display that includes a touch screen display that enables direct interaction with user interface elements on the touch screen display, a contact detected on the touch screen acts as a "focus selector" such that when an input is detected on the touch screen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form that the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0181] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). The intensity of the contact is used as an attribute of the user input, thereby allowing the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control such as a knob or button).

[0182] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the portable computing system 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of the touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, the user of the device is provided with a software setting for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0183] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0184] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture) where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0185] In some embodiments, one or more predefined intensity thresholds are used to determine whether a particular input meets the intensity-based criteria. For example, the one or more predefined intensity thresholds include (i) a contact detection intensity threshold IT0, (ii) a light press intensity threshold IT1, and (iii) a light press intensity threshold IT2. L (iii) Deep compression intensity threshold IT D (e.g., at least initially above I L ) and / or (iv) one or more other intensity thresholds (e.g., below I L The intensity threshold I H ). In some embodiments, the light press strength threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press strength threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press strength threshold (e.g., and higher than a nominal contact detection intensity threshold IT0, contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press strength threshold or the deep press strength threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0186] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and a time-based criterion. For example, for some "deep press" inputs, as long as a delay time passes between satisfying the first strength threshold and satisfying the second strength threshold, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers a second response. The duration of the delay time is typically less than 200ms (e.g., 40ms, 100ms, or 120ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). The delay time helps avoid accidental deep press inputs. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold increases. This temporary increase in the second strength threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to detecting the deep press input is not dependent on time-based criteria.

[0187] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. patent applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0188] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0189] User interface and associated processes

[0190] Attention is now directed to the embodiment of the present invention which can be used in a computer including a laptop computer 300 ( Figure 1A ) and tablet device 100( Figure 1A to Figure 1B ) system, wherein a user interface generated by one device (e.g., laptop computer 300) is presented at another device (e.g., tablet device 100). In the following description, a mode in which such a user interface generated by another device is presented at one device is referred to herein as a companion display mode. Although a laptop computer is used as an illustrative example in the following description, it will be readily understood by those skilled in the art that the techniques described herein are equally applicable to any device running a desktop / laptop computer operating system, or in some cases, operations described as being performed on a laptop computer may also be performed by a tablet device, and vice versa. The following examples illustrate one or more embodiments.

[0191] Figure 1A The laptop computer 300 is shown to have a connection 194 (e.g., a wired or wireless connection), is associated with (e.g., logged into) the same user account as the tablet device 100, and has established a trusted connection with the tablet device (e.g., a trust prompt, such as the trust prompt described below, has been accepted by the user of the device). The laptop computer includes a display 302, which can also be a touch-sensitive display. In addition, in some embodiments, the laptop computer may also include a dynamic function row 304 for displaying additional information (additional details regarding such dynamic function rows 304 are provided in U.S. patent application serial number 15 / 655,707, which is hereby incorporated by reference in its entirety). In addition, the laptop computer also includes a keyboard 305. Relative to the tablet device 100, the tablet device 100 includes a touch-sensitive display 103 that can be capacitive sensing, and the device 100 is also capable of receiving input from an input device such as a stylus or a user's finger. Figure 1A Also shown is a selection operation performed with cursor 190 (e.g., by hovering or performing a right-click) on a maximize button 196 (e.g., a button displayed in a corner of a user interface window between two other buttons, which may also be rendered green). In the example below, a laptop computer may display a photo application window 189 (this is an example for explanation purposes, but other application user interfaces may also be displayed and used with the companion display modes discussed herein).

[0192] like Figure 1AAs shown, input at maximize button 196 causes menu 191 to be displayed, which includes selectable options that each control a display attribute associated with photo application window 189. For example, selectable option 192, when selected, causes photo application window 189 to be sent to tablet device 100 for display on touch-sensitive display 103. Other selectable options displayed in menu 191 control corresponding display attributes of photo application window 189 (e.g., a "full screen" selectable option, when selected, causes photo application window 189 to be displayed in full screen mode on display 302; a "send to device 2" selectable option, when selected, causes photo application window 189 to be displayed on device 2, and a "snap to corner or edge" selectable option, when selected, causes photo application window 189 to be displayed to cover the left side, right side, top, or bottom of display 302 of laptop device 300).

[0193] In some embodiments, both devices 100 and 300 are running their own respective operating systems, and when companion display mode is activated, the tablet device will continue to run its operating system, but will then receive information from device 300 that allows device 100 to display a user interface generated by device 300 (in some cases, device 100 also stops displaying any user interface elements associated with its operating system when companion display mode is initiated). In some embodiments, both devices may run the same operating system (e.g., two desktop devices running a mobile operating system or two laptop devices running a desktop operating system).

[0194] Figure 1B Shows the selected Figure 1A, causing the tablet device to enter companion display mode. The companion display mode allows the laptop computer 300 to utilize the tablet device as an additional monitor / display system, including sending a user interface to the tablet device (and also receiving information from the tablet device 100 about interacting with a user interface generated by the device 300 that is presented on the device 100 during the companion display mode). In some embodiments, when the companion display mode is operating on the tablet device 100, the touch-sensitive display 103 will display a companion display mode user interface 419, a control strip 197 that displays at least one control for controlling companion display mode functions (e.g., the control strip 197 helps the user change the display properties of the companion display mode without having to navigate hidden menus to change the companion display mode properties), and a tablet device dynamic function row 198, which may include application-specific controls that control functions within, for example, the photo application 189 or another application associated with the device 300. In some embodiments, the dynamic function row 198 includes controls that are the same as those provided on a touch-sensitive secondary display of a laptop computer (additional details regarding such controls are provided in application serial number 15 / 655,707, which is hereby incorporated by reference in its entirety).

[0195] Figure 1B The placement of the control strip 197 and the tablet dynamic function row 304 in the touch-sensitive display 103 is merely exemplary, as these control strips can be oriented in many different ways. For example, the control strip can also be placed on the right side of the touch-sensitive display 103. In another example, the control strip can be placed at the top or bottom of the touch-sensitive display 103. When the control strip is placed in different positions, the size of the companion display mode user interface 419 can be adjusted to fill the blank area of ​​the display. Users can choose to display the control strip 197 and the tablet dynamic function row 198 in different positions based on their preferences, which can be set in the companion display mode settings user interface 750 (such as Figure 7W In addition, the display of the control strip 197 or the dynamic function row 198 of the tablet device can also be turned off. When this happens, the companion display mode user interface 419 resizes to fill the now empty area previously occupied by the control strip 197 or the dynamic function row 198 displayed on the tablet device 100.

[0196] Figures 4A to 4QQ30 is a schematic diagram of a laptop computer display 302 and a touch-sensitive display 103 of a tablet device, which is used to illustrate exemplary user interfaces according to some embodiments. The user interfaces in these figures are used to illustrate the methods and / or processes described below. It will be understood by those of ordinary skill in the art that the following user interfaces are merely examples, and the user interfaces shown in each of these figures may be called in any particular order. In addition, it will be understood by those of ordinary skill in the art that different layouts with additional or fewer affordances, user interface elements, or graphics may be used in various situations. It should also be understood that any of the following exemplary user interfaces may correspond to separate embodiments, and no particular order need be followed.

[0197] Figure 4A 193 (e.g., the same ICLOUD account from APPLE INC. (Cupertino, CA), such as in Figure 4A 404 on the displays of the two devices and have an established connection 194 (e.g., a wired or wireless connection). When the two devices are logged into the same user account and have an established connection, the companion display mode may not yet be available until the devices have a trusted connection. For context, Figure 4A Small representations of laptop computer 300 and tablet device 100 are shown, with their respective displays shown in larger form below the associated small representations. Figure 4A A laptop computer and a tablet device are shown respectively running their respective operating systems, and their displayed user interfaces 406 and 407. The display 302 of the laptop computer 300 shows a task bar 412 with some applications available on the laptop computer, and the display of the tablet device 100 shows a separate task bar 411 with some applications available on the tablet computer.

[0198] Laptop computer 300 and tablet device 100 are both connected to the same Wi-Fi wireless network to illustrate that the devices have an established connection 194. Figure 4A , both devices are connected to the same wireless network "ABC Wi-Fi", as shown at 418 within the user interface of laptop computer 406 and 410 within the user interface of tablet device 407. In some embodiments, the users may not need to be on the same Wi-Fi network, and there may be other forms of connection between the two devices, such as near field communication (NFC), Bluetooth, or other short range communication protocols.

[0199] Figure 4BA prompt 413 is shown overlaid on top of the user interface of the tablet device 407, asking whether to "Trust this computer?" The prompt includes two selectable user interface elements: "Trust" 413-1 and "Don't Trust" 413-2. The prompt 413 is activated when the devices form a connection with each other (e.g., through a universal serial bus (USB) cable, or via a wireless communication protocol).

[0200] Figure 4C Input 414 is shown received at a "trusted" user interface element 413-1 within prompt 413. In response, a trusted connection is created between the two devices, as illustrated by the addition of a bidirectional arrow including the label "trusted 195" between the laptop computer 300 and the tablet device 100. Figure 4D As shown. Bidirectional arrows 193, 194, and 195 each represent fixed connection criteria that must be met before the companion display mode is available for use. In some embodiments, more or fewer criteria may be utilized, for example, the devices may only need to be logged into the same user account and have a connection without additional trust criteria. In some embodiments, when the devices have established a trusted connection, if the devices are later connected to each other again, the prompt will no longer be displayed.

[0201] Figure 4D A mouse pointer 190 (also referred to herein as a cursor 190) is shown on a display of a laptop computer 302 (e.g., an arrow pointing to an element that can be selected using a mouse click or other similar interaction). The pointer can be used to select an icon (e.g., photo application icon 416) for an application (e.g., photo application) from a taskbar 412 of the laptop computer. Once a selection is made, the laptop computer 300 executes the photo application and displays the photo application window 189, as shown. Figure 4E shown.

[0202] Specifically, Figure 4E Photos application window 189 is shown overlaying a portion of another user interface within user interface 406 of laptop computer 300 . Figure 4E Also shown is a dynamic function row 304 of the laptop computer that updates to provide controls based on the state of the application focused on the display of the laptop computer 300. For example, the photos application window 189 is in focus, so application-specific user interface elements associated with the photos application window are displayed (e.g., "Photos A" 421, "Photos B" 422, "Photos C" 423, and "Photos D" 424). In some embodiments, the dynamic function row also includes controls for controlling system-level functions of the laptop computer 300, such as an escape control 304, a volume control, and a brightness control, such as Figure 4E shown.

[0203] Next, in Figure 4F , pointer 190 is moved to hover over a maximize button 196 (e.g., a button that controls display properties of the photo application window, which may be displayed in green) associated with photo application user interface 189. In some embodiments, if the tablet device has established a trusted connection with the laptop computer, the appearance of the maximize button may be different. As shown, maximize button 196 is positioned between the close window button and the minimize window button, but the maximize button may be placed elsewhere and not combined with the other two buttons.

[0204] In response to selection of the maximize button 196 by the pointer 190, Figure 4G A user interface 191 is shown subsequently displaying the display properties of the window 189. In some embodiments, the selection may be made by performing a right-click on the maximize button 196 with the pointer 190, or may occur when the cursor 190 hovers over the maximize button 196 for a predetermined period of time (e.g., greater than 0.5 seconds). The user interface 191 may include a number of selectable options, some examples of which are placing the photo application window 189 in "full screen mode" on the laptop computer display 302, sending the photo application window 189 to another device (e.g., "Send to Device 1" 191, "Send to Device 2"), or snapping the photo application window 189 to another portion of the laptop computer display. In some embodiments, when snapped to a corner, the photo application window 189 may be displayed on half of the laptop computer's display 302, on the right or left side.

[0205] The pointer 190 may be moved to select one of the options, for example, option 192 ("Send to Device 1") within the user interface 191. Figure 4H As shown. User interface 191 may include more than one device to which the user interface can be sent. As long as the devices have met the security connection criteria (e.g., one or more of a trusted connection, a trusted user account), those devices can be displayed within user interface 191. User selection of selectable option 192 (e.g., "Send to Device 1") causes tablet device 100 to enter companion display mode. Companion display mode allows the tablet device to act as a secondary display for a laptop computer. In some embodiments, the display can be a screen extension or another desktop computer used and managed by the laptop computer 300 operating system.

[0206] Now go to Fig. 4IIn response to the pointer selecting a selectable option 192 (e.g., "Send to Device 1"), the user interface of the photo application stops being displayed on the laptop computer and is sent to be displayed on the tablet device 100. In addition, when the photo application window 189 from the laptop computer is sent to the tablet device 100, the tablet device enters the companion display mode and displays the companion display mode user interface 419. When operating in the companion display mode, a control strip 197 including controls for changing the display properties of the companion display mode is displayed. The controls within the control strip 197 are represented by user interface elements 427-437, and each user interface element has a function for controlling an aspect of the companion display mode. For example, the control strip 197 may include: a shift function 428, an option function 429, a command function 430, a software keyboard (e.g., a virtual keyboard) 431, a rotate display function 432, a rotate content function 433, functions 434 and 435 for manually placing the tablet computer to the left or right side of the laptop computer, a function 436 for bringing up the taskbar from the laptop computer, and a function 437 for exiting the companion display mode. In some embodiments, a toggle key 427 is also displayed, which allows the user to switch between different functions of the control strip 197. It should also be understood that the control strip can be placed in different locations, such as horizontally at the top or bottom of the display, or vertically on the left or right side of the display. In addition, in some embodiments, the control strip 197 is displayed next to and outside the user interface 419 (and not on top of it), such as for Fig. 7A As shown in the control strip 197, the control strip is displayed on the left side of the user interface 419.

[0207] Reference now Figure 4J , shows input made by pointer 190 at on-screen keyboard function 431, which causes on-screen keyboard 438 to be displayed. Although pointer 190 is shown in this example, input by a finger, stylus, or any other type of input device may be made in a variety of different situations relative to input received at control strip 197. In some embodiments, on-screen keyboard 438 may cause a displayed user interface (e.g., photo application window 189) to resize to avoid overlapping with on-screen keyboard 438. In another embodiment, on-screen keyboard 438 may overlap with a displayed user interface (e.g., photo application window 189). Although pointer 190 is in Figure 4J 100 is shown in conjunction with the companion display mode, but it should be understood that input can also be made by stylus 484 or finger without displaying any pointer 190 (e.g., an example of such interaction without a pointer can be seen in FIG. 100 with input 711). Figure 7F and with input 715 Figure 7JFor example, in some embodiments, touch-sensitive device inputs such as drag gestures, tap gestures, pinch gestures, unpinch gestures, and rotate gestures using fingers can be used with the companion display mode, and the pointer 190 is not displayed during such touch-sensitive device inputs. In some embodiments, the pointer 190 is not displayed for inputs provided over the control strip 197 or the dynamic function row 198.

[0208] like Figure 4K As shown, the user can provide another input at the on-screen keyboard function 431 to cause the on-screen keyboard 438 to be removed from the display of the tablet device 100. In some embodiments, if the keyboard causes the user interface element to be resized, once the on-screen keyboard 438 stops being displayed, the resized user interface element will be resized to its original size.

[0209] Figure 4L Selection of the display function 432 at the rotation control strip 197 by the pointer 190 is shown. In some embodiments, the degree of rotation may be predefined in the user's settings. In such embodiments, the direction of rotation may also be predefined in the user's settings. Additionally, there may be two buttons for rotating the display, one for clockwise direction and one for counterclockwise direction.

[0210] Figure 4M Shown is a response to selection of the rotate display function 432. When the rotate display function 432 is selected, the companion display mode user interface 419 rotates in a counterclockwise direction of 90 degrees. Figure 4M Also shown is a second selection of functionality for rotating the user interface of the tablet device.

[0211] Figure 4N A response to a second selection of the rotate display function 432 is shown, which causes the user interface 419 to rotate in a 90 degree clockwise direction. In some embodiments, when the rotate function is selected, the user may be able to specify how the user interface is to be rotated. Figure 4N A third selection for the rotate display function 432 is also shown.

[0212] Fig.4O A response to the third selection of the rotate display function 432 is shown with the accompanying display mode user interface 419 rotating in a 90 degree clockwise direction. Fig.4O Also shown is a fourth selection of functionality for rotating the user interface of the tablet device.

[0213] Figure 4P A response to a fourth selection of the rotate display function 432 is shown with the accompanying display mode user interface 419 being rotated 90 degrees in a counter-clockwise direction.

[0214] Next, in Figure 4Q , the rotation content function 433 is selected via point 190. Content may refer to the user interface of an application or content (e.g., an image) within a user interface. For illustrative purposes, the user interface of a photo application is rotated.

[0215] Figure 4R The photo application window 189 is shown in response to Figure 4Q 90 degrees in a counterclockwise direction by using the input shown in . The rotate content function 433 can also be used to rotate content within a user interface. For example, a photo within the photo application window 189 can be rotated. In some embodiments, the degree of rotation can be predefined in the user settings. In such embodiments, the direction of rotation can also be predefined in the user's settings. Figure 4R A second input at the rotate content function 433 is also shown.

[0216] Figure 4S A response to a second input at the rotate content function 433 is shown, where the content is rotated 90 degrees in a clockwise direction. In some embodiments, the user can specify how the content is rotated when the rotate content function is selected (e.g., by performing a right-click operation on the rotate content function 433 to display a more detailed rotation function). In addition, there may be two buttons for rotating the content within the control strip 197, one for causing a clockwise rotation and one for causing a counterclockwise rotation.

[0217] Figure 4T An input is shown above functionality 435 for manually arranging the tablet display to the right of the laptop display. In some embodiments, this ensures that the user's cursor and other user interface elements travel between the displays in a logical manner consistent with the user's expectations. In some cases, the user may find it useful to override the default arrangement of the displays by using buttons 434 and 435. Inputs above buttons 434 and 435 result in rearranging the displays. Figure 4T , Figure 4T-1 and Figure 4T-2 In the example of , having the tablet display to the right of the laptop display allows the user to move the pointer from the left edge of the tablet to the right edge of the laptop display, which Figure 4T-1 and Figure 4T-2 Shown in.

[0218] Figure 4U 4 shows a response to selection of function 434 for manually deploying the tablet to the left side of the laptop computer, where the tablet computer display is now to the left of the laptop computer display in the companion display mode arrangement. Figure 4U-1 and Figure 4U-2As shown, movement of pointer 190 travels from the right edge of the tablet computer display to the left edge of the laptop computer display.

[0219] Figure 4V An input is shown above function 435 for manually deploying the tablet to the right side of the laptop computer, which causes the tablet computer display to move back to the right side of the laptop computer display, as shown in FIG. Figure 4W shown.

[0220] Figure 4X An input above function 436 is shown to bring up a task bar associated with the laptop computer's operating system (e.g., Figure 4X 412 shown on the display of the laptop computer in FIG.

[0221] In response to the input above function 436, Figure 4Y-1 and Figure 4Y-2 The taskbar 412 is shown moving into position on the display of the tablet device 100. The taskbar 412 from the laptop is moving to Figure 4Y-1 (which is referred to as the taskbar 439 when displayed on a tablet 100) because it floats above the control strip 197. Figure 4Y-2 , the taskbar 439 has moved to its display position above the control strip 197. In some embodiments, the taskbar 439 can move the content on the tablet computer display to make room for the taskbar. Alternatively, the taskbar 439 can cover a portion of the content displayed on the tablet computer display.

[0222] To remove the taskbar 439 from the tablet display, additional input is provided above the user interface object 436, such as Figure 4Y-2 As shown, then Figure 4Z The taskbar is no longer displayed in 439.

[0223] Figure 4AA The photo application window 189 is shown restored to its original position once the task bar 439 generated by the laptop disappears.

[0224] Figure 4BB Selection at toggle switch 427 with pointer 190 is shown for switching between different functional modes of control strip 197. In embodiments where dynamic function row 198 is displayed simultaneously with control strip 197 on the tablet display, toggle switch 427 may not be displayed within control strip 197 at all.

[0225] In response to Figure 4BB The input at the switch 427, Figure 4CCA dynamic function row 198 is shown subsequently displayed, which allows control of application-specific functions of the currently displayed application (e.g., for selecting a photo within the photos application window 189).

[0226] Figure 4DD An upward swipe gesture on a display of the tablet device 100 is shown. While the tablet device is operating in companion display mode, the user can still access gestures that are processed by the tablet device's operating system. Thus, in some embodiments, the tablet device distinguishes between different input types (e.g., stylus or finger) that are processed by one of the tablet device's operating system or the laptop device's operating system. Figure 4DD The upward swipe gesture shown begins anywhere at the bottom edge of display 440 and moves in an upward direction substantially perpendicular to the bottom edge of the display. Gesture 441 travels until it reaches a predetermined distance away from the edge (e.g., slightly below the midpoint of the tablet display), as shown. Figure 4DD As shown, at this time, the contact associated with the swipe gesture 441 is lifted off the tablet computer display.

[0227] Figure 4EE Shows the reception Figure 4DD In response to the upward swipe gesture 441 of the tablet device. In response to the upward swipe gesture 441, an application switching user interface is displayed at the tablet computer display 103. The application switching user interface includes representations of recently used applications (e.g., a representation of "Message Application" 446), and representations of one or more desktops from the laptop computer (e.g., representations of the desktops of "Racing Game" 443, "Desktop 2" 444, and "Desktop 1" 445). These representations are all displayed simultaneously within the same application switching user interface, allowing the user to easily switch between recently used applications from the tablet device or switch to using various desktops from the laptop device. In addition, a taskbar 411 of the tablet device is also displayed at the bottom of the touch-sensitive display, which can be used to execute additional applications that may be different from those shown in the multitasking view.

[0228] Input 447 (eg, a tap input where contact touches representation 445 and then lifts off of the tablet display) is received at a representation of "Desktop 1" 445, which was a previously displayed desktop running within companion display mode.

[0229] Figure 4FFThe device is shown returning to a display of a user interface associated with "Desktop 1" of a laptop computer. After returning to "Desktop 1" of the companion display mode, a second upward swipe gesture is received, wherein the second upward swipe gesture 442 travels a longer distance than the first swipe gesture 441 described above. The second upward swipe gesture begins anywhere at the bottom edge and moves in an upward direction substantially perpendicular to the bottom edge of the display. In some embodiments, the gesture ends beyond the midpoint of the tablet computer display.

[0230] Figure 4GG The home screen of the tablet device is shown as being displayed in response to an upward swipe gesture 442. The home screen includes a taskbar 411 of the tablet device, and multiple representations of applications that can be launched on the tablet device. In addition, within the multiple representations, there is a representation for returning to a companion display mode 443. In some embodiments, the icon 443 is only available when the tablet device is capable of operating in a companion display mode. The representation for returning to the companion display mode 443 can be placed in the next available position of the representation for displaying the application on the home screen of the tablet device. Compared to the multiple representations, in some embodiments, the icon 443 only appears when the companion display mode is utilized, and the icon 443 does not correspond to an application installed on the tablet device through an application store, etc.

[0231] Alternatively or in addition to displaying icon 443 adjacent to multiple representations on the home screen, icon 443 may also be displayed within task bar 411, such as Figure 4HH In some embodiments, the icon 443 for returning to the companion display mode may be temporary and remain unchanged as long as the device is capable of operating in the companion display mode.

[0232] Fig. 4II An input 444 is shown over the representation for returning to the companion display mode 443 in the taskbar 411. Alternatively, the input may be made over the representation for returning to the companion display mode when the companion display mode 443 is displayed with multiple representations rather than within the taskbar.

[0233] Figure 4JJ The tablet device is shown ceasing to display its home screen and returning to the companion display mode user interface 419. The companion display mode user interface 419 shows the same photo application window 189 discussed with reference to some of the preceding figures, and also again displays the control strip 197 having a plurality of functions for controlling display properties of the companion display mode. In some embodiments, the pointer 190 is placed in the same position that it was previously in before the tablet device 100 entered its home screen.

[0234] like Figure 4KK-1As shown, the task bar of the tablet device can be initially called out by a short swipe gesture (e.g., a swipe gesture that travels less than a quarter of the vertical length of the touch-sensitive display of the tablet device). The short swipe gesture is shown by an upward swipe gesture 4002.

[0235] In some embodiments, such as Figure 4KK-2 As shown, the tablet device's taskbar 411 can overlay a portion of the companion display mode user interface 419. After bringing up the tablet device's taskbar 411, a swipe gesture 448 starting over the email application 444 can be used to activate a split screen view in which two applications are displayed side by side on the tablet device display.

[0236] Figure 4LL 444. Once the split screen view request is received at the representation of the email application 444, the tablet device's email application occupies half of the display, with the other half of the display being used to display the companion display mode user interface. In some embodiments, the two interfaces are divided by dividing line 467. Figure 4LL As shown, the companion display mode user interface 419 can be resized to fit the email application user interface 450 of the tablet device. Within the companion display mode, the photo application window 189 and the control strip 197 are both reduced in size. In such embodiments where the size of the control strip is reduced, the control strip 197 can then include an expanded more functions icon 449 to allow the user to still interact with the control strip functions that disappeared due to the resizing. In other embodiments, the control strip 197 can be displayed in the bottom portion of the tablet display and can span between two different user interfaces so that its horizontal width does not need to be reduced.

[0237] like Figure 4LL As shown, the email application user interface includes emails 451-1, 451-2 and 451-3, and there is a write new email icon 452 at the lower right corner of the email application user interface 450. In addition, input 453 is Figure 4LL 4 is shown above the compose new email icon 452 in the lower right corner of the email application user interface 450.

[0238] Figure 4MM The resulting new message user interface 454 is shown within the email application 450. The new message user interface includes a "Send To" section 454-1 for indicating to whom the email is to be sent, a "Subject" section 454-2 for stating the subject of the email, and a "Body" section 454-3 for including the body of the email.

[0239] Notably, in some embodiments, the companion display mode allows for bidirectional drag-and-drop operations between different operating systems (e.g., between a mobile operating system of a tablet device and a desktop operating system of a laptop device). This allows files (e.g., images, text documents, etc.) to be dragged from a user interface generated by one device (e.g., the companion display mode user interface 419 generated by the laptop computer 300) to a user interface generated by another device (e.g., the email application user interface 450 generated by the tablet device 100). In other words, a user can drag a file from one operating system (e.g., a desktop computer / laptop operating system) to another operating system (e.g., a tablet device operating system) using a single gesture.

[0240] Figure 4MM Also shown is a swipe gesture 458 received that starts over the representation of "Photo H" 457 and moves to the right. The purpose of such a gesture is to copy or move the image to the body portion 454-3 of the email.

[0241] Figure 4NN The swipe gesture 458 is shown continuing to eventually end within the body portion of the email 451-3. Figure 4NN , "Photo H" 457 crosses the dividing line 467 between two user interfaces (e.g., companion display mode user interface 419 and email application user interface 450). Thus, the user can quickly transfer files from one user interface generated by the laptop computer and another user interface generated by the tablet device 300. In some embodiments, the representation of the photo within the photo application window 189 can change its appearance as a result of being selected and dragged (e.g., "Photo H" 459 has a dashed line around it when it is dragged over the email application).

[0242] Figure 4OO "Photo H" 457 is shown passing through dividing line 467 and entering the user interface of the email application 450. Specifically, "Photo H" is dragged to the body portion 454-3 of the email.

[0243] Figure 4PP Shown once the gesture ends (e.g., the contact associated with the gesture is lifted from the tablet display) and "Photo H" is placed within the body portion 454-3 of the email. In some embodiments, "Photo H" will no longer be a representation, and an expanded "Photo H" 459 will be displayed within the body portion 454-3 of the email.

[0244] Figure 4QQ An input is shown above the send 456 user interface element for sending an email to the address included within the "Send To" portion 454-1 of the email.

[0245] Figure 4RR New message user interface 454 is shown replaced with a previous user interface, which is an inbox user interface within email application 450 . Figure 4RR A user input 462 is shown selecting an email 451 - 2 of the email form “friend@e-mail.com”, wherein a portion of the subject or body of the email is displayed.

[0246] Figure 4SS The user interface is shown updated to show an email associated with the representation of email 451-2. Within the email, the body portion of the received email 461 is shown, and a receipt image 463 is included within the body portion of the received email.

[0247] Figure 4TT Swipe gesture 464 is shown starting from receipt image 463 and moving toward dividing line 467. As described above, when a user is able to transfer files from one user interface generated by one device (e.g., a tablet device email application) to another user interface generated by another device (e.g., a laptop companion display mode), an improved human-computer interface is enabled because the user avoids wasting time transferring content between two devices in an inefficient manner.

[0248] Figure 4UU A continuation of the swipe gesture 464 is shown. The receipt image 463 changes to a reduced image size 465 for easier dragging around. The receipt image 463 may change its appearance state in response to the drag gesture, which is shown by the dotted line now surrounding the receipt image 463.

[0249] Figure 4VV The receipt image 465 is shown passing through the dividing line 467 and entering the user interface of the companion display mode 419. Specifically, the receipt image is dragged to the photo application window 189.

[0250] Figure 4WW The receipt is shown now entirely within the companion display mode user interface 419 and no longer interacting with the tablet device's email application user interface 450.

[0251] Figure 4XX A swipe gesture 468 is shown starting from dividing line 467, which allows the mail application user interface 450 to be dismissed, as shown in FIG. Figure 4YY to Figure 4ZZ shown.

[0252] Figure 4YY Also shown is that a swipe gesture 468 can be used to adjust the size of the user interface in the split-screen view (in Figure 4YY In the example shown in , the companion display mode user interface 499 is resized to a larger size and the email application user interface 450 is resized to a reduced size).

[0253] Figure 4ZZ The resulting user interface is shown when swipe gesture 468, which began at dividing line 467, ends near the edge of the display. As discussed, email application user interface 450 is dismissed (e.g., no longer displayed), and the accompanying display mode user interface 419 once again occupies substantially all of the display (e.g., all of the display except for the status bar portion at the top of the display). Figure 4ZZ Also shown is the user scrolling the photo application window 189 via a swipe gesture 471 that may be performed by a finger. Scroll bar 470 indicates where the user interface is currently located and is updated in response to the finger swipe gesture 471.

[0254] When operating in companion display mode, where the possibility exists for interacting with two different operating systems (e.g., an operating system of a laptop computer (e.g., companion display mode) and a device operating system of a tablet computer), it may be advantageous to allow only some inputs that correspond to only one operating system and not the other. For example, in some cases, finger inputs may be more suitable for controlling a tablet device operating system, while stylus inputs may be more suitable for controlling a user interface generated by the laptop computer 300. Nonetheless, a user may have become accustomed to some inputs made using fingers to manipulate content, and not allowing such inputs may be frustrating to the user. Therefore, finger inputs, while typically directed to the operating system of the tablet device, may also be mapped to the operating system of the laptop device to avoid user frustration and confusion. Therefore, in some embodiments, certain finger inputs may be received by the companion display mode and may be used to manipulate content within the companion display mode (e.g., the operating system of the laptop computer).

[0255] Figure 4AAA Shown is a photo application scrolled through, and a set of new photos (e.g., photos IL) displayed in response to the scroll. In addition, scroll bar 470 moves in response to a swipe gesture received within photo application window 189.

[0256] Figure 4 BBBA two-finger tap gesture 472 is shown received at a representation of "Photo E" 473, and the two-finger tap gesture may allow a user to access secondary options (e.g., a menu associated with a right-click function). In response to the two-finger tap gesture, a right-click menu is displayed, which includes: a "get info" user interface element 474, which, when selected, causes the device to display an independent user interface that displays information about "Photo E" 473; a rotate user interface element 475, which, when selected, causes the photo to rotate a predetermined amount; an "edit" user interface element 476, which, when selected, causes the photo to enter an edit mode; and a "delete" user interface element 477, which, when selected, causes the photo to be removed from the photos window 189.

[0257] Figure 4CCC An input 478 is shown received at the edit user interface element 476, which causes the photo to enter an edit mode. The photo edit mode may cause the photo to expand on the tablet display.

[0258] Figure 4DDD The resulting user interface that appears when the representation of "Photo E" 473 is placed in an editing mode is shown. In the photo editing mode, the thumbnail representation of "Photo E" 473 is no longer displayed, but the full "Photo E" 473-1 is displayed.

[0259] Figure 4EEE 479 is shown to be in edit mode, the photo can be manipulated through a two-finger pinch gesture using contacts 480-1 and 480-2 received over "Photo E" 479. In response to such a pinch gesture, the size of "Photo E" is adjusted to a reduced display size, such as Figure 4EE shown.

[0260] Figure 4FFF Photo E 479 is shown receiving a two-finger spread gesture from contacts 481 - 1 and 481 - 2 , which is used to expand (eg, enlarge) Photo E 479 .

[0261] Figure 4GGG Shown are changes resulting from receiving a two-finger expand gesture at "Photo E" 479, where "Photo E" 479 is expanded to a new, larger display size.

[0262] Figure 4HHH Two contacts 482-1 and 482-2 at "Photo E" are shown, each moving in a clockwise direction and rotating around each other. Such input causes "Photo E" 479 to rotate in a clockwise direction. If the two finger inputs 482-1 and 482-2 move in a counterclockwise direction, then "Photo E" will rotate in a counterclockwise direction instead.

[0263] Figure 4III Shown is a response to receiving two contacts 482-1 and 482-2 each rotating around each other in a clockwise direction at "Photo E." As shown, "Photo E" rotates an amount depending on the distance traveled by the clockwise rotation of the two contacts 482-1 and 482-2.

[0264] in addition, Figure 4III The two contacts 482-1 and 482-2 at "Photo E" are shown rotating around each other in a counterclockwise direction, which will cause "Photo E" to rotate in a counterclockwise direction, which is then Figure 4JJJ Shown in.

[0265] Figure 4KKK The same companion display mode user interface 419 is shown with the photo application window 189, and also shows that the stylus 484 can be used as an input device. When the stylus is physically in contact with the tablet computer display 103 and dragged along the tablet computer display 103, or moved a predefined distance above the display (e.g., a hovering movement of a cursor above the display but not in contact with the display), the pointer 190 follows the movement of the stylus. In some embodiments, the stylus is used to control the movement of the pointer 190, but the user's finger does not control the movement of the pointer 190. In addition, Figure 4KKK A tap input 4004 by the stylus 484 at the search bar 4003 is shown. In response to the tap input by the stylus 484, a left-click operation may be performed.

[0266] Figure 4 LLL Shown is a response to receiving a tap input 4004 by stylus 484 for performing a left-click operation at search bar 4003. In response to such input, search bar 4003 expands to display previously conducted searches, such as "dogs" 485 and "signatures" 486. These previously conducted searches may be selected, and the displayed photos (or representations of photos) will include photos relevant to the search criteria.

[0267] Figure 4 MMM Stylus 484 is shown, as well as pointer 190 moved by stylus 484 to cause selection of a previously made search for "Signature" 486. In response to this selection, which may be made by a tap input by stylus 484, the "Signature" 486 box turns gray to show that it has been selected.

[0268] Figure 4NNN Shown is a response to selecting a previously conducted search for “signature” 486. Such a selection causes the photo application window 189 to display photos and videos associated with the previously conducted search for “signature.” The previously displayed photos are no longer displayed unless they are associated with a signature. Figure 4NNNAlso shown is movement of the stylus 484 causing the pointer 190 to move to select Figure 4NNN A single-click gesture may be made with the stylus to indicate that a left-click is desired on top of “Signature Video A” 487 to open “Signature Video A” 487. It is noteworthy that “Signature Video A” 487 is a video accessed from the laptop computer 300 rather than from the tablet device 100.

[0269] Figure 4OOO "Signature Video A" 487 is shown playing on a tablet device and using the speakers of the device (e.g., tablet device) on which the video is displayed. Although the video content is generated by laptop computer 300, viewing "Signature Video A" 487 on one device (e.g., tablet device 100) but listening to the associated audio of "Signature Video A" 487 on another device (e.g., laptop computer 300) may confuse the viewer. Figure 4OOO It is shown that the laptop's speakers 311 may be muted, while the tablet device 100 plays only the associated audio of "Signature Video A" 487 on its speakers 111. In other embodiments, the speakers of both the laptop and tablet may be used together.

[0270] Figure 4PPP A downward swipe gesture 489 is shown starting from the upper right corner of the tablet computer display to trigger display of a control center user interface 4005 having a plurality of controls for controlling functionality of the tablet device. The control center user interface 4005 may include controls such as: a Bluetooth toggle key 491 for turning the tablet device's Bluetooth on or off; a Wi-Fi toggle key 492 for turning the tablet device's Wi-Fi on or off; a Do Not Disturb toggle key 493 for putting the device into a mode in which notifications are actually displayed to the user; a Low Power Mode toggle key 494 for preserving battery life; a Lock Orientation toggle key for allowing the device user interface to rotate or not rotate based on sensor data; and a Text Size icon 496 for entering a user interface to quickly adjust the size of text within the tablet device's user interface. In some embodiments, the control center user interface is displayed as an overlay to the companion display mode user interface (e.g., the control center user interface is displayed as an overlay to the companion display mode user interface). Figure 4PPP In some embodiments, if an application switching user interface was previously displayed (an example of which is shown in 4QQQ), the control center user interface can overlay or include the application switching user interface, such as the companion display mode user interface shown in FIG. Figure 4EE described.

[0271] Figure 5A-1 to Figure 5E-2Schematic diagrams of a display of a laptop computer and a display of a tablet device are used to illustrate exemplary user interfaces according to some embodiments. The user interfaces in these figures are used to illustrate the methods and / or processes described below. It will be understood by those of ordinary skill in the art that the following user interfaces are merely examples. In addition, it will be understood by those of ordinary skill in the art that different layouts with additional or fewer affordances, user interface elements, or graphics may be used in various situations.

[0272] Figure 5A-1 to Figure 5E-2 4 shows how the same input can result in two different responses, depending on whether the input is made by a user's finger or by a stylus device 484. The following figures are shown together to show that the same input is received at the same device, and depending on the input type (e.g., stylus or finger), different responses occur.

[0273] Figure 5A-1 to Figure 5E-2 The enlarged display of the laptop device 100 is omitted to emphasize the difference between the gestures as they appear on the tablet device. Nevertheless, a representation of the laptop computer 100 is shown connected to the tablet device 300 to indicate that the devices are still operating in the companion display mode.

[0274] Figure 5A-1 and Figure 5A-2 Both show a downward swipe gesture made from the top center area of ​​the display. Figure 5A-1 A downward swipe gesture 500 is shown using a stylus 484 (eg, an APPLE PENCIL designed by APPLE INC. (CUPERTINO, CA)). Additionally, the gesture moves in a downward direction substantially perpendicular (eg, within + / - 5 degrees of vertical) to the top of the display. Figure 5A-2 A downward swipe gesture 502 is shown as a user's finger 501 moves in a downward direction as indicated by directional arrow 502 .

[0275] Figure 5B-1 and Figure 5B-2 Both show responses to a downward swipe gesture. Figure 5B-1 As shown, in response to determining that the gesture is performed using a stylus, the cursor 190 moves to a second position on the display according to the illustrated movement of the stylus. Figure 5B-2 As shown, in response to determining that the gesture is performed using the user's finger, the notification center user interface 503 of the tablet device is displayed at the top of the companion display mode. The notification center user interface of the tablet device includes a time and date indicator 504 and may contain multiple notifications (e.g., news application user interface element 502 and instant messaging application user interface element 503).

[0276] Figure 5C-1 and Figure 5C-2Both show an edge swipe gesture moving from the right side of a tablet display to the left. Figure 5C-1 Stylus 484 is shown contacting the right edge of tablet computer display 103, and as stylus 484 contacts the display, pointer 190 is shown at the location where contact is made (in some embodiments, pointer 190 may also follow the stylus as it hovers over the tablet computer display). After contact, the stylus moves in left direction 506 to display the laptop computer 300 notification center user interface ( Figure 5D-1 ). Figure 5C-2 The user's finger is shown making contact 507 near the right edge of the tablet computer display 103, and then moving in a left direction 508 to reveal a slide ( Figure 5D-2 ).

[0277] Figure 5D-1 and Figure 5D-2 Both show the response to the edge swipe gesture moving to the left. Figure 5D-1 As shown, in response to determining that the gesture is provided by the stylus device, the laptop computer 300 notification center user interface is displayed above the companion display mode user interface. The notification center user interface may include multiple notifications. Figure 5D-2 As shown, in response to determining that the gesture is provided by the user's finger, a slide over the user interface is displayed for partially covering a portion of the companion display mode with the previously executed application 510. For example, the slide over the user interface may include "Racing Game" 5003.

[0278] Figure 5E-1 and Figure 5E-2 Both show a bottom edge swipe gesture starting from the bottom edge of a tablet display. Figure 5E-1 , a bottom edge swipe gesture is performed using stylus 484 (contacted by the stylus near the bottom edge of the tablet display and moved in an upward direction 511), while in Figure 5E-2 , a bottom edge swipe gesture is performed using the user's finger (contact 512 is made near the bottom edge of the display and moves in an upward direction 513).

[0279] Depending on which input object is used to provide the bottom edge swipe gesture, the results are different. For example, Figure 5F-1 4 shows that in response to a bottom edge swipe provided using a stylus, a taskbar 412 from the laptop computer 300 is displayed within the companion display mode on the tablet computer display. Figure 5F-2 As shown, if a bottom edge swipe gesture is provided using the user's finger, the task bar 411 is displayed instead.

[0280] In some embodiments, when the stylus is used as an input object at the tablet computer display when the desktop is operating in the companion display mode, information about the input provided by the stylus is sent to the laptop for processing; and when the user's finger is used as an input object at the tablet computer display when the tablet computer is operating in the companion display mode, the tablet computer processes the input without sending information to the laptop. In this way, the user is provided with an intuitive way to interact with the features available in two different operating systems, thereby improving the human-computer interface.

[0281] Now turn your attention to FIG. 6A to FIG. 6L , which shows a schematic diagram of a user interface used in conjunction with an annotation mode for marking content (e.g., where the content is an image, screenshot, etc.). In some embodiments, the annotation mode allows a user to send content from one device to a different device for marking on the different devices. The annotation mode is entered when the first device instructs the second device to enter the annotation mode. Instructing the device to enter the annotation mode allows the user to utilize features of the second device that may not be present on the first electronic device (e.g., the use of a touch-sensitive display and a stylus that may not be available at the first device). The user can mark content using multiple marking options (e.g., pencil marking, pen marking, highlight marking, etc.).

[0282] FIG. 6A to FIG. 6L The user interfaces shown in are used as examples to illustrate the methods and / or processes described below. One of ordinary skill in the art will appreciate that the following user interfaces are merely examples. In addition, one of ordinary skill in the art will appreciate that different layouts with additional or fewer affordances, user interface elements, or graphics may be used in various situations.

[0283] Fig. 6A A screenshot is shown taken on a laptop computer 300 while the laptop computer is connected to a tablet device 100. Similar to the above description regarding the companion display mode, one or more secure connection criteria (user account 193, connection 194, and trust 195) may be used to determine when content may be sent to another device for annotation mode. Fig. 6A 601 in the user interface element. Also shown is a tablet device 300 that can operate in a locked state (e.g., a state in which the tablet display is not displaying anything). In some embodiments, the tablet device 100 can operate in an unlocked state and can additionally display user interfaces for some applications available on the tablet device. When the tablet device 100 is operating in any of the states discussed, activating the annotation mode on the device 300 immediately replaces any interface currently on the tablet display.

[0284] Figure 6BInput 602 on a screenshot user interface element 601 is shown made within a laptop computer display.

[0285] After selecting the screenshot UI element 601 using input 602, Figure 6C A user interface 1601 is shown for displaying a full view of the screenshot 601 and then displayed on the laptop display. The user interface 1601 includes a user interface element 603 for entering the annotation mode, and Figure 6C Providing input 604 at user interface element 603 is shown.

[0286] Upon selecting user interface element 603 for entering annotation mode, the laptop may then display a submenu with multiple devices that can be used with annotation mode. In some embodiments, when the user has approved another device (e.g., device 100) for use with companion display mode (or for use with annotation mode), a single input over user interface element 603 does not bring up the submenu, but instead initiates sending the content (e.g., the screenshot discussed above) to the approved device.

[0287] exist Fig.6D , another device has not yet been approved, so in this example, a submenu 606 is presented on the laptop display showing a number of devices that can be used with the annotation mode. Fig.6D An input 607 is shown at the "Edit on Device 1" selectable option 605, which, when selected, causes the laptop computer 300 to send the screenshot 601 to the tablet device 100 (e.g., device 1) for use with the annotation mode. For purposes of this example, the annotation mode is shown on the tablet device 100, but this may be done with other devices or with multiple devices simultaneously (e.g., a user requests to enter the annotation mode on multiple devices simultaneously). In addition, the devices shown within the submenu may be displayed based on those devices that have satisfied one or more fixed connection criteria (examples of these fixed connection criteria are described above).

[0288] Fig. 6E It is shown that in response to input 607 at the "Edit on Device 1" selectable option 605, the tablet device then enters an annotation mode for the screenshot 601. Fig. 6E, the annotation mode user interface 699 presented on the tablet display includes the same content as that displayed on the laptop computer 300 (e.g., screenshot 601), which allows the user to provide input on the tablet and see those inputs reflected on the tablet display and the laptop display in a substantially simultaneous manner (e.g., after providing input on the tablet, the input is reflected on the laptop display in less than 0.25 seconds). Screenshot 601 is an image capture of the user interface currently displayed on the laptop computer 300. As shown, the annotation mode also includes a plurality of marking tools for marking content (e.g., screenshot 601) on the tablet device. These marking tools include a pencil / pen tool 608, an eraser tool 609, and other similar tools (these tools may be displayed on either or both of the laptop computer and the tablet).

[0289] Likewise Fig. 6E As shown, stylus 484 may be used to draw line segment 611 on top of screenshot 601 .

[0290] At substantially the same time as the input is received on the tablet device 100, the tablet device sends information to allow the update to be displayed at the laptop computer 300, so that the laptop computer 300 can also display the tagged content. FIG. 6E to FIG. 6H , additional input is provided by stylus 484 at the top of screen shot 601 (respectively at FIG. 6G to FIG. 6H ), and these inputs result in markup annotations being added on top of screenshot 601, and these markup annotations can be displayed at both the tablet and the laptop.

[0291] Figure 6H An input is shown pointing to a "done editing" user interface element 618. Upon selection of the "done editing" user interface element 618, the device invoked to operate in annotation mode will return to its previous state (e.g., screen off state, locked state, unlocked state, or previous user interface state, etc.).

[0292] Fig.6I A response to selection 617 of a "done editing" user interface element 618 is shown. Upon selection of the "done editing" user interface element 618, the tablet device 100 returns to its previous state, which in this example is a locked state. Additionally, in response to selection of the "done editing" user interface element 618, the laptop displays a screenshot with a markup annotation added at the tablet device 100.

[0293] Figures 6J to 6LAn embodiment is shown in which a user of a first computing device (eg, laptop computer 300 ) is able to take screenshots at laptop computer 300 and annotate them on tablet device 100 . Figure 6J Shown in use Figure 6J The user interface elements indicated by numeral 690 are shown in the new screenshot. The new screenshot 691 is an image capture of the user interface currently displayed on the laptop computer 300. Figure 6J Also shown is input 691 received at or on the new screenshot 690. Figures 6J to 6L As shown, in this embodiment, the user does not need to select a device on which annotation will be performed because (i) the user previously selected a device for annotation and the laptop computer 300 stored the selection. Thereafter, if the selected device (e.g., tablet device 100) is available, the laptop computer 300 will automatically invoke the annotation mode on the previously selected device (e.g., tablet device 100) without human intervention.

[0294] Figure 6K A response to input 691 received at or on a second screenshot 690 is shown. In response to input 691, screenshot 690 is displayed within user interface 1601 on laptop computer 300. The user may then Figure 6K Input 692 at user interface element 603 is shown selecting annotation mode.

[0295] Figure 6L The tablet device 100 is shown in FIG. 600 from the closed state (eg, Figure 6K ) transitions to the annotation mode user interface 699. It is worth noting that due to the previous selection of the device for annotation (e.g., Fig.6D 1) and is stored so that the prompt for selecting a device to annotate is no longer shown (assuming the previously selected device is connected and available for use). In other words, the laptop computer 300 will select the device to annotate based on the previous selection stored at the laptop computer 300 (e.g., Fig.6D to select the device that enters annotation mode without human intervention.

[0296] 7A to 7Wis a schematic diagram of a laptop computer display and a tablet computer display, which is used to illustrate an exemplary user interface used in conjunction with a companion display mode according to some embodiments. The user interfaces in these figures are used to illustrate the methods and / or processes described below. One of ordinary skill in the art will understand that the following user interfaces are merely examples. In addition, one of ordinary skill in the art will understand that different layouts with additional or fewer affordances, user interface elements, or graphics may be used in various situations.

[0297] As described above, in some embodiments, control strip 197 can be displayed outside the window borders of user interface 419, and dynamic function row 198 can be displayed in addition to control strip 197, and the dynamic function row can also be displayed outside the window borders of user interface 419. In some cases, displaying control strip 197 and dynamic function row 198 outside the window borders of UI 419 helps avoid user confusion during companion display mode. An example of this display arrangement is shown in FIG. Fig. 7A , which shows that both the control strip 197 and the dynamic function row 198 are displayed outside the window border 419. Figures 4A to 4QQ The functionality discussed for the companion display mode also applies to 7A to 7W The user interface shown in the figure. Fig. 7A As shown, the companion display mode user interface 419 can be displayed with a simulated circular edge 752 that simulates the physical edge of a mobile device 753, which is also circular.

[0298] Figure 7B An exemplary arrangement of the control strip 197 and the dynamic function row 198 is shown, wherein the control strip 197 is alternatively displayed on the right side of the user interface 419 (using reference Figure 7W The user may also use other display layout options as described in the settings interface). Figure 7B It is also shown that the control strip 197 may include additional functions for controlling the accompanying display mode. These functions include keyboard modifier keys such as the shift key 428, the option key 429, the command key 430, and the control key 702.

[0299] These functions also include an undo or redo key 703, which is shown in the control strip 197. Such functions can be used to redo an input or to redo an input. When a user performs multiple inputs, the undo or redo function can be performed more quickly rather than manually deleting the input information or reinserting previously input information.

[0300] These functions also include cursor functions accessed from within the control strip 197, where the cursor functions include: right click 704, left click 705, and center click 706. Accessing these functions allows the user to save time when interacting with the user interface presented in conjunction with the companion display mode (otherwise, the user may waste time searching aimlessly for the desired function).

[0301] Because the three-dimensional image can rotate along three independent axes, it may be difficult to interact with the three-dimensional image. Therefore, the functions accessible through the control strip 197 may also include a yaw-pitch-roll function 707, which is extended to lock the three-dimensional image to rotate along a specific one or more axes. In addition, Figure 7B The control strip 197 may include a menu function 708 for displaying or hiding a top-level status menu bar 730 generated by the laptop device.

[0302] The remaining functionality available in the control strip 197 (as described in the dynamic function row 198) is discussed above, and for the sake of brevity, those descriptions are not repeated here.

[0303] Figure 7C The dynamic function row 198 is shown to also be displayed in different locations, including at the top portion of the tablet computer display, such as Figure 7C Another example of the location of the control strip 197 (right side of the tablet display) and the dynamic function row 198 (top of the tablet display) is shown in Fig.7D Shown in.

[0304] Fig. 7E An input 709 is shown pointing to a menu function 708 for displaying or hiding a top-level status menu bar 730 generated by the laptop. Fig. 7E , selecting menu function 708 from within control strip 197 causes menu bar 730 to be hidden, as shown in FIG. Figure 7F shown.

[0305] Figure 7F An input 711 is shown on a shift function 710 displayed within the control strip 197. In some embodiments, the selection can be made by pressing and holding (so that the function is activated only while the function is still held) or double-clicking (locking the function into a persistent activation mode and activating without requiring the function to be held). In this example, the input 711 is a double-click input over the shift function 710, which results in a persistent selection of the shift key in the control strip 197.

[0306] Go to Figure 7G, the shift key is now in a persistent selection state, just as if the user is holding down the shift key on a physical or virtual keyboard. As shown, the shift key 710 appears with a modified visual appearance (e.g., a darkened or shaded state) to indicate a persistent selection. Figure 7G Also shown is the stylus 484 used to select "Photo A" 712 with a tap on the screen. Since the shift key is selected in the control strip; selecting an item will highlight it and may allow the user to select multiple items to interact with.

[0307] Figure 7H It is shown that when the shift key is in a persistent selection state, a second input is received at "Photo B" 713, and both "Photo A" 712 and "Photo B" 713 will change appearance states to show that they have been selected (e.g., a corresponding dotted line around each selected photo).

[0308] Fig.7I It is shown that when the shift key is in a persistent selection state, a third input is received at "Photo C" 714, and then the photos "Photo A" 712, "Photo B" 713 and "Photo C" 714 will change appearance states to show that they are all selected (e.g., corresponding dotted lines around each selected photo).

[0309] Figure 7J It is shown that when the shift key is in a persistent selection state, input 715 is received over the left-click function 705 displayed within the control strip 197. In some embodiments, input 715 can be made by pressing and holding (so that the function is activated only while the function is still held) or double-clicking (locking the function into a persistent activation mode and activating without the function being held). By placing the shift function and the left-click function of the control strip 197 in a persistent selection state, the input at the tablet computer takes into account both persistent selections.

[0310] For example, in Figure 7K , input from stylus 484 is received and causes a selection box 716 to be drawn around the items the user wishes to select. The selection box begins at the location where the stylus first contacts the display and ends at the location where the stylus is lifted from the display. Since the shift key is also selected, the previously selected photos (e.g., "Photo A" 712, "Photo B" 713, and "Photo C" 714) remain selected in addition to anything selected using the selection box 716.

[0311] Figure 7L"Photo D" 717 is shown surrounded by a selection box 716. As the stylus 484 moves along the display, the selection box 716 follows the stylus. Because the selection box surrounds "Photo D" 717, the appearance state of "Photo D" 717 changes similar to how the appearance state of previous photos changes in response to being selected (e.g., the outer dashed line around "Photo D").

[0312] Figure 7M "Photo D" 717 and "Photo E" 718 are shown surrounded by a selection box 716. As the stylus 484 moves along the display, the selection box 716 follows the stylus. Because the selection box surrounds "Photo E" 718, the appearance state of "Photo E" 718 changes similar to how the appearance state of the previous photo changes in response to being selected.

[0313] Figure 7N "Photo D" 717, "Photo E" 718, and "Photo F" 719 are shown surrounded by selection box 716. Selection box 716 follows stylus 484 as it moves along touch-sensitive display 103. Because the selection box surrounds "Photo F" 719, the appearance state of "Photo F" 719 changes similar to how the appearance state of the previous photos changes in response to being selected.

[0314] Fig.7O An input 720 directed to the shift function 710 and an input 721 directed to the left-click function 705 are shown (and each of these inputs results in a persistent selection state disabling the shift and left-click functions of the control strip 197). Fig.7O Also shown is the photo in the photo application user interface 189 remaining selected.

[0315] Figure 7P Stylus 484 is shown interacting with scroll bar 722 to scroll downward through photo application window 189. A tap of stylus 484 causes pointer 190 to perform a left-click function, and while the stylus remains in contact with scroll bar 722 (e.g., not lifted from scroll bar 722), scroll bar 722 moves in a downward direction in accordance with the movement of stylus 484.

[0316] Figure 7Q The response to the scrolling of the photo application performed by stylus 484 is shown, and the photo application window 189 now displays other photos that were not previously displayed (e.g., "Photo G", "Photo H", and "3D Photo" 724). In addition, Figure 7QStylus 484 is shown making a left click selection of "3D Photo" 724 using cursor 190. When no modifier key (e.g., shift, control, command, option) is selected, a left click will open the selected photo to fill a larger portion of the photo application window 189.

[0317] Figure 7R "3D Photo" 724 is shown as having been selected and now filling a larger portion of the photos application window 189. When in this expanded view, the previously displayed photos (e.g., "Photo D" 717, "Photo E" 718, and "Photo F" 719, "Photo G," and "Photo H") are no longer displayed. In this example, "3D Photo" 724 is a three-dimensional photo that can be manipulated in three-dimensional space. In some embodiments, an axis 725 may be provided in the lower left portion of the photos application window 189 to help the user interact with the three-dimensional photo. When interacting with the three-dimensional photo, it may be difficult to orient the photo to a desired orientation because the three-dimensional object can rotate about multiple axes.

[0318] Within the "3D Photo" 724, there is a three-dimensional object 726, which in this example is a cylinder. The cylinder is currently viewed from the top, which shows a circular cross-section, but can be rotated to a different view as desired by the user.

[0319] Figure 7S Input 727 is shown from the yaw-pitch-roll user function 707 within the control strip 197. Selecting the yaw-pitch-roll user function 707 allows the user to lock in which axis the three-dimensional object 726 can be manipulated. For example, if the user selects to only allow manipulation of the three-dimensional object 726 in both the yaw axis and the pitch axis, the three-dimensional photo will not be manipulable along the roll axis.

[0320] Figure 7T An example is shown where the user has first selected the yaw-pitch-roll user function 707 to bring up additional controls yaw 707-1, pitch 707-2, and roll 707-3. These controls, when selected, allow the user to move the three-dimensional object around a selected axis (e.g., the yaw axis) based on input on the touch-sensitive surface, such as a drag input that rotates the object around the selected axis based on the magnitude and / or direction of the drag input. A selection 727 is made at the additional control for yaw 707-1 to lock the rotation to the yaw axis.

[0321] Figure 7UA click-and-drag movement by stylus 484 is shown for moving three-dimensional object 726 of "3D Photo" 724. Since the degree of freedom is locked to the yaw axis in the example shown, three-dimensional object 726 rotates only about the yaw axis. Although input 728 has a directional component that could cause three-dimensional object 726 to rotate along a different axis besides the yaw axis, three-dimensional object 726 will only rotate along the yaw axis. Other components of the input that do not involve yaw rotation will not be used to rotate the 3D Photo.

[0322] Figure 7V The input 728 is shown continuing to the second position 729 and thus the "3D photo" 724 is rotated along the yaw axis. Thus, the three-dimensional object 726 is rotated along the yaw axis and the top of the cylinder and the sides of the cylinder are displayed due to the rotation about the yaw axis.

[0323] Figure 7W It shows that Figure 7V The input 728 is shown being completed, and the resulting new orientation of the three-dimensional object 726 within the "3D photo" 724 is also completed. Figure 7W The laptop computer 300 is shown with a companion display mode settings menu 750 displayed on the laptop computer display. The companion display mode settings menu 750 is displayed in response to the pointer 190 inputting at the settings menu icon 751. The settings menu 750 includes a number of modifications for adjusting the companion display mode.

[0324] Specifically, the companion display mode settings menu 750 includes a check box 731 for enabling or disabling the dynamic function row 198 on the tablet device. In some embodiments, when the dynamic function row on the tablet device is disabled, the resolution of the companion display mode user interface 419 is adjusted to expand and fill the space previously filled by the dynamic function row 198. In some embodiments, when the dynamic function row on the tablet device is enabled, two additional check boxes are displayed within the settings menu. The two additional check boxes are associated with displaying the dynamic function row at the top of the display 732 or at the bottom of the display 733 on the tablet device (in some embodiments, options for displaying the dynamic function row on the left or right side of the tablet display may also be presented, but in this illustrated example, options are selected from the left or right side of the tablet display). Figure 7W omitted).

[0325] exist Figure 7WIn the example shown in FIG. 7 , the companion display mode settings menu 750 includes a check box 734 for enabling or disabling the control strip 197 on the tablet device. In some embodiments, when the control strip 197 of the tablet device is disabled, the resolution of the companion display mode user interface 419 is adjusted to expand and fill the space previously filled by the control strip. When the control strip on the tablet device is enabled, some of the settings that can be configured are: "Show on right" 735, "Show on left" 736, "Auto-hide" 737, "Show modifier keys" 738, "Show persistent yaw, pitch, and roll controls" 739, "Show keyboard controls 740", "Show layout controls" 741, "Show taskbar controls" 742, "Show menu bar" 743, "Show rotation controls" 744, "Show toggle keys" 745. In addition, the companion display mode settings menu 750 includes a status indicator 746 (e.g., "iPad currently connected to Ari"), a disconnect user interface element 747 for ending the companion display mode, an add more displays 748 for adding additional displays for use with the companion display mode, and an audio source user interface element 749 for bringing up a submenu for controlling where audio is played when operating in the companion display mode. In some embodiments, the "Show on right" 735 check box is used to select the location on the tablet device where the control strip 197 is displayed, and the "Show on left" 736 check box is also used to select the location on the tablet device where the control strip 197 is displayed. With respect to "Show on right" 735 and "Show on left" 736, only one check box can be selected at a time because the control strip can only be displayed in one location at a time (in some embodiments, the menu 750 also includes a check box for selecting whether to display the control strip 197 on the top portion or the bottom portion of the tablet display).

[0326] In some embodiments, an "auto hide" 737 checkbox enables the control strip to appear only when the user interacts with the side of the display where the control strip is located. In one example, when the control strip is hidden, the companion display mode user interface 419 is resized to fill the area previously occupied by the now hidden control strip 197. In some embodiments, a "show modifier keys" 738 checkbox allows the user to decide whether they want modifier keys to be displayed within the control strip 197 (e.g., the shift key, option key, command key, and control key). In some embodiments, the settings menu may include a submenu that allows selection of which specific modifier keys to display.

[0327] In some embodiments, the "Show persistent yaw, pitch, and roll" 739 check box allows the user to decide whether the additional controls yaw 707-1, pitch 707-2, and roll 707-3 should always be displayed, or whether they should be displayed only when the yaw-pitch-roll user function 707 is selected. In some embodiments, the "Show keyboard" 740 check box is used to display a function for displaying a virtual keyboard on the tablet device 100 within the control strip 197. In some embodiments, the "Show placement controls" 741 check box is used to display a function for manually placing the tablet to the left or right of the laptop on the control strip 197; the "Show taskbar controls" 742 check box is used to display a toggle key for displaying the taskbar 412 at the tablet device 100 on the control strip 197; and the "Show menu bar" 743 check box is used to display a toggle key for displaying the menu 730 in the companion display mode on the control strip 197.

[0328] In some embodiments, a "Show rotation controls" 744 checkbox is used to display controls for rotating the display, as well as controls for rotating content within the companion display mode user interface 419. Finally, in some embodiments, a "Show toggle keys" checkbox 745 is used instead of having two separate control strips (e.g., control strip 197 and dynamic function row 198), and allowing the user to switch between control strip 197 and dynamic function row 198 at the same location.

[0329] The following reference methods 800 to 1100 provide Figures 4A to 7W Additional description of .

[0330] FIG. 8A to FIG. 8G 800 is a flowchart of a method 800 for entering a companion display mode on a first device (e.g., a tablet device running a mobile operating system) and then simultaneously displaying a user interface generated by a second device (e.g., a laptop device running a desktop operating system) and a control strip including companion display mode specific functions. The method 800 is performed at a first electronic device including a touch-sensitive display (801). Some operations in the method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0331] In method 800 described below, an exemplary tablet device (which runs a mobile operating system) can operate as an extended display of another device (which runs a desktop operating system), and the desktop operating system can generate a user interface that is then presented on the exemplary tablet device. For the purpose of describing method 800 below, device 100 can be interchangeably referred to as a first electronic device and a tablet device (or simply a tablet computer), and device 300 can be interchangeably referred to as a second electronic device and a laptop electronic device (or simply a laptop computer). In other specific implementations, these devices can switch positions and perform operations attributed to another device in the examples below.

[0332] As described below, method 800 (and associated interface) enables quick access to companion display mode functionality. Fig. 8A As shown, method 800 initially includes receiving (802) an instruction (e.g., a communication with instructions from a laptop computer) to operate a first electronic device in a companion display mode in which a user interface generated by a second electronic device (e.g., a laptop computer) is displayed at the first electronic device. As shown in FIG. 5A , the second electronic device 300 is separated from the first electronic device 100.

[0333] Likewise Fig. 8A As shown, the first electronic device may be a tablet device with a touch-sensitive display, and the second electronic device may be a personal computer (804). In some embodiments, the tablet device may be a computer that executes a mobile operating system (such as 4A to 4H and Figure 4GG to Figure 4II In response to receiving (806) an instruction to operate in a companion display mode, the method includes simultaneously displaying (808) on the touch-sensitive display of the first electronic device: a user interface generated by the second electronic device (e.g., Figure 4A 406 shown on the device 300 in FIG); and a plurality of user interface objects (eg, reference Figures 4A to 4QQ and 7A to 7W The plurality of UI objects 427-437 described above, which are shown as a narrow rectangular strip (referred to as control strip 197) of the display of device 100, include a first user interface object and a second user interface object, the first user interface object being associated with a first function among a plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode, and the second user interface object being associated with a different second function among the plurality of functions. As an example, the first user interface object and the second user interface object may be any one of UI objects 427-437.

[0334] In some embodiments, the companion display mode initially uses the first electronic device (e.g., an electronic tablet device) as an extended monitor for the second electronic device (e.g., a laptop or desktop computer). In this example, the user interface associated with the second electronic device is the background image of the desktop view of the second electronic device (e.g., the user interface of the device 100 shown in Figure 407 can be the background image of the second electronic device 300). In some embodiments, the companion display mode initially uses the first electronic device as a mirrored monitor for the second electronic device. In this example, the user interface associated with the second electronic device is the user interface of any application currently open on the second electronic device. In some embodiments, when the user interface generated by the second electronic device is displayed at the first device, the user interface is not displayed within the application installed on the first device; instead, the companion display mode module 180 of the mobile operating system of the first device (e.g., a module that is not downloadable from an application store) provides the user interface based on data received from the second device.

[0335] In some embodiments, the functions for the companion display mode are specific to when the first electronic device is operating in the companion display mode, and therefore, these functions are not available at the first electronic device when the first electronic device is operating in other display modes. In some embodiments, the multiple user interface objects are displayed in a control strip area (e.g., the control strip 197 described above), which is below the user interface associated with the second electronic device. In other embodiments, the control strip area covers the user interface associated with the second electronic device. The control strip can be a narrow rectangular section of the touch-sensitive display that spans from one edge of the touch-sensitive display to the other edge, for example, Fig. 4I As shown, the control strip area 197 spans along the entire bottom edge of the touch-sensitive display from the left edge to the right edge.

[0336] Additionally, when users interact with the auxiliary display, they often have to navigate through complex menu sequences to adjust the display according to their needs at different points in time. Figure 4I to Figure 4PPP and 7A to 7WThe plurality of user interface objects shown (e.g., objects 428-437, 702-708 displayed within a narrow rectangular strip of the display of device 100) are displayed immediately (and without any other human intervention), allowing the user to avoid additional inputs to access a menu for controlling the touch-sensitive display of the first electronic device. Reducing the number of inputs required to control the touch-sensitive display enhances the operability of the device and makes the human-machine interface more efficient (e.g., by helping the user reduce the number of inputs the user needs to make), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0337] In some embodiments, in response to receiving an instruction to operate in a companion display mode (806), the method includes (e.g. Figure 8B The determining operation (814) further includes arranging a touch-sensitive display of the first electronic device relative to the second electronic device in response to determining the position of the first electronic device relative to the second electronic device.

[0338] In some embodiments, the rightmost edge of the second electronic device is adjacent to the leftmost edge of the first electronic device, which allows the user to move the cursor between the two displays in a continuous and predictable manner. When the devices are in this exemplary arrangement, a cursor that leaves the display at the leftmost edge of the second electronic device will reappear on the rightmost edge of the display of the first electronic device, such as Figure 4U-1 and Figure 4U-2 shown.

[0339] In some embodiments, one or more data points are used to determine the position of the first device. Exemplary data points include data provided via sensors located at one or both devices (e.g., Bluetooth, Wi-Fi, and near field communication (NFC)), and data about which side the first electronic device is physically connected to the second electronic device (e.g., plugged into a Universal Serial Bus (USB) port on the right or left side of the display (in other words, a laptop detects which port a tablet device is connected to and determines which side the port is on)). Arranging the displays of a primary monitor and a secondary monitor can often force a user to waste time trying to locate menus and then physically rearrange the monitors until the desired arrangement is achieved (often the user must reposition the devices to find the correct placement for the desired arrangement). Allowing devices to communicate with each other to determine their orientation relative to each other and to arrange user interfaces on both devices without the user having to interact with menus allows the user to quickly interact with the first electronic device in companion display mode. Automatically determining the orientation of the display (without any other human intervention) enhances the operability of the device and makes the human-machine interface more efficient (e.g., by allowing the user to not have to go to a menu to set the arrangement of the display), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0340] In some embodiments, in response to receiving an instruction to operate in a companion display mode (806), the method includes (e.g. Figure 8B ): Displaying (816) on the touch-sensitive display of the first electronic device a toggle key (e.g., Fig. 4I 427 in the toggle key); and in response to detecting an input at the toggle key: stopping displaying the plurality of user interface objects; and displaying the different user interface object at the touch-sensitive display of the first electronic device (e.g., Figure 4BB After the input at toggle key 427 in the control strip is replaced by a Figure 4CC In some embodiments, operation (816) also includes, in response to input at a different user interface object (e.g., a photo user interface object displayed within the dynamic function row 198), method 800 includes performing an operation at the second electronic device (e.g., causing a photo associated with the selected photo user interface object to be displayed on the display of the laptop computer).

[0341] In some embodiments, the toggle key may be used to switch between more than two modes, and those additional modes may include user-predefined modes, or dynamic modes that automatically adjust their displayed functionality based on open applications and user requests, such as Figure 4BB to Figure 4CC As shown. Changing the functionality of a predefined user interface typically involves entering a menu and changing what is displayed. Allowing the user to switch between multiple different menus that address the various needs raised by the user reduces the complexity of adjusting the control strip within the menu settings for each different purpose. Providing this interaction at the auxiliary device enhances the operability of the device and makes the human-computer interface more efficient (for example, by allowing the user to quickly and seamlessly switch between multiple control strip functions), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the tablet display may display the control strip 197 and the dynamic function row 198 ( Fig. 7A ) both, and in such embodiments, the toggle key may be omitted from the display within the control strip 197.

[0342] Referring to operation (816), method 800 further includes (at Figure 8B ): The operation for performing the operation at the second electronic device (818) includes performing the operation within a different user interface displayed on the second electronic device.

[0343] Additionally, in some embodiments, the plurality of functions (corresponding to user interface objects displayed within control strip 197) do not include a volume function or a brightness function, which can help avoid user confusion because such functions may control features of a laptop computer rather than features of a tablet computer.

[0344] See now Figure 8C , the method operation (808) may include: the multiple functions for controlling the touch-sensitive display of the first electronic device when the first electronic device operates in the companion display mode are a predefined set of functions, including one or more functions associated with the functions of the second device (820).

[0345] In some embodiments, there may be multiple functions for controlling the touch-sensitive display (e.g., each function may be accessed by providing input at control strip 197), and the corresponding multiple functions may be a predefined set of functions associated with the touch-sensitive display. In some embodiments, there may be some functions for controlling the second electronic device, which are displayed as selectable options at control strip 197. In addition, the use of a predefined set of functions helps to further enhance the operability of these devices and make the human-computer interface on these devices more efficient (e.g., by allowing users to quickly and seamlessly access those functions that they find most useful and that they will have to look for more frequently than other functions). In some embodiments, the predefined set of functions is selected by the user when setting up a user interface (such as Figure 7W The settings are defined in the setup user interface shown in the figure.

[0346] When users interact with an auxiliary display, they typically must navigate through a complex menu sequence to adjust the display according to their needs at different points in time. To overcome this problem, method 800 (eg, Figure 8C In response to detecting the request, method 800 includes displaying a selectable option for stopping displaying one of the plurality of user interface objects in a configuration options menu for the companion display mode (e.g., the configuration options menu is displayed in the first electronic device or the second electronic device). Figure 7W ). The method 800 may also include: detecting an input pointing to a selectable option; and in response to detecting the input, stopping displaying the one user interface object of the plurality of user interface objects on the touch-sensitive display of the first electronic device.

[0347] As an example, Figure 7W As shown, the menu may allow the user to: enable or disable display of the control strip (e.g., using check box 734, Figure 7W ); adjust the orientation of the control strip (e.g., using one of the check boxes 735-736, Figure 7W ); add or remove keyboard modifier keys on the control strip (e.g., using check box 738, Figure 7W ), add or remove the rotation feature on the control strip (for example, using check box 744, Figure 7W ); enable or disable the dynamic feature row (e.g., using checkbox 731, Figure 7W ); and adjusting the orientation of the dynamic function row (e.g., using check boxes 732-733, Figure 7W ).

[0348] Allowing the user to adjust the predefined set of functions allows the user to adjust these functions to better suit their needs. When the user has predefined a customized set of functions, they will not need to waste time searching additional menus, which enhances the operability of the device and makes the human-machine interface more efficient (e.g., by allowing the user to quickly and seamlessly set up an auxiliary display specific to their needs without having to navigate multiple display menus to set up the display), which additionally reduces power usage and extends the battery life of the device by enabling the user to more quickly and efficiently find the functions they need on the tablet device when the tablet device is operating in the companion display mode.

[0349] Return to Figure 8C In method 800, the first user interface object associated with the first function may be a function for changing the arrangement of displays relative to each other (824). In one example, the user interface object may allow different arrangement options for placing the display of the first electronic device relative to the display of the second electronic device, and an example of changing the arrangement options is shown in FIG. Figure 4T to Figure 4U . In some cases, when a user wishes to change the arrangement of displays relative to each other, the user must navigate a complex, confusing, and difficult to remember menu sequence to achieve this. Allowing the user to select a display arrangement using a single input at multiple user interface objects (displayed at the control strip 197 on the tablet computer display) allows the user to quickly change the display arrangement without having to perform the laborious task of navigating through a menu sequence. Thus, this enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to avoid menus and quickly override the display arrangement on the control strip), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0350] Next go to Fig.8D , method 800 may include one or more of operations 826, 828, and 830. Referring to operation 826, method 800 may include: when operating in the companion display mode: receiving a request to enter a split screen view on the first electronic device (e.g., a gesture 448 for dragging an application icon out of the taskbar 411 to trigger activation of the split screen view, Figure 4KK-2 Operation 826 also includes: in response to receiving a request to enter the split screen view on the first electronic device, simultaneously displaying the user interface generated by the second electronic device and the user interface for the application installed on the first electronic device side by side on substantially the entire touch-sensitive display (for example, displaying the mail application and the user interface for the companion display mode side by side, such as Figure 4LL shown).

[0351] It is often difficult for a user to interact with two separate devices at the same time. Responding to a request to enter a split-screen view by simultaneously displaying (on substantially the entire display of the first device) a user interface generated by the second device and a user interface for an application installed on the first device side by side allows the user to easily view and subsequently interact with content associated with two different devices (and two different operating systems). In this way, the operability of the device is enhanced (e.g., the user is able to interact with two devices at once, rather than switching back and forth between the two devices) and allows for continuous interaction with the two devices.

[0352] exist Fig.8D In the method, the method optionally further includes: while simultaneously displaying (828) a user interface generated by the second electronic device and a user interface for an application being executed on the first electronic device, detecting a gesture that includes dragging content from within the user interface generated by the second electronic device to the user interface of the application being executed on the first electronic device (e.g., Figure 4MM In the example, a gesture of a contact of dragging a photo H object) and in response to dragging the content to a user interface of an application being executed on the first electronic device (eg, Figure 4NN to Figure 4PP The method also includes performing an operation corresponding to content within the user interface of the application on the first electronic device (e.g., content or a representation of the content displayed in the user interface of the application on the first electronic device, such as Figure 4PP for “Photo H”).

[0353] When two user interfaces are displayed on one device, each of which is driven by an independent operating system on two different devices, it may be inconvenient to quickly transfer files between the two devices. Allowing the user to drag a file from one user interface generated by one operating system to another user interface generated by a second operating system greatly speeds up the process of transferring files. Allowing files to be transferred between two active operating systems enhances the operability of the device and makes the human-computer interface more efficient (e.g., allowing the user to not waste time looking for (or calling) a complex way to send files between two operating systems (e.g., devices)), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0354] exist Fig.8DIn the method, the method optionally includes: while simultaneously displaying (830) a user interface generated by a second electronic device and a user interface for an application being executed on a first electronic device, detecting a gesture, the gesture comprising a contact that drags content from within the user interface of an application being executed on the first electronic device to the user interface generated by the second electronic device; and in response to the contact being lifted after the contact drags the content to the user interface generated by the second electronic device, performing an operation corresponding to the content within the user interface generated by the second electronic device. The above example is in Figure 4SS to Figure 4XX , where a receipt object 463 is dragged from an email application on a tablet device to a photo application window 189 of the companion display mode.

[0355] As described above, allowing content to be transferred between two active operating systems enhances the operability of the device and makes the human-computer interface more efficient (e.g., allowing the user to not have to find (or call) a way to send files between two operating systems (e.g., devices)), which further reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0356] Go to Fig. 8E The method 800 further includes: based on determining that the user interface generated by the second electronic device includes content associated with the audio, playing (831) the audio (e.g., Figure 4OOO In some embodiments, the audio playback using the speakers of the first device may alternatively or additionally use other speakers associated with the second electronic device, such as integrated speakers for the second electronic device or external speakers for the second electronic device. In some embodiments, the user can specify which device they want the audio to play on, or can choose to use a speaker external to the first and second devices (e.g., selected from the companion display mode settings menu 750, such as Figure 7W ) speaker shown in .

[0357] like Fig. 8E As shown, method 800 includes: detecting (832) a contact and movement of the contact in an upward direction away from an edge of the touch-sensitive display; (e.g., the upward direction of movement away from the edge is substantially perpendicular to the upward direction of the edge (e.g., the contact moves along a path within a range of + / - 5% away from a straight line extending at a 90% angle away from the edge, such as Figure 4FF In response to detecting the contact and the movement of the contact, displaying: a home screen of the first electronic device (eg, Figure 4GG), an icon representing a user interface generated by the second electronic device, and other icons representing applications installed on the first electronic device (e.g., Figure 4GG and Figure 4HH Icon 443 in both).

[0358] When interacting with a multifunction device, there may be times when a user wants to interact with the home screen user interface (or other native user interface) of the first electronic device rather than the user interface generated by the second electronic device. However, switching back and forth between the two user interfaces may be inconvenient or require searching for the user interface behind a series of other user interfaces associated with other applications. Allowing a representation of the companion display mode to appear on the home screen of the device when the device is operating in the companion display mode can greatly facilitate conveniently returning from the home screen to the user interface generated by the second electronic device.

[0359] In method 800, if Fig. 8E As shown, the first user interface object associated with the first function of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode may be a function for rotating the display (for example, the function may be activated by Figure 4L to Figure 4P The method further includes: detecting (834) a selection of the first user interface object; and in response to detecting the selection, rotating a user interface generated by the second electronic device within the touch-sensitive display (e.g., an exemplary rotation in Figure 4L to Figure 4P ).

[0360] Allowing the user to adjust the rotation of the screen from the control strip allows the user to quickly adjust the secondary display to their needs. Providing this interaction at the tablet device enhances the operability of these devices and makes the human-machine interface on these devices more efficient (e.g., by allowing the user to quickly and seamlessly rotate their secondary display without having to navigate multiple display menus to rotate the display), which additionally reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0361] In method 800, if Fig. 8E As shown, the user interface object associated with the first function of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode is a function for stopping operation in the companion display mode (e.g., Fig. 4I 437 in the user interface object), and the method includes: detecting (836) a selection of the first user interface object; and in response to the selection, stopping displaying the user interface generated by the second electronic device.

[0362] In some embodiments, when exiting the companion display mode, the first electronic device may be restored to a previous state that the first device was in use before invoking the companion display mode. Allowing the user to easily turn off the companion display mode with a single input reduces the complexity of exiting the companion display mode. Providing this interaction at the auxiliary device enhances the operability of these devices and makes the human-machine interface on these devices more efficient (e.g., by allowing the user to quickly and seamlessly exit the companion display mode without accessing any menus), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0363] refer to Figure 8F In method 800, the first user interface object associated with the first function of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode is a function for bringing up a task bar associated with the second electronic device (e.g., Fig. 4I and Figure 7B ), and the method includes: detecting (838) a selection of the first user interface object; and in response to the selection, displaying a task bar (e.g., task bar 439 in 4X, 4Y-1, and 4Y-2, which can float into view when selected) that covers a portion of the user interface generated by the second electronic device.

[0364] When operating two separate devices, users may have to switch between devices to open their desired application, which wastes time. Allowing the user to bring up the taskbar from two separate devices on a single device eliminates the need to switch between devices to open the desired application. Providing this interaction at a tablet electronic device enhances the operability of these devices and makes the human-machine interface on these devices more efficient (e.g., by allowing the user to quickly and seamlessly open applications from two devices on a single device), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0365] like Figure 8F As shown, the method 800 may further include: when the first electronic device is operated in the companion display mode (840): receiving a request to display an application switching user interface at the first electronic device (eg, Figure 4DD ) and, in response to the request, simultaneously displaying: representations of recently used applications (e.g., Figure 4EE and representations of various user interfaces and / or desktops generated by the second electronic device (e.g., representations of “Desktop 1” 445 and “Desktop 2” 444, Figure 4EE ).

[0366] In some embodiments, the first electronic device may display multiple representations of different user interfaces from the second electronic device, including different desktop views established at the second electronic device, and displaying recently used applications from the first electronic device. Providing an application switching user interface that includes representations of such recently used applications and representations of user interfaces generated by the second electronic device creates a simple single application switching user interface. The single application switching user interface provides users with easy access to all user interfaces available on their devices (including user interfaces for both installed applications and user interfaces associated with companion display modes), which enhances the operability of the device (e.g., by displaying multiple accessible user interfaces in a single location between the two devices), which additionally reduces power usage and extends the battery life of the device by enabling the user to locate the desired user interface without having to waste time searching for the user interface.

[0367] exist Figure 8F In the method, the method further includes: simultaneously displaying (842) a desktop (eg, Figure 4EE In some embodiments, the desktop established at the second electronic device can be a desktop that does not include a user interface generated by the second electronic device; in other words, the user interface generated at the second device is associated with some other desktops instead. In some embodiments, the application switching user interface includes representations of all available desktops established at the second device to allow the user to easily and seamlessly start using one of the other desktops at the first device. Making representations of desktops established at the second device available at the application switching user interface of the first device enhances the operability of these devices and makes the human-computer interface on these devices more efficient (for example, by allowing the user to quickly select a desktop representation).

[0368] like Figure 8F As shown, in method 800, the request includes (844): four or more contacts on the touch-sensitive display, and movement of four or more contacts on the touch-sensitive display toward each other. In one example, the contact is formed by four or five simultaneously detected contacts together to perform the following functions: displaying an application switching user interface, navigating to a home screen, and / or switching between different applications or different groups of applications. It is very convenient to allow a user to enter an application switching view using an intuitive gesture from one hand. Providing this interaction at an auxiliary device enhances the operability of these devices and makes the human-computer interface on these devices more efficient (for example, by allowing a user to quickly enter a multitasking view using one hand).

[0369] In method 800, if Figure 8F As shown, the request includes a contact and movement of the contact in an upward direction starting from an edge of the touch-sensitive display and moving away from the edge of the touch-sensitive display (846). In some embodiments, movement of the contact in an upward direction perpendicular to the edge of the touch screen display may need to exceed a first distance threshold (e.g., the distance is equal to or greater than the first threshold) in order to bring up the multitasking user interface, and an example of the above is Figure 4EE In some embodiments, there may be a second distance threshold that is longer than the first distance threshold. When the edge gesture exceeds the second distance threshold, the application switching view is not displayed and the second function is executed. The second function may be to display the home screen of the first electronic device, such as Figures 4FF to 4GG As shown. The threshold that is exceeded is determined by the distance from the initial contact to the location where the lift-off from the display occurs. In some embodiments, the upward direction of movement away from the edge is an upward direction substantially perpendicular to the edge (e.g., the contact moves along a path within a range of + / - 5% away from a straight line extending at a 90% angle away from the edge). It is very convenient to allow a user to enter a multitasking view with a single contact gesture. Providing this interaction at an auxiliary device enhances the operability of these devices and makes the human-computer interface on these devices more efficient (e.g., by allowing a user to quickly enter a multitasking view).

[0370] Now go to Figure 8G , the plurality of user interface objects may be displayed within a first portion of the touch-sensitive display, and the method 800 includes: in response to receiving (848) an instruction to operate in a companion display mode, displaying in a second portion of the touch-sensitive display different from the first portion: a plurality of application-specific user interface objects, each respective application-specific user interface object controlling a function available within an application centralized on the second electronic device. As an example, the plurality of application-specific user interface objects may be displayed within the dynamic function row 198, such as Fig. 7A shown.

[0371] In some embodiments, the first portion is a narrow rectangular strip of the display in which the UI object is displayed (e.g., the first area or portion of the tablet display used to display the control strip 197, 7A to 7W ), and the second portion is another non-overlapping narrow rectangular strip of the display in which the application-specific UI objects are displayed (e.g., a second area or portion of a tablet computer display for displaying dynamic function row 198, 7A to 7W ). In some embodiments, the first portion is referred to as a sidebar or a control strip, and the second portion is referred to as a touch bar or a dynamic function row. Figure 7WThe companion display mode settings configuration menu discussed may be used to allow reconfiguration of the position of the first portion and the second portion on the touch-sensitive display.

[0372] like Figure 8G As shown, in method 800, the user interface generated by the second electronic device is displayed on the touch-sensitive display in a third portion of the touch-sensitive display, and the third portion includes corners adjacent to the first portion and the second portion, each of the corners having a visual appearance that mimics a physical corner of the first electronic device. Fig. 7A , which shows that the companion display mode user interface 419 has rounded corners that simulate physical corners that are also rounded.

[0373] Figure 8G Also shown is an operation 852, which includes: detecting (852) a press and hold input directed to a first user interface object displayed on the touch-sensitive display of the first electronic device; after detecting the press and hold input: detecting a first input directed to a user interface generated by a second application; in response to detecting the first input, performing an operation within the user interface generated by the second application; detecting a double-click input directed to the first user interface object displayed on the touch-sensitive display of the first electronic device; (e.g., Figure 7J In response to detecting the double-click input, changing the visual appearance of the first user interface object to indicate that it is selected (e.g., Figure 7K ); while the first user interface object remains selected: detecting a second input and a third input within the user interface generated by the second electronic device; in response to detecting the second input and the third input, performing an operation within the user interface generated by the second electronic device. In this way, the user can use a press and hold input to perform an operation associated with the modifier key on a single-use basis, and can also use a double-click input to persistently activate the operation associated with the modifier key.

[0374] In some embodiments, when a user interface generated by a laptop device is displayed on a tablet device, a notification user interface can be pulled on top of the user interface. For example, detecting contact and movement of the contact in a downward direction perpendicular to the edge of the touch-sensitive display, and in response to detecting the contact and movement of the contact, overlaying on top of the user interface generated by the second electronic device gesture (e.g., swiping down) to cover the user interface including the notification user interface element (e.g., as shown in FIG. 1 ). Figure 5A-2 to Figure 5B-2 In some embodiments, selection of the notification user interface element results in launching an application associated with the notification on the first electronic device, and launching the application may replace the displayed user interface generated by the second electronic device.

[0375] It should be understood that the specific order of operations described in method 800 is only an example and is not intended to indicate that the order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of many ways to reorder the operations described herein.

[0376] The following describes a method 900 for performing different operations (e.g., mobile or desktop operating system operations) depending on which type of input object is used at a first electronic device operating in a companion display mode. The method 900 is performed at a first electronic device including a touch-sensitive display. Some operations in the method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0377] The touch-sensitive display may be included in a first electronic device (e.g., such as Figure 1A Method 900 is performed at a tablet electronic device (shown in FIG. 1 ). In some embodiments of method 900, the method includes operating (902) the first electronic device in a companion display mode in which a user interface generated by a second electronic device is displayed at the first electronic device. The second electronic device is separated from the first electronic device. The method also includes: when operating in the companion display mode (904): displaying (906) a user interface generated by the second electronic device on a touch-sensitive display of the first electronic device (for example, in various examples, the user interface can operate in full screen mode, minimized mode, or partially extended mode); detecting (908) a gesture using an input object at the first electronic device; method 900 also includes, in response to detecting the first gesture (910): performing a first operation on the touch-sensitive display based on determining that the input object is one or more user fingers. Moreover, similarly in response to detecting the first gesture, performing a second operation different from the first operation on the touch-sensitive display based on determining that the input object is a stylus. Figures 5A-1 to 5F-2 Examples of different gestures and responses to those gestures based on the type of input object used are shown.

[0378] In some embodiments, the stylus can be a passive device detected by the capacitive touch-sensitive display of tablet computer 100 or an active device that communicates with the device, or the stylus can have both active and passive characteristics.

[0379] Allowing a single gesture to have multiple purposes depending on the input device (e.g., finger or stylus) allows the user to perform more operations than would normally be possible and enables efficient interaction with the companion display mode. Increasing the number of operations that can be performed relative to a set number of gestures can enhance the operability of the device and make the human-computer interface more efficient (e.g., by helping the user reduce the number of gestures required for the user to perform an operation in one of two different operating systems), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0380] Next go to Fig. 9B , additional optional features of method 800 are shown. Fig. 9B As shown, in method 900, the first operation may be an operation performed by the first electronic device (912). In some embodiments, the first electronic device performing the first operation includes: the first device displays a notification user interface (e.g., a notification interface) in response to receiving a downward swipe gesture provided by a user's finger on the touch-sensitive display. Figure 5A-2 As shown, a gesture 501 is received from a user's finger, and in response, Figure 5B-2 The notification center user interface is then shown displayed on the tablet computer display. Other examples of the first operation are provided below. In some embodiments, the first electronic device is running a mobile operating system, such as 4A to 4H and Figure 4GG to Figure 4II shown.

[0381] Users are already accustomed to providing finger-based touch inputs at devices with touch-sensitive displays. Therefore, having these finger-based touch inputs processed by the first electronic device meets the user's expectations and thus ensures a consistent user experience when interacting with the tablet device when the tablet device is used in companion display mode, which enhances the operability of the device and makes the human-machine interface more efficient.

[0382] In method 900, if Fig. 9B As shown, the second operation is an operation performed by the second electronic device. An example of the second electronic device performing the second operation includes the second device moving a cursor in response to receiving a downward swipe gesture provided by a stylus on the touch-sensitive display (the example is in Figure 5A-1 , where a downward swipe movement of the stylus causes Figure 5B-1 Other examples of the second operation are provided below. In some embodiments, the second electronic device is running a desktop / laptop computer operating system, such as FIG. 4D to FIG. 4EIn some embodiments, the first electronic device receives input provided by the stylus and sends the input to the second electronic device for processing. In response, the second electronic device sends an updated user interface (or data for presenting an updated user interface) to be displayed on the first electronic device.

[0383] Allowing input provided by the stylus to be processed at the second electronic device helps avoid confusion with gestures provided using the user's finger (in some cases, the same gesture may be provided using either input object, but different responses will be provided depending on which input object is used). By providing different responses to different input objects, the user is able to conveniently interact with the user interface presented in conjunction with the companion display mode using their finger or stylus, thereby ensuring an improved human-computer interface for use with the companion display mode.

[0384] In some embodiments, such as Fig. 9B As shown in , a gesture includes a contact near the top edge of the touch-sensitive display and movement of the contact in a downward direction away from the top edge of the touch-sensitive display (e.g., Figure 5A-1 and Figure 5A-2 In some embodiments, the first operation includes displaying a user interface generated by the first electronic device on the touch-sensitive display (e.g., displaying a Figure 5B-2 The second operation includes displaying a status bar generated by the second electronic device on the touch-sensitive display (e.g., displaying a status bar generated by the second electronic device on the touch-sensitive display if the status bar was previously hidden). Figure 5B-1 Display a status bar at the top of the display in , or if a status bar is already displayed, move the pointer 190 instead).

[0385] In some embodiments, the gesture must travel a predefined distance from the edge of the touch-sensitive display (e.g., meet or exceed a threshold), such as Figure 5A-1 and Figure 5A-2 In some embodiments, the downward direction of movement away from the top edge is substantially perpendicular to the downward direction of the top edge (e.g., the contact moves along a path within + / - 5% away from a straight line extending at a 90% angle away from the top edge).

[0386] Requiring a user to remember and maintain multiple gestures for interacting with a touch-sensitive display can be frustrating to the user and can cause the user to forget gestures and subsequent features. Allowing a single gesture to have multiple purposes (such as displaying a notification center or displaying a menu bar) reduces the requirement for the user to learn how to use the device and to have to remember different inputs. Increasing the number of operations that can be performed according to a set number of gestures enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user remember shortcuts built into the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0387] Depending on where the gesture begins, the response and change in the tablet computer display may vary. For example, in some embodiments, when the contact near the top edge is also near the corner of the touch-sensitive display, the user interface generated by the first electronic device is to set the user interface (e.g., Figure 4PPP to Figure 4QQ ; however, when the contact near the top edge is a predefined distance from the corner of the touch-sensitive display, the user interface generated by the first electronic device is a user interface including an electronic notification (e.g., Figure 5B-2 Notification center 503)(916).

[0388] Move to Fig. 9C In some embodiments, the gesture includes contact near the right edge of the touch-sensitive display of the first electronic device and movement of the contact in a leftward direction away from the right edge of the touch-sensitive display (e.g., Figure 5C-1 506 and Figure 5C-2 507), performing the first operation includes displaying a user interface (eg, Figure 5D-2 The “racing game” 5003 in the first electronic device is overlaid on a portion (e.g., less than the entire user interface) of the user interface produced by the second electronic device, and performing a second operation includes displaying a user interface including an electronic notification (e.g., from the first electronic device) on the touch-sensitive display of the first electronic device. Figure 5D-1 The notification user interface 509 of the laptop computer 100 in FIG. 1 is overlaid on a portion (eg, less than the entire user interface) of a user interface generated by a second electronic device (922).

[0389] In some embodiments, the gesture must travel a predefined distance from the edge of the touch-sensitive display (e.g., meeting or exceeding a threshold distance of 0.5 inches). In some embodiments, the leftward direction of movement away from the edge is a leftward direction substantially perpendicular to the right edge (e.g., the contact moves along a path that is within + / - 5% of a straight line extending at a 90% angle away from the right edge).

[0390] Requiring a user to remember and maintain multiple gestures for interacting with a touch-sensitive display may be annoying and may cause the user to forget the gestures and the features they produce. Allowing a single gesture to have multiple purposes (such as overlaying an application user interface or a notification user interface on a user interface being executed on the first electronic device) reduces the requirement for the user to learn how to use the device and to have to remember different inputs. Increasing the number of operations that can be performed according to a set number of gestures enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user remember shortcuts built into the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0391] In some embodiments, and also as Fig. 9C As shown, the gesture includes a contact near the bottom edge of the touch-sensitive display and movement of the contact in an upward direction away from the bottom edge of the touch-sensitive display (e.g., Figure 5E-1 Bottom edge swipe gesture in 511 and Figure 5E-2 ), performing a first operation includes overlaying a first task bar on a touch-sensitive display of the first electronic device over a user interface generated by the second electronic device, the first task bar including user interface objects each for opening an application installed on the first electronic device; (e.g., Figure 5F-2 411 in the task bar), and performing the second operation includes moving the second task bar (eg, Figure 5F-1 The taskbar 412 in the figure is overlaid on the user interface generated by the second electronic device, and the second taskbar includes a first user interface object for displaying the user interface of the first application on the second electronic device and a second user interface object (924) for displaying the user interface of the second application on the second electronic device.

[0392] In some embodiments, the overlay of the taskbar may cause some of the content originally displayed at the location where the taskbar is displayed to be relocated to a new location. In such an embodiment, when the taskbar is no longer displayed, the taskbar content will return to its previous position. In one example, the control strip is moved when the taskbar is displayed. In some embodiments, the upward direction moving away from the bottom edge is substantially perpendicular to the upward direction of the bottom edge (e.g., the contact moves along a path within a range of + / -5% away from a straight line, which extends at a 90% angle away from the edge). Requiring users to remember and maintain multiple gestures for interacting with a touch-sensitive display may be annoying and may cause users to forget gestures and the features they produce. Allowing a single gesture to have multiple purposes (such as overlaying two separate taskbars on a user interface being executed on a first electronic device) reduces the requirement for users to learn how to use the device and have to remember different inputs. Increasing the number of operations that can be performed according to a set number of gestures enhances the operability of the device and makes the human-machine interface more efficient (e.g., by helping users remember built-in shortcuts in the device), which in addition reduces power usage and extends the battery life of the device by enabling users to use the device more quickly and efficiently.

[0393] Reference now Fig.9D In some embodiments, the gesture includes movement of the input object over a second user interface generated by the second electronic device, and performing the first operation includes scrolling content within the user interface generated by the second electronic device (e.g., by Figure 4AAA contact 471 (e.g., by a finger) in scrolling the photo application window 189), and performing a second operation includes moving a cursor within a user interface generated by the second electronic device (e.g., in response to Figure 5A-1 and Figure 5B-1 Move the cursor 190)(926) by using the stylus 484 in the image.

[0394] Requiring a user to remember and maintain multiple gestures for interacting with a touch-sensitive display may be annoying and may cause the user to forget the gestures and the features they produce. Allowing a single gesture to have multiple purposes (such as moving a cursor or scrolling content) reduces the requirement for the user to learn how to use the device and to have to remember different inputs to perform within the operating system for the first device and the second device. Increasing the number of operations that can be performed according to a set number of gestures enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user remember shortcuts built into the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0395] In some embodiments, the second gesture is performed while the stylus is hovering over the touch-sensitive display, wherein while the stylus is hovering over the touch-sensitive display, the stylus remains within a threshold distance of the touch-sensitive display but does not touch the touch-sensitive display (928).

[0396] An example of a threshold distance at which a hovering stylus may be detected is 1-2 inches from the display, or another suitable value of up to 4 inches from the display. In some embodiments, the device registers the hovering stylus only when the stylus is a predefined distance above the touch-sensitive display. In some embodiments, moving the cursor closer to the display may cause a different operation to be performed. In some embodiments, a visual cue will appear on the display (e.g., a small circle around the tip of the stylus) when the stylus is within a predefined range. In another embodiment, the size of the visual cue may decrease as the stylus begins to leave the predefined distance above the touch-sensitive display.

[0397] Allowing the stylus to control the movement of the cursor / pointer when the stylus is hovering over the display provides additional control options without covering the user interface with the stylus, which enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to see the entire display and find where to place the cursor), which improves the human-computer interface.

[0398] In some embodiments, such as Fig.9D As shown in , the gesture includes a two-finger tap from a user's fingers on a touch-sensitive display (e.g., as Figure 4 BBB In one embodiment, the first electronic device may be used to perform a right-click operation within a user interface generated by a second electronic device, and performing (930) a first operation includes displaying, on a touch-sensitive display of the first electronic device, a menu associated with a right-click operation within a user interface generated by a second electronic device. In one example, the right-click operation includes a drop-down menu having functions related to the user interface at the location where the two-finger tap gesture was detected. The cursor may be repositioned to the location where the two-finger tap gesture occurred, and in other embodiments, the cursor may remain at the same location. Allowing a user to use a gesture including a two-finger tap enables access to a right-click function that would otherwise be difficult to access and would cause frustration to the user. Therefore, the gesture enhances the operability of the device and improves the human-computer interface.

[0399] In some embodiments, the gesture includes a single click on the touch-sensitive display, and performing (932) the second operation includes performing an operation associated with the single click within a user interface generated by the second electronic device (e.g., Figure 4KKK to Figure 4NNN ). Allowing a user to use a gesture including a single stylus tap enables access to a left-click function that would otherwise be difficult to access and cause user frustration. Thus, the gesture enhances the operability of the device and improves the human-machine interface.

[0400] In some embodiments, the gesture includes two contacts over content on the touch-sensitive display, followed by rotational movement of the two contacts relative to each other (e.g., Figure 4HHH 482-1 and 482-2); performing (934) the first operation includes rotating the content in the user interface generated by the second electronic device (for example, rotating Figure 4III In one example, when the two contacts on the touch-sensitive display are lifted off, the rotated content returns to its original orientation. Allowing the user to use a gesture that includes rotating two contacts about a common location enables access to a rotate content function that would otherwise be difficult to access and cause user frustration. Thus, the gesture enhances the operability of the device and improves the human-machine interface.

[0401] In some embodiments, such as Fig.9E As shown, input provided by the stylus is processed by sending data to a second electronic device, and the method includes: detecting (936) input at the stylus corresponding to a double-click on the stylus, and in response, stopping sending data about the input provided by the stylus to the second electronic device.

[0402] In some embodiments, a double-click comprises two consecutive tap inputs on the stylus, wherein a first tap is received and then a second tap is received within a predetermined time threshold (e.g., 50ms or 60ms) after that time. In some embodiments, the stylus is used to provide input processed by a second electronic device, but the user may also be interested in enabling the stylus to work with other tablet device features (including certain drawing features). Therefore, it is desirable to allow the user to perform a double-click on the stylus to return the stylus to work with tablet device features, thereby allowing the user to easily switch back and forth between using the stylus to provide input at the second device (e.g., cursor control input) or using the stylus to perform operations processed by the first device (e.g., drawing input).

[0403] In some embodiments, performing (938) the second operation includes sending to the second electronic device stylus orientation data used by an application executed at the second electronic device so that a user interface generated by the second electronic device changes as it is displayed at the second electronic device. In some embodiments, the stylus orientation data includes pressure applied by the user on the display with the stylus, coordinates of the stylus on the display, and acceleration and velocity of the stylus strokes.

[0404] In some embodiments, the gesture is a pinch or spread gesture; and performing (940) the first operation includes performing a touch-sensitive display of the first electronic device according to the pinch or spread gesture (e.g., Figure 4EEE to Figure 4GGGThe exemplary pinch and expand gesture shown in the figure adjusts the size of the content within the user interface generated by the second electronic device. In some embodiments, when there is a lift-off of the two contacts on the touch-sensitive display, the resized content will be restored to its original size.

[0405] Allowing the user to use a gesture including pinching or spreading two fingers enables access to a resize content function that would otherwise be difficult to access and cause user frustration. Thus, the gesture enhances the operability of the device and improves the human-machine interface.

[0406] It should be understood that the specific order in which the operations described in method 900 are described is only an example and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0407] The following describes a method 1000 for providing selectable options for changing display properties of a user interface (e.g., maximizing a user interface at a display of the first device) and for sending the user interface to a second electronic device (e.g., a tablet device) at a first electronic device (e.g., a laptop device). The method 1000 is performed at a first electronic device that includes a touch-sensitive display. Some operations in the method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0408] In some embodiments, in a first electronic device (e.g., Figure 1A The method 1000 (1001) is performed at a laptop computer as shown. The method may include: displaying 1002 a user interface associated with an application on a display, the user interface being associated with controlling user interface elements (e.g., FIG. 4G to FIG. 4E 196) is displayed together with a maximize button 196 of the control user interface for changing (only) display properties of the user interface (for example, the display properties may include: the size of the user interface, the display position, etc.); detecting (1004) an input pointing to (hovering or right clicking) a control user interface element. The method may optionally include that the input above the control user interface element is performed by a right click or hover operation at the first electronic device (1006). In some embodiments, the hover operation may require that the input remain above the control user interface element for a specific period of time (for example, 0.5 seconds) without contacting the display during the period of time.

[0409] Additionally, after operation 1004, in response to detecting the input, and while continuing to display the user interface (1008), method 1000 includes simultaneously displaying on the display (1010): a first selectable option (e.g., Figure 4Gand a second selectable option (e.g., selectable option 192, displayed in ) for requesting that a user interface including content from the user interface be displayed at a second electronic device different from the first electronic device. Figure 4G ).

[0410] In some embodiments, the first electronic device is a laptop or desktop computer running a desktop / laptop operating system, and the second electronic device is an electronic tablet device operating system, such as FIG. 4A to FIG. 4G The display attributes include the size of the user interface and the position of the user interface within the display. In some embodiments, the first selectable option and the second selectable option are displayed in a drop-down menu below the control user interface element. The selectable options may be displayed in response to a hover input or a right-click input above the control user interface element, such as Figure 4F The inputs shown in the control user interface element 196 are shown above.

[0411] Changing the arrangement of a user interface running on multiple displays may sometimes involve performing repeated drag operations or using multiple keyboard commands to achieve the desired orientation. Allowing a user to select a single control user interface element that populates a list of multiple selectable options for changing display properties (e.g., maximizing a window, sending to another display, etc.) ensures that such display properties are changed with a minimum number of inputs. Reducing the number of inputs to change these display properties enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping the user reduce the number of inputs required to send the user interface to some other device), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0412] In some embodiments, method 1000 includes: receiving (1016) a selection of a first selectable option; and in response to receiving the selection of the first selectable option, changing a display attribute of a user interface on a display of the first electronic device (e.g., in response to an input ( Figure 4G ), and then the UI window is displayed at full screen size).

[0413] In some embodiments, changing (1020) the display properties of the user interface includes maximizing the user interface to fill substantially the entire display of the first electronic device. In some embodiments, the user interface is maximized to fill the entire display, and all menu bars and task bars are hidden, leaving only the user interface associated with the application. In some embodiments, maximization means that the user interface fills the entire display except for the portion where the menu bar and task bar are still displayed. Allowing the user to adjust the display properties without having to interact with a separate button enhances the operability of the device by reducing the number of inputs required to perform an operation (e.g., by making the control user interface element a multifunctional user interface element), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0414] Display properties include the size of the user interface and the location of the user interface within the display. Allowing the user to adjust display properties without having to interact with separate buttons enhances the operability of the device by providing additional control options without cluttering the user interface with additional displayed controls (e.g., by making the control user interface elements multi-function user interface elements), which additionally reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0415] In some embodiments, instead of utilizing a maximize button as a control user interface element, a minimize button may be used alternatively. In such embodiments, a first selectable option may minimize the user interface window. Moreover, in such embodiments, changing (1018) the display properties of the user interface includes minimizing the user interface of the first electronic device. In some embodiments, an animation may be associated with the minimization process. In another embodiment, when the second electronic device is running in a companion display mode, the active window may be minimized to an icon displayed on a task bar at the second electronic device. Allowing the user to adjust the display properties without having to interact with a separate button enhances the operability of the device by reducing the number of inputs required to perform an operation (e.g., by making the control user interface element a multifunctional user interface element), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0416] In some embodiments, and now with reference to Fig. 10C , method 1000 optionally includes: receiving (1022) a selection of a second selectable option; and in response to receiving the selection of the second selectable option: ceasing to display the user interface including the content from the user interface on the display at the first electronic device; and sending to the second electronic device a message displaying the content from the user interface (e.g., as received from Figure 4H to Figure 4I ) instructions as shown in the transition.

[0417] In some embodiments, the instructions for displaying content from the user interface include instructions for adjusting the size of the user interface to fit on the second electronic device, such as FIG. 4H to FIG. 4I As shown. In one example, after ceasing to display a user interface including content from a user interface, a second user interface is shown that works with the user interface that was stopped from being displayed on a first electronic device (e.g., a laptop computer). Typically, moving a user interface from one display to a second display involves dragging the user interface from one location to another, and may involve multiple drag gestures to obtain a desired orientation of the user interface. Allowing the user to make a single selection of a second selectable option allows the user to quickly send the desired user interface to a second display (e.g., a second electronic device), thereby ensuring the minimum input required to send the user interface for display at a new touch-sensitive display. Reducing the number of inputs to send a user interface to other devices enhances the operability of the device and makes the human-computer interface more efficient (e.g., by helping to reduce the number of gestures required for a user to move the interface), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0418] In some embodiments, based on determining that the second electronic device has met the secure connection criteria (e.g., Figure 4G The second selectable option (1024) is displayed based on a trusted connection 195 in the secure connection criteria. Exemplary criteria for the secure connection criteria include one or more criteria selected from the following: a criteria that is satisfied when the first device and the second device have an active communication link (wired or wireless), a criteria that is satisfied when the first electronic device and the second electronic device are registered to the same user account, and a criteria that is satisfied when the user has provided an indication that the first electronic device and the second electronic device are trusted devices. Limiting the selectable options to devices that are physically or wirelessly connected to the first electronic device ensures that devices that are unavailable but still registered to the same user account are not shown. Performing operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by not requiring the user to determine whether the device is available), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0419] In some embodiments, and also as Fig. 10CAs shown, when the first electronic device and the second electronic device have an active communication link, it is determined that the second electronic device has met the security connection criteria (1026). In some embodiments, the wireless connection is established through multiple networks (e.g., a Wi-Fi network, a Bluetooth network, or a near field communication (NFC) network). In some embodiments, the first electronic device can continuously scan for other devices on one of these networks, and when detected, the selectable options available in the user interface will be updated.

[0420] In some embodiments, when the first electronic device and the second electronic device are registered to the same user account (e.g., Figure 4A , Ari's user account is registered on both the tablet device 100 and the laptop computer 300), it is determined that the second electronic device has met the security connection criteria (1028). In some embodiments, the user account is an ICLOUD provided by APPLE INC. (Cupertino, CA). TM In such an embodiment, the user is required to log into their user account on both devices before activating the feature, such as Figure 4A In some embodiments, when the first connection is made, the electronic device displays a prompt prompting the user to log into their user account to allow the companion display mode to operate.

[0421] In some embodiments, after the user has provided an indication that the first electronic device and the second electronic device are trusted devices (e.g., FIG. 4B to FIG. 4C After the trust prompt 413 in the second electronic device, it is determined that the second electronic device has met the secure connection criteria. In some embodiments, the user may be prompted to trust the other device on the first electronic device (e.g., a laptop computer), the second electronic device (e.g., a tablet device), or both devices. In some embodiments, the confirmation of trusting the other device occurs immediately after the physical connection between the two devices is made.

[0422] It should be understood that the specific order of operations described in method 1000 is only an example and is not intended to indicate that the order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of many ways to reorder the operations described herein.

[0423] Next, a method 1100 is described of receiving a request to annotate content at a first electronic device (e.g., a laptop device), and then determining whether to display a selectable option that allows selection of which device to annotate the content or whether to send an instruction to a second device to display the content in an annotation mode (which is available and previously approved). The method 1100 is performed at a first electronic device that includes a touch-sensitive display. Some operations in the method 1100 are optionally combined, and / or the order of some operations is optionally changed.

[0424] Method 1100 (1101) may be performed at a first electronic device including a display device, and method 1100 includes: receiving (1102) a request to annotate content on the first electronic device. In some embodiments, the request to annotate content is a screenshot of the content on the first electronic device (e.g., Fig. 6A Screen shot 601) (1104) of the request.

[0425] In some embodiments, a screenshot is captured by a combination of key inputs on a keyboard on a second electronic device. In other embodiments, a screenshot of the region is captured by a user selecting the region with a cursor. When a user captures a screenshot, they typically attempt to share a certain content on their screen with other people, so it is convenient to allow a user to quickly enter an annotation mode with minimal input to annotate the screenshot. Reducing the number of inputs required to perform an operation enhances the operability of the device, and makes the user-device interface more efficient (e.g., by providing a shortcut to the annotation mode under certain conditions (such as capturing a screenshot), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0426] Continuing from operation 1102, in response to receiving (1106) the request, method 1100 includes: based on determining that a second electronic device different from the first electronic device is available for displaying content in an annotation mode, and that using the second electronic device to display content in an annotation mode has not been previously approved, displaying a selectable option via a display device that, when selected, causes the first electronic device to send an instruction to display the content in an annotation mode at the second electronic device (1108) (e.g., a prompt to select an electronic device for annotation mode, such as Fig.6D In some embodiments, annotation is a mode in which inputs are received at certain locations over content and then used to annotate the content (including drawing lines, circles, handwriting, shapes, etc.).

[0427] In addition, in response to receiving the request, method 1100 includes: based on determining that the second electronic device is available for displaying content in the annotation mode, and using the second electronic device to display content in the annotation mode has not been previously approved (for example, the second electronic device has been selected as a device for annotating content displayed on the first electronic device), sending a request to the second electronic device to automatically display the content in the annotation mode (for example, FIG. 6E to FIG. 6H Instructions (1110) are executed without further user intervention (shown in screen shot 601 on tablet device 100).

[0428] In some embodiments, if the user has indicated that the second electronic device is approved to display content in annotation mode, it will waste time and require redundant input to continuously ask the user to re-approve the second device. Therefore, by determining whether the second device is available to display content in annotation mode and whether the second device has been previously approved to respond to a request to annotate content, it is ensured that the user does not waste time providing additional input to re-approve the second electronic device. In this way, the human-computer interface is improved and continuous interaction with two different devices is possible.

[0429] In some embodiments of method 1100, the method includes: in response to receiving the request: based on determining that the second electronic device is not available for entering the annotation mode: stopping (1112) displaying the selectable options; and abandoning sending an instruction to the second electronic device to display the additional content in the annotation mode. Not showing the selectable options is convenient for the user because it indicates that the device is not available for the annotation mode. Providing this improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by not confusing the user which devices are available), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0430] Continue to refer Fig. 11B , method 1100 may also include: in response to receiving the request: based on determining that multiple electronic devices are available for displaying additional content in annotation mode and that use of the multiple electronic devices to display additional content in annotation mode has been previously approved, sending (1114) instructions to one or more of the multiple electronic devices to automatically display the additional content in annotation mode without further user intervention.

[0431] Many devices have varying degrees of usability (e.g., some devices accept input from precision input devices, while others provide larger displays for use). Therefore, it may be convenient for a user to be able to annotate across multiple devices. Performing operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by not requiring the user to exit annotation mode and make a second selection of a selectable option to use a second device), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0432] Reference now Fig. 11CIn some embodiments, displaying selectable options includes displaying a plurality of selectable options, each respective selectable option, when selected, causing (1116) the first electronic device to send an instruction to display the content in an annotation mode at a respective electronic device different from the first electronic device and the second electronic device. In some embodiments, the selectable options may display a different representation of each available device. For example, if a mobile device such as a smart phone is set for annotation mode, the selectable options may be a visual representation of the smart phone.

[0433] Selecting multiple devices to send instructions to display content in annotation mode can be inconvenient and inefficient. Having a simplified menu containing multiple selectable options representing devices is convenient because it places all options in a single location without having to navigate to a settings page. Reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly interact with devices approved for annotation mode), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0434] In some embodiments, the plurality of selectable options each correspond to a corresponding electronic device (1118) that is physically or wirelessly connected to the first electronic device. In some embodiments, the wireless connection may be made via a wired connection such as a universal serial bus (USB) or a wireless connection such as Bluetooth, Wi-Fi, and near field communication (NFC). Limiting the selectable options to devices that are physically or wirelessly connected to the first electronic device ensures that devices that are not available but are still registered to the same user account are not shown. Performing operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by not requiring the user to determine whether the device is available), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0435] In some embodiments, both the first electronic device and the second electronic device are associated with the same user account as the first electronic device (e.g., FIG. 6A to FIG. 6L The user account 193, connection 194 and trusted connection 195) are associated (1120). In some embodiments, the user account is ICLOUD provided by APPLE INC. (Cupertino, CA). TM Account. Registering a device to a user account includes logging into the user account on the electronic device.

[0436] When multiple devices are executed by a computer, for security purposes, it is important that the device connected to the first electronic device is not an unknown device. Allowing the first electronic device to enter annotation mode with a device that shares the same user account as the first electronic device helps ensure that a secure connection exists between the two devices and does not require the user to select which device they think is on the same account as them. Performing operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (for example, by determining whether the same user account is associated with two devices and not displaying devices that are not associated with the same user account), which additionally reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0437] Now turn your attention to Fig.11D , which shows that method 1100 may optionally include: before sending an instruction to the second electronic device to automatically display content in annotation mode without further user intervention, based on a determination that the second electronic device can be used to display content in annotation mode and that use of the second electronic device to display content in annotation mode has been previously approved, abandoning (1122) displaying an optional option.

[0438] In some embodiments, more than one device may be automatically placed in annotation mode without further user intervention. Optionally, when entering annotation mode, the user may choose to proceed only on a preferred device. If the user has indicated that the second electronic device is approved to display content in annotation mode, it would be redundant for the user to repeatedly approve the second electronic device. Therefore, since the device performs an operation when a set of conditions have been met without further user input, it is ensured that the user does not waste time reselecting the second electronic device by forgoing displaying an optional option based on determining that the second electronic device can be used to display content in annotation mode. By this method, the operability of the device is enhanced by reducing the number of inputs required to perform an operation (e.g., enter annotation mode).

[0439] In some embodiments, when the second electronic device receives an instruction to display content in annotation mode, the second electronic device is in a locked state (1124) (this example is in FIG. 6A to FIG. 6E, where content for use in annotation mode is sent from the laptop to the desktop device even though the tablet device is initially in a locked state). Allowing the second electronic device to remain in a locked state, but also allowing instructions to be sent to display content in annotation mode can speed up the interaction process without including security. Providing a restricted access mode enhances the operability of the device and makes the user-device interface more efficient (e.g., allowing the user to quickly interact with the device without having to exit a locked state), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0440] In some embodiments, method 1100 may include receiving (1126) data regarding an annotation to content provided at a second electronic device, and in response, updating content displayed on the first electronic device to include the annotation (e.g., FIG. 6E to FIG. 6H The annotation may be made by a stylus (e.g., FIG. 6E to FIG. 6H The stylus 484 in is provided at the second electronic device.

[0441] In some embodiments, it is known that providing annotations with a stylus may cause certain annotation settings to appear on the first electronic device. When interacting with the same content on two independent devices, it may confuse the user if the input does not cause a change at both devices. Therefore, requiring annotations from the second electronic device to be transferred to the first electronic device, and vice versa, helps prevent the user from making redundant inputs. In this way, the user will not make duplicate inputs due to the lack of visual feedback on one of the devices, which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0442] Finally, if Fig.11D As shown, in some embodiments, the content is an image including underlying content (e.g., from Figure 6B Screen shot 601 of is an exemplary image, and other images may be photographs, document images, and other similar content), and annotations made in annotation mode modify the appearance of the image without modifying the underlying content (1128).

[0443] When a user annotates an image, it is common to emphasize a portion of the image or make some other comment on the image. Allowing the user to change the appearance of an image (e.g., making a note on top of it) without including the underlying content makes it easier to create annotations. For example, if a user is using an eraser function and wants to erase only the annotation, it would be inconvenient and unintuitive if the eraser also erased the underlying image they were trying to annotate. Providing additional control options without cluttering the user interface with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by not allowing the user to edit the underlying content), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0444] It should be understood that the specific order in which the operations described in method 1100 are described is only an example and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0445] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that other technical personnel in the field can best use the present invention and the various described embodiments with various modifications suitable for the specific purpose contemplated.

Claims

1. A method for displaying a user interface, comprising: At a first electronic device including a touch-sensitive display: receiving an instruction to operate the first electronic device in a companion display mode in which a user interface generated by a second electronic device is displayed at the first electronic device, wherein the second electronic device is separate from the first electronic device; In response to receiving the instruction to operate in the companion display mode: Simultaneously displaying on the touch-sensitive display of the first electronic device: a user interface generated by the second electronic device; and a plurality of user interface objects including a first user interface object associated with a first function of a plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode and a second user interface object associated with a second, different function of the plurality of functions; and When the first electronic device is operated in the companion display mode: receiving, at the first electronic device, a request to display an application switching user interface, and In response to the request, the following is displayed: Representation of recently used applications; and A representation of the user interface generated by the second electronic device. 2 . The method of claim 1 , wherein the first electronic device is a tablet device having a touch-sensitive display and the second electronic device is a personal computer.

3. A method according to claim 2, wherein the multiple functions for controlling the touch-sensitive display of the first electronic device when the first electronic device operates in the companion display mode are a predefined set of functions, and the predefined set of functions includes one or more functions associated with the functions of the second electronic device.

4. The method according to claim 3, comprising: detecting a request to modify settings associated with the companion display mode; in response to detecting the request, displaying within a configuration options menu for the companion display mode a selectable option for ceasing to display one of the plurality of user interface objects; detecting input directed to the selectable option; as well as In response to detecting the input, ceasing to display the one of the plurality of user interface objects on the touch-sensitive display of the first electronic device.

5. The method according to any one of claims 1 to 4, comprising: In response to receiving the instruction to operate in the companion display mode: determining a position of the first electronic device relative to the second electronic device; In response to determining the position of the first electronic device relative to the second electronic device, the touch-sensitive display of the first electronic device is arranged relative to the second electronic device.

6. A method according to any one of claims 1-4, wherein the first user interface object associated with the first function is a function for changing the arrangement of the touch-sensitive display of the first electronic device and the display of the second electronic device relative to each other.

7. The method according to any one of claims 1 to 4, comprising: When operating in the companion display mode: Receiving a request to enter a split-screen view on the first electronic device; In response to receiving the request to enter the split screen view on the first electronic device, the user interface generated by the second electronic device and the user interface of the application installed on the first electronic device are simultaneously displayed side by side on the touch-sensitive display.

8. The method according to claim 7, comprising: When the user interface generated by the second electronic device and the user interface of the application being executed on the first electronic device are displayed simultaneously: detecting a gesture, the gesture comprising a contact to drag content from within the user interface generated by the second electronic device to the user interface of the application being executed on the first electronic device; as well as In response to the contact being lifted off after the contact drags the content to the user interface of the application being executed on the first electronic device, an operation corresponding to the content within the user interface of the application on the first electronic device is performed.

9. The method according to claim 8, wherein: detecting a gesture while simultaneously displaying the user interface generated by the second electronic device and the user interface of the application being executed on the first electronic device, the gesture comprising contact to drag content from within the user interface of the application being executed on the first electronic device to the user interface generated by the second electronic device; as well as In response to the contact being lifted off after the contact drags the content to the user interface generated by the second electronic device, an operation corresponding to the content within the user interface generated by the second electronic device is performed.

10. The method according to any one of claims 1 to 4, comprising: When displaying the user interface generated by the second electronic device: Based on determining that the user interface generated by the second electronic device includes content associated with audio, the audio is played using a speaker of the first electronic device.

11. The method of any one of claims 1-4, comprising, in response to receiving the instruction to operate in the companion display mode: displaying, on the touch-sensitive display of the first electronic device, a toggle key for replacing the plurality of user interface objects with different user interface objects at the touch-sensitive display of the first electronic device; as well as In response to detecting input at the toggle key: ceasing to display the plurality of user interface objects; as well as displaying the different user interface object at the touch-sensitive display of the first electronic device; In response to input at the different user interface object, an operation is performed at the second electronic device. 12 . The method of claim 11 , wherein performing the operation at the second electronic device comprises performing the operation within a different user interface displayed on the second electronic device.

13. The method according to any one of claims 1 to 4, comprising: detecting a contact and movement of the contact in an upward direction moving away from an edge of the touch-sensitive display; In response to detecting the contact and movement of the contact, displaying: a main screen of the first electronic device, an icon representing the user interface generated by the second electronic device, and Other icons representing applications installed on the first electronic device.

14. The method of any one of claims 1 to 4, wherein the first user interface object associated with the first of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode is a function for rotating the display, the method comprising: detecting a selection of the first user interface object; as well as In response to detecting the selection, rotating the user interface generated by the second electronic device within the touch-sensitive display.

15. The method according to any one of claims 1 to 4, wherein: The user interface object associated with the first function of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode is a function for ceasing operation in the companion display mode, and the method comprises: detecting a selection of the first user interface object; and In response to the selection, the user interface generated by the second electronic device stops being displayed.

16. The method according to any one of claims 1 to 4, wherein: the first user interface object associated with the first of the plurality of functions for controlling the touch-sensitive display of the first electronic device when the first electronic device is operating in the companion display mode is a function for invoking a task bar associated with the second electronic device, and The method comprises: detecting a selection of the first user interface object; and In response to the selection, the task bar is displayed to cover a portion of the user interface generated by the second electronic device.

17. The method of claim 1, wherein the simultaneous displaying comprises: An additional representation of a desktop established at the second electronic device is displayed along with the representations of recently used applications and the representation of the user interface generated by the second electronic device.

18. The method of claim 1, wherein the request comprises: four or more contacts on the touch-sensitive display, and Movement of the four or more contacts toward each other on the touch-sensitive display.

19. The method of claim 1, wherein the request comprises a contact and movement of the contact in an upward direction starting from an edge of the touch-sensitive display and moving away from the edge of the touch-sensitive display.

20. The method of any one of claims 1-4, wherein the plurality of user interface objects are displayed within a first portion of the touch-sensitive display, and the method comprises: In response to receiving the instruction to operate in the companion display mode, displaying in a second portion of the touch-sensitive display that is different from the first portion: a plurality of application-specific user interface objects, each respective application-specific user interface object controlling a function available within an application centralized on the second electronic device.

21. A method according to claim 20, wherein the user interface generated by the second electronic device is displayed on the touch-sensitive display in a third portion of the touch-sensitive display, and the third portion includes corners adjacent to the first portion and the second portion, each of the corners having a visual appearance that mimics a physical corner of the first electronic device.

22. The method according to any one of claims 1 to 4, comprising: detecting a press and hold input directed to the first user interface object displayed on the touch-sensitive display of the first electronic device; After detecting the press and hold input: detecting a first input directed to the user interface generated by the second electronic device; in response to detecting the first input, performing an operation within the user interface generated by the second electronic device; detecting a double-click input directed to the first user interface object displayed on the touch-sensitive display of the first electronic device; as well as in response to detecting the double-click input, changing the visual appearance of the first user interface object to indicate that it is selected; While the first user interface object remains selected: detecting a second input and a third input within the user interface generated by the second electronic device; and In response to detecting the second input and the third input, the operation is performed within the user interface generated by the second electronic device.

23. A non-transitory computer-readable storage medium storing executable instructions, which, when executed by one or more processors of a first electronic device having a touch-sensitive display, cause the first electronic device to perform the method according to any one of claims 1 to 22.

24. A first electronic device, comprising: one or more processors; Touch-sensitive display; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 22.

25. A computer program product comprising executable instructions which, when executed by one or more processors of a first electronic device having a touch-sensitive display, cause the first electronic device to perform the method according to any one of claims 1 to 22.

Images