System, method and user interface for interacting with multiple application windows
By using display generation components and input devices on handheld electronic devices, an intuitive interactive method of multitasking is realized, which solves the problems of small device screen size and difference in input mechanism, and improves the efficiency and user experience of multitasking.
Patent Information
- Application Number
- CN202011108072.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-24
- Filing Date
- 2020-03-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-03-30
AI Technical Summary
Existing handheld electronic devices have insufficient ability to interact between multitasking and applications, especially due to the small screen size and differences in input mechanisms, which lead to complex and unintuitive user interfaces that are difficult to implement with desktop or laptop computers. multitasking capability.
The intuitive and seamless interactive approach of multitasking is achieved by using display generation components and input devices on electronic devices, including displaying user interfaces of multiple applications, detecting inputs from touch-sensitive surfaces, and performing applications according to predefined standards Switching and window management provide an intuitive user interface to access multiple features or applications.
Improves multitasking efficiency and user satisfaction of handheld electronic devices, provides intuitive and easy-to-use systems and methods, simplifies multi-window interaction and reduces the chance of user error.
Smart Images

Figure CN112346802B_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application filed on March 30, 2020, with application number 202080001784.3 and entitled "System, Method and User Interface for Interacting with Multiple Application Windows". Technical Field
[0002] The embodiments described herein generally relate to electronic devices, and more specifically, to systems and methods for multitasking on electronic devices having display generation components and input devices (e.g., portable multifunction devices with touch-sensitive displays). Background Technology
[0003] Handheld electronic devices with touch-sensitive displays are ubiquitous. While these devices were originally designed for information consumption (e.g., web browsing) and communication (e.g., email), they are rapidly replacing desktop and laptop computers as users' primary computing devices. When using desktop or laptop computers, users are able to perform everyday multitasking by accessing and using different running applications (e.g., cutting and pasting text from a document into an email). Although the new features and applications of handheld electronic devices have grown tremendously, the ability to multitask and exchange between applications on handheld electronic devices requires input mechanisms that are completely different from those on desktop or laptop computers.
[0004] Furthermore, the need for multitasking is particularly acute due to the smaller screens of handheld electronic devices compared to traditional desktop and laptop computers. Some conventional handheld electronic devices attempt to meet this need by recreating the desktop computer interface on the handheld device. However, these attempts fail to consider the significant differences in screen size between desktop computers and handheld electronic devices, and the significant differences between keyboard and mouse interaction on desktop computers and touch and gesture input on handheld electronic devices with touch-sensitive displays. Other attempted solutions require complex input sequences and menu hierarchies, which are even less user-friendly than those offered by desktop or laptop computers. Therefore, there is a need for an intuitive and easy-to-use system and method for simultaneously accessing multiple functions or applications on a handheld electronic device. Summary of the Invention
[0005] The embodiments described herein address the need for systems, methods, and graphical user interfaces that provide intuitive and seamless interaction for multitasking on handheld electronic devices. Such methods and systems optionally complement or replace conventional touch input or gestures.
[0006] According to some implementations, a method is performed in an electronic device that includes a display generating component (e.g., a monitor, projector, head-up display, etc.) and one or more input devices including a touch-sensitive surface (e.g., a touch-sensitive surface coupled to a separate monitor, or a touch screen display that serves as both a monitor and a touch-sensitive surface). The method includes: displaying a first user interface of a first application via the display generation component; while displaying the first user interface of the first application, receiving a first input corresponding to a request to display a second application together with the first application in a corresponding simultaneous display configuration; in response to receiving the first input, displaying a second user interface of the second application and the first user interface of the first application according to the corresponding simultaneous display configuration, wherein at least a portion of the first user interface of the first application is displayed simultaneously with the second user interface of the second application; while displaying the second application and the first application according to the corresponding simultaneous display configuration, receiving a second input including detecting a first contact on a touch-sensitive surface at a position corresponding to the second application and detecting movement of the first contact on the touch-sensitive surface; in response to detecting the second input: replacing the display of the second application with the display of a third application according to a determination that the second input meets a first criterion, so as to display the third application and the first application according to the corresponding simultaneous display configuration; and maintaining the display of the first application according to a determination that the second input meets a second criterion different from the first criterion; and stopping the display of the second application and not displaying the third application.
[0007] According to some implementations, a method is performed on an electronic device that includes display generating components (e.g., a monitor, projector, head-up display, etc.) and one or more input devices (e.g., a camera, remote controller, pointing device, a touch-sensitive surface coupled to a separate monitor, or a touch screen display that serves as both a monitor and a touch-sensitive surface). The method includes: displaying a taskbar via the display generation component, the taskbar comprising a plurality of application icons overlaid on a first user interface of a first application, wherein the plurality of application icons correspond to different applications installed on the electronic device; detecting a first input while displaying the taskbar overlaid on the first user interface of the first application, including detecting a selection of a corresponding application icon in the taskbar; in response to detecting the first input and determining that the first input meets the selection criteria: determining that the corresponding application icon corresponds to the first application and that the first application is associated with a plurality of windows, displaying corresponding representations of the plurality of windows of the first application via the display generation component; maintaining the display of the first user interface of the first application based on the determination that the corresponding application icon corresponds to the first application and that the first application is currently associated with only a single window; and replacing the display of the first user interface of the first application with the display of a second user interface of the second application based on the determination that the corresponding application icon corresponds to a second application different from the first application, regardless of the number of windows associated with the second application at the time the first input was detected.
[0008] According to some embodiments, a method is performed at an electronic device including a display generating component (e.g., a monitor, projector, head-up display, etc.) and one or more input devices (e.g., a keyboard, remote controller, camera, a touch-sensitive surface coupled to a separate monitor, or a touchscreen display that serves as both a monitor and a touch-sensitive surface). The method includes displaying a first user interface via the display generating component that contains a selectable representation of first content, wherein the first content is associated with a first application; while displaying the first user interface containing the selectable representation of the first content, detecting a first input, including detecting an input corresponding to a request to move the selectable representation of the first content to a corresponding position on the monitor; in response to detecting the first input: adjusting the size of the first user interface and displaying a second user interface including the first content adjacent to the first user interface, based on determining that the corresponding position is a first position; and displaying a third user interface including the first content overlaid on the first user interface, based on determining that the corresponding position is a second position different from the first position.
[0009] According to some embodiments, a method is performed at an electronic device including a display generation component and one or more input devices. The method includes: displaying a first user interface containing selectable user interface objects via the display generation component; while displaying the first user interface containing the selectable user interface objects, detecting a first input, including detecting input corresponding to a request to move the selectable user interface objects to a corresponding position on the display; in response to detecting the first input: creating a new window for the first application based on determining that the corresponding position is in a first predefined area of the user interface and that the selectable user interface object is an application icon of a first application; creating a new window for the first application based on determining that the corresponding position is in a second predefined area of the user interface, wherein the second predefined area of the user interface is smaller than the first predefined area of the user interface and that the selectable user interface object is a representation of content associated with the first application; and performing operations corresponding to the selectable user interface object other than creating a new window for the first application based on determining that the corresponding position is in a third area of the user interface, wherein the third area of the user interface is smaller than the first predefined area of the user interface and does not overlap with the second predefined area of the user interface, and that the selectable user interface object is a representation of content associated with the first application.
[0010] According to some embodiments, a method is performed at an electronic device including a display generating component (e.g., a monitor, projector, head-up display, etc.) and one or more input devices (e.g., a camera, remote controller, pointing device, a camera, a touch-sensitive surface coupled to a separate monitor, or a touchscreen display that serves as both a monitor and a touch-sensitive surface). The method includes: simultaneously displaying a taskbar containing multiple application icons, wherein the multiple application icons correspond to different applications, via the display generating component and a first user interface of a first application; while the taskbar is displayed simultaneously with the first user interface of the first application, detecting a first input pointing to an application icon corresponding to a second application in the taskbar, the first input including movement to a first area of the monitor, followed by the end of the first input in the first area of the monitor; in response to detecting the first input: based on determining that the second application is associated with multiple windows, simultaneously displaying a first representation of a first window of the second application and a second representation of a second window of the second application in a second area of the monitor via the display generating component and the first user interface of the first application; and based on determining that the second application is associated with only a single window, simultaneously displaying a user interface of the second application via the display generating component and the first user interface of the first application, wherein the user interface of the second application is displayed in the second area of the monitor.
[0011] According to some implementations, a method is performed on an electronic device that includes display generating components (e.g., a monitor, projector, head-up display, etc.) and one or more input devices (e.g., a camera, remote controller, keyboard, touch-sensitive surface coupled to a separate monitor, or a touchscreen display that serves as both a monitor and a touch-sensitive surface).The method includes: simultaneously displaying a first application view and a second application view via the display generation component in a first simultaneous display configuration of a plurality of simultaneous display configurations, the plurality of simultaneous display configurations including the first simultaneous display configuration, the second simultaneous display configuration, and the third simultaneous display configuration, the first simultaneous display configuration specifying a first arrangement of the application views to be displayed simultaneously, the second simultaneous display configuration specifying a second arrangement of the application views to be displayed simultaneously, different from the first arrangement of the application views to be displayed simultaneously, and the third simultaneous display configuration specifying a third arrangement of the application views to be displayed simultaneously, different from both the first arrangement and the second arrangement of the application views to be displayed simultaneously; detecting a first input, the first input starting at a position pointing to the first application view within the first arrangement of the application views to be displayed simultaneously, and including a first movement, followed by the end of the first input after the first movement is detected; in response to detecting the first movement of the first input, moving the representation of the first application view on the display according to the first movement of the first input, including: when the representation of the first application view is above a first portion of the display, the end of displaying the first input will result in the first simultaneous display configuration being displayed simultaneously. Displaying a first visual indication of the first application view and the second application view; when the representation of the first application view is above a second portion of the display, displaying the end of the first input will cause a second visual indication of the first application view and the second application view to be displayed in a second simultaneous display configuration; and when the representation of the first application view is above a third portion of the display, displaying the end of the first input will cause a third visual indication of the first application view and the second application view to be displayed in the third simultaneous display configuration; and in response to detecting the end of the first input: based on determining that the first input ends when the first application view is above the first portion of the display, displaying the first application view and the second application view in the first simultaneous display configuration; based on determining that the first input ends when the first application view is above the second portion of the display, displaying the first application view and the second application view in the second simultaneous display configuration; and based on determining that the first input ends when the first application view is above the third portion of the display, displaying the first application view and the second application view in the third simultaneous display configuration.
[0012] According to some embodiments, an electronic device includes: a display generating component (e.g., a display, projector, head-mounted display, etc.); one or more input devices (e.g., a touch-sensitive surface, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface); optional one or more tactile output generators; one or more processors; and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some embodiments, a computer-readable storage medium stores instructions therein that, when executed by an electronic device having a display generating component, one or more input devices (e.g., a touch-sensitive surface, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more tactile output generators, cause the device to perform or cause the operation of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a display generating component, one or more input devices (e.g., a touch-sensitive surface, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), optional one or more tactile output generators, memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements of the elements shown in any of the methods described herein, said one or more elements being updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display generating component, one or more input devices (e.g., a touch-sensitive surface, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more tactile output generators; and means for performing or causing the operation of any of the methods described herein. According to some embodiments, an information processing device in an electronic device having a display generating component, one or more input devices (e.g., a touch-sensitive surface, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface), and optional one or more tactile output generators includes means for performing or causing the operation of any of the methods described herein.
[0013] Therefore, improved methods and interfaces are provided for electronic devices having display generation components, one or more input devices (e.g., touch-sensitive surfaces, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surfaces), optional one or more haptic output generators, optional one or more device orientation sensors, and optional audio systems, for interacting with multiple windows on handheld or portable electronic devices, thereby improving the effectiveness, efficiency, and user satisfaction of these devices. Such methods and interfaces can complement or replace conventional methods for multitasking and interacting with multiple windows.
[0014] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description
[0015] To better understand the various embodiments described, reference should be made to the following detailed description section in conjunction with the accompanying drawings, wherein similar reference numerals throughout the drawings refer to corresponding components.
[0016] Figure 1A This is a high-level block diagram of a computing device with a touch-sensitive display according to some implementation schemes.
[0017] Figure 1B This is a block diagram of exemplary components for event handling according to some implementation schemes.
[0018] Figure 1C This is a schematic diagram of a portable multi-functional device with a touch-sensitive display according to some implementation schemes.
[0019] Figure 1D This is a schematic diagram illustrating a computing device having a touch-sensitive surface separate from the display, according to some embodiments.
[0020] Figure 2 This is a schematic diagram of a touch-sensitive display according to some implementation schemes, used to show the user interface of an application's menu.
[0021] Figures 3A to 3C An example of a dynamic intensity threshold according to some implementation schemes is shown.
[0022] Figures 4A1 to 4A50 , Figures 4B1 to 4B51 , Figures 4C1 to 4C48 , Figures 4D1 to 4D19 and Figures 4E1 to 4E28This is a schematic diagram of a touch-sensitive display according to some implementation schemes, illustrating a user interface for interacting with multiple applications and / or windows.
[0023] Figures 5A to 5I It is a flowchart representation of a method for interacting with multiple windows in a corresponding simultaneous display configuration (e.g., a side-pull display configuration) according to some implementation schemes.
[0024] Figures 6A to 6E It is a flowchart representation of a method for interacting with an application icon when displaying an application, according to some implementation schemes.
[0025] Figures 7A to 7H It is a flowchart representation of a method for displaying content along with the currently displayed application, according to some implementation schemes.
[0026] Figure 7I It is a flowchart representation of a method for dragging and dropping objects onto the corresponding area of the display to open a new window, according to some implementation schemes.
[0027] Figures 8A to 8E It is a flowchart representation of a method for displaying an application along with the currently displayed application, according to some implementation schemes.
[0028] Figures 9A to 9J This is a flowchart representation of a method for changing window display configuration using fluid gestures according to some implementation schemes. Detailed Implementation
[0029] This disclosure describes various implementations that facilitate multitasking on portable electronic devices, where conventional multi-window interaction and user interface navigation techniques have proven inefficient, cumbersome, error-prone, and time-consuming. For battery-powered devices with small displays, there is a need for improved user interfaces for interacting with multiple applications, windows, and / or documents.
[0030] In some implementations, a method is described for performing window switching within a subgroup of windows (e.g., a group of pull-out applications or windows) configured to be displayed simultaneously with another full-screen window or application. Subgroups of windows with the same display configuration (e.g., displayed in pull-out mode) are organized in a stack or carousel and can switch in response to gestures that meet predefined criteria. Furthermore, an overlay switcher user interface is provided to offer a consistent way to view and manage subgroups of windows displayed in pull-out mode and to quickly select windows to overlay on the currently displayed full-screen window or application.
[0031] In some implementations, application switching requests and window management requests are integrated into the same input (e.g., a tap input on an application icon when a first application is displayed). Heuristics are used to determine whether to switch to a second application or display the window switcher for the first application. When the active application icon corresponds to the displayed application, the input is considered a request to open the window switcher for that application; and when the active application icon corresponds to an application other than the displayed application, the input is considered a request to switch applications, regardless of the number of windows already open for the first application. Input is ignored if the currently displayed application does not have multiple windows (e.g., optionally with error feedback). This integration of application switching and window switching within the application provides a more efficient interface because the user does not need to keep track of the number of currently open windows for the currently displayed application. Instead, the device automatically provides intuitive responses based on heuristics, thereby improving user interface efficiency and reducing the amount of input required to achieve the desired result.
[0032] In some implementations, an object representing content is dragged from the currently displayed window to a predefined area of the display, and the device opens a new window based on the position of the input or the position of the dragged object when the end of the input is detected. This new window displays the content together with the currently displayed window in a corresponding simultaneous display configuration (e.g., in a side-swipe window or a split-screen window). In some implementations, drag-and-drop operations are also integrated with drag-and-drop operations performed within the original window containing the object representing content or in another simultaneously displayed window. The integration of multiple operations performed within an application window, across two simultaneously displayed windows, in a first type of new window, or in a second type of new window allows users to easily perform different operations based on the end position of the input. This helps reduce the complexity of user interface interactions because fewer gestures need to be implemented, used, and remembered to implement these functions, thereby reducing user errors and improving the efficiency of the user interface.
[0033] In some implementations, when an object is dragged and dropped into different areas of the display, different operations are performed depending on the end position of the input. These operations include opening different types of new windows (e.g., swiping windows or split-screen windows), operations within the object's original window, and operations across two simultaneously displayed windows. For some objects, such as application icons, applicable operations within or across existing windows of the display are uncommon; therefore, it is beneficial to enlarge the placement area for opening new windows by dragging and dropping application icons, relative to the placement area used to open new windows by dragging and dropping objects representing content. This user interface improvement helps reduce user errors without significantly impairing functionality, thereby improving the efficiency of the user interface.
[0034] In some implementations, when a request to open an application in a simultaneous display configuration is received, the application is displayed in the simultaneous display configuration if it is not associated with multiple windows, and a window selector user interface is displayed in the corresponding simultaneous display configuration if the application is associated with multiple windows. Allowing the user to open an application in a simultaneous display configuration, or to open the application using the same input (e.g., dragging the application icon to a side area of the display), based on whether the application is associated with multiple windows, is intuitive and efficient. This helps reduce the number and types of input that the user needs to provide to achieve the desired result, and reduces the chance of user errors.
[0035] In some implementations, in response to input that drags a window to different placement areas defined on the display, the device provides dynamic visual feedback to indicate the resulting display configuration of the window if the end of the input is detected at the current position. The final state of the user interface is not determined until the end of the input is detected, and by ending the input at an appropriate location, the user is given the opportunity to view and understand various possible outcomes before the final display configuration of the window is submitted. The fluid nature of the input and feedback allows multiple results to be achieved using the same gesture, and reduces the chance of user error through the simplicity of the gestures and the continuous visual feedback provided according to the current position of the input.
[0036] The methods and user interface heuristics described in this paper take into account the following factors: (i) the significant difference in screen size between desktop computers and handheld electronic devices, and (ii) the significant difference between keyboard and mouse interaction on desktop computers and touch and gesture input on handheld electronic devices with touch-sensitive displays. Various multitasking functions at different levels, such as across applications, across all windows of a given application, across windows of a given type within a given application, switching between opening new windows or between existing windows, between opening content and opening applications, etc., are achieved without the need for menu navigation or complex input sequences. These methods and user interface heuristics provide intuitive and easy-to-use systems and methods for simultaneously accessing multiple functions or applications on a handheld electronic device.
[0037] Figures 1A to 1D and Figure 2 A description of an exemplary device is provided. Figures 3A to 3C An example of a dynamic intensity threshold is shown. Figures 4A1 to 4A50 , Figures 4B1 to 4B51 , Figures 4C1 to 4C48 , Figures 4D1 to 4D19 and Figures 4E1 to 4E28 These are schematic diagrams of touch-sensitive displays used to illustrate user interfaces interacting with multiple applications and / or windows, according to some implementation schemes, and these diagrams are used to illustrate... Figures 5A to 5I , Figures 6A to 6E , Figures 7A to 7H , Figure 7I , Figures 8A to 8E and 9A to Figure 9J The methods / procedures shown are as follows.
[0038] Exemplary device
[0039] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the various aspects of the embodiments.
[0040] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.
[0041] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” (“a”, “an”) and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in connection with the description. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0042] As used herein, depending on the context, the term "if" is optionally interpreted as meaning "when," "at," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is optionally interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected," or "in response to the detection of [the stated condition or event]."
[0043] The disclosure herein is interchangeable with respect to detecting touch input on, at, above, on top of, or substantially within a specific user interface element or portion of a touch-sensitive display. As used herein, touch input detected “at” a specific user interface element can also be detected “on,” “above,” “on top of,” or “substantially within” the same user interface element. In some embodiments, and as described in more detail below, the desired sensitivity level for detecting touch input is configured by the user of the electronic device (e.g., the user can decide (and configure the electronic device to be operated) that touch input should only be detected when the touch input is entirely within the user interface element).
[0044] As is widely recognized, the use of personally identifiable information should comply with privacy policies and practices that are generally accepted to meet or exceed industry or governmental requirements for protecting user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly explained to users.
[0045] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). IPOD and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).
[0046] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.
[0047] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.
[0048] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.
[0049] The focus is now on implementation schemes for portable electronic devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 (also referred to herein as electronic device 100 or device 100) with a touch-sensitive display 112 according to some embodiments. The touch-sensitive display 112 is sometimes conveniently called a “touchscreen” and is sometimes referred to as a touch-sensitive display system. Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a controller 120, one or more processing units (CPUs) 122, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 (e.g., touch-sensitive surfaces, such as the touch-sensitive display system 112 of device 100) for detecting the intensity of contact on device 100. Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100 or the touchpad of device 100). These components optionally communicate via one or more communication buses or signal lines 103.
[0050] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the tactile surface remains unchanged, movement of the tactile surface can optionally be interpreted or sensed by the user as the “roughness” of the tactile surface. While such interpretations of touch by users will be limited by their individualized sensory perceptions, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a user’s specific sensory perception (e.g., “release click,” “press click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components, which will generate the sensory perception described by a typical (or average) user.
[0051] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.
[0052] Memory 102 optionally includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, or other random access solid-state memory devices), and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 102 optionally includes one or more storage devices located remotely from one or more processors 122. Access to memory 102 by other components of device 100 (such as CPU 122 and peripheral interface 118) is optionally controlled by controller 120.
[0053] Peripheral interface 118 can be used to couple the device's input peripherals and output peripherals to CPU 122 and memory 102. One or more processors 122 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.
[0054] In some implementations, the peripheral interface 118, CPU 122, and controller 120 are optionally implemented on a single chip, such as chip 104. In other implementations, they are optionally implemented on separate chips.
[0055] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates wirelessly with networks and other devices, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). Wireless communication may optionally use any of a variety of communication standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy, and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n).
[0056] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack. The headset jack provides an interface between audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., mono-ear or binaural headphones) and input (e.g., microphone).
[0057] I / O subsystem 106 connects input / output peripherals such as touchscreen 112 and other input control devices 116 on device 100 to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / send electrical signals to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some alternative embodiments, one or more input controllers 160 may optionally be coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing devices such as a mouse. The one or more buttons may optionally include volume up / down buttons for speaker 111 and / or microphone 113. One or more buttons may optionally include a push-down button.
[0058] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects.
[0059] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In one exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the area beneath the user's finger.
[0060] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, or OLED (Organic Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that found in Apple Inc. (Cupertino, California), is used. IPOD and The technology in it.
[0061] Touchscreen 112 optionally has a video resolution of more than 400 dpi. In some embodiments, touchscreen 112 has a video resolution of at least 600 dpi. In other embodiments, touchscreen 112 has a video resolution of at least 1000 dpi. The user optionally uses any suitable object or finger, such as a stylus or finger, to contact touchscreen 112. In some embodiments, the user interface is designed to operate primarily with finger-based contact and gestures. In some embodiments, the device translates finger-based input into precise pointer / cursor positions or commands to perform actions desired by the user.
[0062] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.
[0063] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.
[0064] The device 100 may optionally also include one or more optical sensors 164. Figure 1A An optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device, allowing the user to optionally acquire an image of another video conference participant for use in the video conference while viewing that participant on the touchscreen display.
[0065] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor 165 is shown coupled to a strength sensor controller 159 in I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrostatic sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100 opposite to the touchscreen 112 located on the front of device 100.
[0066] The device 100 optionally also includes one or more proximity sensors 166. Figure 1AA proximity sensor 166 is shown coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).
[0067] The device 100 may optionally also include one or more tactile output generators 167. Figure 1A A haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components; and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., a haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the rear of the device 100, opposite to the touch-sensitive display 112 located on the front of the device 100.
[0068] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 is shown coupled to a peripheral device interface 118. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, information is displayed on a touch-sensitive display in portrait or landscape view based on analysis of data received from the one or more accelerometers. The device 100 may optionally include, in addition to the accelerometer 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information about the location and orientation (e.g., portrait or landscape) of the device 100.
[0069] In some embodiments, software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 stores device / global internal state 157, such as... Figure 1A As shown in the diagram. The device / global internal state 157 includes one or more of the following: active application state, indicating which applications (if any) are currently active; display state, indicating what applications, views, or other information occupy various areas of the touch-sensitive display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and location information regarding the device's position and / or orientation (e.g., the device's orientation). In some embodiments, the device / global internal state 157 communicates with the multitasking module 180 to track applications active in a multitasking mode (also known as shared screen view, shared screen mode, or multitasking mode). In this way, if the device 100 rotates from portrait display mode to landscape display mode, the multitasking module 180 is able to retrieve multitasking state information (e.g., the display area of each application in multitasking mode) from the device / global internal state 157 to reactivate the multitasking mode after switching from portrait to landscape.
[0070] Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
[0071] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with, the 30-pin connector used in some embodiments of Apple Inc.'s iPod devices. In other embodiments, the external port is a multi-pin (e.g., 8-pin) connector that is the same as, similar to, and / or compatible with, the 8-pin connector used in Apple Inc.'s Lightning connectors.
[0072] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.
[0073] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least one subset of intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold for a touchpad or touch-sensitive display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touch-sensitive display hardware. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).
[0074] The touch / motion module 130 optionally detects user gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and in some embodiments, subsequently detecting a finger lift-off (lift-away) event.
[0075] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.
[0076] In some implementations, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from applications, etc., specifying the graphic to be displayed, and also receives coordinate data and other graphic attribute data if necessary, and then generates screen image data for output to the display controller 156. In some implementations, the graphics module 132 retrieves graphics stored along with the multitasking data 176 of each application 136. Figure 1BIn some implementations, the multitasking data 176 stores multiple graphics of different sizes, enabling applications to quickly resize them in screen-sharing mode.
[0077] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 167 to produce haptic output at one or more locations on the device 100 in response to user interaction with the device 100.
[0078] Optionally, the text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., the contact module 137, the email client module 140, the IM module 141, the browser module 147, and any other application that requires text input).
[0079] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).
[0080] Application (“application”) 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:
[0081] • Contacts module 137 (sometimes called address book or contact list);
[0082] • Telephone module 138;
[0083] • Video conferencing module 139;
[0084] • Email client module 140;
[0085] • Instant Messaging (IM) module 141;
[0086] Fitness module 142;
[0087] • Camera module 143 for still images and / or video images;
[0088] • Image management module 144;
[0089] • Browser module 147;
[0090] • Calendar module 148;
[0091] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock market desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5, and other desktop mini-programs obtained by the user, and desktop mini-programs created by the user 149-6;
[0092] • Search module 151;
[0093] • A video and music player module 152, optionally composed of a video player module and a music player module;
[0094] • Memo module 153;
[0095] • Map module 154; and / or
[0096] • Online video module 155.
[0097] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, website creation applications, disk editing applications, spreadsheet applications, JAVA-enabled applications, encryption, digital rights management, speech recognition, desktop applet creator modules 149-6 for creating user-created desktop applets, and speech copying.
[0098] In conjunction with the touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via the telephone module 138, video conferencing module 139, email client module 140, or IM module 141; and so on.
[0099] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in address book 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.
[0100] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.
[0101] Incorporating RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.
[0102] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS protocols for internet-based instant messaging), receiving an instant message, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).
[0103] Incorporating RF circuitry 108, touchscreen 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 146, health module 142 includes executable instructions for: creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (sports devices, such as watches or pedometers); receiving fitness sensor data; calibrating sensors used for monitoring fitness; selecting and playing fitness music; and displaying, storing, and transmitting fitness data.
[0104] In conjunction with touchscreen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or videos (including video streams) and store them in memory 102; modify the characteristics of still images or videos; or delete still images or videos from memory 102.
[0105] Incorporating touchscreen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.
[0106] Combining RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) according to user instructions.
[0107] Combining RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and calendar-related data (e.g., calendar entries, to-dos, etc.) according to user instructions.
[0108] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).
[0109] In conjunction with RF circuit 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134 and browser module 147, the desktop applet creator module (not shown) can optionally be used by the user to create desktop applets (e.g., to transfer user-specified portions of a webpage to a desktop applet).
[0110] In conjunction with the touchscreen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.
[0111] Incorporating touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as the iPod from Apple Inc.
[0112] Combining the touchscreen 112, display controller 156, touch module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.
[0113] In conjunction with RF circuitry 108, touchscreen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally configured to receive, display, modify, and store maps and map-related data (e.g., driving directions; data on shops and other points of interest at or near a specific location; and other location-based data) according to user instructions.
[0114] In conjunction with touchscreen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that allow a user to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos.
[0115] like Figure 1A As shown, the portable multifunction device 100 also includes a multitasking module 180 for managing multitasking operations on the device 100 (e.g., communicating with the graphics module 132 to determine appropriate display areas for applications to be displayed simultaneously). The multitasking module 180 optionally includes the following modules (or instruction sets) or a subset or superset thereof:
[0116] • Application selector 182;
[0117] • Compatibility module 184;
[0118] • Picture-in-picture / overlay module 186; and
[0119] • Multitasking history 188 is used to store information about the user's multitasking history (e.g., frequently used applications in multitasking mode, the latest display area of applications in multitasking mode, applications pinned together for display in split view / multitasking mode, etc.).
[0120] In conjunction with the touchscreen 112, display controller 156, contact module 130, graphics module 132, and contact strength sensor 165, the application selector 182 includes executable instructions to display a power indication corresponding to an application (e.g., one or more of applications 136) and allows a user of device 100 to select the power indication to use in a multitasking / split-screen mode (e.g., a mode in which more than one application is simultaneously displayed and active on touchscreen 112). In some embodiments, the application selector 182 is a taskbar (e.g., taskbar 408 described below).
[0121] Integrating the touchscreen 112, display controller 156, touch module 130, graphics module 132, and application selector 182, the compatibility module 184 includes executable instructions to determine whether a particular application is compatible with multitasking mode (e.g., by checking flags, such as those stored with multitasking data 176 for each application 136). Figure 1B (As shown).
[0122] In conjunction with the touchscreen 112, display controller 156, contact module 130, graphics module 132, and contact strength sensor 165, the PIP / cover module 186 includes executable instructions to determine the reduced size of an application that will be displayed as an overlay on another application, and to determine the appropriate location on the touchscreen 112 for displaying the reduced-size application (e.g., the location to avoid important content within the active application being covered by the reduced-size application).
[0123] Each module and application identified above corresponds to a set of executable instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.
[0124] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.
[0125] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to a main menu, home menu, or root menu. In such embodiments, a touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.
[0126] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A The middle) includes an event classifier 170 (e.g., in operating system 126) and selected from portable multifunction devices 100 ( Figure 1A The corresponding application 136-1 in application 136 (e.g., any of the aforementioned applications stored in memory 102 together with application 136).
[0127] Event classifier 170 receives event information and determines the application 136-1 and application view 175 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 175 to which the event information should be delivered.
[0128] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by application 136-1, a state queue for enabling the user to return to a previous state or view of application 136-1, and a repeat / undo queue for previous actions taken by the user. In some implementations, the multitasking module 180 uses the application internal state 192 to facilitate multitasking operations (e.g., the multitasking module 180 retrieves recovery information from the application internal state 192 to redisplay a previously deactivated application).
[0129] In some implementations, each application 136-1 stores multitasking data 176. In some implementations, multitasking data 176 includes compatibility flags (e.g., flags accessed by compatibility module 184 to determine whether a particular application is compatible with multitasking mode), a list of compatible sizes for displaying application 136-1 in multitasking mode (e.g., 1 / 4, 1 / 3, 1 / 2, or full screen), and graphics of various sizes (e.g., different graphics for each size in the list of compatible sizes).
[0130] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.
[0131] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).
[0132] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.
[0133] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.
[0134] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.
[0135] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that forms an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.
[0136] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.
[0137] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 178). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 181.
[0138] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.
[0139] In some embodiments, application 136-1 includes a plurality of event handlers 177 and one or more application views 175, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 175 in application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 175 includes a plurality of event recognizers 180. In other embodiments, one or more event recognizers among the event recognizers 180 are part of a separate module, which is a higher-level object such as a user interface toolkit (not shown) from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handlers 177 include one or more of the following: a data updater 177-1, an object updater 177-2, a GUI updater 177-3, and / or event data 179 received from an event classifier 170. Event handler 177 optionally utilizes or invokes data updater 177-1, object updater 177-2, or GUI updater 177-3 to update the application's internal state 192. Alternatively, one or more application views in application view 175 include one or more corresponding event handlers 177. Additionally, in some embodiments, one or more of data updater 177-1, object updater 177-2, and GUI updater 177-3 are included in the corresponding application view 175.
[0140] The corresponding event recognizer 178 receives event information (e.g., event data 179) from the event classifier 170 and identifies events from the event information. The event recognizer 178 includes an event receiver 181 and an event comparator 183. In some embodiments, the event recognizer 178 also includes at least one subgroup of metadata 189 and event delivery instructions 190 (which optionally include sub-event delivery instructions).
[0141] Event receiver 181 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device orientation).
[0142] Event comparator 183 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 183 includes event definition 185. Event definition 185 contains the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 177.
[0143] In some implementations, event definition 186 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 183 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 183 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 177, the event comparator uses the result of the hit test to determine which event handler 177 should be activated. For example, event comparator 183 selects the event handler associated with the sub-event and the object that triggered the hit test.
[0144] In some implementations, the definition of the corresponding event 187 also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.
[0145] When the corresponding event recognizer 178 determines that the sub-event sequence does not match any event in event definition 185, the corresponding event recognizer 178 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process ongoing sub-events based on touch gestures.
[0146] In some embodiments, the corresponding event recognizer 178 includes metadata 189 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 189 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 189 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.
[0147] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 178 activates the event handler 177 associated with the event. In some implementations, the corresponding event recognizer 178 delivers event information associated with the event to the event handler 177. Activating the event handler 177 is different from sending (and delaying) the sub-events to the corresponding hit view. In some implementations, the event recognizer 178 issues a flag associated with the identified event, and the event handler 177 associated with that flag acquires the flag and executes a predefined procedure.
[0148] In some implementations, event delivery instruction 190 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined procedure.
[0149] In some implementations, data updater 177-1 creates and updates data used in application 136-1. For example, data updater 177-1 updates phone numbers used in contact module 137, or stores video files used in video or music player module 145. In some implementations, object updater 177-2 creates and updates objects used in application 136-1. For example, object updater 177-2 creates new user interface objects or updates the location of user interface objects. GUI updater 177-3 updates the GUI. For example, GUI updater 177-3 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display. In some implementations, GUI updater 177-3 communicates with multitasking module 180 to adjust the size of various applications displayed in multitasking mode.
[0150] In some implementations, event handler 177 includes, or has access to, a data updater 177-1, an object updater 177-2, and a GUI updater 177-3. In some implementations, data updater 177-1, object updater 177-2, and GUI updater 177-3 are included in a single module of the corresponding application 136-1 or application view 175. In other implementations, they are included in two or more software modules.
[0151] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.
[0152] Figure 1CThis is a schematic diagram of a portable multifunction device (e.g., portable multifunction device 100) with a touch-sensitive display (e.g., touchscreen 112) according to some embodiments. The touch-sensitive display optionally displays one or more graphics within a user interface (UI) 201a. In this embodiment, and in other embodiments described below, a user can select one or more of these graphics by gesturing on the screen, for example, with one or more fingers or with one or more styluses. In some embodiments, selection of one or more graphics occurs when the user (e.g., by lifting a finger off the screen) interrupts contact with one or more graphics. In some embodiments, gestures optionally include one or more tapping gestures (e.g., a series of touches on the screen after lifting), one or more swiping gestures (continuous contact during a gesture along the screen surface, e.g., from left to right, from right to left, up and / or down), and / or scrolling of a finger that has been in contact with device 100 (e.g., from right to left, from left to right, up and / or down). In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, when the gesture used to launch an application is a tap gesture, a swipe gesture that sweeps over an application's functionality (e.g., an icon) may optionally not launch (e.g., open) the corresponding application.
[0153] Device 100 optionally also includes one or more physical buttons, such as a "home" button or a menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.
[0154] In one embodiment, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.
[0155] Figure 1D It is used to show the device (e.g., Figure 1A A schematic diagram of a user interface on a device 100, the device having a touch-sensitive surface 195 (e.g., a tablet or touchpad) separate from a display 194 (e.g., a touchscreen 112). In some embodiments, the touch-sensitive surface 195 includes one or more contact strength sensors (e.g., one or more of contact strength sensors 359) for detecting the intensity of contact on the touch-sensitive surface 195, and / or one or more haptic output generators 357 for generating haptic outputs for a user of the touch-sensitive surface 195.
[0156] Although some embodiments of the following examples are given with reference to input on touchscreen 112 (which combines a touch-sensitive surface and a display), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, such as... Figure 1D As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 1D 195) has a spindle (e.g., on the display (e.g., 194) with the spindle on the display (e.g., Figure 1D The principal axis corresponding to 198 in the middle (e.g., Figure 1D (199 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 1D In the middle, 197-1 corresponds to 196-1 and 197-2 corresponds to 196-2) is in contact with the touch-sensitive surface 195 (e.g., Figure 1D (Referring to 197-1 and 197-2 in the text). Thus, on touch-sensitive surfaces (e.g., ... Figure 1D 195 in the middle) and the display of a multi-functional device (e.g., Figure 1D When 194 is separated, user input detected by the device on the touch-sensitive surface (e.g., touch 197-1 and touch 197-2 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.
[0157] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, single-finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or a mouse and multiple finger touches may optionally be used simultaneously.
[0158] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 1D When an input (e.g., a press input) is detected on a touch-sensitive surface 195 (in some embodiments, the touch-sensitive surface 195 is a touchpad), that particular user interface element is adjusted according to the detected input. This applies to touchscreen displays capable of enabling direct interaction with user interface elements on the touchscreen display (e.g., ...). Figure 1AIn some embodiments of the touch-sensitive display system 112 or touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that transmits the user's expected interaction with the user interface (e.g., by indicating to the device the user interface element with which the user expects to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch-sensitive display), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).
[0159] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact on a tactile surface (e.g., finger contact or stylus contact), or to an alternative (substitute) of the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values that includes at least four different values and more typically hundreds of different values (e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in pressure units) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise not be easily accessible to the user on smaller devices with limited physical area for displaying power indications and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons) for (e.g., on a touch-sensitive display).
[0160] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of the portable computing system 100). For example, the mouse “click” threshold of a touchpad or touchscreen display can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).
[0161] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted, before or after contact begins to move, before contact ends, before or after contact intensity is detected to increase and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more intensity thresholds is used to determine whether to perform one or more actions (e.g., whether to select an option or abandon the action), rather than to determine whether to perform the first or second action.
[0162] In some implementations, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface may receive a series of swipe contacts that transition from a starting position to an ending position (e.g., a drag gesture), where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position may be based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm may be applied to the intensity of the swipe gesture before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.
[0163] In some implementations, one or more predefined intensity thresholds are used to determine whether a particular input meets an intensity-based criterion. For example, the one or more predefined intensity thresholds include (i) a contact detection intensity threshold IT0, and (ii) a light press intensity threshold IT0. L (iii) Threshold of deep pressure intensity IT D (For example, at least initially higher than I) L (iv) and / or one or more other intensity thresholds (e.g., below I) L Intensity threshold I H As used in this article, IT L and I L Refers to the same light press intensity threshold, IT D and I D Refers to the same deep pressure threshold, while IT H and I H This refers to the same intensity threshold. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold IT0, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface, without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.
[0164] In some implementations, the device's response to input detected by the device depends on a criterion based on the contact intensity during the input. For example, for some "light press" inputs, a first response is triggered by the intensity of contact exceeding a first intensity threshold during the input. In some implementations, the device's response to input detected by the device depends on a criterion that includes both the contact intensity during the input and a time-based criterion. For example, for some "deep press" inputs, a second response is triggered by the intensity of contact exceeding the second intensity threshold (greater than the first light press threshold) during the input, provided a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold. The duration of this delay time is typically less than 200 ms (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). This delay time helps avoid accidental deep press inputs. As another example, for some "deep press" inputs, there is a period of decreased sensitivity after the first intensity threshold is reached. During this period of decreased sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to the detected deep press input does not depend on time-based criteria.
[0165] In some implementations, one or more of the input intensity threshold and / or the corresponding output varies based on one or more factors, such as user settings, contact motion, input timing, application operation, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. Patent Applications 14 / 399,606 and 14 / 624,296, the entire contents of which are incorporated herein by reference.
[0166] For example, Figure 3A A dynamic intensity threshold 380, which varies over time based in part on the intensity of the touch input 376, is shown. The dynamic intensity threshold 380 is the sum of two components: a first component 374 that decays over time after a predefined delay time p1 starting from the initial detection of the touch input 376, and a second component 378 that tracks the intensity of the touch input 376 over time. The initial high intensity threshold of the first component 374 reduces the chance of an unintentional "deep press" response, while still allowing an immediate "deep press" response if the touch input 376 provides sufficient intensity. The second component 378 reduces unintentional triggering of the "deep press" response through gradual intensity fluctuations in the touch input. In some embodiments, when the touch input 376 meets the dynamic intensity threshold 380 (e.g., in…),… Figure 3A When the pressure is applied at point 381, a "deep press" response is triggered.
[0167] Figure 3B Another dynamic intensity threshold 386 is shown (e.g., intensity threshold I). D ). Figure 3B Two other intensity thresholds are also shown: the first intensity threshold I H Second intensity threshold I L .exist Figure 3B In the middle, although touch input 384 satisfies the first intensity threshold I before time p2. H Second intensity threshold I L However, a response was not provided until time 382 after a delay of time p2. Similarly, at... Figure 3B In the process, the dynamic intensity threshold 386 decays over time, with the decay occurring from time 382 (triggered by the second intensity threshold I). L The associated response begins after a predefined delay time p1, starting at time 388. This type of dynamic intensity threshold reduction immediately follows the trigger with a lower threshold intensity (such as the first intensity threshold I). H Or the second intensity threshold I L The associated response, or the unexpected triggering of the dynamic intensity threshold I, occurs after or simultaneously with it. D Related responses.
[0168] Figure 3C Another dynamic intensity threshold 392 is shown (e.g., intensity threshold I). D ).exist Figure 3C In the middle, after a delay time p2 has elapsed since the touch input 390 was initially detected, the trigger is activated based on the intensity threshold I. L The associated response. Simultaneously, the dynamic intensity threshold 392 decays after a predefined delay time p1 from when touch input 390 is initially detected. Therefore, the intensity of touch input 390 without release of touch input 390 is triggered in accordance with the intensity threshold I. L The associated response decreases afterward, and then an increase in the intensity of the touch input 390 can trigger a response with a threshold intensity I. D The associated response (e.g., at time 394) even when the intensity of the touch input at 390 is below another intensity threshold, e.g., intensity threshold I. L hour.
[0169] Contact characteristic strength from below the light press strength threshold IT L The intensity increases to within the light press intensity threshold IT L With deep pressure intensity threshold IT D The intensity of this contact is sometimes referred to as a "light press" input. The characteristic intensity of the contact ranges from below the deep press intensity threshold (IT). D The intensity increases to above the deep press intensity threshold IT DThe intensity is sometimes referred to as "deep press" input. The contact feature intensity increases from below the contact detection intensity threshold IT0 to between the contact detection intensity threshold IT0 and the light press intensity threshold IT0. L The intensity of the contact is sometimes referred to as the detection of a contact on the touch surface. The characteristic intensity of the contact decreasing from above the contact detection intensity threshold IT0 to below the contact detection intensity threshold IT0 is sometimes referred to as the detection of a contact being lifted off the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, shaded circles or ellipses are used to represent the intensity of a contact on the touch-sensitive surface. In some examples, unshaded circles or ellipses are used to represent the corresponding contact on the touch-sensitive surface without specifying the intensity of the corresponding contact.
[0170] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., performing the corresponding operation on the "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below a press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below a press input threshold (e.g., performing the corresponding operation on the "upward stroke" of the corresponding press input).
[0171] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above a press input intensity threshold and a subsequent decrease in the intensity of that contact below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., performing the corresponding operation on the "up stroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and optionally the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and an appropriate operation is performed in response to the detection of a press input (e.g., depending on the environment, the contact strength increases or decreases).
[0172] For ease of explanation, the description of an operation triggered optionally in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided: the contact strength increases to above the press input strength threshold; the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold; the contact strength decreases to below the press input strength threshold; or the contact strength decreases to below a hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example describing the operation as being performed in response to detecting a decrease in contact strength to below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength to below a hysteresis strength threshold corresponding to and less than the press input strength threshold. As described above, in some embodiments, the triggering of these operations also depends on meeting a time-based criterion (e.g., a delay time has elapsed between meeting a first strength threshold and meeting a second strength threshold).
[0173] Exemplary user interface and associated process
[0174] Now turn our attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on electronic devices (such as device 100 with a touch-sensitive display or a device with a separate display and a touch-sensitive surface) that have display generation components and one or more input devices.
[0175] Figure 2This is a schematic diagram of a touch-sensitive display according to some embodiments, used to illustrate a user interface for an application's menu. A similar user interface is optionally present in device 100 ( Figure 1A Implemented on [the platform]. In some implementations, the user interface 201a includes the following elements or a subset or superset thereof:
[0176] • One or more signal strength indicators 202 for wireless communications such as cellular signals and Wi-Fi signals;
[0177] ·time;
[0178] Bluetooth indicator 205;
[0179] • Battery status indicator 206;
[0180] • Tray tray 203 with icons for frequently used applications, such as:
[0181] The telephone module 138 has an icon 216 labeled "telephone", which optionally includes an indicator 214 for the number of missed calls or voicemails;
[0182] The email client module 140 has an icon 218 labeled "Mail", which optionally includes an indicator 210 for the number of unread emails;
[0183] The icon 220 labeled "Browser" in browser module 147; and
[0184] The icon 222 labeled "iPod" for the video and music player module 152 (also referred to herein as a video or video browsing application, or iPod (a trademark of Apple Inc.) module 152); and
[0185] • Icons of other applications, such as:
[0186] Icon 224 labeled "Message" in module 141 of the οIM module;
[0187] The icon 226 labeled "Calendar" in the calendar module 148;
[0188] The image management module 144 has an icon 228 labeled "Photo".
[0189] The icon 230 of the camera module 143, which is labeled "camera";
[0190] The icon 232 of the online video module 155 is labeled "Online Video";
[0191] The icon 234 in the Stock Market Desktop Mini Program 149-2, labeled "Stock Market";
[0192] The icon 236 labeled "Map" in map module 154;
[0193] The icon labeled "Weather" in the Weather desktop mini-program 149-1 is 238.
[0194] The icon labeled "Clock" in the alarm clock desktop mini-program 149-4 is 240;
[0195] The icon 242 labeled "Fitness" in the fitness module 142;
[0196] The icon 244 labeled "Memo" in the Memo module 153;
[0197] ο Sets an icon 246 for the application or module, which provides access to the settings of the device 100 and its various applications; and
[0198] Other icons for applications such as the App Store, iTunes, Voice Memos, and Utilities.
[0199] It should be noted that Figure 2 The icon labels shown are merely exemplary. Other labels may be optionally used for various application icons. For example, the icon 242 of the fitness module 142 may alternatively be labeled "Fitness Support," "Workout," "Exercise Support," "Sports," "Exercise Support," or "Health." In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.
[0200] In some implementations, the home screen includes two areas: a tray 203 and an icon area 201. For example... Figure 2 As shown, icon area 201 is displayed above tray 203. However, icon area 201 and tray 203 (also referred to as the "taskbar") may optionally be displayed in a location other than those described herein.
[0201] Tray 203 optionally includes icons of a user's favorite applications on computing device 100. Initially, tray 203 may include a set of default icons. The user can customize tray 203 to include icons other than the default ones. In some embodiments, the user customizes tray 203 by selecting icons from icon area 201 and dragging and dropping the selected icons into tray 203 to add icons to tray 203. To remove an icon from tray 203, the user selects the icon displayed in the favorites area for a threshold amount of time, which causes computing device 100 to display a control for removing the icon. The user's selection of the control causes computing device 100 to remove the icon from tray 203. In some embodiments, tray 203 is replaced by taskbar 4006 (described in more detail below), therefore, the details provided above with reference to tray 203 also apply to taskbar 4006, which supplements the description of taskbar 4006 provided below.
[0202] In this disclosure, the reference to "split-screen mode" refers to a mode in which at least two applications are simultaneously displayed side-by-side on display 112 and the two applications are interactive (e.g., an email application and an instant messaging application on...). Figure 4E1(Displayed in split-screen mode). Split-screen mode is also known as a "side-by-side" display configuration or a "split-screen" display configuration. In some embodiments, the at least two applications displayed simultaneously in split-screen mode may also be "pinned" together, which refers to an association between the at least two applications (stored in the memory of device 100) that causes both applications to open together when either of the at least two applications is launched. In some embodiments, a power indicator (e.g., a drag handle displayed near the top edge of the application window) can be used to unpin the application, instead displaying one of the at least two applications as an overlay on the other, and this overlay display mode is called a side-pull display mode (e.g., the email application and instant messaging application in the side-pull mode shown in Figure 5E2). Side-pull mode is also known as a "side-pull" display configuration or a "side-pull view". The side-pull window may also be referred to as a background full-screen window or an "overlay" of a pair of split-screen windows. In some embodiments, the at least two applications displayed simultaneously in side-pull mode are not "pinned" together; therefore, when one of the at least two applications is displayed, the other application is optionally not displayed at the same time, and alternatively, another application is displayed simultaneously. In some implementations, a power indicator (e.g., a drag handle displayed near the top edge of an application window) can be used to pin applications together and display them in split-screen mode. Users can also use a boundary power indicator displayed within a boundary extending between the at least two applications, while they are displayed in split-screen mode, to unpin or clear one of the at least two applications (e.g., by dragging the boundary power indicator until it reaches the edge of display 112 adjacent to the first application of the at least two applications, which is then cleared and subsequently unpinned). The use of a boundary power indicator (or a gesture at the boundary between two applications) to unpin applications is discussed in more detail in commonly owned U.S. Patent Application No. 14 / 732,618 (e.g., in Figures 37H through 37M and the associated descriptive paragraphs), which is incorporated herein by reference in its entirety. While many of the examples provided herein refer to different applications displayed in split-screen and swipe modes, many of these examples are also valid if the windows of the different applications are replaced with different windows of the same application displayed in split-screen or swipe modes, unless otherwise explicitly stated.
[0203] Figures 4A1 to 4A50 , Figures 4B1 to 4B51 , Figures 4C1 to 4C48 , Figures 4D1 to 4D19 and Figures 4E1 to 4E28This is a schematic diagram of a touch-sensitive display according to some implementation schemes, illustrating a user interface for interacting with multiple applications and / or windows.
[0204] Figures 4A1 to 4A50 The user interface behavior of an application window displayed in a side-swipe mode according to some embodiments is illustrated. Interaction with an overlay switcher user interface that simultaneously displays multiple side-swipe windows corresponding to different applications is also described. The user interface in these figures is used to illustrate the processes described below, including... Figures 5A to 5I The process is described below. For ease of explanation, some embodiments of the implementation will be discussed with reference to the operation performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representation point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operation is optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to a contact detected on the touch-sensitive surface 451 when the user interface shown in the figures, along with the focus selector, is displayed on the display 450.
[0205] For the context described below, in some embodiments, the home screen user interface includes multiple application icons corresponding to different applications installed on the device. Each application icon, when activated by the user (e.g., by tapping an input), causes the device to launch the corresponding application and display the application's user interface on the display (e.g., the default initial user interface or the last displayed user interface). The taskbar is a container user interface object that includes a subgroup of application icons selected from the home screen user interface to provide quick access to a small number of frequently used applications. The application icons included in the taskbar are optionally selected by the user (e.g., via the settings user interface) or automatically selected by the device based on various criteria (e.g., usage frequency or time since last use). In some embodiments, the taskbar is displayed as part of the home screen user interface (e.g., covering the bottom portion of the home screen user interface). In some embodiments, in response to a user request (e.g., a gesture that satisfies taskbar display criteria (e.g., an upward swipe gesture starting from the bottom edge of the touchscreen)), the taskbar is displayed above a portion of another user interface (e.g., the application user interface) independent of the home screen user interface. The application switcher user interface displays a representation of multiple recently opened applications (e.g., arranged in order based on the time the applications were last displayed). When selected (e.g., by tapping the input), a representation of the corresponding recently opened application (e.g., a snapshot of the last displayed user interface of the corresponding recently opened application) causes the device to re-display the last displayed user interface of the corresponding recently opened application on the screen. In some implementations, the application switcher user interface display may correspond to windows with different display configurations of the same or different applications (e.g., full-screen windows, side-swipe windows and split-screen windows, minimized windows and / or draft windows, etc.).
[0206] like Figure 4A1 As shown, the first application window of the first application (e.g., window 4002 of the map application) is displayed on the touchscreen 112 in an independent display configuration (e.g., also a full-screen display configuration), without being displayed simultaneously with another application window of the same application or another application. The first application window 4002 displays a portion of the first user interface of the first application (e.g., a searchable map interface). Input that meets the taskbar display criteria (e.g., an upward swipe input via touch 4004) is detected on the touchscreen 112 (e.g., near the bottom edge of the touchscreen 112). Figures 4A1 to 4A2As shown. In response to detecting input that meets the taskbar display criteria, taskbar 4006 is displayed as a first application window (e.g., window 4002) covering the first application. Taskbar 4006 includes multiple application icons corresponding to different applications (e.g., icon 216 for a phone application, icon 218 for an email application, icon 220 for a browser application, and icon 232 for an online video application). In some embodiments, the taskbar includes application icons for the currently displayed application (e.g., a map application) and one or more recently displayed applications. In some embodiments, the taskbar is temporarily removed from the display in response to input that meets the taskbar clearing criteria (e.g., a downward swipe gesture moving towards the bottom edge of the touchscreen on the taskbar).
[0207] exist Figures 4A4 to 4A7 In some implementation schemes, a second application window (e.g., an online video application) of a second application (e.g., Figure 4A7 Window 4010 in the first application is displayed in a side-pull display configuration as an overlay on the first application window (e.g., window 4002). The second application window displays a portion of the second application's second user interface (e.g., a media player user interface for an online video application). Figure 4A4 As shown, when the first window 4002 of the first application (e.g., a map application) is displayed, input that meets the selection criteria (e.g., a still touch hold input or a light press input via touch 4008) is detected on the application icon 232 of the online video application, and it enables a drag operation to be initiated on the application icon 232 using subsequent movement of the input (e.g., movement of touch 4008 away from its touch press position). Figure 4A5 and Figure 4A6 In this context, a representation of a second application (e.g., representation 4012) is dragged on the touchscreen based on input movement (e.g., movement of touch 4008). When touch 4008 is on the touchscreen display of the first user interface of the first application (e.g., a map application) and above a portion outside a first predefined portion of the touchscreen (e.g., within a predefined area 4014 within a threshold distance of a predefined side edge (e.g., the right edge and / or the left edge)... Figure 4D3 Also known as the predefined region 4308, in Figure 4E8 Also known as the F zone, such as Figure 4A5As shown, the representation 4012 of the second application dragged by touch 4008 has a first appearance (e.g., the same appearance as the original application icon 232), indicating that if input ends at the current position (e.g., a lift of touch 4008 is detected), the dragging operation will be canceled and the display state before input was detected will be restored. When touch 4008 moves on a portion of the touchscreen within a first predetermined portion (e.g., predetermined area 4014), the electronic device displays as shown... Figure 4A6 The visual feedback shown (e.g., the representation of the second application 4012 being stretched) indicates that if the input ends at the current position within the first predetermined portion of the touchscreen, the window of the second application will be displayed together with the first window of the first application in a corresponding simultaneous display configuration (e.g., a side-swipe display configuration, where the window of the second application covers a portion of the first window of the first application). In some embodiments, other visual feedback is provided, such as a reduction in the size of the display of the first window 4002 of the first application on the touchscreen (e.g., revealing the background below around the reduced first window) and / or a change in the visual clarity of the first window 4002 of the first application (e.g., blurring and / or darkening of window 4002), to indicate that the second application (e.g., an online video application) will open together with the currently open application (e.g., a map application) in a side-swipe display configuration. Figure 4A7 As shown, after input ends when touch 4008 is over the first predefined portion 4014 of the touchscreen, the device opens a window of a second application (e.g., window 4010 of an online video application), which covers a portion of the first window of the first application (e.g., window 4002 of a map application) and at least a portion of the first predefined portion 4014 of the touchscreen. In some embodiments, if the second application has no open window or only a single open window when touch 4008 is detected, window 4002... Figure 4A7 The configuration shown is illustrated. In some implementations, if the second application has multiple open windows, a representation of those multiple windows of the second application is displayed (e.g., in a window selector user interface for the second application), and the user selects one of those windows to be displayed alongside the first application in a side-swipe configuration (e.g., by tapping the desired window representation of the second application in the window selector user interface). For example, relative to... Figures 4D1 to 4D19 More details are provided regarding the behavior related to multiple windows of the second application.
[0208] exist Figures 4A8 to 4A11In the middle, another input made by contact 4016 selects a third application (e.g., touch hold input or light press input to the application icon 220 of a browser application) and according to the movement of that input (e.g., movement of contact 4016 after the initial static portion of the input made by contact 4016), in a manner similar to Figures 4A4 to 4A7 The second application (e.g., an online video application) is shown in a way that the representation of the third application (e.g., representation 4018) is dragged on the touchscreen. Figure 4A10 As shown, when touch 4016 is within a predefined area 4014 near the side edge of the touchscreen, the representation 4018 of the third application is stretched and horizontally expanded to indicate that if input ends at the current position, the window of the third application (e.g., a browser application) will be displayed in a side-pull display configuration along with the first window 4002 of the first application (e.g., a map application). Figure 4A11 In response to detecting the end of input via contact 4016 (e.g., detecting the lifting of contact 4016), the device displays a portion of window 4020 of the browser application's overlay map application window 4002. Figure 4A11 As shown, when the currently displayed side window covers the map application window 4002, the browser application window 4020 completely obscures or replaces window 4010.
[0209] In some implementation schemes, Figures 4A1 to 4A11 The interaction shown results in multiple pull-out windows (e.g., window 4010 and window 4020) being added to a list of zero or more pull-out windows stored in the device’s memory. Figures 4A12 to 4A50 It shows from Figure 4A12 The state shown indicates the various interactions with the list of side-swipe windows that begin, such as an application's side-swipe window being displayed as part of a full-screen window that covers another application (e.g., the same or a different application from the one corresponding to the side-swipe window).
[0210] exist Figure 4A12In this context, multiple inputs (e.g., multiple swipe inputs) are represented on the touchscreen (e.g., via different touches 4021, 4022, 4023, 4024, 4025, 4026, 4027, and 4066), corresponding to inputs at different locations and / or with different directions of movement. In some embodiments, these inputs are individual inputs detected on the screen at different times when windows 4020 and 4002 are displayed in a side-swipe mode. In some embodiments, the device detects a single input based on the position and / or direction of movement of the input, and determines the characteristics of the input based on the position and / or direction of movement of the input (e.g., evaluated according to different criteria for performing different operations (e.g., system-level operations (e.g., navigating between applications, switching between side-swipe windows, switching between display configurations, opening documents across applications, etc.) or application-level operations (e.g., activating user interface elements within the user interface of the displayed application, scrolling the user interface within the displayed application, etc.), performing actions such as relative to... Figures 4A13 to 4A50 The different operations described.
[0211] exist Figures 4A13 to 4A14 In Figure 4A12 Subsequently, input via contact 4024 is detected at a location corresponding to the drag handle area of the side-sliding window 4020 (e.g., near the top edge of window 4020), and this input includes movement of contact 4024 along a first direction (e.g., to the left, substantially horizontal) toward the opposite side edge of the display occupied by the side of window 4020. Figure 4A13 As shown, drag the side-sliding window 4020 on the monitor to cover a portion of window 4002. Figure 4A13 Even if window 4020 is dragged away from its original position on the right side of the display, the previously displayed side-sliding window 4010 will not be revealed or displayed at that position on the right side of the display after window 4020 is moved away by drag input via contact 4024. Figure 4A14 In the process, after the input via contact 4024 ends near the left edge of the display (e.g., within a first threshold distance from the left edge of the display and within a second threshold distance from the top edge of the display) (e.g., at... Figure 4E8 If the lifting of contact 4024 is detected in area H or area B, the device displays a portion of window 4020 covering window 4002 on the left side of the display (e.g., displaying a configuration that has been changed previously (e.g., switching sides, but remaining in side pull mode)).
[0212] exist Figure 4A12 After Figure 4A15In the process, input via contact 4025 is detected at a location corresponding to the drag handle area of the side-sliding window 4020 (e.g., near the top edge of window 4020), and this input includes movement of contact 4025 along a second direction (e.g., to the right, slightly downward) toward the side edge of the display (e.g., the side edge on the side occupied by window 4020) and terminating therein. Figure 4E8 In area E as shown. For example... Figure 4A15 As shown, in response to region E ( Figure 4E8 Upon the end of the input in (), the pull-out window 4020 is converted to a side-by-side window 4028, and the full-screen window 4002 is converted to a side-by-side window 4030. Window 4028 and window 4030 are displayed in a side-by-side display configuration (or split-screen mode). In this case, window 4028 and window 4030 are pinned together and will be displayed together in a split-screen configuration when one of the windows is subsequently recalled to the monitor. Furthermore, the pull-out window 4020 is removed from the list of pull-out windows stored in memory and will not be recalled to the monitor as a pull-out window.
[0213] exist Figure 4A12 After Figures 4A16 to 4A18 In this process, input via contact 4021 is detected at a location within the bottom edge area of the touchscreen, and this input includes movement of contact 4021 toward the top edge of the touchscreen in a third direction (e.g., upward). Based on the determination that the input meets application switcher display criteria (e.g., the speed and / or distance of the input meets predefined speed and / or distance thresholds for navigation to the application switcher user interface), such as... Figures 4A16 to 4A18 As shown, an animation sequence is displayed, illustrating the current display state of the screen (e.g., Figure 4A12 ) to display application switcher user interface 4032 (e.g., also known as multitasking user interface) (e.g., Figure 4A18 The transition is as follows: In this animation sequence, the size of the full-screen window 4002 decreases and moves upwards as the touch 4021 moves. The size of the side-swipe window 4020 decreases and moves away from the representation of window 4002, causing them to appear in... Figure 4A16 The transitional user interface 4032' shown no longer overlaps. Figure 4A17 In the transition user interface 4032', other windows stored in the device's memory (e.g., recently opened windows with stored states in the memory) are exposed, including full-screen windows, split-screen windows, and side-swipe windows currently available on the device, which will be invoked to the display with their stored display states. Figure 4A18An application switcher user interface 4032 is shown, which includes representations of full-screen windows (e.g., representation 4002' of window 4002, representation 4034' of full-screen email window 4034), representations of pairs of windows displayed in split-screen mode (e.g., representation 4036' of windows 4030 and 4028 displayed in split-screen mode, and representation 4038' of browser window and email window displayed in split-screen mode), and representations of pull-out windows (e.g., representation 4020' of window 4020, representation 4010' of window 4010, representation 4040' of email pull-out window, and representation 4042' of photo pull-out window).
[0214] In some implementations, windows with different display configurations are grouped and displayed in different areas of the application switcher user interface 4032, and within each group, these windows are ordered according to the corresponding timestamp of when the window was last displayed. For example, in the area including representations for side-swipe windows, window 4020 is the most recently displayed side-swipe window, and its corresponding representation 4020' is displayed at the far left of the row, with representation 4010' for side-swipe window 4010 displayed next to it. The side-swipe windows represented by representations 4040' and 4042' were displayed earlier than the time when window 4010 was last displayed.
[0215] In some implementations, each representation of an application window in the application switcher user interface 4032 displays an identifier for the application of the window (e.g., application name and application icon) and an identifier for the application window (e.g., a window name automatically generated based on the window's content).
[0216] In some implementations, each representation of a window in the application switcher user interface, when activated (e.g., by tapping an input), causes the device to redisplay that window on the display. If the activated representation corresponds to a full-screen window (e.g., window 4002 or window 4034), that window is invoked to the screen in a full-screen independent display configuration without displaying another application on the screen simultaneously. In some implementations, even if the full-screen window is last displayed simultaneously with another side window at the top, the full-screen window is displayed when invoked to the screen from the application switcher user interface 4032, without displaying the side window at the top. In some implementations, when a representation of a side window (e.g., window 4010, window 4020, window 4040, or window 4042) is activated in the application switcher user interface 4032, that side window and another full-screen or split-screen window below it (e.g., window 4002, window 4034, or a pair of windows in a split-screen configuration) are invoked to the display. In some implementations, the window below the side-swipe window is either a full-screen window or the pair of split-screen windows that are displayed immediately before the application switcher user interface 4032 is displayed. In some implementations, the window below the side-swipe window is the last window displayed simultaneously with the side-swipe window. In some implementations, when a representation of a pair of split-screen windows (e.g., representation 4036' or representation 4038') is activated in the application switcher user interface 4032, the pair of split-screen windows are invoked together on the display in split-screen mode.
[0217] exist Figure 4A12 After Figures 4A19 to 4A21 In the display, input via contact 4022 is detected at a location within the bottom edge region of the side-slide window 4020, and this input includes movement of contact 4022 along a fourth direction (e.g., substantially horizontally) toward the edge on one side of the display side-slide window 4020 (e.g., the right edge of the screen). In response to the input via contact 4022, the side-slide window 4020 is dragged toward the right edge of the screen, and removed from the screen after the input ends. During the movement of contact 4022 and window 4020, other windows in the side-slide window stack stored in the device's memory are displayed on the screen. For example, such as... Figure 4A19 and Figure 4A20 As shown, representations of windows 4010, 4040, and 4042 are exposed below window 4020. The order of windows 4020, 4010, 4040, and 4042 corresponds to the order in which these windows were last displayed on the screen. In some embodiments, windows 4020, 4010, 4040, and 4042 are displayed at different depths in a direction perpendicular to the surface plane of the screen (e.g., with decreasing size and reduced sharpness as the distance from the surface plane of the screen increases). This is consistent with... Figures 4A13 to 4A14The situation shown is different, where when window 4020 is dragged toward the other side of the screen, there is no indication that other windows are located below window 4020 (e.g., no other windows are displayed below window 4020). Figure 4A19 and Figure 4A20 In the middle, when window 4020 is dragged toward the right edge of the screen by inputting to point to the bottom edge of window 4020, the next window in the side-sliding window stack (e.g., window 4010) gradually appears and eventually becomes the top window shown as window 4002 covering the entire screen (e.g., window 4002). Figure 4A21 (As shown).
[0218] exist Figure 4A21 After Figures 4A22 to 4A25 In the display, input via contact 4046 is detected at a location within the bottom edge region of the side-slide window 4010, and this input includes movement of contact 4046 along a fourth direction (e.g., substantially horizontally) toward the edge on one side of the display side-slide window 4010 (e.g., the right edge of the screen). In response to the input via contact 4046, the side-slide window 4010 is dragged toward the right edge of the screen, and removed from the screen after the input ends. During the movement of contact 4046 and window 4010, other windows in the side-slide window stack stored in the device's memory are displayed on the screen. For example, such as... Figure 4A23 As shown, representations of windows 4040, 4042, and 4020 are displayed below window 4010. Generally, if an input via contact 4046 is detected within a threshold time interval following an input via contact 4022, the original top window 4020 is randomized to the bottom of the stack (e.g., as shown). Figure 4A23 (As shown), although window 4020 is the most recently displayed window besides window 4010. If input via touch 4046 is detected after window 4010 has been displayed as the top window for more than a threshold amount of time, the stack is sorted based on the last displayed order of the windows, and window 4020 will be inserted into it. Figure 4A23 Between windows 4010 and 4040 in the stack shown. In some embodiments, when the entire stack of side-slide windows is removed from the display, the entire stack of side-slide windows is reordered based solely on the last time these windows were displayed (e.g., as shown). Figures 4A28 to 4A29 (As shown). In Figure 4A24 In the middle, after the input by contact 4046 ends, window 4040 is displayed as a side-pull window that covers the full-screen window 4002.
[0219] exist Figure 4A24 After Figures 4A25 to 4A27In the process, input via contact 4048 is detected at a location within the bottom edge region of the side-slide window 4040, and this input includes movement of contact 4048 along a fifth direction (e.g., substantially horizontally) away from the screen on one side of the display side-slide window 4040 (e.g., the right edge of the screen). In response to the input via contact 4048, the side-slide window 4010, just removed from the display, is dragged back onto the screen, covering window 4040. During the movement of contact 4048 and window 4010, other windows in the side-slide window stack stored in the device's memory are displayed on the screen. For example, as... Figure 4A26 As shown, representations of windows 4040, 4042, and 4020 are shown below window 4010. In some embodiments, windows in a side-slide window stack are arranged on a circular dial, with bottom and top tabs arranged adjacent to each other. A swipe in one direction scrolls through the window around the circular dial in that direction, and a swipe in the opposite direction scrolls through the window in the opposite direction. After the end of input via touch 4048 is detected, window 4010 is displayed as a side-slide window covering the full-screen window 4002, as... Figure 4A27 As shown. This is also the opposite of the case where the top side window is dragged to the other side of the screen using input pointing to the top edge area of the top side window, where no other windows are displayed below the top side window, and no other windows are added above the top side window during the movement of the input. In this case, when the top side window is flicked (e.g., window 4040) or dragged to the left away from the right edge of the screen, another window (e.g., window 4010) is displayed above the dragged window (e.g., window 4040), and at least one window (e.g., windows 4042 and 4020) is displayed below the dragged window (e.g., window 4010).
[0220] exist Figure 4A12 After Figures 4A28 to 4A29 In this embodiment, input via contact 4027 is detected near the left edge of the side-slide window 4020, and this input includes movement of contact 4027 along a sixth direction (e.g., substantially horizontally) toward the edge on one side of the display side-slide window 4020 (e.g., the right edge of the screen). In some embodiments, the device requires input to be detected at or within a threshold distance of the left edge of the window 4020 to trigger an operation to remove the side-slide window overlay from the display. In some embodiments, such as Figure 4A28As shown, during the movement of contact 4027 toward the right edge of the display, window 4020 is gradually dragged away from the display, and visual indications of other windows in the side-slide window stack are shown following the movement of window 4020. After the end of input via contact 4027 is detected, window 4020 is removed from the display, and no other side-slide windows are simultaneously displayed on the display along with background window 4002. Window 4002 is displayed as a full-screen window in an independent display configuration, instead of as a full-screen background window of the side-slide window in a side-slide display configuration. This is consistent with the following in Figure 5A12. Figure 4A50 The opposite is true, where an input via contact 4026, involving movement in a sixth direction (e.g., substantially horizontally toward the right edge of the display), is detected outside the side-pull window 4020, causing the user interface within window 4002 to shift to the right according to the movement of contact 4026, without causing any movement of the side-pull window 4020. This is also consistent with a rightward swipe input via contact 4022 (in... Figure 4A12 After Figures 4A19 to 4A21 (In the middle) the window 4020 slides off the display and the window below 4010 becomes a side window that covers the background window 4002 after the input ends, which is the opposite of the case where the input ends.
[0221] exist Figure 4A29 After Figures 4A30 to 4A32 In this process, input via contact 4052 is detected on the side edge of the display (e.g., on one side of the previously displayed side-sliding window (e.g., window 4020) on the screen), and this input includes movement of contact 4052 away from the side edge on the display along a seventh direction (e.g., substantially horizontally). In response to the detection of input via contact 4052, the last displayed side-sliding window (e.g., window 4020) is dragged back onto the display, covering the currently displayed full-screen window (e.g., window 5004), as... Figure 4A32 As shown. In some implementations, if a window on the display has been switched to another full-screen window in a standalone display configuration (e.g., a full-screen window displayed in response to actions such as tapping an application icon in the taskbar, selecting from a list of open windows of an application after tapping the application icon, or an application switching gesture (e.g., a horizontal swipe along the bottom edge of the currently displayed standalone window)), then if input via touch is detected on the side edge of the display (including horizontal movement of the touch from the side edge onto the screen), the last displayed side-pull window (e.g., window 4020) is dragged back onto the display, covering the currently displayed full-screen window (e.g., a full-screen window other than window 4002). Figure 4A31 In the middle, when window 4020 is dragged back onto the display as contact 4052 moves to the left, the representation of other windows in the side-sliding window stack is shown below window 4020.
[0222] In some implementation schemes, with Figure 4A12 After Figures 4A30 to 4A32 The scenario shown is the opposite: input via touch is detected in a region at a threshold distance away from the side edge of the display (e.g., the side edge on one side of the screen where the previously displayed pull-out window (e.g., window 4020) was shown), and this input includes movement of touch 4052 away from that side edge on the display along a seventh direction (e.g., substantially horizontally). In response to the detection of input via touch, the last displayed pull-out window (e.g., window 4020) will not be dragged back onto the display. Instead, the input causes the execution of an operation in an application (e.g., a map application) corresponding to the input, such as moving a searchable map user interface displayed in window 4002 relative to the display based on the movement of the touch.
[0223] exist Figures 4A33 to 4A34 In Figure 4A12 Subsequently, input via touch 4023 is detected at the bottom edge of the side-sliding window (e.g., window 4020), and this input includes movement of touch 4023 on the display in an eighth direction (e.g., upward). In response to the detection of input via touch 4023 and upon determining that the movement of touch 4023 meets a preset criterion (e.g., exceeding a threshold amount of movement in the eighth direction, or exceeding a threshold speed in the eighth direction), the device displays a transitional user interface 4053, which includes a representation of the side-sliding window 4020 (e.g., representation 4020') that moves according to the movement of touch 4023. In some embodiments, a background window (e.g., window 4002) is visually obscured (e.g., blurred and darkened) below the representation of the side-sliding window in the transitional user interface 4053. In some embodiments, when the representation of the top side-slide window is dragged around the display according to the movement of contact 4023, the representations of other side-slide windows in the side-slide window stack (e.g., representations 4010', 4040', and 4042') are shown below the representation of the top side-slide window (e.g., representation 4020'). In some embodiments, the representation of the side-slide window is dynamically updated (e.g., resized) according to the current position of the representation (and contact 4023) on the display. Figure 4A34 In this process, the removal of contact 4023 has been detected, and the device displays either the side-slide window switcher user interface or the overlay switcher user interface 4054 only for the side-slide windows currently stored in the memory within the side-slide window stack. In some embodiments, the representation of the side-slide windows in the side-slide window stack is displayed and can be individually selected within the overlay switcher user interface 4054. The overlay switcher user interface 4054 behaves similarly to the application switcher user interface (e.g., ...). Figure 4A18In the application switcher user interface 4032), the side window is displayed because tapping the icon covering the side window in the switcher user interface 4054 causes the side window to be displayed. Figure 4A34 As shown, in some embodiments, the representations of the side windows in the stored side window overlay are unfolded against the background and do not overlap with each other. In some embodiments, the representation of a side window is a scaled-down image of that side window. In some embodiments, due to limitations in display size and the total number of side windows in the overlay, some representations of the side windows are not displayed. For example, in Figure 4A34 In the overlay, there are a total of four side-slide windows, and the representation of one of those windows (e.g., representation 4042') is initially only partially visible in the overlay switcher user interface 4054. If additional side-slide windows exist in the overlay, the representations of those additional side-slide windows will not initially be visible in the overlay switcher user interface 4054. In some embodiments, instead of displaying the representations of the side-slide windows in a fully expanded configuration in the overlay switcher user interface, these representations are displayed in the overlay, where the lower-layer representation is offset from the representation of the top side-slide window by a different amount.
[0224] Figure 4A35 The overlay switcher user interface 4054 is displayed, including a representation of the side window currently in the side window stack. The representation on the touchscreen (e.g., via different touches 4056, 4058, 4059, 4060, 4062, and 4064) corresponds to multiple inputs (e.g., tap inputs and swipe inputs) at different locations, input types, and / or different directions of movement. In some embodiments, these inputs are individual inputs detected at different times on the screen while the overlay switcher user interface 4054 is displayed. In some embodiments, the device detects a single input based on the input's location, input type, and / or direction of movement, and determines the characteristics of the input based on the input's location, input type, and / or direction of movement (e.g., evaluation based on different criteria used to perform different operations (e.g., different system-level operations, such as navigating or browsing within the overlay switcher user interface, exiting the overlay switcher user interface to display a previously displayed or selected window, closing a window in the side window stack, etc.), performing actions such as relative to... Figures 4A36 to 4A42 The different operations described.
[0225] exist Figure 4A35 After Figures 4A36 to 4A37In this embodiment, input via contact 4056 is detected on one of the displayed representations (e.g., representation 4010'), and this input includes movement of contact 4056 on the display in a ninth direction (e.g., horizontally (e.g., to the right)). In response to the detection of input via contact 4056 and based on the determination that the input meets preset criteria (e.g., the position of contact 4056 is on a representation in the side-slide window, and the direction of movement of contact 4056 is horizontal), the device scrolls the overlay switcher user interface 4054 to reveal representations in the side-slide window that are not currently displayed or are not fully displayed in the overlay switcher user interface. In some embodiments, a representation displayed near one side of the display (e.g., representation 4020') gradually moves off the display, and a representation on the other side of the display gradually enters the display according to the movement of contact 4056, such as... Figure 4A35 and Figure 4A36 As shown. In Figure 4A37 In some embodiments, the representation that moves away from the display is added to the end of the overlay (e.g., the end of which is connected to the beginning of the overlay, similar to a circular turntable), and is re-displayed on the other side of the display as the contact 4056 continues to move in the same direction. In some embodiments, the device does not require the contact 4056 to be detected on the representation of the side pull-out window in the overlay switcher user interface 4054; scrolling of the overlay switcher user interface 4054 is performed as long as the input includes movement in the horizontal direction exceeding a threshold amount. In some embodiments, the scrolling direction is determined based on the direction of movement of the contact on the display.
[0226] exist Figure 4A35 After Figures 4A38 to 4A39 In the process, input via touch 4058 is detected on one of the displayed representations (e.g., representation 4010'), and this input includes movement of touch 4058 on the display along a tenth direction (e.g., vertically (e.g., upward)). In response to the detection of input via touch 4058, a representation is removed from the overlay switcher user interface 4054, and the side window represented by the removed representation is removed from the side window stack stored in memory. In other words, the side window corresponding to the removed representation is "closed". Figure 4A39 In this context, representations of other windows that are not closed (e.g., representations 4042', 4040', and 4020') remain displayed in the overlay switcher user interface 4054.
[0227] exist Figure 4A35 After Figure 4A40In the process, a tap input via contact 4059 is detected on representation 4010' of window 4010; and in response to the detection of the tap input via contact 4059, the device stops displaying the overlay switcher user interface and displays the full-screen background window in a side-slide mode along with the side-slide window 4010. In some embodiments, the full-screen background window is the last full-screen window displayed (e.g., window 4002), regardless of whether the full-screen window is ultimately displayed along with the selected side-slide window. In some embodiments, the full-screen background window is the last full-screen window displayed along with the selected side-slide window (e.g., window 4002).
[0228] exist Figure 4A35 After Figure 4A41 In the process, a tap input via touch 4060 is detected on representation 4040' of window 4040; and in response to the detection of the tap input via touch 4060, the device stops displaying the overlay switcher user interface 4054 and displays the full-screen background window in a side-slide mode along with the side-slide window 4040. In some embodiments, the full-screen background window is the last full-screen window displayed (e.g., window 4002), regardless of whether the full-screen window is ultimately displayed along with the selected side-slide window. In some embodiments, the full-screen background window is the last full-screen window displayed along with the selected side-slide window (e.g., window 4002 or another window different from window 4002).
[0229] exist Figure 4A35 After Figure 4A42 In the process, a tap input via contact 4062 is detected on representation 4020' of window 4020; and in response to the detection of the tap input via contact 4062, the device stops displaying the overlay switcher user interface 4054 and displays the full-screen background window in a side-slide mode along with the side-slide window 4020. In some embodiments, the full-screen background window is the last full-screen window displayed (e.g., window 4002), regardless of whether the full-screen window is ultimately displayed along with the selected side-slide window. In some embodiments, the full-screen background window is the last full-screen window displayed along with the selected side-slide window (e.g., window 4002).
[0230] also, Figure 4A42The displayed state is also displayed in response to a tap input detected via touch 4064 on a portion of the overlay switcher user interface 4054 that is not occupied by any representation of the side-slide window. In some embodiments, the overlay switcher user interface 4054 includes a power off indicator, and a tap input detected on the power off indicator also causes the device to stop displaying the overlay switcher user interface 4054 and redisplay the last displayed user interface state (e.g., window 4020 of the overlay window 4002 in side-slide mode).
[0231] Figure 4A42 After Figures 4A43 to 4A46 The diagram illustrates the detection of a swipe input via contact 4066 within the bottom edge region of the display, where the movement of contact 4066 is substantially horizontal (e.g., excluding or including minimal vertical movement compared to horizontal movement). In response to the edge swipe input, and based on determining that the edge swipe input meets application switching criteria (e.g., distance and speed criteria for meeting application switching criteria), window 4002 is dragged off the screen and replaced by window 4034, which is the last full-screen window displayed before window 4002. Figures 4A43 to 4A45 As shown, when the background full-screen window changes, the side-swipe window 4020 is unaffected by input made via touch 4066. After the input made via touch 4066 ends, the side-swipe window 4020 covers the window 4034 in side-swipe mode, as... Figure 4A46 As shown. In some implementations, Figures 4A43 to 4A46 The process shown can also be found from Figure 4A12 The user interface shown is now accessible. In some implementations, Figure 4A12 The user interface shown does not include the taskbar (e.g., after the taskbar is removed by swiping down on it). In some embodiments, window 4034 is a full-screen window of another application (e.g., an email application) that is different from the full-screen window that was initially displayed below the side-swipe window 4020. In some embodiments, window 4034 is a full-screen window of the same application as the full-screen window that was initially displayed below the side-swipe window 4020.
[0232] exist Figure 4A46In the context of window 4034, another input via touch 4068 is detected on a document (e.g., an email message in an email message list in an email application) represented by touch 4034. The initial portion of the input via touch 4068 has met the criteria for initiating a drag operation on the document (e.g., the input is a tap-hold input that remains substantially still for at least a threshold amount of time after a touch is pressed on the document, or the input is a press input with a contact intensity exceeding a threshold intensity greater than a nominal touch detection intensity threshold), and the document has been selected (e.g., as indicated by the document's visual highlighting).
[0233] exist Figure 4A47 In the middle, the document is dragged on the monitor according to the movement of contact 4068, represented by 4070. Figure 4A48 When contact 4068 is within a predefined area of the display (e.g., a predefined area 4014 for opening a side-pull window by placing an application icon on it, or a scaled-down version of the predefined area 4014), the document's representation is converted (e.g., converted to representation 4044') to a state that displays a preview of the new side-pull window, which displays the document in the document's local application.
[0234] exist Figure 4A49 In the middle, after the input ends (e.g., the lifting of contact 4068 is detected within the predefined area 4014 or a smaller version of the predefined area 4014), the document is opened in the side window of the local application of the document (e.g., the side window of an email application), and the side window 4044 displaying the document becomes the top side window that covers the background full-screen window 4034.
[0235] In some implementations, if the input ends at a different location on the display, other operations can be performed. For example, in some implementations, if the input ends in an area of the display corresponding to opening a new window in split-screen view mode, the document will be opened in a new window displayed side-by-side with a resized version (e.g., a narrower version) of the email application window 4034. In some implementations, if the input ends in an area of the display above a side window but outside a predefined area for opening a new window for the document, and the side window presents an acceptable placement for the document, the document will be inserted into that placement position in the side window (e.g., inserted into another document, message, or storage location shown in the side window). In some implementations, if the input ends outside the side window, the document will be placed in an acceptable placement position (if any) in window 4034 corresponding to the end position of the input, or will return to its original position if no acceptable placement position is available.
[0236] exist Figure 4A12 After Figure 4A50 In the process, input via touch 4026 is detected outside the side-slide window 4020, including movement of touch 4026 in a corresponding direction. In response to the movement of the touch, the device performs an operation within the application corresponding to the background full-screen window 4002, for example, shifting a map based on the movement of touch 4026. Because the initial position of touch 4026 is outside the side-slide window 4020, the application-level operation is initiated and continues even when the touch later moves above the area displaying the side-slide window 4020.
[0237] Figures 4B1 to 4B51 The diagram illustrates user interface behavior in response to a user's request to switch applications by selecting an application icon, according to some embodiments. The request to switch applications is integrated with a request to view a window switcher user interface for the applications using the same gesture. The device automatically determines whether to switch applications or display the window switcher user interface for the currently displayed application based on whether the currently displayed application currently has more than one window. According to some embodiments, user interaction with a window switcher user interface that simultaneously displays multiple windows corresponding to the respective applications is also described. The user interface in these figures is used to illustrate the processes described below, including... Figures 6A to 6E The process is described below. For ease of explanation, some embodiments of the implementation will be discussed with reference to the operation performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representation point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operation is optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to a contact detected on the touch-sensitive surface 451 when the user interface shown in the figures, along with the focus selector, is displayed on the display 450.
[0238] Figures 4B1 to 4B4 This illustrates the interaction where a user selects an application icon to open the corresponding application when the application is currently displayed.
[0239] like Figure 4B1As shown, a full-screen window 4102 of an email application is displayed on touchscreen 112. In this example, the full-screen window 4102 is displayed in a full-screen independent display configuration, and no other windows are displayed simultaneously on the screen. In some embodiments, the device has the same response as described below, regardless of whether the full-screen window 4102 is displayed in an independent configuration or as a background window for a pull-out window in pull-out mode (e.g., a pull-out window of the same or a different application). Figure 4B1 In this context, input via touch 4104 is detected at the location on the screen corresponding to the first application icon (e.g., application icon 218 of an email application) in the taskbar 4006 covering the full-screen window 4102. In response to detecting this input, and based on determining that the input meets selection criteria (e.g., the input meets criteria for detecting tap input on an application icon (e.g., position and time criteria)), the device determines whether the selected icon corresponds to the application of the currently displayed window. In this case, both the currently displayed window (e.g., window 4102) and the selected application icon (e.g., application icon 218) correspond to the email application. In response to determining that both the currently displayed window (e.g., window 4102) and the selected application icon (e.g., application icon 218) correspond to the email application, the device determines whether the application is associated with multiple windows (e.g., multiple open windows are saved in memory as "open" windows, which can be recalled to the screen using the saved last displayed state). In this case, based on determining that the email application currently has more than one open window, the device displays a window switcher user interface 4018. Figure 4B4 This window switcher user interface simultaneously displays representations of multiple open windows associated with the email application. This is in contrast to the case where the email application's application icon is activated by input that meets the selection criteria, if the email application is not the currently displayed application (e.g., when another application is currently displayed or when the system user interface is currently displayed, such as the home screen user interface).
[0240] exist Figures 4B2 to 4B3 In response to determining that the input made via touch 4104 has met the selection criteria and that the currently displayed window and the selected application icon correspond to the same application, and that the application is associated with multiple windows, an animation transition is displayed. The animation transition shows the currently displayed full-screen window 4102 shrinking and becoming a representation of window 4102 (e.g., a scaled-down image) 4102', and representations of other windows (e.g., ... Figure 4B4The side-swipe email window 4106 (represented by 4016') and the email window and photo window (represented by 4110') shown in split-screen mode appear on the screen overlaying the background of the window switcher user interface 4108. Figure 4B4 In this context, after the input via touch 4104 ends and the animation transition is complete, a window switcher user interface 4108 is displayed, replacing the full-screen window 4102 of the email application on the screen. In this case, the window switcher user interface 4108 is displayed in a state with representations of all saved windows associated with the email application, overlaid on a background (e.g., a blurred or darkened image of the full-screen window 4102). These displays include representations of all full-screen windows (e.g., representation 4102' of the full-screen window 4102), representations of all side windows (e.g., representation 4106' of the side window 4106), and representations of all windows displayed in split-screen mode (e.g., representation 4102' of the email window displayed in split-screen mode with a photo window). When each representation in the window switcher user interface 4108 is activated by an input that meets the selection criteria (e.g., a tap input), the device stops displaying the window switcher user interface and displays the window corresponding to the selected representation, completing the task to return to the previously displayed window (e.g., if the representation of the originally displayed window was selected) or switch to a different window of the same application (e.g., if the representation of a window other than the originally displayed window was selected).
[0241] Similarly, Figure 4B4 As shown, a shutdown enable indicator 4114 is provided in the window switcher user interface 4108. When activated by input that meets the selection criteria (e.g., a tap), the shutdown enable indicator causes the device to stop displaying the window switcher user interface 4108 and redisplay the full-screen window 4102. A new window enable indicator 4112 is also provided in the window switcher user interface 4108. When activated by input that meets the selection criteria (e.g., a tap), the new window enable indicator 4112 causes the device to stop displaying the window switcher user interface 4108 and display a new window for the email application (e.g., the default window (e.g., email inbox user interface, draft email user interface, new message user interface, etc.)).
[0242] exist Figure 4B4 In the process, input via touch 4118 is detected on representation 4102' of the originally displayed full-screen window 4102. In response to detecting input via touch 4118 and determining that the input meets a first criterion (e.g., the input is a tap), the device stops displaying the window switcher user interface 4108 and displays the full-screen window 4102, as shown. Figure 4B5 As shown.
[0243] Figure 4B5 The illustration shows that while a full-screen window 4102 of an email application is displayed, input via touch 4120 is detected on the application icon 224 of a messaging application. Based on determining that the input via touch 4120 meets a first criterion (e.g., the input is a tap), the device determines whether the application icon 224 and the currently displayed window 4102 correspond to the same application. Based on determining that the application icon 224 and the currently displayed window 4102 do not correspond to the same application, the device stops displaying the full-screen window 4102 and displays the full-screen window 4122 corresponding to the messaging application (e.g., the default window of the messaging application (e.g., the last full-screen window displayed by the messaging application)). Figure 4B6 As shown. In Figures 4B5 to 4B6 In the example shown, the user's request to switch applications is satisfied, regardless of whether the messaging application or the email application is currently associated with multiple windows, because the user selects an application icon that is different from the currently displayed application.
[0244] Figures 4B7 to 4B8 It shows the relationship with Figures 4B1 to 4B4 The opposite of what is shown. In Figures 4B7 to 4B8 In the example shown, a full-screen window 4122 of the messaging application is displayed on the touchscreen 112. In some embodiments, the device has the same response as described below, regardless of whether the full-screen window 4102 is displayed in an independent configuration or as a background window for a pull-out window in pull-out mode (e.g., a pull-out window of the same or a different application). Figure 4B7In the context of the message application, input via touch 4124 is detected at the location of the application icon 224 of the message application in the taskbar 4006 covering the full-screen window 4122. In response to detecting this input, and based on determining that the input meets selection criteria, the device determines whether the selected application icon corresponds to the application of the currently displayed window. In this case, both the currently displayed window (e.g., window 4122) and the selected application icon (e.g., application icon 224) correspond to the message application. In response to determining that both the currently displayed window (e.g., window 4122) and the active application icon (e.g., application icon 224) correspond to the message application, the device determines whether the application is associated with multiple windows (e.g., multiple open windows are stored in memory as "open" windows, which can be recalled to the screen using the saved last displayed state). In this scenario, based on the determination that the messaging application does not currently have more than one open window, the device provides one or more outputs (e.g., outputs corresponding to visual feedback, audio feedback, and / or haptic feedback) to indicate that an application switching operation or window switcher display operation will not be initiated in response to input made via touch 4124. Figure 4B8 In the middle, the application icon 224 shakes in response to input via touch 4124, optionally in conjunction with an audio or haptic alarm, to indicate that the currently displayed window and the selected application icon correspond to the same application and that the application is not associated with multiple windows, and to indicate that no application switching or window switcher display operation will be performed.
[0245] Figure 4B8 After Figures 4B9 to 4B13 The diagram illustrates a process for opening an additional window within a messaging application, such that more than one window remains associated with the messaging application at the end of the process. Other methods exist for opening additional windows within a messaging application, and... Figures 4B9 to 4B13 The process shown is just one of many ways to open an additional window in an application.
[0246] like Figure 4B9 As shown, input via touch 4128 is detected at the location of representation 4130 on the full-screen window 4122 corresponding to the session with Greg Kane. In response to the detection of input via touch 4128, and based on determining that the initial portion of the input meets object movement criteria (e.g., time or intensity criteria for detecting tap-hold input or press input to initiate a drag operation on an object (e.g., document, user interface object, content item, etc.), the device displays representation 4130 in a highlighted state. Figures 4B10 to 4B11In this context, based on the movement of contact 4128 detected after the initial portion of the input via contact 4128 meets the object movement criterion, another representation 4132 (e.g., a copy of representation 4130) of the session with Greg Kane is dragged on the display. Figure 4B12 In the middle, when 4132 is dragged to a predefined area 4308 near the right edge of the display (e.g., also in...), Figure 4C28 As shown, predefined region 4308 is Figure 4A6 When content in a side-swipe window of the messaging application is opened in the predefined regions 4014 and 4E8 (reduced width versions of the F area), the device provides visual feedback (e.g., the size of the full-screen window 4122 is reduced and converted to a scaled-down representation 4122' of window 4122, thereby revealing the background below the scaled-down representation 4122', and representation 4132 is simultaneously stretched and horizontally expanded) to indicate that if input ends at this time, the side-swipe window of the messaging application will be displayed as the full-screen window 4122 covering the right side of the screen. In some embodiments, the visual feedback also includes visually blurring the resized full-screen window and displaying the application icon corresponding to the full-screen window on the visually blurred window. In some embodiments, the application icon of the messaging application is shown on representation 4132. Figure 4B13 In this process, a side-swipe window 4136 of the messaging application is opened and displayed on the right side of the display, covering a portion of the full-screen window 4122 of the messaging application. Within the side-swipe window 4136, the conversation with Greg Kane is displayed. In other words, the content object (e.g., the conversation with Greg Kane) that was dragged to the right side of the screen (e.g., dragged into a predefined area 4308 to open content in the side-swipe window) of the application corresponding to the content object (e.g., the side-swipe messaging window) is opened. After the input via touch 4128 is completed, there is now more than one window associated with the messaging application, including the full-screen window 4122 and the side-swipe window 4136.
[0247] exist Figure 4B13 After Figures 4B14 to 4B17 In the process, another input via touch 4138 is detected on the application icon 228 in the taskbar 4006, and this other input causes a side-swipe window to open in the Photos application. There are many ways to open a new window in an application. Figures 4B14 to 4B17 The process shown is just one of many ways to open a new window. In this example, the new window is the first window opened in the Photos application. Figure 4B14As shown, when the full-screen window 4122 and the side-swipe window 4136 of the messaging application are displayed in side-swipe mode, input via touch 4138 is detected at the location on the display corresponding to the application icon 228 of the photo application. Figures 4B15 to 4B16 In the process, after the initial portion of the input satisfies the object movement criteria for initiating a drag operation on the application icon, the representation 4140 of the photo application is dragged on the display based on the movement of contact 4138 detected after the initial portion of the input satisfies the object movement criteria. Figure 4B16 In the middle, when contact 4138 drags the representation 4140 of the photo application into a predefined area 4014 for opening a side-pull window on the right side of the display (e.g., the area 4014 for opening the side-pull application window by placing the application icon is wider than...), Figure 4B12 The area 4308 used to open the contents of a new side-drag window indicates that 4140 has been stretched and expanded horizontally to indicate that the placement area for opening the side-drag window of the dragged application has been reached. Figure 4B17 In the process, after the input ends in the predefined area 4014 (e.g., after lifting off contact 4138 in the predefined area 4014), the side pull window 4142 of the photo application is displayed as a top side pull window that covers the full-screen window 4122.
[0248] Figures 4B18 to 4B19 Shows something like Figures 4B1 to 4B4 The situation shown, and with Figures 4B5 to 4B6 as well as Figures 4B7 to 4B8 The situations shown are the opposite.
[0249] exist Figures 4B18 to 4B19 In the example shown, the full-screen window 4122 of the messaging application is displayed together with the side-swipe window 4142 of the photos application on the touchscreen 112. In some embodiments, the device has the same response as described below, regardless of whether the full-screen window 4122 is displayed in a standalone display configuration or as a background window for side-swipe windows in side-swipe mode (e.g., side-swipe windows of the same or different applications). Figure 4B18In this context, input via touch 4144 is detected at the location of the application icon 224 of the messaging application in the taskbar 4006 covering the full-screen window 4122. In response to detecting this input, and based on determining that the input meets a first criterion, the device determines whether the selected icon corresponds to the application of the currently displayed window. In this case, both the currently displayed window (e.g., window 4122) and the selected application icon (e.g., application icon 224) correspond to the messaging application. In response to determining that both the currently displayed window (e.g., window 4122) and the active application icon (e.g., application icon 224) correspond to the messaging application, the device determines whether the messaging application is associated with multiple windows (e.g., multiple open windows are stored in memory as "open" windows, which can be recalled to the screen using the saved last displayed state). In this case, based on determining that the messaging application indeed has more than one open window at this time (e.g., because window 4136 is also open, such as...), the device determines that the messaging application does indeed have more than one open window at this time (e.g., because window 4136 is also open, such as...). Figure 4B15 As shown), the device displays a window switcher user interface 4018 that simultaneously displays representations of multiple open windows associated with the messaging application. This is different from the situation where the messaging application's application icon is activated by input that meets selection criteria, but the messaging application is not the currently displayed application (e.g., as shown). Figures 4B5 to 4B6 The opposite of the case shown is that the application switching operation is performed immediately in response to the input. This is also the opposite of the case where the application corresponding to the active application icon is the currently displayed application but has only a single open window, and neither application switching nor the display of a window switcher occurs (e.g., as shown). Figures 4B7 to 4B8 (As shown).
[0250] like Figure 4B19As shown, after the input via touch 4144 ends, the window switcher user interface 4108 is displayed, replacing the full-screen window 4122 of the messaging application and the side-swipe window 4142 of the photos application. In this case, the window switcher user interface 4108 is displayed with representations of all saved windows associated with the messaging application, overlaid on a background (e.g., a blurred or darkened image of the full-screen window 4122). These representations include representations of all full-screen windows (e.g., representation 4122' of full-screen window 4122), representations of all side-swipe windows (e.g., representation 4136' of side-swipe window 4136), and representations of all windows displayed in split-screen mode (e.g., none at this time). Each representation in the window switcher user interface 4108, when activated by input that meets the selection criteria (e.g., tap input), causes the device to stop displaying the window switcher user interface and display the window corresponding to the selected representation, completing the task to return to the previously displayed window (or optionally, the simultaneously displayed window) or switch to a different window of the same application (e.g., window 4136'). In the window switcher user interface 4108, the same new window enable indicator 4112 and close enable indicator 4114 are displayed. When activated, the new window enable indicator 4112 causes the device to open a new window for the messaging application. When activated, the close enable indicator 4114 causes the device to stop displaying the window switcher user interface 4108 and redisplay the full-screen window 4122 and the side-swipe window 4142. In some embodiments, each application has its own copy of the window switcher user interface, with customizations (e.g., user interface objects, functionality, and appearance) configured within the application. In some embodiments, the window switcher user interface is a system user interface displayed in different states corresponding to the respective applications that invoke the window switcher user interface.
[0251] Figures 4B20 to 4B21 Interaction with the new window display 4112 in the window switcher user interface 4108 is illustrated. Figure 4B20 In this context, input via touch 4146 is detected at a location corresponding to the new window indication 4112. In response to this input, and based on determining that the input meets selection criteria (e.g., the input is a tap), the device displays a new window for the messaging application. In this embodiment, the new window is the default window for the messaging application (e.g., window 4148 displaying a new message template for composing a new message with a new recipient and a list of existing conversations).
[0252] Figures 4B22 to 4B23 This demonstrates navigation to another user interface within full-screen window 4148 without opening a new window. Figure 4B22In the context of the conversation with Mary Ford, input via touch 4152 is detected at the location corresponding to the representation 4150. In response to this input, and based on determining that the input meets a first criterion (e.g., the input is a tap), the user interface in window 4148 is transformed, and the new message template in the window is replaced with the conversation with Mary Ford, such as... Figure 4B23 As shown. To improve clarity, window 4148 is relabeled as window 4154 to indicate that the window's content has changed, but no new window is opened in the messaging application. Alternatively, navigation actions within the messaging application cause window 4148 to close and window 4154 to open within the messaging application.
[0253] Figures 4B24 to 4B27 The diagram illustrates a process, according to some embodiments, for opening a window in a Photos application in split-screen mode while simultaneously converting a full-screen window in a Messages application into a split-screen window. In this process, the window in the Photos application is a newly opened window, while the window in the Messages application is not a newly opened window but rather an existing window that has been resized.
[0254] like Figure 4B24 As shown, when the full-screen window 4154 of the messaging application is displayed, input via touch 4156 is detected at the location corresponding to the application icon 228 of the photos application. In response to detecting this input and determining that an initial portion of the input meets a second criterion (e.g., a criterion for initiating a drag operation on an object at the location of the input), the device highlights the application icon 228 to indicate that the criterion for initiating the drag operation has been met. Figure 4B25 In this context, the photo application's representation 4158 is dragged based on the movement of contact 4156 detected after the initial input portion has met the second criterion. Figure 4B26 In the middle, the representation 4158 of the photo application is dragged to a predefined area 4162 near the left edge of the display (e.g., in...). Figure 4E8This area (also referred to as Area A) is used to open windows in split-screen mode. In some embodiments, the predefined area 4162 for opening windows in split-screen mode is closer to the left edge of the display than the predefined area 4014 (for opening windows in side-pull mode) which is closer to the right edge of the display. In response to determining that contact 4156 is within the predefined area 4162 for opening the application window in split-screen mode, the device provides visual feedback to indicate that if the input ends at this time, the window of the dragged application will be opened in split-screen mode. In some embodiments, the visual feedback includes, for example, resizing the full-screen window 4154 in the horizontal direction to reveal the background on one side of the display in which the new window will be displayed. In some embodiments, when the full-screen window 4154 is resized, the content of the full-screen window is visually obscured (e.g., blurred or darkened), with the application icon of the corresponding application displayed on the visually obscured window. In some implementations, visual feedback includes, for example, stretching the representation 4158 of the application and reducing the horizontal width of the representation 4158 such that the representation 4158 does not overlap with the representation 4154' of the messaging application window, which has a reduced width. Figure 4B27 Upon detection of the end of input via touch 4156, a new window 4166 in the Photos application opens in split-screen mode on the left side of the display. Additionally, the full-screen window 4154 of the Messaging application is resized and displayed simultaneously with the new window 4166 of the Photos application in split-screen mode. For clarity, window 4154 is relabeled as 4164 to indicate that it has been resized and converted from a full-screen window to a split-screen window, but no new window is opened in the Messaging application. In some embodiments, the aforementioned window resizing operation in the Messaging application is accomplished by closing the full-screen window 4154 and opening the split-screen window 4164 in the Messaging application. Window 4166 and window 4164 are associated (e.g., fixed) as a pair of split-screen windows and are represented together by a single representation in the application switcher user interface (e.g., application switcher user interface 4032). Furthermore, in some embodiments, each window in the pair of split-screen windows is also counted as an open window of the corresponding application in the window switcher user interface corresponding to the respective application. In some implementations, the pair of split-screen windows is represented by a single representation in the window switcher user interface. In some implementations, when (e.g., by tapping an input) a single representation of the pair of split-screen windows is selected, the pair of split-screen windows is recalled from the application switcher user interface and / or the window switcher user interface together to the display.
[0255] Figures 4B28 to 4B31 The diagram illustrates window switching operations using a window switcher user interface according to some implementation schemes. For example... Figure 4B28As shown, the Photo application window 4166 and the Messages application window 4164 are displayed side-by-side in split-screen mode. Input via touch 4168 is detected on the application icon 224 corresponding to the Messages application. Based on determining that the input meets selection criteria (e.g., the input is a tap input), based on determining that one of the currently displayed windows (e.g., windows 4166 and 4164) and the active application icon correspond to the same application (e.g., both window 4164 and application icon 224 correspond to the Messages application), and based on determining that the application (e.g., the Messages application) of the active application icon is associated with multiple windows, the device displays a window switcher user interface 4108 in a state corresponding to the Messages application (e.g., at this time, a representation of multiple windows associated with the Messages application is displayed), as shown. Figure 4B29 As shown. In Figure 4B29 In the diagram, representation 4122' is displayed for the full-screen window 4122, representation 4136' is displayed for the side-swipe window 4136, and representation 4168' is displayed for the split-screen window 4164 (e.g., the same representation is also used for the split-screen window 4166 in the window switcher user interface of the Photos application). Figure 4B30 In the message application's window switcher user interface 4108, input via touch 4170 is detected on representation 4122'. In response to this input and based on the determination that the input meets selection criteria (e.g., the input is a tap), the device stops displaying the message application's window switcher user interface 4108 and redisplays the message application's full-screen window 4122 on the screen in a stand-alone display configuration, as shown below. Figure 4B31 As shown. At this time, the process of switching from the window switcher user interface 4108 is completed. Figure 4B28 The window switching operation from split-screen window 4164 to full-screen window 4122 is shown in the figure.
[0256] Figures 4B32 to 4B33 Similar to some implementation schemes are shown. Figures 4B5 to 4B6 The scenario shown involves an application switching operation from the first application to the second application in response to the selection of the application icon of the second application, regardless of the number of windows associated with the second application.
[0257] Figure 4B32The illustration shows that while a full-screen window 4122 of a messaging application is displayed, input via touch 4172 is detected on the application icon 218 of an email application. Based on the determination that the input via touch 4172 meets selection criteria (e.g., the input is a tap), the device determines whether the application icon 218 and the currently displayed window 4122 correspond to the same application. Based on the determination that the application icon 218 and the currently displayed window 4122 do not correspond to the same application, the device stops displaying the full-screen window 4122 and displays the full-screen window 4102 corresponding to the email application (e.g., the default window of the email application (e.g., the last full-screen window displayed by the email application)). Figure 4B33 As shown. In Figures 4B32 to 4B33 In the example scenario shown, the user's request to switch applications is satisfied regardless of whether the email application or the messaging application is currently associated with multiple windows, because the user has activated an application icon that is different from the currently displayed application.
[0258] Figures 4B34 to 4B35 exist Figure 4B33 Then, and showed something similar to Figures 4B1 to 4B5 The example shown depicts an application switcher user interface displayed in response to activating the application icon of the currently displayed application by tapping the input. Figure 4B34 When the email application window 4102 is displayed on the screen, input via touch 4174 is detected on the email application's application icon 218. Based on determining that the input meets selection criteria (e.g., the input is a tap), and based on determining that the active application icon and the currently displayed window correspond to the same application, and based on determining that the application has more than one window, the device displays the window switcher user interface 4108 of that application (e.g., the email application), such as... Figure 4B35 As shown. In Figure 4B35 At this point, all windows associated with the email application are displayed in the window switcher user interface 4108. Each window representation displays an application icon and a unique name automatically generated based on the window's content to distinguish windows with similar or identical content.
[0259] According to some implementation plans Figures 4B32 to 4B35A double-click (e.g., two consecutive inputs satisfying selection criteria, and optionally, an interval less than a threshold time) causes the device to switch from displaying a first application to displaying a second application and to display a window switcher user interface for that second application. In some embodiments, the intermediate state of displaying the second application is not displayed, and the device directly switches from displaying the first application to displaying the window switcher user interface for the second application in response to a double-click input. Then, in response to input selecting a window from that window switcher user interface or an existing window switcher user interface, the device switches from displaying the window switcher user interface for the second application to displaying the window of the second application (e.g., by selecting to close a power indicator or a new window power indicator, tapping outside the window's power indicator, etc.).
[0260] Figures 4B36 to 4B37 An example process is illustrated, wherein input via touch 4176 is detected on application icon 228 in taskbar 4006, which is overlaid on window switcher user interface 4108. In some embodiments, taskbar 4006 is initially hidden when window switcher user interface 4108 is displayed and is recalled to the screen by input that meets taskbar display criteria (e.g., an upward swipe gesture starting from the bottom edge of the touchscreen). In response to detecting the input and determining that the input meets selection criteria (e.g., a tap input), the device stops displaying window switcher user interface 4108 and displays window 4178 of the application corresponding to the active application icon 228 (e.g., a photos application), such as... Figure 4B37 As shown.
[0261] Figures 4B38 to 4B42 An example process is shown of using an application's window switcher user interface 4108 to switch from a first window (e.g., a full-screen window (e.g., window 4178)) to a second window (e.g., a side-swipe window (e.g., window 4142)) of the application (e.g., a photo application) according to some embodiments.
[0262] like Figure 4B38 As shown, when the full-screen window 4178 of the Photos application is displayed, input via touch 4180 is detected on the application icon 228 of the Photos application. In response to the input via touch 4180, the device displays a window switcher user interface 4108 corresponding to the state of the Photos application, including representations of multiple windows currently associated with the Photos application (e.g., representation 4168' of full-screen window 4168, representation 4142' of side-swipe window 4142, and representation 4178' of full-screen window 4178). Figure 4B40In the process, input via touch 4182 is detected on the display 4142' of the side-swipe window 4142. In response to detecting this input and determining that the input meets selection criteria (e.g., the input is a tap), the device stops displaying the window switcher user interface 4108 and displays the side-swipe window 4142, as shown. Figure 4B41 or Figure 4B42 As shown. In Figure 4B41 In some embodiments, the side-sliding window 4142 is displayed simultaneously with the previous window (e.g., a full-screen window or a pair of split-screen windows) that was last displayed with the side-sliding window 4142, which shares the same background (e.g., window 4122 was last displayed with, for example, a pair of split-screen windows). Figure 4B18 (The side-pull window 4142 is displayed together). Figure 4B42 In some implementations, before the display window switcher user interface 4108 (e.g., window 4178 is the last full-screen window displayed immediately before the display window switcher user interface 4108), the side window 4142 is displayed simultaneously with the last full-screen window (e.g., a full-screen window or a pair of split-screen windows).
[0263] Figures 4B43 to 4B43 Another example process for invoking the window switcher user interface 4108 for an application, according to some implementations, is shown. According to some implementations, although... Figures 4B43 to 4B43 The example shown illustrates that the window switcher user interface 4108 of the Photos application is invoked by input detected when the Photos application is displayed, but this example procedure is used to invoke the application's window switcher user interface 4108 regardless of whether the application is the currently displayed application (e.g., when input is initially detected, another application or system user interface may be displayed).
[0264] like Figure 4B43 As shown, when an application (e.g., the Photos application or another application different from the Photos application) and taskbar 6004 are displayed, input via touch 4183 is detected on the application icon in the taskbar (e.g., the application icon 228 of the Photos application). In response to detecting the input and determining that the input meets menu display criteria (e.g., the input is a tap-to-hold input or a press-to-hold input), the application icon 228 is highlighted to indicate that the input has met the menu criteria. Figure 4B44 In response to the detection of the end of input (e.g., in response to the detection of the lifting of contact 4183), a menu 4182 is displayed, displaying selectable options 4184 for window management corresponding to the selected application icon (e.g., a photo application). Figure 4B44As shown, the selectable options include a first option to display all windows associated with the Photos application in the window switcher user interface, a second option to open a new window (e.g., a new default window) within the Photos application, and a third option to close all windows associated with the Photos application. Figure 4B45 In the first selectable option for displaying all windows, input via touch 4186 is detected. In response to detecting this input and determining that the input meets selection criteria (e.g., the input is a tap), the device displays a window switcher user interface 4108 including a representation of all windows associated with the Photos application, such as... Figure 4B46 As shown.
[0265] exist Figure 4B4 , Figure 4B19 , Figure 4B29 , Figure 4B35 and Figure 4B40 In the window switcher user interface 4108 shown, a new window enablement 4112 is displayed, and when activated (e.g., by tapping input), the new window enablement initiates the process of opening a new window of the application, corresponding to the currently displayed window switcher user interface. In some embodiments, the newly opened window is the application's default new window. In some embodiments, a second version of the window switcher user interface 4108 displays two different new window enablements: one for opening a new document in a new window, and another for opening an existing document in a new window. In some embodiments, the device selects which version of the window switcher user interface 4108 to use depending on whether the application corresponding to the window switcher user interface is a document editor application (e.g., word processing application, spreadsheet application, presentation editor application, drawing application, PDF document generation application, publishing application, etc.) or a document creation application (e.g., browser application, email application, instant messaging application, photo application, etc.). Figures 4B47 to 4B50 Two different versions of the new window display representation are shown in a second version of a window switcher user interface 4108 according to some implementation schemes.
[0266] like Figure 4B47 As shown, a full-screen window (4188) of the Notes application is displayed. The Notes application is a quality document editor application because users can frequently create and edit documents, and reopen previously created and edited documents for further editing. Figure 4B47As shown, when the Memos application window 4188 is displayed, input via touch 4190 is detected on the application icon 244 of the Memos application in the taskbar 6004. In response to the detected input and based on the determination that the input meets the selection criteria, the device displays a window switcher user interface 4108 corresponding to the Memos application, such as... Figure 4B48 As shown. In Figure 4B48 In this context, the version of the window switcher user interface 4108 displayed for the Notes application includes representations of windows associated with the Notes application (e.g., representation 4188' for a full-screen window 4188 and representation 4192' for a side-swipe window of the Notes application). In addition to representations of open windows of the Notes application, the window switcher user interface also includes an "Open" enable representation 4194 for opening an existing document in a new window of the Notes application, and a "New" enable representation 4196 for opening a new document in a new window of the Notes application. Figure 4B48 The window switcher user interface 4108 shown indicates inputs made via contact 4198 and inputs made via contact 4200.
[0267] exist Figure 4B49 In response to detecting input via touch 4200 on the "New" indicator 4196 and determining that the input meets a first criterion (e.g., the input is a tap), the device stops displaying the window switcher user interface and displays a new window 4202 displaying a new memo document (e.g., a new document created based on a default memo template in a memo application, opened in an editable state with a keyboard covering the document). In some embodiments, instead of directly opening a new document based on a default new document template, the device displays a document creation user interface including selectable options corresponding to different new document formats and / or different new document templates. Once the user selects a corresponding new document format and / or new document template, the device creates and opens the new document in a new window of the application according to the selected document format and / or document template.
[0268] exist Figure 4B50 In response to detecting input via touch 4198 on the "Open" indicator 4194 and determining that the input meets a first criterion (e.g., the input is a tap), the device stops displaying the window switcher user interface 4108 and displays a new window 4204 with a document selector user interface for a memo application. In some embodiments, the document selector user interface includes selectable options corresponding to different existing folders and documents that can be opened in the application (e.g., a memo application). For example, as Figure 4B50As shown, a list of existing memos is displayed in the document selector user interface of a memo application. Once the user selects a corresponding memo from the existing memos, the device opens the selected document (e.g., the previously created selected memo) in a new window of the application (e.g., the memo application). In some embodiments, the application is a document management application and is configured to open documents corresponding to different applications. In this case, the document selector of the document management application optionally displays representations of documents corresponding to different applications in its document selector user interface and invokes the different application corresponding to the selected document to open the selected document in response to the user's selection input.
[0269] Figure 4B51 The home screen user interface 4205 is displayed, which includes multiple application icons corresponding to different applications installed on the device. In response to input that meets menu display standards (e.g., tap-to-hold or tap-to-hold input followed by release; long-touch-to-hold input without release; or deep-press input without release), a quick action menu 4206 is displayed. In the quick action menu 4206, selectable options corresponding to in-application operations (e.g., displaying recent photos, displaying favorite folders, searching for photos, etc.) are displayed in menu 4182. Figure 4B44 The selectable options shown in the diagram are displayed simultaneously, including a first option for displaying all windows associated with the Photos application in the window switcher user interface, a second option for opening a new window (e.g., a new default window) in the Photos application, and a third option for closing all windows associated with the Photos application.
[0270] Figures 4C1 to 4C48The diagram illustrates a process for dragging and dropping objects (e.g., user interface objects representing content items or application icons) at different locations (e.g., side areas) on a display, according to some embodiments. In some embodiments, placing the object corresponding to the content item in different areas on the display optionally causes the device to perform different operations based on various location-based criteria (e.g., a comparison of the final position of the dragged object predicted based on the end position of the drag input, the position of the object when the drag input ends, or a comparison based on past movement of the input relative to different predefined areas on the display). In some implementations, operations performed in response to placing an object corresponding to a content item in different areas of the display include: (1) displaying the content item or its representation at different locations within the same window (e.g., performing object movement or object copying within the same application window); (2) displaying the content item or its representation at locations in different windows simultaneously displayed with the object's original window (e.g., performing object movement or object copying between two simultaneously displayed windows (e.g., windows of the same application or two different applications); (3) opening the content item and displaying it in a new window with the object's original window in a first simultaneous display configuration (e.g., displaying the content item in a new side-sliding window of the local application corresponding to the content item, covering the object's original window); (4) opening the content item and displaying it in a new window with the object's original window in a second simultaneous display configuration (e.g., resizing the object's original window and displaying the content item in a new split-screen window of the local application corresponding to the content item, alongside the object's original window). (5) Open the content item and display it in a new window with the object's original window in a third simultaneous display configuration (e.g., display the content item in a draft window that covers the center portion of the object's original window and optionally visually blur the object's original window); (6) Open the content item and display it in a new window with the object's original window in a fourth simultaneous display configuration (e.g., display the content item in a minimized window that is visible simultaneously with the content item's original window); and / or (7) open the content item and display it in a new full-screen window based on the drag input or the position or projection position of the dragged object at the end of the drag input (e.g., open the content item in a new full-screen window, replacing the object's original window on the display (and replacing other windows displayed simultaneously on the display). In some embodiments, a predefined area on the display is used to determine whether to open a new window (e.g., a side window or a split window) for the application when dragging and dropping the application icon (e.g., Figure 4B12 and Figure 4C28Regions 4308 and 4310 in the image are relative to a predefined region (e.g., ) used to determine whether to open a new window for a content item when an object representing a content item is dragged and dropped on the display. Figure 4B16 Area 4014 and Figure 4B26 The area 4162 in the figure is reduced. For example, in some embodiments, the predefined area for placing the application icon to create a side window of the application is wider than the predefined area for placing objects representing content items to create a side window for displaying those content items. Similarly, in some embodiments, the predefined area for placing the application icon to create a split-screen window of the application is wider than the predefined area for placing objects representing content items to create a split-screen window for displaying those content items. The user interface in these figures is used to illustrate the process described below, including Figures 7A to 7H as well as Figure 7I The process is described below. For ease of explanation, some embodiments of the implementation will be discussed with reference to the operation performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representation point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operation is optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to a contact detected on the touch-sensitive surface 451 when the user interface shown in the figures, along with the focus selector, is displayed on the display 450.
[0271] Figures 4C1 to 4C5 This illustrates a process for opening content items in a side pane via drag-and-drop operation, according to some implementation schemes. Figures 4C1 to 4C5 In this process, the object representing the content item is dragged from the first window displayed on the monitor and placed in a first predefined area near the side edge of the monitor (e.g., Figure 4C3 In the predefined area 4308 shown, and therefore, the content item is opened in a new side window of the application corresponding to that content item. This is in contrast to the predefined area used to place the application icon and open the side window of the application corresponding to that application icon (e.g., Figure 4A5 , Figure 4B16 Compared to the predefined area 4014 in the first window, the size of the first predefined area for placing content items is reduced (e.g., with a reduced width and / or a reduced distance from the corresponding side edge of the display). This makes more area available for operations relative to content items in the first window and optionally relative to content items in a second window displayed simultaneously with the first window.
[0272] like Figure 4C1As shown, a full-screen window 4122 of the messaging application is displayed (e.g., in a standalone configuration). Input via touch 4302 is detected at a location corresponding to an object 4304 representing a first content item (e.g., a conversation with Greg Kane). The initial portion of the input via touch 4302 satisfies object movement criteria for initiating a drag operation on object 4304 or a copy of object 4304 representing the first content item (e.g., the initial portion of the input via touch 4302 satisfies a touch hold time threshold or a light press intensity threshold), and the device highlights object 4304 to indicate that the criteria for initiating a drag operation on that object have been met.
[0273] exist Figure 4C2 In this context, based on the movement of touch 4302 detected after the second criterion is met, a representation 4306 of the first content item (e.g., a copy of object 4304) is dragged on the display. In some embodiments, representation 4306 has a first appearance indicating that there is no acceptable placement available for the object in the portion of window 4122 outside the first predefined area 4308, and that if input ends at this point, no object move or object copy operation will be performed relative to the first content item in window 4122.
[0274] exist Figure 4C3 In the process, after the object movement criteria are met, the representation 4306 of the first content item is dragged to a position within the first predefined area 4308 based on the movement of contact 4302. In some embodiments, the representation 4306 presents a second appearance (e.g., the representation is stretched and horizontally expanded), which indicates that if input ends at this time, the first content item will be displayed in a new side-swipe window of the application (e.g., a messaging application) corresponding to the first content item. Figure 4C4 In some embodiments, in addition to altering the appearance of the representation 4306 of the first content item, the device also provides additional visual feedback when the representation is dragged to a position within a first predefined area 4308 to indicate that the current position of the input and / or representation 4306 meets positional criteria for opening the first content item in a side-slide window. In some embodiments, the additional visual feedback includes reducing the overall size of the first window 4122 to display the representation 4122' of the first window 4122, and revealing a background 4134 below the representation 4122'.
[0275] exist Figure 4C5 In response to the detection of the end of input via contact 4302 (e.g., detection of the lifting of contact 4302), a first content item is displayed in a new side-pull window 4136 of the messaging application, covering the first window 4122.
[0276] Figure 4C4 After Figures 4C6 to 4C7 This demonstrates how the input to contact 4302 is continuously evaluated against positional criteria corresponding to different predefined areas on the display for different operations performed after the input ends (e.g., moving an object within the same window, moving an object to a different window, opening content in a new side pane, opening content in a new split-screen window, etc.), and if the input is to end at the current position, visual feedback is dynamically updated to indicate the corresponding possible outcome. Figures 4C6 to 4C7 In the event that the input via touch 4302 has ended, the movement of touch 4302 drags the representation 4306 of the first content object from the first predefined area 4308 to a position outside the first predefined area 4308 in the center portion of the display. Therefore, visual feedback is dynamically updated to indicate that the positional criteria for opening the first content item in the side-pull window are no longer met, and if input is made at this time (e.g., in...), Figure 4C7 If the indicated time expires, no object move or object copy operation will be performed.
[0277] Figures 4C8 to 4C11 This illustrates a process for opening content items in a split-screen window via drag-and-drop operations, according to some implementation schemes. Figures 4C8 to 4C11 In this process, the object representing the content item is dragged from the first window displayed on the monitor and placed in a second predefined area near the side edge of the monitor (e.g., Figure 4C10 In the predefined area 4310 shown, and therefore, the content item is opened in a new split-screen window of the application corresponding to that content item. This is in contrast to the predefined area used to place the application icon and open the split-screen window of the application corresponding to that application icon (e.g., Figure 4B26 Compared to the first predefined area 4162, the second predefined area 4310 for placing content items is smaller in size (e.g., having a reduced width and / or a reduced distance from the corresponding side edge of the display). The second and first predefined areas on the same side of the display are optionally adjacent to each other and share a common boundary. For example, the second predefined area is defined by a first boundary line at a first distance from the side edge of the display, and the first predefined area is defined by the first boundary line and a second boundary line at a second distance (greater than the first distance) from the side edge of the display. In some embodiments, a third predefined area outside the first (and second) predefined areas is used to determine whether an operation is performed relative to the first content item within the first window, rather than opening a new window for the first content item.
[0278] like Figure 4C8As shown, a full-screen window 4122 of a messaging application is displayed (e.g., in a standalone configuration). Input via touch 4312 is detected at a location corresponding to object 4304 representing a first content item (e.g., a conversation with Greg Kane). The initial portion of the input via touch 4312 satisfies object movement criteria for initiating a drag operation on object 4304 or a copy of object 4304 representing the first content item (e.g., the initial portion of the input via touch 4312 satisfies a touch hold time threshold or a light press intensity threshold), and the device highlights object 4304 to indicate that the criteria for initiating a drag operation on that object have been met. In some embodiments, touch 4312 may be the same as touch 4302, and input via touch may trigger different operations depending on the position of the input at the end of the input (e.g., Figures 4C1 to 4C7 or Figures 4C8 to 4C15 (Those operations described herein). In some embodiments, contact 4312 and contact 4302 are different contacts corresponding to two different inputs detected at different times on the same window displaying the same user interface state.
[0279] exist Figure 4C9 In this context, based on the movement of contact 4312 detected after the second criterion is met, a representation 4306 of the first content item (e.g., a copy of object 4304) is dragged on the display. In some embodiments, representation 4306 has a first appearance that indicates that there is no acceptable placement available for objects in portions of window 4122 outside of the first predefined area 4308 (and the second predefined area 4310), and that if input ends at this point, no object move or object copy operation will be performed relative to the first content item in window 4122.
[0280] exist Figure 4C10 In the process, after the second criterion is met, the representation 4306 of the first content item is dragged to a position within the second predefined area 4310 based on the movement of contact 4312. In some embodiments, the representation 4306 presents a third appearance (e.g., the representation is further elongated and laterally contracted), which indicates that if input ends at this point, the first content item will be displayed in a new split-screen window of the application corresponding to the first content item (e.g., a messaging application), where the split-screen window of the messaging application is converted from the full-screen window 4122. Figure 4C10In some embodiments, in addition to altering the appearance of the representation 4306 of the first content item, the device also provides additional visual feedback when the representation is dragged to a position within the second predefined area 4310 to indicate that the input and / or the current position of the representation 4306 meets the positional criteria for opening the first content item in a split-screen window. In some embodiments, the additional visual feedback includes reducing the width of the first window 4122 to display the representation 4122' of the first window 4122, and revealing a background 4134 below the representation 4122' on one side of the display on which the representation 4306 is currently located.
[0281] exist Figure 4C11 In response to the detection of the end of input via contact 4312 (e.g., detection of the lifting of contact 4312), the first content item is displayed side-by-side with another split-screen window 4314 converted from the first window 4122 in a new split-screen window 4316 of the messaging application.
[0282] exist Figure 4C12 In the context of displaying the split-screen windows 4314 and 4316, input via touch 4320 is detected on the close indicator 4318 of split-screen window 4316. In response to detecting input via touch 4320 and determining that the input meets selection criteria (e.g., the input is a tap), split-screen window 4316 closes, and split-screen window 4314 is converted back to an independent full-screen window 4122, as... Figure 4C13 As shown.
[0283] Figure 4C10 After Figures 4C14 to 4C15 This demonstrates how the input to contact 4312 is continuously evaluated against positional criteria corresponding to different predefined areas on the display for different operations performed after the input ends (e.g., moving an object within the same window, moving an object to a different window, opening content in a new side pane, opening content in a new split-screen window, etc.), and if the input is to end at the current position, the visual feedback is dynamically updated to indicate the corresponding possible outcome. Figures 4C14 to 4C15 In the process, before the end of input via touch 4312 is detected, the movement of touch 4312 drags the representation 4306 of the first content object from the second predefined area 4310 to a position outside the first predefined area 4308 in the center portion of the display, and therefore, visual feedback is dynamically updated to indicate that the position criteria for opening the first content item in the split-screen window are no longer met, and if input is made at this time (e.g., in...), Figure 4C15 If the indicated time expires, no object move or copy operation will be performed. As long as the end of the input is not detected, Figure 4C2 , Figure 4C3 , Figure 4C4 , Figure 4C6 , Figure 4C7 , Figure 4C9 , Figure 4C10 , Figure 4C14 as well as Figure 4C15 The dynamic visual feedback shown can be displayed in any order and repeated any number of times based on the current position of the contact. Furthermore, it will be displayed separately. Figure 4C5 , Figure 4C11 and Figure 4C13 The final state of the screen shown depends on whether the final end position of the input is in the first predefined area 4308, the second predefined area 4310, or a third predefined area outside the first and second predefined areas (as well as any other predefined areas used to open new windows in various display modes (e.g., full screen, draft mode, minimized mode, side-pull windows on different sides of the display, split screen on different sides of the display, etc.)).
[0284] Figures 4C16 to 4C17 The diagram illustrates input made via touch 4322 at the location of object 4304 corresponding to the first content item in representation window 4122 (e.g., a session from GregKane). In response to detecting input via touch 4322 and determining that the input meets selection criteria (e.g., the input is a tap), the device navigates to another user interface in the messaging application without opening a new window. For clarity, the window showing the new user interface is labeled window 4324, as shown below. Figure 4C17 As shown. In some embodiments, operations corresponding to navigation of the user interface within the application are implemented by closing the current window displaying the current user interface and opening a new window with the new user interface. In some embodiments, contact 4322 may be the same as contact 4302 and / or 4312, and input via contact may trigger different operations (e.g., ...) depending on the position of the input when it finally ends and the type of the input (e.g., drag input or tap input). Figures 4C1 to 4C7 and / or Figures 4C8 to 4C15 (Those operations described herein). In some embodiments, contact 4322, contact 4312, and contact 4302 are different contacts corresponding to different inputs detected at different times on the same window displaying the same user interface state.
[0285] Figures 4C18 to 4C23 This shows content items associated with different applications (e.g., email applications), similar to... Figures 4C1 to 4C17 The example processes shown are... relative to... Figures 4C1 to 4C17 The examples shown illustrate many aspects applicable to Figures 4C18 to 4C23 The example shown.
[0286] Figures 4C18 to 4C21This illustrates a process, according to some implementations, of opening another content item in a split-screen window via drag-and-drop. Figures 4C18 to 4C21 In this process, the object representing the content item is dragged from the first window displayed on the monitor and placed in a second predefined area (e.g., near the side edge of the monitor, or the right edge) of the monitor. Figure 4C20 In the predefined area 4310 shown, and therefore, the content item is opened in a new split-screen window of the application corresponding to the content item.
[0287] like Figure 4C18 As shown, a full-screen window 4102 of an email application is displayed (e.g., in a standalone configuration). Input via touch 4328 is detected at a location corresponding to object 4326 representing a second content item (e.g., an email message from MobileFind). The initial portion of the input via touch 4328 meets object movement criteria for initiating a drag operation on object 4326 or a copy of object 4326 representing the second content item (e.g., the initial portion of the input via touch 4328 meets a touch hold time threshold or a light press intensity threshold), and the device highlights object 4326 to indicate that the criteria for initiating a drag operation on that object have been met.
[0288] exist Figure 4C19 In this context, based on the movement of contact 4328 detected after the second criterion is met, a representation 4330 of the second content item (e.g., a copy of object 4326) is dragged on the display. In some embodiments, representation 4330 has a first appearance that indicates that there is no acceptable placement available for objects in portions of window 4102 outside of the first predefined area 4308 (and the second predefined area 4310), and that if input ends at this point, no object move or object copy operation will be performed relative to the first content item in window 4102.
[0289] exist Figure 4C20 In the process, after the second criterion is met, the representation 4330 of the second content item is dragged to a position within the second predefined area 4310 based on the movement of contact 4328. In some embodiments, the representation 4330 presents a second appearance (e.g., the representation is stretched), which indicates that if input ends at this time, the second content item will be displayed in a new split-screen window of the application corresponding to the second content item (e.g., an email application), wherein the split-screen window of the email application is converted from the full-screen window 4102. Figure 4C20In some embodiments, in addition to altering the appearance of the representation 4330 of the second content item, the device also provides additional visual feedback when the representation is dragged to a position within the second predefined area 4310 to indicate that the input and / or the current position of the representation 4330 meets the positional criteria for opening the second content item in a split-screen window. In some embodiments, the additional visual feedback includes reducing the width of the full-screen window 4102 to display the representation 4102' of the window 4102, and revealing a background 4134 below the representation 4102' on one side of the display on which the representation 4330 is currently located.
[0290] exist Figure 4C21 In response to the detection of the end of input via touch 4330 (e.g., detection of touch 4330 being lifted off), the second content item is displayed side-by-side with another split-screen window 4332 converted from window 4102 in a new split-screen window 4334 of the email application.
[0291] Figure 4C22 and Figure 4C23 from Figure 4C18 , Figure 4C19 and Figure 4C20 Continuing from any of them, and showing an example situation where the second content item is opened in a new side-pull window 4336 of the email application, covering the full-screen window 4102. For example Figure 4C22 As shown, after the object movement criteria are met, the representation 4330 of the second content item is dragged to a position within the first predefined area 4308 based on the movement of contact 4328. In some embodiments, the representation 4330 presents a third appearance (e.g., compared to the state shown in 4C20, the representation is stretched to a lesser extent and expanded laterally), which indicates that if input ends at this point, the second content item will be displayed in a new side-swipe window of the application (e.g., an email application) corresponding to the second content item. Figure 4C22 In some embodiments, in addition to altering the appearance of the representation 4330 of the second content item, the device also provides additional visual feedback when the representation is dragged to a position within the first predefined area 4308 to indicate that the current position of the input and / or representation 4330 meets the positional criteria for opening the second content item in a side-slide window. In some embodiments, the additional visual feedback includes reducing the overall size of window 4102 to display representation 4102' of window 4102, and revealing background 4134 below representation 4102'.
[0292] exist Figure 4C23 In response to the detection of the end of input via contact 4328 (e.g., detection of the lifting of contact 4328), a second content item is displayed in a new side-pull window 4336 of the email application, covering window 4102.
[0293] Figures 4C23 to 4C24 The diagram illustrates the detection of input via touch 4338 on power indicator 4340 to create a new draft email in an email application. In response to the detection of input via touch 4338 and based on the determination that the input meets selection criteria (e.g., the input is a tap input), the device opens a new draft window containing the new draft email (e.g., a new reply email to the email shown in side-swipe window 4336 (e.g., because power indicator 4340 is part of side-swipe window 4336)). Figure 4C24 As shown. In some implementations, the new draft window 4342 can be accessed through other user interaction processes. Figure 4C24 The configuration shown is as follows (e.g., opening an existing draft email in a side window or split-screen window, and displaying the existing draft email in draft mode by dragging the window to the center of the display).
[0294] exist Figures 4C24 to 4C26 In the draft window 4342, input via contact 4344 is detected on the drag handle 4344, and this input includes movement of contact 4344 toward the side edge (e.g., the right edge) of the display. In response to detecting this input and based on determining the current position of contact 4346 within a first predefined area 4308, the representation 4348 of the draft window 4342 is displayed with an appearance (e.g., an elongated application icon, which is also horizontally expanded) indicating that if the input is to end at the current position, the draft window 4342 will be converted into a side-swipe window covering the original background window 4102. In some embodiments, visual feedback also includes reducing the overall size of the background window 4102 to display representation 4102', and revealing background 4134 below representation 4102'. Figure 4C26 In the middle, contact 4344 and indication 4348 are in the predefined region 4014 (or Figure 4E8 When the end of input is detected within area F of the draft window 4342, the draft window 4342 is converted into a side-slide window 4348 that overlays the background window 4102. The side-slide window 4348 displays the draft email reply to John. According to some implementations, relative to... Figures 4E1 to 4E28 More detailed descriptions are provided of other relevant examples of dragging the currently displayed window and switching a window in one display configuration to a window in another display configuration.
[0295] Figures 4C27 to 4C40 Various examples are shown in which, after a drag operation is initiated on a content object, the final result of the input is determined based on the position touched when the input ends or the position of the dragged object (e.g., after the end of the input is detected).
[0296] exist Figure 4C27 In this example, the display is generally divided into several areas, including a first predefined area 4308, a second predefined area 4310, a third predefined area 4354, a fourth predefined area outside the first, second, and third predefined areas of window 4102 and outside the search input field 4355, and a fifth predefined area corresponding to the search input field 4355 in window 4102. In this example, the area of window 4102 outside the search input field 4335 does not correspond to any operation that can be performed on the dragged content item in response to the end of drag input. However, in some implementations, window 4102 may include sub-areas where operations can be performed relative to the dragged content item (e.g., moving the dragged item within the sub-area, copying the dragged item to a folder within the sub-area, sending the dragged item to another user (e.g., placing the content item via a "send" button), deleting the dragged item (e.g., placing the content item on a virtual recycle bin in the window), printing the dragged item (e.g., placing the content item on a printer icon shown in the window), etc.).
[0297] exist Figure 4C27 In the document, input via touch 4350 has been detected at a location corresponding to object 4352 (e.g., image "Attachment 1") representing the document. In response to the detection of an initial portion of the input that meets the object movement criteria (e.g., a tap to hold input or a press of input without lifting the touch), the device displays visual feedback (e.g., highlighting object 4352) to indicate that the criteria for initiating a drag operation on the document have been met by the initial portion of the input.
[0298] exist Figure 4C28 In this implementation, a first movement of contact 4350 is detected after the initial portion of the input meets the object movement criteria, and the representation 4356 of the document is dragged on the display according to the movement of contact 4350. When contact 4356 is above a portion of window 4102 outside the first predefined area 4308, the second predefined area 4310, and the third predefined area 4354, the appearance of representation 4356 indicates that no acceptable placement position is available at that location, and no operation will be performed relative to the document if the input is to end at the current location. In some embodiments, if an acceptable placement position is available at the current location, the device will provide visual feedback to indicate that an operation will be performed relative to the document if the input ends at the current location (e.g., changing the appearance of representation 4356 in a manner that indicates a specific operation to be performed when the end of input is detected at that location).
[0299] exist Figure 4C29In the input, after the initial portion meets the second criterion, a second movement of contact 4350 is detected, and a representation 4356 of the document is dragged on the display based on the movement of contact 4350 to the search input field 4355. The appearance of representation 4356 changes (e.g., from an icon to a filename) to indicate that an acceptable placement position is available at that location, and if the input ends at the current position, a search is performed based on the filename of the document.
[0300] exist Figure 4C30 In the input process, after the initial portion of the input meets the second criterion, a third movement of contact 4350 is detected, and a representation 4356 of the document is dragged on the display based on the movement of contact 4350 to a third predefined area 4354 in the side-pull window 4348. The appearance of representation 4356 changes (e.g., its size decreases, and a preview of the document (e.g., image 4358) is displayed in the side-pull window 4348) to indicate that an acceptable placement position is available at that location, and if the input ends at the current position, the content of the document is inserted into a draft email.
[0301] exist Figure 4C31 When contact and representation 4356 are within the third predefined area 4354, the end of input is detected. Therefore, a document (e.g., image 4358) is inserted at the insertion point in the draft email shown in the side window 4348.
[0302] exist Figure 4C32 In the input, after the initial portion satisfies the object movement criterion, a fourth movement of contact 4350 is detected, and a representation 4356 of the document is dragged on the monitor based on the movement of contact 4350 to the first predefined area 4308 in the side-pull window 4348. If the input is to end at the current position, the appearance of representation 4356 changes (e.g., with...). Figure 4C28 Compared to the appearance shown, it is elongated and horizontally expanded to indicate that the document will open in a new side-pull window.
[0303] exist Figure 4C33 When the touch and indication 4356 are within the first predefined area 4354, the end of the input is detected. Therefore, the document (e.g., image 4358) is opened in a new side-pull window 4360 of the photo application (e.g., a native application for image documents), covering the full-screen window 4102 of the email application.
[0304] In some implementations, a fifth movement of contact 4350 is detected after the initial portion of the input meets the second criterion, and a representation 4356 of the document is dragged on the display based on the movement of contact 4350 to a second predefined area 4310 in the side-pull window 4348. If the input is to end at the current position, the appearance of representation 4356 changes (e.g., with...). Figure 4C32 Compared to the appearance shown, it is further elongated and horizontally contracted to indicate that the document will be opened in a new split-screen window. If the end of input is detected when the touch and indication 4356 is within the second predefined area 4310, the document (e.g., image 4358) will be opened in a new split-screen window of the photo application (e.g., the native application of the image document), side by side with the split-screen window converted from the full-screen window 4102 of the email application.
[0305] In some implementations, the positions of the contacting and dragged objects are continuously evaluated, and the positions of the contacting / draggled objects are compared with the different predefined areas (e.g., relative to the above-mentioned areas). Figure 4C27 , Figure 4C28 , Figure 4C29 , Figure 4C30 and Figure 4C32 The visual feedback is dynamically updated by comparing the inputs. Before the end of the input is detected, Figures 4C27 to 4C30 as well as Figure 4C32 The displayed state can be repeated any number of times in any order based on the current position of the input.
[0306] Figures 4C34 to 4C40 The diagram illustrates operations performed relative to a content object in response to the completion of a drag operation performed on the content object, according to some implementation schemes.
[0307] exist Figure 4C34 In this configuration, a side-swipe window 4348 is displayed to cover the full-screen window 4102. Input via touch 4366 is detected at a location corresponding to an object 4364 representing a webpage (e.g., a hyperlink). The initial portion of the input via touch 4366 has satisfied the object movement criterion, and the device highlights the object 4364 to indicate that the criteria for initiating a placement operation on the object 4364 have been met.
[0308] exist Figure 4C35 In response to the first movement of contact 4366 detected after the object movement criterion has been met, representation 4368 is dragged on the display according to the movement of contact 4366. For example... Figure 4C35As shown, when the contact and representation 4368 is above a portion of the display where an object representing a webpage is not rendered (e.g., in an area outside the first predefined area 4308, the second predefined area 4310, the third predefined area 4354, and the search input field 4355), the representation 4368 has a first appearance to indicate that if the input ends at this time, no object move or object copy operation will be performed relative to the object in the email application.
[0309] exist Figure 4C36 In the process, after the initial portion of the input meets the object criteria, a second movement of contact 4366 is detected, and a representation 4368 of the webpage is dragged on the display based on the movement of contact 4366 to the search input field 4355. The appearance of representation 4368 changes (e.g., from an icon to the URL or title of the webpage) to indicate that an acceptable placement position is available at that location, and if the input ends at the current location, a search is performed based on the URL or title of the webpage.
[0310] exist Figure 4C37 In this process, after the initial portion of the input meets the object movement criteria, a third movement of contact 4336 is detected, and the representation 4368 of the webpage is dragged on the display according to the movement of contact 4336 to a third predefined area 4354 in the side-slide window 4348. The appearance of representation 4368 changes (e.g., its size decreases, and a website address (e.g., URL) or a preview of the webpage is displayed in the side-slide window 4348) to indicate that an acceptable placement position is available at that location, and if the input ends at the current location, the URL or content of the webpage is inserted into the draft email. In some embodiments, if the end of the input is detected when contact and representation 4336 are within the third predefined area 4354, the URL or content of the webpage is inserted at the insertion point in the draft email shown in the side-slide window 4348.
[0311] exist Figure 4C38 In the input, after the initial portion of the input meets the object movement criteria, a fourth movement of contact 4336 is detected, and a representation 4368 of dragging the webpage on the display is generated based on the movement of contact 4336 to the first predefined area 4308 in the side-pull window 4348. If the input is to end at the current position, the appearance of representation 4368 changes (e.g., with...). Figure 4C35 Compared to the appearance shown, it is elongated and horizontally expanded to indicate that the webpage will open in a new side-pull window of the browser application.
[0312] exist Figure 4C39When the touch and indication 4368 are within the first predefined area 4308, the end of the input is detected. Therefore, the document (e.g., a webpage) is opened in a new side-pull window 4372 of the browser application (e.g., a native application of the webpage), overriding the full-screen window 4102 of the email application.
[0313] exist Figure 4C40 In the process, after the initial portion of the input meets the object movement criterion, a fifth movement of contact 4336 is detected, and a representation 4368 of dragging the webpage on the display is generated based on the movement of contact 4336 to the second predefined area 4310 on the display. The appearance of representation 4368 changes (e.g., with...) Figure 4C38 Compared to the shown appearance, it is further elongated and horizontally contracted to indicate that if input is to end at the current position, a webpage will open in a new split-screen window. In some embodiments, the size of the background full-screen window 4102 is adjusted (e.g., the width is reduced) to create space to display the new split-screen window. The background 4134 is exposed behind the representation 4102' of the resized window 4102. In some embodiments, the side-sliding window 4348 displayed on the same side of the display as representation 4368 is shifted to the other side of the display. Figure 4C41 In this context, the end of input is detected when touch 4336 and indication 4368 are within the second predefined area 4310, and the webpage opens in a new split-screen window 4376 of the browser application (e.g., a local application of the webpage), side-by-side with the split-screen window 4374, which is converted from the full-screen window 4102 of the email application. In some embodiments, the side-swipe window 4348 is shifted to the other side of the display, such as... Figure 4C41 As shown. In some implementations, the side-sliding window 4348 remains on the same side of the display as before (e.g., the right side), with the pair of split-screen windows 4374 and 4376 serving as the background.
[0314] In some implementations, the position of the contacted and dragged object is continuously evaluated, and the position of the contacted / dragged object is compared with the different predefined areas mentioned above (e.g., relative to...). Figure 4C35 , Figure 4C36 , Figure 4C37 , Figure 4C38 and Figure 4C40 The visual feedback is dynamically updated by comparing the inputs. Before the end of the input is detected, Figure 4C35 , Figure 4C36 , Figure 4C37 , Figure 4C38 and Figure 4C40 The displayed state can be repeated any number of times in any order based on the current position entered.
[0315] In the example above, the content object is dragged to the area of the display that includes the side-sliding window. In some implementations, the same predefined areas 4308 and 4310 exist on the display and function in the same manner as described above, regardless of whether a side-sliding window or a split-screen window exists in those predefined areas.
[0316] Figures 4C42 to 4C46 The diagram illustrates a predefined area for opening a new side window or a new split-screen window by dragging and dropping an application icon, according to some embodiments. This predefined area is extended relative to a predefined area for opening a new side window or a new split-screen window by dragging and dropping an object representing a content item (e.g., a document or other content).
[0317] like Figure 4C42 As shown, input via touch 4378 is detected on the application icon 220 of the browser application. The initial portion of this input meets the object movement criterion, and the device highlights the application icon 220 to indicate that a drag operation can be initiated on the application icon 220 by moving the touch 4378.
[0318] exist Figure 4C43 In the process, a first movement of contact 4378 is detected, and based on the movement of contact 4378 detected after the object movement criterion is met in the initial part of the input, a representation 4380 of dragging the application icon 220 (e.g., for a browser application) on the display is executed. Figure 4C43 As shown, when contact 4378 is at any location within the extended first predefined region 4308' (e.g., with...), Figures 4C35 to 4C31 Compared to region 4308, the device provides visual feedback (e.g., indicating that 4380 is stretched and expanded horizontally, and the overall size of the background window 4102 is reduced, thus revealing the background 4134) to indicate that if the end of input is detected at the current location, a new side-swipe window of the browser application will open. Figure 4C44 In this context, when the contact is within an extended first predefined area 4308' (e.g., optionally in an area outside the original first predefined area 4308), the end of the input via the contact is detected, and a new side-pull window 4382 of the browser application is displayed, overriding the full-screen window 4102 of the email application. In some embodiments, if the browser application is associated with more than one window, the device optionally opens the window selector user interface 4508 of the browser application (e.g., as shown in the image). Figure 4D5 (as shown), instead of a browser application's pull-out window. Relative to... Figures 4D1 to 4D19 More details were described.
[0319] exist Figure 4C45In the process, a second movement of contact 4378 is detected, and based on the movement of contact 4378 detected after the object movement criterion is met in the initial part of the input, a representation 4380 of dragging the application icon 220 (e.g., for a browser application) on the display is executed. Figure 4C45 As shown, when contact 4378 is anywhere within the extended second predefined area 4310', the device provides visual feedback (e.g., indicating that 4380 further elongates and shrinks laterally, and the width of the background window 4102 decreases, thus revealing the background 4134) to indicate that if the end of input is detected at the current location, a new split-screen window of the browser application will be opened. Figure 4C46 When the contact is within the extended second predefined area 4310' (e.g., optionally, in an area outside the original second predefined area 4310 and within the original first predefined area 4308), the end of the input via the contact is detected, and a new split-screen window 4384 of the browser application is displayed, side-by-side with a new split-screen window 4186 converted from the full-screen window 4102 of the email application. In some embodiments, if the browser application is associated with more than one window, the device optionally opens the browser application's window selector user interface 4508 (e.g., as shown in the image). Figure 4D19 (as shown), instead of a browser application's split-screen window. Relative to... Figures 4D1 to 4D19 More details were described.
[0320] As described above, the extended second predefined area 4310' is defined by the side edge of the display and a boundary shifted a certain distance away from the side edge, which is greater than the distance between the boundary of the first predefined area 4310 and the same side edge of the display. The extended first predefined area 4308' is defined by the boundary of the extended second predefined area and a new boundary shifted a certain distance away from the side edge, which is greater than the shifted distance of the boundary of the second predefined area 4310'. As a result of these boundary adjustments, the width of the extended first predefined area 4308' is greater than the width of the first predefined area 4308, and the width of the extended second predefined area 4310' is greater than the width of the second predefined area 4310. This allows application icons to be more easily placed on predefined areas on the display to open new windows of the desired type, since object movement and object copying operations are rare or unimplemented for application icons in background windows.
[0321] Figures 4C47 to 4C48 This illustrates that, according to some implementations, the same result can be achieved using a quick actions menu, in addition to opening content items in a new window (e.g., a new side window, a new split-screen window) via drag-and-drop operations performed on the object. Figure 4C47As shown, input via touch 4386 is detected on object 4326 representing an email from MobileFind. The initial portion of the input meets menu display criteria (e.g., a tap-to-hold input time threshold and / or a press-to-input intensity threshold is met), and the device highlights object 4326 to indicate that the menu display criteria have been met. In some embodiments, if movement of the touch is not detected before the touch is lifted, the object movement criteria used to initiate the drag operation are also used to determine whether a quick action menu will be presented upon touch lifting. Figure 4C48 In the process, the end of input is detected without contact movement (e.g., the lifting of contact 4386 is detected), and in response, a quick action menu 4388 is displayed adjacent to object 4326, wherein the menu includes at least a first selectable option for opening the content item represented by object 4326 in a full-screen window of the local application of the content item (e.g., open in application), a second selectable option for opening the content item in a new side-pull window (e.g., open as a side-pull window), and a third selectable option (e.g., open as a split-screen window).
[0322] In some implementations, when a first selectable option is activated by input that meets the selection criteria (e.g., a tap), if the local application of the content item is not the currently displayed application, the device optionally switches to the local application of the content item and displays the content item in a new full-screen window of that local application. If the local application of the content item is the same as the application currently displaying the object representing the content item, the content item is opened in either the currently displayed window including the object or a new full-screen window of the currently displayed application, depending on the implementation. In some implementations, the operation performed in response to the activation of the first selectable option is the same as the operation performed when input that meets the selection criteria (e.g., a tap) is detected on the object representing the content item.
[0323] In some implementations, when the second selectable option is activated by input that meets the selection criteria (e.g., tap input), the device displays the content item in a new side pane of the content item's local application (e.g., such as...). Figure 4C23 (As shown). In some implementations, the operation performed in response to the activation of the second selectable option is the same as the operation performed when an input that meets the object movement criteria initiates a drag operation on the object and ends in the first predefined area 4308 on the display.
[0324] In some implementations, when a third selectable option is activated by input that meets the selection criteria (e.g., a tap input), the device displays the content item in a new split-screen window of the content item's native application (e.g., such as...). Figure 4C21 (As shown). In some implementations, the operation performed in response to the activation of a third selectable option is the same as the operation performed when an input that meets the object movement criteria initiates a drag operation on the object and ends in the second predefined area 4310 on the display.
[0325] Figures 4D1 to 4D19 The accompanying diagram illustrates user interface behavior according to some embodiments when an application icon is dragged and dropped into a predefined area on the display to open the application in a corresponding simultaneous display configuration (e.g., pull-out mode or split-screen mode) with the currently displayed full-screen window. Specifically, according to some embodiments, when the application corresponding to the dragged application icon has multiple windows associated with it, a window selector user interface area is displayed to allow the user to select the desired window of the application to open in simultaneous display mode. Other user interface interactions with the window selector user interface are also described. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 8A to 8E The process is described below. For ease of explanation, some embodiments of the implementation will be discussed with reference to the operation performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representation point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operation is optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to a contact detected on the touch-sensitive surface 451 when the user interface shown in the figures, along with the focus selector, is displayed on the display 450.
[0326] Figures 4D1 to 5D 5 illustrates a heuristic according to which, if multiple windows are associated with an application, when the application icon of the application is dragged over a predefined area of the display (e.g., predefined areas 4308', 4310') for opening the application in a simultaneous display configuration, a window selector area is displayed to allow the user to select a window from the multiple windows to open in the simultaneous display configuration; and according to some embodiments, if a single window is associated with the application, the single window associated with the application is displayed in the simultaneous display configuration instead of the window selector area.
[0327] like Figure 4D1As shown, when the full-screen window 4122 is displayed, input via touch 4502 is detected on the application icon 220 of the browser application in the taskbar 4006. Movement of touch 4502 is detected after the initial portion of the input meets the criteria for initiating a drag operation on the application icon (e.g., the input is a tap-to-hold or a press-to-hold input). In response to the movement of touch 4502, a representation 4504 of dragging the application icon 220 on the display is executed based on the movement of touch 4502, as shown. Figure 4D2 As shown. In Figure 4D3 When contact 4502 is dragged to a position within a predefined area for opening the side-pull window (e.g., an expanded first predefined area 4308'), the device provides visual feedback indicating that the position criteria for opening the side-pull window have been met, and if the input ends at the current position, the application will open in the side-pull window.
[0328] exist Figure 4D3 After Figure 4D4 In the first branch of the heuristic, when the application whose application icon 220 is dragged (e.g., a browser application) is currently associated with zero windows (e.g., the application is not open) or only a single window (e.g., only one recently opened window is saved in memory), the device opens the application in a side-swipe window 4506 that covers a portion of the background window 4122 (e.g., on the right side of the screen). In some embodiments, if the application is associated with zero windows, the side-swipe window 7506 displays the application's default start user interface. In some embodiments, if the application is currently associated with a single window, the side-swipe window 7506 displays the last displayed user interface or content in that single window. In some embodiments, the single window saved in memory does not necessarily have to be a side-swipe window. In some embodiments, the single window saved in memory is converted from a full-screen window or split-screen window to a side-swipe window before being displayed in response to input made via touch 4502.
[0329] exist Figure 4D3 After Figure 4D5In the second branch of the heuristic, when the application whose application icon 220 is being dragged (e.g., a browser application) is currently associated with multiple windows (e.g., multiple recently opened windows are stored in memory), the device opens a window selector user interface area 4508 (e.g., in a side-swipe window or overlay) that covers a portion of the background window 4122 (e.g., on the right side of the screen). In some implementations, all windows associated with the application can be used to view and select in the window selector user interface area 4508 (e.g., initially displayed, or displayed in response to scrolling or browsing input), regardless of the display configuration (e.g., full-screen, split-screen window, side-swipe window, draft window, minimized window, etc.).
[0330] exist Figure 4D5 In this embodiment, the window selector user interface area 4508 includes representations of windows associated with the application corresponding to the dragged application icon (e.g., representation 4510 of a first window of a browser application and representation 4512 of a second window of a browser application). The window representations include an application identifier and a unique name corresponding to each of these windows. In some embodiments, the window names are automatically generated by the device based on the content displayed in the window (e.g., title, username, subject line, etc. of a document, email, message, webpage, image, etc.). Each window representation includes a close capability representation (e.g., capability representations 4518 and 4520) for closing the window individually without closing other saved windows of the application. In some embodiments, the window selector user interface area 4508 includes a close capability representation 4514 for closing the window selector user interface area 4508 without closing saved windows of the application. In some embodiments, the window selector user interface area 4508 includes a capability representation for closing all windows associated with the application without closing the window selector user interface area 4508. In some implementations, the window selector user interface area 4508 includes a display representation 4516 for opening a new window of the application. In some implementations, the new window is displayed in a side-pull mode immediately after it is opened. In some implementations, a representation of the new window is first displayed in the window selector user interface area 4508, and in response to another user input that selects the representation of the new window, the new window is displayed only in a side-pull mode. Figures 4D6 to 4D17 Some features of the window selector user interface area 4508 are described according to some implementation schemes.
[0331] exist Figures 4D6 to 4D8In the window selector user interface area 4508, input via touch 4522 is detected on representation 4512 of window 2 of the browser application. This input includes movement of touch 4522 toward the right edge of the display. In response to the detected movement of touch 4522, representation 4512 is dragged off the display, and the window corresponding to representation 4512 closes, as... Figures 4D7 to 4D8 As shown. In Figure 4D8 In this context, only the representation of window 1 of the browser application 4510 is preserved in the window selector user interface area 4508.
[0332] Figures 4D8 to 4D9 In some embodiments, it is shown that if all windows shown in the window selector user interface area 4508 except for one window (e.g., window 1) are closed, the device stops displaying the window selector user interface area 4508 and displays a single hold window of the application in a pull-out mode (e.g., as a pull-out window 4506). Figure 4D9 As shown, user input is used to select a representation of the last held window without the need for additional selection. In some implementations, the window selector user interface area remains displayed, and user input to select the representation of the last held window (e.g., tap input) opens the last held window in a side-pull mode.
[0333] Figures 4D10 to 4D11 The device detects input via touch 4524 on a representation of a window in a window associated with the application (e.g., representation 4510), and in response to detecting the input and determining that the input meets a first criterion (e.g., the input is a tap input), the device stops displaying the window selector user interface area 4508 and displays the selected window (e.g., window 1) in a side-pull mode (e.g., as a side-pull window 4506).
[0334] Figures 4D12 to 4D13 The following are examples of implementation schemes. Figures 4D6 to 4D8 The window shown is an alternative way to close the window. For example... Figure 4D12 As shown, input via touch 4526 is detected at the location corresponding to the close indication 4520 of window 2 represented in the window selector user interface area 4508. In response to this input and based on determining that the input meets a first criterion (e.g., the input is a tap), the device stops displaying the indication 4512 of window 2 and closes window 2 of the browser application, as... Figure 4D13 As shown.
[0335] exist Figures 4D13 to 4D14In some implementations, if all windows shown in the window selector user interface area 4508 except for one window (e.g., window 1) are closed, the device stops displaying the window selector user interface area 4508 and displays a single hold window of the application in a pull-out mode (e.g., as a pull-out window 4506). Figure 4D14 As shown, user input is used to select a representation of the last held window without the need for additional selection. In some implementations, the window selector user interface area remains displayed, and user input to select the representation of the last held window (e.g., tap input) opens the last held window in a side-pull mode.
[0336] exist Figures 4D15 to 4D17 In some implementations, a series of inputs use a close enable indicator on the window representation in the window selector user interface area 4508 to individually close all windows represented in the window selector user interface 4508. For example... Figure 4D15 As shown, a tap input via touch 4528 is detected on the close indication display 4520 of window 2. In response to the input via touch 4528, the representation 4512 of window 2 is removed from the window selector user interface area 4508, and the corresponding window is closed (e.g., removed from memory). Figure 4D16 In the middle, another tap input via touch 4530 is detected on the close indication indicator 4518 of window 1. In response to the detection of input via touch 4530, indication 4524 is removed from the window selector user interface area 4508, and the corresponding window is closed (e.g., removed from memory), as... Figure 4D17 As shown.
[0337] In some implementations, interaction with the window selector user interface area (e.g., as described above) is performed... Figures 4D6 to 4D8 and Figures 4D15 to 4D17 After all windows associated with the application are closed (as described in the text), the window selector user interface area 4508 optionally remains on the display, such as Figure 4D17As shown. In some embodiments, a user can use the power indicator 4516 to open additional new windows and have them displayed in the window selector user interface area 4508. In some embodiments, after all windows in the window selector user interface area 4508 have been closed, user input (e.g., a tap input on the close power indicator 4514, or a horizontal swipe input originating outside the window selector user interface area 4508 that continues on the window selector user interface area 4508) is required to remove the area from the display. In some embodiments, after all windows displayed in the window selector user interface area 4508 have been closed, the device stops displaying the window selector user interface area 4508 without requiring input to close the window selector user interface area 4508.
[0338] Figures 4D18 to 4D19 A similar window selector user interface area 4534 is shown, according to some embodiments, when the application icon of a browser application is dragged and dropped into a second predefined area 4310' for opening the application in split-screen mode (if multiple windows are associated with the application). The window selector user interface area 4534 is optionally displayed side-by-side with the background window to indicate to the user that the selected window from the window selector user interface area 4534 will be displayed in the split-screen view along with the split-screen window 4532, which is converted from the full-screen background window 4122.
[0339] As in Figure 4D2 or Figure 4D3 After Figure 4D18 As shown, movement of touch 4502 has dragged representation 4504 into an extended second predefined area 4310' on the display, which is used to open a new split-screen window of the application on the right side of the display. In some embodiments, if input via touch 4502 ends when touch and representation 4504 are in the extended second predefined area 4310', and the browser application is associated with zero windows or a single window at this time, the device displays a new default window or single window in a split-screen configuration along with the split-screen window 4532 converted from the background window 4122. In some embodiments, if input via touch 4502 ends when touch and representation 4504 are in the extended second predefined area 4310', and the browser application is associated with multiple windows at this time, the device displays the window selector user interface area 4534 in a split-screen configuration along with the split-screen window 4532 converted from the background window 4122.
[0340] like Figure 4D19 As shown, according to some implementations, the window selector user interface area 4534 is similarly configured relative to... Figures 4D5 to 4D17The described window selector user interface area 4508. For example, the user selector user interface area 4534 includes representations of the same group (e.g., representations 4510 and 4512 of recently opened windows saved by a browser application) and power-off representations (e.g., single power-off representations 4518 and 4520, power-off representation 4514, new window power-off representation 4516, etc.). According to some embodiments, the user interface interactions described relative to the window selector user interface area 4508 also apply to the window selector user interface area 4534.
[0341] Figures 4E1 to 4E28 The diagram illustrates user interface behavior, according to some embodiments, in response to input that drags a representation of a window to different positions on the display and releases it to different placement areas on the display. For example... Figures 4E1 to 4E28 As shown, before the end of input is detected, dynamic visual feedback is provided to indicate the result of the input based on the current position of the input compared to multiple predefined placement areas on the display and the dragged representation of the window. In some embodiments, a drag operation performed on a window displayed in a corresponding simultaneous display configuration (e.g., side-pull display configuration, split-screen display configuration, minimized display configuration, draft mode display configuration, etc.) causes the window to be displayed in the same simultaneous display configuration, a different simultaneous display configuration, or an independent display configuration, depending on the position of the window's representation when the end of input is detected, such as relative to different placement areas corresponding to different simultaneous display configurations and independent display configurations (e.g., ...). Figure 4E8 The evaluation was conducted in the placement area shown in the figure. Figures 4E9 to 4E17 The device display according to some implementations is shown to indicate various intermediate states that may result in various final states if the input is to end at the current position. Figures 4E9 to 4E17 The accompanying figures also illustrate the dynamic nature of the visual feedback, based on the movement of the input and its current position relative to different placement areas on the display, before the end of input is detected, and how the intermediate state can be repeated any number of times in any order. The user interface in these figures is used to illustrate the process described below, including... Figures 9A to 9J The process is described below. For ease of explanation, some embodiments of the implementation will be discussed with reference to the operation performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representation point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operation is optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to a contact detected on the touch-sensitive surface 451 when the user interface shown in the figures, along with the focus selector, is displayed on the display 450.
[0342] Figures 4E1 to 4E7 Seven different initial states of a window (e.g., an email application window) are illustrated. For ease of explanation, the window in this example is labeled differently based on its current display configuration. The same content is displayed in the window, and the window's display configuration changes from one configuration to another due to drag-and-drop operations performed on the window. In some implementations, the initial configuration of the window includes any of a variety of configurations, including a left-side pull-out window, a right-side pull-out window, a background window with the left side covered by the pull-out window, a background window with the right side covered by the pull-out window, a right-side split-screen window, a left-side split-screen window, a draft window, a background window of the draft window, a minimized window, a full-screen window displayed simultaneously with the minimized window, a standalone full-screen window, etc. In some implementations, the final configuration of the window includes any of a variety of configurations, including a left-side pull-out window, a right-side pull-out window, a background window with the left side covered by the pull-out window, a background window with the right side covered by the pull-out window, a right-side split-screen window, a left-side split-screen window, a draft window, a background window of the draft window, a minimized window, a full-screen window displayed simultaneously with the minimized window, a standalone full-screen window, etc. The number of transitions between possible initial and final configurations is too numerous to list individually here. For illustrative purposes, representative initial and final states of possible window display configurations are described according to some embodiments. In some embodiments, either window in a pair of simultaneously displayed windows can be the subject of a drag-and-drop operation to change the window's display configuration to another state. In some embodiments, windows can be converted from an independent display configuration to a simultaneously displayed configuration, and vice versa. In some embodiments, the drag handles of simultaneously displayed windows switch between a first display state (e.g., active) and a second display state (background) depending on which of the simultaneously displayed windows has input focus.
[0343] like Figures 4E1 to 4E7 The image shows seven example starting states of the display configuration for an email application window.
[0344] exist Figure 4E1 In the example start state A of the display configuration of the email application window, the email application window is a split-screen window (e.g., window 4602) displayed simultaneously with the split-screen window 4604 of the messaging application. The email application split-screen window 4602 is displayed on the left side of the display. Input via touch 4610 is detected on the drag handle 4606 of the split-screen window 4602, and the drag handle 4606 is displayed in an active state (e.g., solid color, light color). The drag handle 4608 of the concurrently displayed split-screen window that does not have input focus is displayed in a background state (e.g., semi-transparent, dark color).
[0345] exist Figure 4E2 In the example start state B of the email application window display configuration, the email application window is a side-swipe window (e.g., window 4614) displayed simultaneously with the full-screen background window 4612 of the messaging application. The side-swipe window 4614 of the email application is displayed on the left side of the display, overlaying the background window 4612 of the messaging application. Input via touch 4610 is detected on the drag handle 4606 of the side-swipe window 4614, and the drag handle 4606 is displayed in an active state (e.g., solid color, light color). The drag handle 4608 of the simultaneously displayed full-screen background window 4612, which does not have input focus, is displayed in a background state (e.g., semi-transparent, dark color). For clarity, the same drag handle label is used when the window corresponding to the drag handle changes from one configuration to another.
[0346] exist Figure 4E3 In the example start state C of the email application window display configuration, the email application window is a draft window (e.g., window 4615) overlaid on the full-screen background window 4612 of the messaging application. The email application draft window 4615 is displayed in the center area of the display and shows an editable draft of the email document. Input via touch 4610 is detected on the drag handle 4606 of the draft window 4615, and the drag handle 4606 is displayed in an active state (e.g., solid color, light color). The drag handle 4608 of the background window 4612, which does not have input focus, is displayed in a background state (e.g., semi-transparent, dark color).
[0347] exist Figure 4E4 In the example start state D of the email application window display configuration, the email application window is a minimized window (e.g., window 4616) displayed on the periphery of the full-screen window 4612 of the messaging application. The minimized window 4616 of the email application does not display the content of the email application. Input via touch 4610 is detected on the minimized window 4615, which does not have a visible drag handle. The drag handle 4608 of the full-screen window 4612, which is displayed simultaneously without input focus, is displayed in a background state (e.g., semi-transparent, dark).
[0348] exist Figure 4E5In the example start state E of the display configuration for the email application window, the email application window is a split-screen window (e.g., window 4602) displayed simultaneously with the split-screen window 4604 of the messaging application. The split-screen window 4602 of the email application is displayed on the right side of the display. Input via touch 4610 is detected on the drag handle 4606 of the split-screen window 4602, and the drag handle 4606 is displayed in an active state (e.g., solid color, light color). The drag handle 4608 of the concurrently displayed split-screen window that does not have input focus is displayed in a background state (e.g., semi-transparent, dark color).
[0349] exist Figure 4E6 In the example start state F of the email application window display configuration, the email application window is a side-swipe window (e.g., window 4614) displayed simultaneously with the full-screen background window 4612 of the messaging application. The side-swipe window 4614 of the email application is displayed on the right side of the display, covering the background window 4612 of the messaging application. Input via touch 4610 is detected on the drag handle 4606 of the side-swipe window 4614, and the drag handle 4606 is displayed in an active state (e.g., solid color, light color). The drag handle 46...
Claims
1. A method comprising: At an electronic device including a display generating component and one or more input devices: displaying, via the display generating component, a first user interface including a selectable representation of first content, wherein the first content is associated with a first application, and the first user interface is a user interface of a second application different from the first application; while displaying the first user interface including the selectable representation of the first content, detecting a first input, including detecting an input of dragging the selectable representation of the first content from within the first user interface of the second application to a corresponding location; as well as In response to detecting the first input of dragging the selectable representation of the first content from within the first user interface of the second application to a corresponding location: According to determining that the corresponding position is the first position, adjusting the size of the first user interface and displaying a second user interface of the first application program including the first content and adjacent to the first user interface of the second application program; as well as Based on determining that the corresponding position is a second position, the second position is in a third user interface of a third application overlaid on the first user interface of the second application, displaying a second representation of the first content within the third user interface of the third application overlaid on the first user interface of the second application, wherein the second position is different from the first position.
2. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the first position, reducing a size of the first user interface.
3. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the second position and based on determining that the first user interface is not displayed in a simultaneous display configuration with another application, reducing the size of the first user interface by a first amount.
4. The method according to claim 3, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the first position and based on determining that the first user interface is not displayed in a simultaneous display configuration with another application, reducing the size of the first user interface by a second amount that is greater than the first amount, and wherein the size of the first user interface is reduced by different amounts on two opposite sides of the first user interface.
5. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Changing the appearance of the selectable representation of the first content according to the current position of the selectable representation includes: Based on determining that the current position of the selectable representation is the first position, displaying the selectable representation of the first content in a first appearance; and Based on determining that the current position of the selectable representation is the second position, the selectable representation of the first content is displayed in a second appearance that is different from the first appearance.
6. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the first position and based on determining that the first user interface is not displayed in a simultaneous display configuration with another application, revealing a portion of the background behind the first user interface to indicate that if termination of the first input is to be detected, a new user interface including the first content will be displayed simultaneously with the first user interface.
7. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the first position: displaying, concurrently with the selectable representation of the first content, a first application identifier of the first application for opening the first content; and visually obscuring the selectable representation of the first content, without visually obscuring the first application logo.
8. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the first position, the selectable representation of the first content is resized such that a gap exists between the selectable representation of the first content and the resized first user interface.
9. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: Based on determining that the current position of the selectable representation is the second position, visually obscuring the selectable representation of the first content without visually obscuring the first user interface.
10. The method according to claim 1, comprising: In response to detecting the first input and before detecting termination of the first input: in response to determining that the current position of the selectable representation is the first position, displaying first visual feedback to indicate that if termination of the first input is detected at the current time, the first content will be displayed simultaneously with the first user interface in the window; as well as Based on determining that the current position of the selectable representation is not the first position, ceasing to display the first visual feedback to indicate that the first content will not be displayed simultaneously with the first user interface in a window if termination of the first input is detected at a current time.
11. The method according to claim 1, comprising: while displaying the first user interface including the selectable representation of the first content, detecting a second input, including detecting an input that satisfies an activation criterion; as well as In response to detecting the second input, display of the first user interface is replaced with display of a fourth user interface including the first content.
12. The method of claim 1 , wherein detecting the first input comprises: detecting a tap-and-hold input that enables a drag operation to be performed on the selectable representation in the first user interface; as well as A drag input is detected subsequent to the tap-and-hold input, the drag input moving the selectable representation or a copy of the selectable representation from an original position of the selectable representation in the first user interface to a predefined side portion of a display area provided by the display generation component.
13. The method according to claim 1, comprising: In response to detecting the first input: According to determining that the corresponding position is a third position different from the first position and the second position, the display of the first user interface is maintained without displaying the first content.
14. The method of claim 1, wherein the first user interface is a user interface of an email application and the first content is an email message.
15. The method of claim 1, wherein the first user interface is a user interface of an email application and the first content is an attachment to an email message.
16. The method of claim 1, wherein the first user interface comprises a simultaneous display of a file list of a file management application and a user interface of the second application, and wherein the first content is a document listed in the file list of the file management application.
17. The method according to claim 1, comprising: while displaying the first user interface including the selectable representation of the first content, detecting a third input, including detecting an input that satisfies a menu display criterion; as well as In response to detecting the third input, one or more selectable options are displayed for performing an operation regarding the first content, including a first selectable option that, when activated, causes the electronic device to display the first content in a new window together with the first user interface.
18. The method according to claim 1, comprising: while displaying the third user interface overlying a portion of the first user interface, detecting a fourth input, including detecting an input corresponding to a request to move the third user interface relative to the first user interface; as well as In response to detecting the fourth input and based on determining that the fourth input satisfies a window closing criterion, ceasing to display the third user interface while maintaining display of the first user interface.
19. The method according to claim 1, comprising: In response to detecting the first input: Based on determining that the corresponding position is the first position, a close enable indication is displayed simultaneously with the second user interface, wherein the close enable indication closes the second user interface when activated and restores the first user interface to a size before the second user interface is displayed.
20. The method according to claim 1, comprising: In response to detecting the first input: Based on determining that the corresponding position is the first position, a send enable indication is displayed simultaneously with the second user interface, wherein the send enable indication closes the second user interface when activated and displays a user interface for sending the first content to a recipient.
21. The method according to claim 1, comprising: In response to detecting the first input: Based on determining that the corresponding position is a third position, an operation corresponding to the first content in the first application is performed.
22. The method according to claim 1, comprising: In response to detecting the first input: Based on determining that the corresponding position is a fourth position, an operation corresponding to the first content in the second application is performed.
23. A computer-readable storage medium storing executable instructions, which, when executed by an electronic device having one or more processors and a display generation component, causes the electronic device to perform the method according to any one of claims 1 to 22.
24. An electronic device comprising: one or more processors; Display generated components; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 22.
