Multi-participant real-time communication user interface

By dynamically adjusting participant representation and real-time communication session notifications, the complexity and inefficiency of multi-participants' real-time communication interface in the prior art is solved, achieving a more efficient user interface and longer battery life.

CN110457104BActive Publication Date: 2024-12-06APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910704856.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-06-12
Filing Date
2018-09-28
Publication Date
2024-12-06
Estimated Expiration
2038-09-28

AI Technical Summary

Technical Problem

The prior art user interface for real-time communication of multiple participants is complex and inefficient, resulting in wasted user time and device energy, especially in battery-driven devices.

Method used

Provides a faster and more efficient user interface and method to display the instant messaging user interface of the instant messaging application, including message areas, notifications of activity status for real-time communication sessions, and dynamically adjust the size of participant representation based on the number of participants and location, reducing user cognitive burden and saving power.

Benefits of technology

Improves the efficiency and user experience of real-time communication among multiple participants, reduces the burden on users and devices, especially in battery-driven devices to extend battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110457104B_ABST
    Figure CN110457104B_ABST
Patent Text Reader

Abstract

The present invention is entitled "Multi-Participant Real-Time Communication User Interface". The present invention provides a method, the method comprising: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface comprising a message area, the message area comprising a plurality of messages in a message conversation between three or more participants. Receive first data indicating that a real-time communication session available to the three or more participants is active. In response to receiving the first data, display a notification indicating that the real-time communication session is active. While displaying the notification, receive second data indicating that the real-time communication session between the three or more participants is no longer active. In response to receiving the second data, stop displaying or update the notification indicating that the real-time communication session is active to indicate that the real-time communication session is no longer active.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with the application date of September 28, 2018, application number 201811136766.7, and name “Multi-Participant Real-Time Communication User Interface”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application is related to U.S. Provisional Patent Application Serial No. 62 / 668,229, filed on May 7, 2018, entitled “MULTI-PARTICIPANT LIVE COMMUNICATION USER INTERFACE,” the contents of which are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for real-time communications involving multiple participants. Background Art

[0005] For personal and business reasons, users are increasingly using electronic devices to communicate virtually with others. Most electronic devices are capable of providing real-time communication between a user and multiple participants. For example, some electronic devices provide interactive interfaces for displaying and controlling real-time communication. Summary of the Invention

[0006] However, some technologies for real-time communication involving multiple participants using electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may include multiple keystrokes or button presses. These technologies require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Thus, the present technology provides electronic devices with faster, more efficient methods and interfaces for real-time communications involving multiple participants. Such methods and interfaces optionally supplement or replace other methods for real-time communications involving multiple participants. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces conserve power and increase the time between battery charges.

[0008] In some embodiments, a method is performed at an electronic device having a display. The method may include: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including a plurality of messages in a message conversation between three or more participants; receiving first data indicating that a real-time communication session available to the three or more participants is active; in response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, ceasing to display the notification or updating the notification to indicate that the real-time communication session is no longer active.

[0009] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including a plurality of messages in a message conversation between three or more participants; receiving first data indicating that a real-time communication session available to the three or more participants is active; in response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, stopping displaying the notification or updating the notification to indicate that the real-time communication session is no longer active.

[0010] In some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for the following operations: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including a plurality of messages in a message conversation between three or more participants; receiving first data indicating that a real-time communication session available to the three or more participants is active. In response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, stopping displaying the notification or updating the notification to indicate that the real-time communication session is no longer active.

[0011] In some embodiments, the electronic device includes a display, one or more processors, and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area, the message area including a plurality of messages in a message conversation between three or more participants; receiving first data indicating that a real-time communication session available to the three or more participants is active. In response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, stopping displaying the notification or updating the notification to indicate that the real-time communication session is no longer active.

[0012] In some embodiments, the electronic device includes: a display; a device for displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area, the message area including multiple messages in a message conversation between three or more participants; a device for receiving first data indicating that a real-time communication session available to the three or more participants is active; a device for performing the following operations in response to receiving the first data indicating that the real-time communication session between the three or more participants is active: displaying a notification indicating that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, a device for performing the following operations: receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and a device for performing the following operations: in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, stopping displaying the notification or updating the notification to indicate that the real-time communication session is no longer active.

[0013] In some embodiments, a method is performed at an electronic device having a display. The method may include: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving the request to display the user interface for the real-time communication session between the two or more participants, displaying the real-time communication user interface, including simultaneously displaying representations of multiple participants in the real-time communication session, wherein displaying the representations of the multiple participants in the real-time communication session includes: based on a determination that the number of participants in the real-time communication session includes a number of participants greater than a threshold, simultaneously displaying representations of a first plurality of participants other than a user of the electronic device at one or more sizes greater than a corresponding threshold size, while displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at sizes smaller than the corresponding threshold size; and based on a determination that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying representations of participants other than the user of the electronic device at one or more sizes greater than the corresponding threshold size, while not displaying representations of participants other than the user of the electronic device at one or more sizes smaller than or equal to the corresponding threshold size.

[0014] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving the request to display the user interface for the real-time communication session between the two or more participants, displaying the real-time communication user interface, including simultaneously displaying representations of multiple participants in the real-time communication session, wherein displaying the representations of the multiple participants in the real-time communication session includes: based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, simultaneously displaying representations of a first plurality of participants other than the user of the electronic device at one or more sizes greater than a corresponding threshold size, while displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying representations of participants other than the user of the electronic device at one or more sizes greater than the corresponding threshold size, while not displaying representations of participants other than the user of the electronic device at one or more sizes less than or equal to the corresponding threshold size.

[0015] In some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving the request to display the user interface for the real-time communication session between the two or more participants, displaying the real-time communication user interface, including simultaneously displaying representations of multiple participants in the real-time communication session, wherein displaying the representations of the multiple participants in the real-time communication session includes: based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, simultaneously displaying representations of a first plurality of participants other than the user of the electronic device at one or more sizes greater than a corresponding threshold size, while displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying representations of participants other than the user of the electronic device at one or more sizes greater than the corresponding threshold size, while not displaying representations of participants other than the user of the electronic device at one or more sizes less than or equal to the corresponding threshold size.

[0016] In some embodiments, an electronic device includes a display, one or more processors, and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving the request to display the user interface for the real-time communication session between the two or more participants, displaying the real-time communication user interface, including simultaneously displaying multiple participant representations in the real-time communication session, wherein displaying the multiple participant representations in the real-time communication session includes: based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, simultaneously displaying representations of a first plurality of participants other than the user of the electronic device at one or more sizes greater than a corresponding threshold size, while displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying representations of participants other than the user of the electronic device at one or more sizes greater than the corresponding threshold size, while not displaying representations of participants other than the user of the electronic device at one or more sizes less than or equal to the corresponding threshold size.

[0017] In some embodiments, the electronic device includes: a display; a device for receiving a request to display a user interface for a real-time communication session between two or more participants; and a device for the following operations in response to receiving the request to display the user interface for the real-time communication session between the two or more participants: displaying the real-time communication user interface, including simultaneously displaying multiple participant representations in the real-time communication session, wherein displaying the multiple participant representations in the real-time communication session includes: a device for the following operations, based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold: simultaneously displaying representations of a first plurality of participants other than the user of the electronic device at one or more sizes greater than a corresponding threshold size, and displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size; and a device for the following operations, based on determining that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying representations of participants other than the user of the electronic device at one or more sizes greater than the corresponding threshold size, and not displaying representations of participants other than the user of the electronic device at one or more sizes less than or equal to the corresponding threshold size.

[0018] In some embodiments, a method is performed at an electronic device having a display. The method may include: displaying a real-time video communication user interface that includes simultaneously displaying a representation of a first participant in a real-time video communication session, a representation of a second participant in the real-time video communication session, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving data indicating that an activity level of the second participant in the real-time video communication session has increased above an activity level of the first participant in the real-time video communication session; and in response to receiving the data indicating that the activity level of the second participant in the real-time video communication session has increased above an activity level of the first participant in the real-time video communication session: changing the size and / or position of the representation of the first participant in the real-time video communication user interface; changing the size and / or position of the representation of the second participant in the real-time video communication user interface; and changing the size and / or position of the representation of the third participant in the real-time video communication user interface.

[0019] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying a real-time video communication user interface that includes simultaneously displaying a representation of a first participant in a real-time video communication session, a representation of a second participant in the real-time video communication session, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving data indicating that an activity level of the second participant in the real-time video communication session has increased above an activity level of the first participant in the real-time video communication session; and in response to receiving the data indicating that the activity level of the second participant in the real-time video communication session has increased above an activity level of the first participant in the real-time video communication session: changing the size and / or position of the representation of the first participant in the real-time video communication user interface; changing the size and / or position of the representation of the second participant in the real-time video communication user interface; and changing the size and / or position of the representation of the third participant in the real-time video communication user interface.

[0020] In some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying a real-time video communication user interface that includes simultaneously displaying a representation of a first participant in a real-time video communication session, a representation of a second participant in the real-time video communication session, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving data indicating that an activity level of the second participant in the real-time video communication session has increased to a level higher than an activity level of the first participant in the real-time video communication session; and in response to receiving the data indicating that the activity level of the second participant in the real-time video communication session has increased to a level higher than an activity level of the first participant in the real-time video communication session: changing the size and / or position of the representation of the first participant in the real-time video communication user interface; changing the size and / or position of the representation of the second participant in the real-time video communication user interface; and changing the size and / or position of the representation of the third participant in the real-time video communication user interface.

[0021] In some embodiments, the electronic device includes a display, one or more processors, and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a real-time video communication user interface, which includes simultaneously displaying a representation of a first participant in the real-time video communication session, a representation of a second participant in the real-time video communication session, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving data indicating that the activity level of the second participant in the real-time video communication session has increased to a level higher than the activity level of the first participant in the real-time video communication session; and in response to receiving the data indicating that the activity level of the second participant in the real-time video communication session has increased to a level higher than the activity level of the first participant in the real-time video communication session: changing the size and / or position of the representation of the first participant in the real-time video communication user interface; changing the size and / or position of the representation of the second participant in the real-time video communication user interface; and changing the size and / or position of the representation of the third participant in the real-time video communication user interface.

[0022] In some embodiments, the electronic device includes: a display; a device for displaying a real-time video communication user interface, displaying the real-time video communication user interface including simultaneously displaying a representation of a first participant in the real-time video communication session, a representation of a second participant in the real-time video communication session, and a representation of a third participant in the real-time video communication session; a device for performing the following operations: while displaying the real-time video communication user interface, receiving data indicating that the activity level of the second participant in the real-time video communication session has increased to be higher than the activity level of the first participant in the real-time video communication session; and a device for performing the following operations in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased to be higher than the activity level of the first participant in the real-time video communication session: changing the size and / or position of the representation of the first participant in the real-time video communication user interface; changing the size and / or position of the representation of the second participant in the real-time video communication user interface; and changing the size and / or position of the representation of the third participant in the real-time video communication user interface.

[0023] In some embodiments, a method is performed at an electronic device having a display. The method may include: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including a plurality of messages between participants in a message conversation including three or more participants; and an affordance separate from the message area for starting a real-time video communication session in a real-time video communication application; detecting an input corresponding to selection of the affordance; and in response to detecting the input corresponding to selection of the affordance, initiating a process in the real-time video communication application for starting a real-time video communication session including the participants in the message conversation.

[0024] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including a plurality of messages between participants in a message conversation including three or more participants; and an affordance separate from the message area for starting a real-time video communication session in a real-time video communication application; detecting an input corresponding to selection of the affordance; and in response to detecting the input corresponding to selection of the affordance, initiating a process for starting a real-time video communication session in the real-time video communication application, the real-time video communication session including the participants in the message conversation.

[0025] In some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including a plurality of messages between participants in a message conversation including three or more participants; and an affordance separate from the message area for starting a real-time video communication session in a real-time video communication application; detecting an input corresponding to selection of the affordance; and in response to detecting the input corresponding to selection of the affordance, initiating a process for starting a real-time video communication session in the real-time video communication application, the real-time video communication session including the participants in the message conversation.

[0026] In some embodiments, the electronic device includes a display, one or more processors, and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including multiple messages between participants in a message conversation including three or more participants; and an enable representation separated from the message area for starting a real-time video communication session in a real-time video communication application; detecting an input corresponding to a selection of the enable representation; and in response to detecting an input corresponding to a selection of the enable representation, initiating a process for starting a real-time video communication session in the real-time video communication application, the real-time video communication session including the participants of the message conversation.

[0027] In some embodiments, the electronic device includes: a display; a device for displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including multiple messages between participants in a message conversation including three or more participants; and an enable representation separate from the message area for starting a real-time video communication session in a real-time video communication application; a device for detecting an input corresponding to selection of the enable representation; and a device for the following operations in response to detecting the input corresponding to selection of the enable representation: initiating a process for starting a real-time video communication session in the real-time video communication application, the real-time video communication session including the participants of the message conversation.

[0028] In some embodiments, a method is performed at an electronic device having a display. The method may include: displaying a real-time video communication user interface of a real-time video communication application, the real-time video communication user interface including: a plurality of participant affordances corresponding to respective one or more participants, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new session affordance for starting a new real-time video communication session; detecting input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a process to start a new real-time video communication session based on input corresponding to selection of the first participant affordance, the new real-time video communication session including the group of three or more participants; and initiating a process for selecting two or more participants and starting a new real-time video communication session based on input corresponding to selection of the new session affordance, the new real-time video communication session including the participants associated with the device and the two or more selected participants.

[0029] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying a real-time video communication user interface of a real-time video communication application, the real-time video communication user interface including: a plurality of participant enable representations corresponding to respective one or more participants, the plurality of participant enable representations including a first participant enable representation corresponding to a group of three or more participants; and a new session enable representation for starting a new real-time video communication session; detecting an input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a process to start a new real-time video communication session according to an input corresponding to selection of the first participant enable representation, the new real-time video communication session including the group of three or more participants; and initiating a process for selecting two or more participants and starting a new real-time video communication session according to an input corresponding to selection of the new session enable representation, the new real-time video communication session including the participants associated with the device and the two or more selected participants.

[0030] In some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: displaying a real-time video communication user interface of a real-time video communication application, the real-time video communication user interface including: a plurality of participant affordances corresponding to respective one or more participants, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new session affordance for starting a new real-time video communication session; detecting an input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a process to start a new real-time video communication session based on an input corresponding to selection of the first participant affordance, the new real-time video communication session including a group of three or more participants; and initiating a process for selecting two or more participants and starting a new real-time video communication session based on an input corresponding to selection of the new session affordance, the new real-time video communication session including the participants associated with the device and the two or more selected participants.

[0031] In some embodiments, the electronic device includes a display, one or more processors and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a real-time video communication user interface of a real-time video communication application, the real-time video communication user interface including: a plurality of participant enable representations corresponding to the corresponding one or more participants, the plurality of participant enable representations including a first participant enable representation corresponding to a group of three or more participants; and a new session enable representation for starting a new real-time video communication session; detecting an input corresponding to the following request, i.e., initiating a process for starting a new real-time video communication session; and in response to detecting the input: initiating a process for starting a new real-time video communication session according to an input corresponding to the selection of the first participant enable representation, the new real-time video communication session including a group of three or more participants; and initiating a process for selecting two or more participants and starting a new real-time video communication session according to an input corresponding to the selection of the new session enable representation, the new real-time video communication session including the participants associated with the device and the two or more selected participants.

[0032] In some embodiments, the electronic device includes: a display; a device for displaying a real-time video communication user interface of a real-time video communication application, the real-time video communication user interface including: multiple participant enable representations corresponding to corresponding one or more participants, the multiple participant enable representations including a first participant enable representation corresponding to a group of three or more participants; and a new session enable representation for starting a new real-time video communication session; a device for detecting an input corresponding to the following request, namely, initiating a process for starting a new real-time video communication session; and a device for the following operations in response to detecting the input: initiating a process for starting a new real-time video communication session based on the input corresponding to the selection of the first participant enable representation, the new real-time video communication session including a group of three or more participants; and initiating a process for selecting two or more participants and starting a new real-time video communication session based on the input corresponding to the selection of the new session enable representation, the new real-time video communication session including the participants associated with the device and the two or more selected participants.

[0033] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0034] Thus, a faster and more efficient method and interface is provided for devices for real-time communication involving multiple participants, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace other methods for real-time communication involving multiple participants. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0036] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0037] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0038] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0039] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0040] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0041] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0042] Figure 5A A personal electronic device according to some embodiments is shown.

[0043] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0044] Figure 5C-5D Example components of a personal electronic device with a touch-sensitive display and an intensity sensor are shown according to some embodiments.

[0045] Figure 5E-5H Exemplary components and user interfaces of a personal electronic device according to some embodiments are shown.

[0046] Figures 6A to 6AF An exemplary user interface according to some embodiments is shown.

[0047] 7A to 7F Exemplary methods according to some embodiments are shown.

[0048] Figures 8A to 8BK An exemplary user interface according to some embodiments is shown.

[0049] Figures 9A to 9K Exemplary methods according to some embodiments are shown.

[0050] Figures 10A to 10S An exemplary user interface according to some embodiments is shown.

[0051] Figures 11A to 11F Exemplary methods according to some embodiments are shown.

[0052] Figures 12A to 12N An exemplary user interface according to some embodiments is shown.

[0053] 13A to 13D Exemplary methods according to some embodiments are shown.

[0054] 14A to 14I An exemplary user interface according to some embodiments is shown.

[0055] Figures 15A to 15C Exemplary methods according to some embodiments are shown. DETAILED DESCRIPTION

[0056] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0057] Electronic devices need to provide efficient methods and interfaces for real-time communications involving multiple participants. Such technologies can reduce the cognitive burden on users participating in real-time communications involving multiple participants, thereby improving productivity. Furthermore, such technologies can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0058] under, Figure 1A to Figure 1B 、 Figure 2 、 Figure 3 、 Figures 4A to 4B and Figures 5A to 5H A description of exemplary devices for executing techniques for real-time communication sessions between multiple participants is provided. Figures 6A to 6AF An exemplary user interface for managing event notifications is shown. 7A to 7F A flow chart illustrating a method of conducting a real-time communication session between multiple participants according to some embodiments. Figures 6A to 6AF The user interface in the 7A to 7F The process is described below.

[0059] Figures 8A to 8BK An exemplary user interface for a real-time communication session between multiple participants is shown. Figures 9A to 9K A flow chart illustrating a method of conducting a real-time communication session between multiple participants according to some embodiments. Figures 8A to 8BK The user interface in the Figures 9A to 9K The process is described below.

[0060] Figures 10A to 10S An exemplary user interface for a real-time communication session between multiple participants is shown. Figures 11A to 11F A flow chart illustrating a method of conducting a real-time communication session between multiple participants according to some embodiments. Figures 10A to 10S The user interface in the Figures 11A to 11F The process is described below.

[0061] Figures 12A to 12N An exemplary user interface for a real-time communication session between multiple participants is shown. 13A to 13D A flow chart illustrating a method of conducting a real-time communication session between multiple participants according to some embodiments. Figures 12A to 12N The user interface in the 13A to 13D The process is described below.

[0062] 14A to 14I An exemplary user interface for a real-time communication session between multiple participants is shown. Figures 15A to 15C A flow chart illustrating a method of conducting a real-time communication session between multiple participants according to some embodiments. 14A to 14I The user interface in the Figures 15A to 15C The process is described below.

[0063] Although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first touch may be named a second touch, and similarly, a second touch may be named a first touch, without departing from the scope of the various embodiments described. Both the first touch and the second touch are touches, but they are not the same touch.

[0064] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0065] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0066] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). equipment, equipment, and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0067] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0068] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0069] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0070] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and external ports 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0071] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0072] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0073] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 100 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0074] Memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0075] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0076] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with a network and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as via a short-range communication radio. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11d), IEEE 802.11e, IEEE 802.11f, IEEE 802.11g, IEEE 802.11g), IEEE 802.11f, IEEE 802.11g ... 11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0077] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves audible to the human ear. The audio circuit 110 also receives an electrical signal converted from the sound wave by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0078] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripherals interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, a depth camera controller 169, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in the ).

[0079] A quick press of the push button optionally unlocks the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. patent application Ser. No. 11 / 322,549, filed Dec. 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," (U.S. Pat. No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0080] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0081] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0082] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® Touch Controller from Apple Inc. (Cupertino, California). and iPod The technology used in

[0083] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, while a touch-sensitive trackpad does not provide visual output.

[0084] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following applications: (1) U.S. patent application No. 11 / 381,313, entitled “Multipoint Touch Surface Controller,” filed on May 2, 2006; (2) U.S. patent application No. 10 / 840,862, entitled “Multipoint Touchscreen,” filed on May 6, 2004; (3) U.S. patent application No. 10 / 903,964, entitled “Gestures For Touch Sensitive Input Devices,” filed on July 30, 2004; (4) U.S. patent application No. 11 / 048,264, entitled “Gestures For Touch Sensitive Input Devices,” filed on January 31, 2005; (5) U.S. patent application No. 11 / 050,264, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices,” filed on January 18, 2005 (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, entitled “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, entitled “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, entitled “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 228,768, filed March 3, 2006, entitled “Multi-Functional Hand-Held Computer with a Touch Screen Interface”. All of these applications are incorporated herein by reference in their entirety.

[0085] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, a finger, or the like, to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the user's desired action.

[0086] In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0087] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0088] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user while viewing other video conference participants on the touch screen display for video conferencing. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0089] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1A A depth camera sensor is shown coupled to a depth camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of objects (e.g., faces) within the scene from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map for different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located on the front of the device 100, enabling the user to optionally capture an image of the user with depth information for video conferencing while viewing other video conference participants on the touchscreen display, and to capture selfies with depth map data. In some embodiments, the depth camera sensor 175 is located on the rear of the device, or on both the rear and front of the device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), enabling the depth camera sensor 175 to be used in conjunction with the touchscreen display for both video conferencing and still and / or video image acquisition.

[0090] In some embodiments, a depth map (e.g., a depth map image) includes information (e.g., values) related to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position of its corresponding two-dimensional pixel in the Z axis of the viewpoint. In some embodiments, a depth map is composed of pixels, where each pixel is defined by a value (e.g., 0 to 255). For example, a value of "0" represents the pixel that is farthest from the viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in a "three-dimensional" scene, and a value of "255" represents the pixel that is closest to the viewpoint in a "three-dimensional" scene. In other embodiments, a depth map represents the distance between objects in a scene and the plane of the viewpoint. In some embodiments, a depth map includes information about the relative depths of various features of an object of interest in the field of view of a depth camera (e.g., the relative depths of the eyes, nose, mouth, ears of a user's face). In some embodiments, a depth map includes information that enables the device to determine the outline of an object of interest in the z direction.

[0091] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100 opposite touch screen display 112 located on the front of device 100.

[0092] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: No. 11 / 241,839, entitled “Proximity Detector In Handheld Device”; No. 11 / 240,788, entitled “Proximity Detector In Handheld Device”; No. 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; No. 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and No. 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals”, which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0093] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile output on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0094] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0095] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state indicating which applications, if any, are currently active; a display state indicating what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state including information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture.

[0096] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0097] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external ports are connected to (trademark of Apple Inc.) devices.

[0098] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a trackpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single-finger contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multiple-finger contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0099] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, a mouse "click" threshold for a touchpad or touchscreen can be set to any one of a large range of predefined thresholds without changing the touchpad or touchscreen display hardware. In addition, in some embodiments, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0100] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0101] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0102] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0103] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0104] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts module 137, email client module 140, IM module 141, browser module 147, and any other application requiring text input).

[0105] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone module 138 for use in location-based dialing; to the camera module 143 as picture / video metadata; and to applications that provide location-based services, such as the weather widget, the local yellow pages widget, and the map / navigation widget).

[0106] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0107] Contacts module 137 (sometimes referred to as address book or contact list);

[0108] Telephone module 138;

[0109] Video conferencing module 139;

[0110] Email client module 140;

[0111] Instant messaging (IM) module 141;

[0112] Fitness support module 142;

[0113] A camera module 143 for still and / or video images;

[0114] Image management module 144;

[0115] Video player module;

[0116] Music player module;

[0117] Browser module 147;

[0118] Calendar module 148;

[0119] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6;

[0120] A widget creator module 150 for forming user-created widgets 149-6;

[0121] Search module 151;

[0122] Video and music player module 152, which combines the video player module and the music player module;

[0123] Notepad module 153;

[0124] Map module 154; and / or

[0125] Online video module 155.

[0126] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0127] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications through the telephone module 138, video conferencing module 139, email client module 140, or instant messaging module 141; and so on.

[0128] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0129] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the touch / motion module 130, the graphics module 132, the text input module 134, the contacts module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0130] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0131] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messaging, or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0132] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0133] In combination with the touch screen 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0134] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing) or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0135] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0136] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0137] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., the Yahoo! desktop widget).

[0138] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page into a desktop widget).

[0139] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0140] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0141] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0142] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0143] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos. Additional descriptions of online video applications can be found in U.S. provisional patent application 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. patent application 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0144] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as separate software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0145] In some embodiments, device 100 is a device that performs operations on a predefined set of functions on it solely through a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0146] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0147] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., an application corresponding to any one of the modules 137 to 151, 155, 380 to 390).

[0148] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0149] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information ready for display by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0150] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (via audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0151] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0152] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0153] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0154] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0155] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0156] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0157] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0158] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In other embodiments, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0159] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 is part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0160] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0161] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0162] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0163] In some embodiments, event definition 186 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0164] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0165] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0166] In some embodiments, the corresponding event identifier 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event identifiers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event identifiers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0167] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and performs a predefined process.

[0168] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0169] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0170] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0171] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0172] Figure 2A portable multifunction device 100 with a touch screen 112 is shown according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and in other embodiments described below, a user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figures) or using one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0173] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0174] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts speech input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0175] Figure 33 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The tactile output generator 167 described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference Figure 1A The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores data related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0176] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the memory device mentioned previously.Each module in the above-mentioned modules corresponds to the instruction set for performing the above-mentioned functions.Above-mentioned modules or programs (for example, instruction set) need not be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0177] Attention is now turned to an embodiment of a user interface, optionally implemented on, for example, portable multifunction device 100 .

[0178] Figure 4A An exemplary user interface is shown for a menu of applications on portable multifunction device 100 according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0179] One or more signal strength indicators 402 of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0180] Time 404;

[0181] Bluetooth indicator 405;

[0182] Battery status indicator 406;

[0183] A tray 408 with icons of commonly used applications, such as:

[0184] o An icon 416 labeled "Phone" for the phone module 138, which optionally includes an indicator 414 of the number of missed calls or voice messages;

[0185] o An icon 418 of the email client module 140 labeled "Mail," which optionally includes an indicator 410 of the number of unread emails;

[0186] o An icon 420 labeled "Browser" for the browser module 147; and

[0187] o An icon 422 labeled “iPod” for the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0188] Icons for other apps, such as:

[0189] o Icon 424 labeled "Messages" of the IM module 141;

[0190] o An icon 426 labeled “Calendar” of the calendar module 148;

[0191] o Icon 428 labeled "Photos" of the image management module 144;

[0192] o An icon 430 labeled “Camera” of the camera module 143;

[0193] ○ Icon 432 labeled “Online Video” of the online video module 155;

[0194] ○ Icon 434 labeled “Stock Market” of the Stock Market desktop applet 149-2;

[0195] o An icon 436 labeled “Map” of the map module 154;

[0196] ○ Icon 438 labeled “Weather” of the weather widget 149-1;

[0197] ○ Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0198] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0199] o An icon 444 of the Notes module 153 labeled "Notes"; and

[0200] o An icon 446 labeled “Settings” for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0201] It should be pointed out that Figure 4A The icon labels shown are exemplary only. For example, icon 422 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0202] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet or trackpad 355) of the device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0203] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B , 460 corresponds to 468 and 462 corresponds to 470 ) at contact with touch-sensitive surface 451 (e.g., Figure 4B 460 and 462 in FIG. 4 ). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of the multi-function device ( Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0204] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0205] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include a body 502 relative to the devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in . In some embodiments, device 500 has a touch-sensitive display screen, referred to hereinafter as touch screen 504. Alternatively or in addition to touch screen 504, device 500 also has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of the contact (e.g., touch) applied. The one or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on their intensity, which means that touches of different intensities can invoke different user interface operations on device 500.

[0206] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0207] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0208] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a Figure 1A 、 Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. The I / O portion 514 can be connected to a touch screen 504, which can have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, the I / O portion 514 can be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, the input mechanism 508 is optionally a button.

[0209] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to I / O portion 514.

[0210] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including processes 700, 900, 1100, 1300, and 1500. 7A to 7F 、 Figures 9A to 9K 、 Figures 11A to 11F 、 13A to 13D as well as Figures 15A to 15C ). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5Bcomponents and configurations, but may include other components or additional components in a variety of configurations.

[0211] As used herein, the term "indicator" refers to an indication that is optionally provided on device 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figures 5A to 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0212] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in FIG, the particular user interface element is adjusted according to the detected input. In the case of a touch screen display (e.g., a touch screen display) that enables direct interaction with user interface elements on the touch screen display Figure 1A touch-sensitive display system 112 or Figure 4A In some implementations of the touch screen 112 in FIG. 1 , a detected contact on the touch screen acts as a “focus selector” such that when input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0213] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0214] Figure 5C Detecting multiple contacts 552A-552E on touch screen 504 using multiple intensity sensors 524A-524D is shown. Figure 5C Also included is an intensity graph that shows the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are both 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are both 7 intensity units. In some implementations, the cumulative intensity is the sum of the intensity measurements of multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, which is a portion of the cumulative intensity. Figure 5D554. The cumulative strengths assigned to contacts 552A-552E based on their distances from the center of force 554 are shown. In this example, each of contacts 552A, 552B, and 552E is assigned a strength of the contact of 8 strength units of the cumulative strength, and each of contacts 552C and 552D is assigned a strength of the contact of 4 strength units of the cumulative strength. More generally, in some implementations, each contact j is assigned a corresponding strength Ij that is a fraction of the cumulative strength A according to a predefined mathematical function Ij=A·(Dj / ΣDi), where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i=1 to the last) from the center of force. Reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figure 5C-5D In some embodiments, the characteristic intensity of the contact is based on one or more intensities of the contact. In some embodiments, the intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included in the Figure 5C-5D To help readers.

[0215] In some embodiments, a portion of a gesture is identified for determining the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic strength.

[0216] The intensity of the contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface, without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0217] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0218] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the intensity of the contact (or multiple contacts) to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact to above the press input intensity threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the press input intensity threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0219] Figure 5E-5HDetection of a gesture is shown, the gesture comprising contact 562 with an intensity from below Figure 5E Light press intensity threshold in L ”) increases to a value higher than Figure 5H The deep press intensity threshold in D ”). On the displayed user interface 570 including application icons 572A-572D displayed in the predefined area 574, a cursor 576 is displayed above the application icon 572B corresponding to application 2, and a gesture performed using contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch screen 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines that the intensity of the contact 562 is above the deep press intensity threshold (for example, “IT D ”) reaches a peak above. Contact 562 is maintained on touch-sensitive surface 560. In response to detecting the gesture, and according to the intensity rising to the deep press intensity threshold (e.g., “IT D ”) above contact 562, displays reduced-scale representations 578A-578C (e.g., thumbnails) of documents recently opened for Application 2, as shown in FIG. Figure 5F-5H In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in the Figure 5E-5H To help readers.

[0220] In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, as shown in FIG. Figure 5F As the animation progresses, representation 578A moves upward and representation 578B is displayed near application icon 572B, as shown. Figure 5G Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. Figure 5H In some embodiments, the animation progresses according to the intensity of contact 562, as shown in FIG. Figure 5F-5G , where representations 578A-578C appear and decrease as the intensity of contact 562 moves toward a deep press intensity threshold (e.g., “IT D In some embodiments, the intensity according to which the animation progresses is the characteristic intensity of the contact. The reference may be performed using an electronic device similar to or identical to device 100, 300, or 500. Figure 5E-5H The operation described.

[0221] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0222] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0223] As used herein, an "installed application" refers to a software application that has been downloaded to an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched (e.g., become open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer program that extracts program portions from the downloaded software package and integrates the extracted portions with the computer system's operating system.

[0224] As used herein, the term "open application" or "executing application" refers to a software application that has persisted state information (e.g., as part of device / global internal state 157 and / or application internal state 192). An open or executing application is optionally any of the following types of applications:

[0225] The active application, which is currently displayed on the display of the device on which the application is being used;

[0226] Background applications (or background processes) that are not currently displayed but whose processes are being processed by one or more processors; and

[0227] • A suspended or dormant application that is not running but has state information stored in memory (volatile and non-volatile, respectively) that can be used to resume execution of the application.

[0228] As used herein, the term "closed application" refers to a software application that does not retain state information (e.g., the state information of a closed application is not stored in the device's memory). Thus, closing an application includes stopping and / or removing the application's application process and removing the application's state information from the device's memory. Generally speaking, while in a first application, opening a second application does not close the first application. When the second application is displayed and the first application ceases to be displayed, the first application becomes a background application.

[0229] Attention is now turned to an embodiment of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0230] Figures 6A to 6AF An exemplary user interface for initiating and participating in a real-time communication session (e.g., from an instant messaging application) according to some embodiments is shown. The user interface in these figures is used to illustrate an example of a user interface including 7A to 7F The process is described below.

[0231] Figure 6AA device 600 with a touch-sensitive display 602 is shown. In some embodiments, the device 600 includes one or more features of devices 100, 300, and 500. In some embodiments, the device 600 also includes a camera 603, which includes at least an image sensor capable of capturing data representing a portion of a spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, the camera 603 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, the camera 603 is capable of capturing other types of data such as depth data. For example, in some embodiments, the camera 603 also uses techniques based on speckle, time of flight, parallax, or focus to capture depth data. The image data captured by the device 600 using the camera 603 includes data corresponding to a portion of the spectrum of the scene within the camera's field of view. In addition, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data contains data sufficient to determine or generate depth data for the data of that portion of the spectrum.

[0232] In some examples, the electronic device 600 includes a depth camera (e.g., as part of camera 603), such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate the object during image capture by the visible light camera and the depth camera (e.g., an IR camera), and information from the depth camera and the visible light camera is used to determine a depth map of different portions of the object captured by the visible light camera. In some embodiments, the depth map (e.g., a depth map image) includes information (e.g., values) related to the distance of objects in the scene from the viewpoint (e.g., a camera). In one embodiment of the depth map, each depth pixel defines the location of its corresponding two-dimensional pixel in the Z axis of the viewpoint. In some examples, the depth map is composed of pixels, where each pixel is defined by a value (e.g., 0 to 255). For example, a value of "0" represents the pixel farthest from the viewpoint (e.g., a camera) in the "three-dimensional" scene, and a value of "255" represents the pixel closest to the viewpoint in the "three-dimensional" scene. In other examples, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depths of various features of the object of interest in the field of view of the depth camera (e.g., the relative depths of the eyes, nose, mouth, ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the outline of the object of interest in the z direction. In some embodiments, the lighting effects described herein are displayed using disparity information from two cameras (e.g., two visible light cameras) for backward images, and depth information from the depth camera is used in combination with image data from the visible light camera for forward images (e.g., selfie images). In some embodiments, the same user interface is used when determining depth information using two visible light cameras and when determining depth information using the depth camera, thereby providing the user with a consistent experience even when using completely different technologies to determine the information used to produce the lighting effects. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects selection of a camera switching indicator and switches from a front-facing camera (e.g., a depth camera and a visible light camera) to a rear-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), while maintaining display of user interface controls to apply the lighting effect and replacing the display of the field of view of the front-facing camera with the field of view of the rear-facing camera (or vice versa).

[0233] like Figure 6A, device 600 displays an instant messaging user interface 604 of an instant messaging application. Instant messaging user interface 604 includes a message area 606 that includes messages 608A to 608D in a message conversation between a group of participants ("dream team") including three or more participants.

[0234] While displaying instant messaging user interface 604, device 600 receives data indicating that a real-time communication session is active (e.g., an invitation to join a real-time communication session), wherein the real-time communication session is available for participants in the "dream team" group. In some embodiments, the real-time communication session is an audio communication session or an audio / video communication session.

[0235] In response to receiving data indicating that the real-time communication session is active, device 600 displays a notification indicating that the real-time communication session is active. Figure 6B As shown, the notification includes notification 614 as a message from participant Pablo in message area 606, indicating that Pablo has initiated a real-time communication session. Notification 614 indicates the type of real-time communication session with the text "Video" and a video camera icon. Notification 614 also indicates the number of participants currently active in the real-time communication session ("1 person active"). In some embodiments, notification 614 indicates the names of one or more participants currently active in the real-time communication session. Notification 614 also includes a real-time communication join enable indication 614A, which, when selected, causes device 600 to initiate a process for joining the real-time communication session represented by notification 614. Techniques for joining a real-time communication session are described in more detail below.

[0236] Alternatively, in response to receiving data indicating that the real-time communication session is active, device 600 generates an audio output indicating that the real-time communication session is active. In some embodiments, the audio output is different from the audio output associated with the instant messaging application (e.g., notification of received messages) and the audio output associated with the phone application (e.g., ringtone for an incoming call). In some embodiments, the audio output is a mixture of the audio associated with the message notification and the audio associated with the incoming call notification. In some embodiments, the audio output shares audio characteristics (e.g., common pitch, note, rhythm) with both the audio output of the phone call and the audio output of the message notification.

[0237] Also in response to receiving data indicating that the real-time communication session is active, the device 600 displays the real-time communication enable indication 610. In some embodiments, the real-time communication enable indication 610 is displayed before the real-time communication session is initiated (e.g., in a grayed-out state) and is visually emphasized (e.g., highlighted) or modified (e.g., to a non-grayed-out state) when the real-time communication session is initiated. In some embodiments, selection of the real-time communication enable indication 610 or the option enable indication 612 initiates the process for joining the real-time communication session. In some embodiments, the notification 614 and / or the real-time communication enable indication 610 is displayed without audio and / or tactile output (e.g., a phone ringing output) (e.g., because it is being displayed in an active message conversation that the user may already be following).

[0238] like Figure 6C As shown, as additional messages are added to the message conversation in the message area 606, the notification 614 moves upward to different locations in the message area 606. In some embodiments, the device 600 moves the notification 614 to a location outside the message area (e.g., the notification 614 transitions from a message to a banner at the top of the display 602 outside the message area 606).

[0239] In some embodiments, the device 600 moves the notification 614 in response to meeting a notification relocation criterion. In some embodiments, the notification relocation criterion includes one or more criteria selected from the following: a predetermined time has passed (e.g., a time has passed after the notification 614 was displayed), the instant messaging user interface 604 has ceased to be displayed, the instant messaging application has been closed, and / or all or a portion of the notification 614 has been hidden in the message area 606 (e.g., due to new incoming messages filling the message area 606). In some embodiments, the notification 614 is moved to the bottom, right, or left side of the display 602. In some embodiments, the notification 614 is a banner notification displayed near the top of the display 612 in direct response to receiving data indicating that a real-time communication session is available.

[0240] In some embodiments, the device 600 receives reminder data indicating a reminder to join a real-time communication session (e.g., a reminder sent by a participant in a conversation or after a predetermined time period). Figure 6DAs shown, in response to receiving data indicating a reminder to join a real-time communication session, the device 600 displays a full-screen notification 616 and outputs an audio output 618. In some embodiments, the audio output 618 is different from the audio output associated with the instant messaging application (e.g., a notification of a received message) and the audio output associated with the phone application (e.g., a ringtone for an incoming call). In some embodiments, the audio output 618 is a mixture of the audio associated with the message notification and the audio associated with the incoming call notification. In some embodiments, the audio output 618 is shorter than the audio output associated with the phone application (e.g., a ringtone for an incoming call) and longer than the audio output associated with the instant messaging application (e.g., a notification of a received message). In some embodiments, the ringtone for an incoming call includes multiple repetitions of the corresponding audio output, and the audio output 618 is a single repetition of the corresponding audio output. In some embodiments, the audio output 618 shares audio characteristics (e.g., common tones, notes, rhythms) with both the audio outputs of the phone call and the message notification. In some embodiments, the device 600 outputs a tactile output. As Figure 6D As shown, full-screen notification 616 includes the name of the group ("Dream Team"), the names of the participants currently active in the real-time communication session ("PABLO" and "STEPHEN"), and the name of the participant who initially initiated the real-time communication session or initiated the reminder ("PABLO"). In some embodiments, full-screen notification 616 includes the names of the participants who have been invited to join the real-time communication session (e.g., according to the order in which they were added to the group). Full-screen notification 616 also includes a reply affordance 620 for joining the real-time communication session (e.g., for joining the real-time communication session immediately or displaying a menu with options for joining the real-time communication session), a "Not now" affordance 622 for dismissing full-screen notification 616, and a message affordance 624 for returning to the instant messaging user interface 604 or sending a message to the "Dream Team" group. The full-screen notification also includes an image (e.g., a video of the user of device 600 from camera 603) (e.g., as a preview of the user's image that will be seen by other participants in the real-time communication session once joined).

[0241] like Figure 6D As shown, device 600 receives (e.g., detects) user input 650A (e.g., a tap) corresponding to selection of the “not now” affordance 622. Figure 6E As shown, in response to user input 650A, device 600 stops displaying full screen notification 616 and returns to instant messaging user interface 604.

[0242] like Figure 6EAs shown, device 600 receives (e.g., detects) user input 650B (e.g., a tap) corresponding to selection of option affordance 612. In response to receiving user input 650B, device 600 expands the header of instant messaging user interface 604 to include audio affordance 626A, video join affordance 626B, and group details affordance 626C.

[0243] like Figure 6F As shown, device 600 receives (e.g., detects) user input 650C (e.g., a tap) corresponding to the selection of the video join enable indication 626B. In some embodiments, device 600 directly joins the real-time communication session in response to receiving user input 650B (e.g., without requiring additional input). In some embodiments, device 600 joins the real-time communication session and transmits video and audio simultaneously. In some embodiments, device 600 joins the real-time communication session and transmits only audio (e.g., without transmitting video). In some embodiments, in response to the selection of the audio enable indication 626A, device 600 joins the real-time communication session with only audio or initiates a separate conference call for the participants of the group.

[0244] like Figure 6G As shown, in response to receiving user input 650C, device 600 displays a real-time communication interface 628 of a real-time communication application. Real-time communication interface 628 includes an indication of the group ("Dream Team"), an image (e.g., a video of the user of device 600 from a camera on the front of device 600 (e.g., camera 603), an indication of the participants currently active in the real-time communication session ("PABLO" and "STEPHEN"), an indication of the participant who initiated the real-time communication session ("Video Call from PABLO"), and a call control menu 630. Call control menu 630 includes a process affordance 631, an effect affordance 632, a call affordance 634, and a menu affordance 636.

[0245] like Figure 6G As shown, device 600 receives (e.g., detects) user input 650D on menu enable representation 636. Figure 6HAs shown, in response to user input 650D, device 600 expands call control menu 630 to display additional information and controls related to the real-time communication session. In some embodiments, device 600 expands call control menu 630 in response to an upward swipe initiated near process enable indication 631. When expanded, call control menu 630 also includes an audio on / off enable indication 638, an audio source menu enable indication 640, a video on / off enable indication 642, a group name 644 (with an indication of the number of participants in the group), a group message enable indication 646, a list 648 including enable indications 648A to 648E of the participants corresponding to the group, and an add participant enable indication 652. In some embodiments, in response to receiving a selection of group message enable indication 646, device 600 launches and / or displays instant messaging user interface 604 (e.g., Figure 6F ) to send messages to the participants of the group.

[0246] like Figure 6H As shown, the affordances of the participants in list 648 include an indication of the communication status of the corresponding participant with respect to the real-time communication session. Indication 648A indicates that participant Pablo is connected to the real-time communication session and is providing video data (e.g., a real-time media stream including video and audio data). Indication 648B indicates that participant Stephen is connected and providing audio data (e.g., no video). Indications 648C and 648D respectively indicate that participants Marcel and Allison are not connected to the real-time communication session (e.g., not providing video and audio data). Unconnected participants have either never joined the real-time communication session or have previously joined and then left the real-time communication session. Exemplary participant communication status types include audio only (e.g., a participant is communicating using only audio); video (e.g., a participant is talking using video and audio); video paused (e.g., a participant's video is paused); video not decoded (e.g., the participant's video stream cannot be processed due to latency issues, issues with the video stream format, etc.); left (e.g., a participant has left the real-time communication session); and waiting to join (e.g., a participant has been invited to the real-time communication session but has not yet joined the real-time communication session).

[0247] In some embodiments, participants in a real-time video communication include participants of the group that are currently connected to the communication session and participants of the group that are currently not connected to the real-time video communication session. That is, all participants of the group are referred to as participants of the real-time communication session regardless of their connection status. When a real-time communication session is active, participants can join (connect) or leave (disconnect) the real-time communication session.

[0248] like Figure 6IAs shown, device 600 receives (e.g., detects) user input 650E (e.g., a tap) corresponding to selection of video on / off affordance 642. Figure 6J As shown, in response to receiving user input 650E, device 600 modifies video on / off enable indication 642 to indicate a video off state and Figure 6I The video camera representation in the changes to Figure 6J 6. The call affordance 634 is modified to indicate an audio-only state by using the representation of the phone in . In the audio-only state, device 600 joins the real-time communication session with only audio (e.g., device 600 does not provide video data).

[0249] like Figure 6J As shown, device 600 receives (e.g., detects) user input 650F (e.g., a tap) corresponding to selection of call affordance 634. Figure 6K As shown, in response to receiving user input 650F, device 600 initiates a connection to join the real-time communication session in audio-only mode. Figure 6K As shown, device 600 stops displaying expanded call control menu 630 (e.g., device 600 reduces call control menu 630) and displays connect screen 654. In some embodiments, device 600, in response to user input 650C on video join affordance 626B, initiates a connection to join the real-time communication session and displays connect screen 654. In some embodiments, notification 614 includes an affordance (e.g., a "join" icon on notification 614 or the entire notification 614 itself is selectable). In some embodiments, device 600, in response to selection of notification 614, initiates a connection to join the real-time communication session and displays connect screen 654.

[0250] like Figure 6L As shown, after device 600 joins the real-time communication session, device 600 displays a real-time communication user interface 656, which includes a representation 658 of the user of device 600, a representation 670A of participant Pablo (e.g., the participant who initiated the real-time communication session), and a representation 670B of participant Stephen. According to the communication status indicated in list 648, the representation 670A of participant Pablo includes a real-time video stream. Since participant Stephen only provides audio data, the representation 670B of participant Pablo includes an avatar. In some embodiments, the layout of the representations on the real-time communication user interface 656 is based on the number of participants currently connected to the real-time communication session (e.g., two participants and the user of device 600). Figure 6L The layout shown in is referred to as a 2-to-1 layout, where the first position corresponds to the position of representation 670A and the second position corresponds to the position of representation 670B.

[0251] In some embodiments, when the device 600 joins a real-time communication session, the device 600 determines whether the number of participants (e.g., active or total (active and invited)) exceeds a predetermined threshold number. Based on determining that the number of participants exceeds the threshold, the device 600 displays an overflow area with overflow representations (e.g., thumbnails), as described in more detail below. Alternatively, based on determining that the number of participants does not exceed the threshold, the device 600 does not display an overflow area with overflow representations.

[0252] When device 600 has joined a real-time communication session, device 600 transmits the real-time media stream to other participants (e.g., participants who have joined the real-time communication session). When a participant associated with another device has joined the real-time communication session and device 600 has also joined, device 600 receives the real-time media stream corresponding to the participant.

[0253] like Figure 6L As shown, when the real-time communication user interface 656 is displayed, the device 600 receives (e.g., detects) user input 650G (e.g., a tap) on the display 602. Figure 6M As shown, in response to receiving user input 650G, device 600 displays call control menu 630. Representations 670A and 670B are reduced in size, and representations 670B and 658 are moved upward on display 602 so that call control menu 630 does not overlap with the representations. Figure 6L In contrast, when the configuration shown in FIG. 670A and 670B is vertically aligned and non-overlapping, Figure 6M When the call control menu 630 is displayed in FIG, representations 670A and 670B are offset from each other and overlap.

[0254] like Figure 6N As shown, device 600 receives (e.g., detects) user input 650H (e.g., a tap) on menu affordance 636 in call control menu 630. Figure 6O As shown, in response to receiving user input 650H, device 600 expands call control menu 630 (e.g., as described with respect to Figure 6H In some embodiments, device 600 displays a reminder affordance in call control menu 630 at the representation of the participant not connected to the real-time communication session. Figure 6OAs shown, list 648 indicates that participants Marcel and Allison are not connected to the real-time communication session. Because participants Marcel and Allison (e.g., devices associated with participants Marcel and Allison) are not connected to the real-time communication session, the expanded call control menu 630 includes ringtone enable indications 662 and 664 at indications 648C and 648D of participants Marcel and Allison, respectively, in list 648. In response to selection of ringtone enable indication 662 or 664, device 600 causes a notification (e.g., an audio output, such as a call or ringtone) associated with the participant corresponding to the selected ringtone enable indication to be provided at the device. In some embodiments, the notification is a new notification that is different from a previous notification sent to the participant. In some embodiments, the new notification is more intrusive than the previous notification (e.g., an audio ringtone, tactile output, or full-screen notification compared to a banner notification without audio or tactile output). In this way, a user can remind participants who have not yet joined the real-time communication session that the real-time communication session is active and available for participants to join (e.g., similar to a user of device 600 being prompted as shown). Figure 6D full-screen notification reminder method as shown).

[0255] like Figure 6O As shown, device 600 receives (e.g., detects) user input 650I (e.g., a tap) corresponding to selection of video on / off enable indication 642. In response to receiving user input 650I, device 600 provides a real-time video stream (e.g., from a camera on device 600) to the real-time communication session and modifies video on / off enable indication 642 to indicate that video is on, as shown. Figure 6P As shown. Figure 6Q , device 600 receives (e.g., detects) user input 650J (e.g., a downward swipe) on display 602 to reduce call control menu 630. Figure 6R As shown, when call control menu 630 is zoomed out, representation 658 corresponding to the user of device 600 includes video from camera 603 of device 600 .

[0256] like Figure 6S As shown, device 600 receives (e.g., detects) user input 650K (e.g., a tap) corresponding to selection of call affordance 634. In response to user input 650K, device 600 disconnects from the real-time communication session. Figure 6TAn embodiment of an instant messaging user interface 604 of an instant messaging application is shown displayed after a device 600 has been disconnected from a real-time communication session. Although the device 600 has been disconnected from the real-time communication session, the real-time communication session remains active. In some embodiments, the real-time communication session remains active even if the participant who initiated the real-time video communication session leaves the session as long as at least one of the participants is connected. Figure 6T As shown, device 600 maintains display of notification 614 and real-time communication affordance 610, indicating that the real-time communication session is active. Notification 614 is updated to indicate the duration of the real-time communication session (e.g., three minutes and forty-two seconds, corresponding to the amount of time since the real-time communication session was initiated).

[0257] While displaying notification 614 indicating that the real-time communication session is active, device 600 receives data indicating that the real-time communication session is no longer active (e.g., has ended). In some embodiments, the real-time communication session ends when no participants are connected to the real-time communication session (e.g., all participants who have initiated or joined the real-time communication session are no longer connected). In response to receiving data indicating that the real-time communication session is no longer active, device 600 stops displaying notification 614 and / or updates notification 614 to indicate that the real-time communication session is no longer active. Figure 6U An embodiment of an instant messaging user interface 604 of an instant messaging application is shown displayed after a real-time communication session has ended (e.g., in response to receiving data indicating that the real-time communication session is no longer active). Figure 6U As shown, notification 614 is updated to include an indication that the real-time communication session is no longer active and the real-time communication enable representation 610 is removed (e.g., the text "Call Ended"). In some embodiments, the real-time communication enable representation 610 and / or notification 614 are grayed out to indicate that the real-time communication session is no longer active. In some embodiments, notification 614 is no longer selectable after the real-time communication session ends. In some embodiments, after the real-time communication session ends, notification 614 remains selectable and can be selected to initiate a new real-time communication session with the group. In some embodiments, if device 600 has not yet joined the real-time communication session, device 600 updates notification 614 to indicate that the user missed the real-time communication session (e.g., "missed call").

[0258] like Figure 6U As shown, device 600 receives (e.g., detects) user input 650L (e.g., a tap) corresponding to selection of return affordance 611. Figure 6VAs shown, in response to receiving user input 650L, device 600 displays a user interface 671 of an instant messaging application, which includes a list 673 of affordances 673A to 673G for accessing corresponding message conversations. Affordance 673D corresponds to a message conversation between a group of three participants (Matthew, Danielle, and the user of device 600). Affordance 673D includes a real-time communication indicator 675, which indicates that there is an active real-time communication session for the group. In contrast, because the real-time communication session of "Dream Team" is not active, affordance 673A corresponding to "Dream Team" does not include real-time communication indicator 675.

[0259] Now turn Figure 6W , device 600 is shown displaying an interface in addition to instant messaging user interface 604. Figure 6W , device 600 displays lock screen 668, indicating that device 600 is in a locked state. Lock screen 668 includes notifications 677A and 677B. Notification 677A indicates that a message from user John Appleseed was received five minutes ago. Notification 677B represents a real-time communication session that is no longer available to join (e.g., a missed video call). The missed video call notification includes an indication of the invited participants (John, Amy, and Rodrigo), the status of the real-time communication session ("Missed"), and the time when the real-time communication session was initiated or ended ("1 hour ago"). In the locked state, device 600 does not display active applications.

[0260] While in the locked state, the device 600 receives data indicating that a real-time communication session is available to join. In some embodiments, the response of the device 600 to receiving data indicating that a real-time communication session is available to join depends on the context of the device 600, the available real-time communication sessions, or other factors. In some embodiments, based on an available real-time communication session between two participants (e.g., a one-to-one communication session), the device 600 displays a first type of notification. Figure 6X As shown, the first type of notification includes a full-screen notification 672 and an audio output 674. In some embodiments, the audio output 674 is different from the audio output associated with the phone call notification and the audio output associated with the message notification. In some embodiments, the audio output 674 shares audio characteristics (e.g., a common pitch, note, rhythm) with both the audio outputs for the phone call and the message notification. Alternatively, based on a real-time communication session available between three or more participants (e.g., a group real-time communication session), the device 600 displays a second type of notification. Figure 6YAs shown, the second type of notification includes a banner notification 676 (e.g., not full screen) and a tactile output 678 (e.g., but no audio output). In some embodiments, based on a real-time communication session available between three or more participants (e.g., a group real-time communication session), the device 600 initially displays a full-screen notification (e.g., similar to a banner notification 676). Figure 6E ), and then displays (e.g., transitions to) a banner notification 676. In some embodiments, notification 676 includes an affordance that, when selected, initiates a process for joining the corresponding real-time communication session (e.g., as described above with reference to FIG. Figures 6F to 6L described above).

[0261] In contrast to notification 677B, which corresponds to a missed video call, notification 676 corresponds to an active real-time communication session. Notification 676 includes an indication of the group ("Dream Team"), the participant (Pablo) who performed the action that triggered the presentation of the notification, and the time the real-time communication session was initiated ("Now"). In some embodiments, notification 676 is updated to accurately reflect the time the real-time communication session was initiated and the status of the real-time communication session. In some embodiments, when the real-time communication session represented by notification 676 ends, the text of notification 676 is updated from "Joined this video call from PABLO" to "Missed video call from PABLO."

[0262] In some embodiments, based on determining that an active application is not displayed (e.g. Figure 6W In some embodiments, device 600 displays a non-full-screen notification (e.g., a banner notification) based on determining that an active application (e.g., an instant messaging application) is being displayed.

[0263] In some embodiments, based on determining to alert the user of device 600, device 600 outputs an alert (e.g., a notification or audio or tactile output); and based on determining not to alert the user of device 600, device 600 refrains from outputting an alert. In some embodiments, the determination to alert the user is based on interference criteria (e.g., whether the device is in a locked state or unlocked state). In some embodiments, based on whether device 600 is in a locked state (e.g., Figure 6Y ), the device outputs an alert (e.g., tactile output 678); and according to the device 600 being in an unlocked state (e.g., as Figure 6Z ), device 600 displays a notification (e.g., banner notification 676) and refrains from providing an alert (e.g., no tactile output).

[0264] In some embodiments, the interference criteria indicates whether the user is looking toward device 600 (e.g., based on data from sensors on the front of device 600). Figure 6AA A user 684 is shown holding the device 600. The user's gaze 680 is not directed toward the device 600. The device 600 determines whether the user 684 is not looking toward the device 600, and based on determining that the user 684 is not looking toward the device 600, the device 600 provides a non-visual notification 682 (e.g., a tactile or audio output). Figure 6AB As shown, the user's gaze 680 is directed toward the device 600. Based on determining that the user 684 is looking at the device 600, the device 600 avoids providing a non-visual notification (e.g., the device 600 displays a banner notification without tactile or audio output). In some embodiments, the device 600 uses an image sensor configured to recognize the user's face to determine whether the user is looking at the device 600.

[0265] Go to Figures 6AC to 6AF , device 600A includes a display 602A (e.g., a touch-sensitive display), a rotatable input mechanism 601A, and a mechanical button 601B. In some embodiments, device 600A includes features of device 100, 300, 500, or 600. In some embodiments, device 600A communicates with device 600 (e.g., via two-way wireless communication). In some embodiments, device 600A is in a paired relationship with device 600.

[0266] In some embodiments, device 600A receives data (e.g., from device 600) indicating that a real-time communication session is active with three or more participants available. Figure 6ACAs shown, in response to receiving data indicating that the real-time communication session is active, device 600A displays user interface 604A, which includes notification 605, real-time communication joining enable indication 607 and release enable indication 609. In some embodiments, notification 605 includes the features of the above-mentioned notification 614 (e.g., participant name, number of participants, name of the participant that triggered the notification). In some embodiments, device 600A generates audio and / or tactile output in response to receiving data indicating that the real-time communication session is active. In some embodiments, device 600A generates audio and / or tactile output different from the audio and / or tactile output generated by device 600 in response to receiving data indicating that the real-time communication session is active. In some embodiments, in response to receiving initial data indicating that the real-time communication session is active, a notification (e.g., display, audio, and / or tactile output) that the real-time communication session is active is provided at device 600 (e.g., not at device 600A), and in response to receiving reminder data indicating a reminder that the real-time communication session is active, the notification is provided at both device 600 and device 600A. In some embodiments, in response to receiving initial data indicating that the real-time communication session is active, a notification (e.g., display, audio, and / or tactile output) that the real-time communication session is active is provided at both device 600 and device 600A, and in response to receiving reminder data indicating a reminder that the real-time communication session is active, the notification is provided at device 600 (e.g., not at device 600A).

[0267] In some embodiments, device 600A receives user input corresponding to selection of real-time communication join enable indication 607 (e.g., a tap on real-time communication enable indication 617). In response to receiving selection of real-time communication join enable indication 607, device 600A causes device 600 and / or device 600A to join the real-time communication session or initiate the process of joining the real-time communication session. In some embodiments, in response to receiving selection of real-time communication join enable indication 607, device 600A causes device 600 to display real-time communication user interface 628 ( Figure 6G )、654( Figure 6K ) or 656( Figure 6L ).

[0268] Figure 6AD FIG2 shows a user interface 604A displayed on device 600A after joining a real-time communication session. In some embodiments, device 600A includes a microphone for a user to provide audio input to the real-time communication session via device 600A. Figure 6AD, the user interface 604A includes a volume control 611, an indication 613 of the participants (“PABLO” and “2 others”) and the duration of the real-time communication session (“9:45”), an end call indication 615, an audio on / off indication 617, and a menu indication 619.

[0269] exist Figure 6AE , device 600A receives (e.g., detects) user input 650M (e.g., an upward swipe gesture initiated at or near menu affordance 619). Figure 6AF As shown, in response to receiving user input 650M, device 600A modifies indication 613 ("Participant 4") and displays keyboard enable indication 621, audio source enable indication 623 (e.g., using device 600 to participate in a real-time communication session), and a list 625 of participants in the real-time communication session (e.g., joined and / or invited participants). In some embodiments, device 604A, in response to the user input 650M, modifies indication 613 ("Participant 4") and displays keyboard enable indication 621, audio source enable indication 623 (e.g., using device 600 to participate in a real-time communication session). Figure 6AC 625 upon receiving user input 650M on user interface 604 (e.g., before joining a real-time communication session). In some embodiments, list 625 includes an indication of the communication status of the participants. In some embodiments, list 625 includes a reminder affordance that can be selected to send a reminder notification to participants who have not yet joined the real-time communication session.

[0270] 7A to 7F Flowchart showing a method according to some embodiments. Method 700 is performed at a device having a display (e.g., 100, 300, 500, 600). Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0271] As described below, method 700 provides, among other things, an intuitive way to provide interactive notifications (e.g., 614, 672, 676) for a real-time communication session between multiple participants. The method reduces the cognitive burden on a user of providing notifications for a real-time communication session between multiple participants using innovative techniques, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to quickly interact with notifications makes the real-time communication session between multiple participants faster and more efficient, thereby saving power and increasing the time between battery charges.

[0272] At block 702, the device displays an instant messaging user interface (e.g., 604) of an instant messaging application. The instant messaging user interface (e.g., 604) includes a message area (e.g., 606). The message area includes a plurality of messages (e.g., 608A to 608D) in a message conversation between three or more participants.

[0273] At box 704, the device receives first data indicating that a real-time communication session (e.g., an audio communication session or an audio / video communication session) is available for three or more participants and is in an active state. In some embodiments, the participants of the real-time communication session have a "currently joined" state (e.g., providing real-time media streams to other participants; also referred to as "connected" or "active") or "currently not joined" (e.g., a participant who has never joined a session or has previously joined a session and then left the session; also referred to as "unconnected" or "invited"). In some embodiments, the participants of the real-time communication session include participants who have currently joined the real-time communication session and participants who have not currently joined the real-time communication session (e.g., 648A to 648D). When the real-time communication session is in an active state, the participant joins (connects) or leaves (disconnects) the real-time communication session. When a participant associated with another device has joined the real-time communication session, the device receives the real-time media stream corresponding to the participant. When the user of the device has joined the real-time communication session, the device transmits the real-time media stream to other participants (e.g., participants who have joined the real-time communication session). In some embodiments, the real-time communication session remains active as long as at least one of the participants is connected (eg, even if the participant who initiated the real-time communication session leaves the session).

[0274] At box 706, in response to receiving first data indicating that a real-time communication session between three or more participants is active, the device displays a notification indicating that the real-time communication session is active (e.g., 614). In some embodiments, the notification indicating that the real-time communication session is active includes a real-time communication join enable indication (e.g., 614A). In some embodiments, the notification includes an indication of the number of participants in the real-time communication session (e.g., the number of participants active in the session; the number of participants invited to the session). In some embodiments, the notification includes an indication of the participant (e.g., name, initials, photo, or avatar), which performs an action that triggers the presentation of the notification on the display (e.g., the participant initiating the real-time communication session, or selecting a "ring again" button, etc. to remind the user of the device to join the real-time communication session). In some embodiments, in response to receiving first data indicating that a real-time communication session between three or more participants is active, the device generates an audio output (e.g., 618) indicating that the real-time communication session is active via one or more audio output devices of the device, wherein the audio output indicating that the real-time communication session is active is different from the audio output associated with the phone call notification and the audio output associated with the message notification. In some embodiments, the audio output indicating that the real-time communication session is active shares audio characteristics (e.g., common pitch, note, rhythm) with the audio output of the phone call and message notification. Displaying a notification indicating that the real-time communication session is active (e.g., 614) and / or generating an audio output indicating that the real-time communication session is active (e.g., 618) provides feedback to the user about the current state of the real-time communication session, and provides visual feedback to the user that if the user activates the notification (e.g., when the notification includes joining an affordance), certain operations associated with the notification and / or the real-time communication session will be performed. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0275] Optionally, at block 704, the device displays a notification (e.g., 606) indicating that the real-time communication session is active at a first location in the message area. In some embodiments, the notification (e.g., 614) indicating that the real-time communication session is active includes a real-time communication join affordance (e.g., 614A).

[0276] At block 710 , while displaying a notification (eg, 614 ) indicating that the real-time communication session is active, the device receives second data indicating that the real-time communication session between three or more participants is no longer active.

[0277] Optionally, at box 712, the device moves a notification indicating that the real-time communication session is active from a first location (e.g., the bottom of the message area 606) to a second location (e.g., the middle or top of the message area 606, or the top, bottom, right, and / or left side of the display (602)). In some embodiments, moving the notification (e.g., 614) occurs in response to satisfying notification repositioning criteria. In some embodiments, the notification repositioning criteria include one or more criteria, such as a predetermined amount of time has passed (e.g., a time has passed after the notification was displayed), the instant messaging user interface (e.g., 604) has ceased to be displayed, the instant messaging application has been closed, the entire notification (e.g., 614) or a portion of the notification (e.g., 614) has been hidden in the message area (e.g., 606) due to new incoming messages filling the message area (e.g., 606), etc. A notification indicating that a real-time communication session is active provides the user with feedback on the current state of the real-time communication session, provides the user with visual feedback indicating that certain actions associated with the notification and / or the real-time communication session will be performed (if selected), and reduces the number of inputs required to perform the actions. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, reducing the number of inputs required to perform an action also enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing the user with more available inputs and reducing the number of inputs), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0278] Optionally, at boxes 714 and 716, the device receives a first user input (e.g., 650C) corresponding to a selection of a real-time communication join enable indication (e.g., 614A or 626B). In response to receiving the first user input corresponding to a selection of a real-time communication join enable indication, the device joins the real-time communication session (e.g., joins the real-time communication session and transmits video and audio simultaneously from the electronic device and / or joins the real-time communication session and transmits only audio from the electronic device (without transmitting video)). Joining the real-time communication in response to receiving a selection of the real-time communication join enable indication included in the notification reduces the number of inputs required to perform the operation of joining the real-time communication session. Reducing the number of inputs required to perform the operation also enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing more user-available inputs and reducing the number of inputs required to join the real-time communication session), which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0279] Optionally, at blocks 718 and 720, the device receives reminder data (e.g., Figure 6D) (e.g., a reminder sent by a participant in a conversation or after a predetermined time period). In response to receiving reminder data indicating a reminder to join a real-time communication (e.g., a reminder sent by a participant in a conversation), the device generates an audio output (e.g., 618) (e.g., a phone ringing output) via one or more audio output devices of the device indicating that the real-time communication session is active. In some embodiments, the audio output (e.g., 618) is generated in conjunction with redisplaying the notification (e.g., 616). In some embodiments, the display of the notification is ongoing when the reminder data is received. In some embodiments, the audio output indicating that the real-time communication session is active is different from the audio output associated with the phone call notification and the audio output associated with the message notification. In some embodiments, the audio output indicating that the real-time communication session is active shares audio characteristics (e.g., common pitch, note, rhythm) with both the audio outputs for the phone call and the message notification. In some embodiments, the initial audio output indicating that the real-time communication session is active is different (e.g., shorter, quieter, or lower intensity) from the reminder audio output indicating that the real-time communication session is active (e.g., the audio output generated based on the participant selecting the "ringing" affordance). In some embodiments, the initial audio output indicating that the real-time communication session is active is a single repetition of the repeated audio output for the phone call and the reminder audio output indicating that the real-time communication session is active is multiple repetitions of the repeated audio output for the phone call, or is the same as the audio output for the phone call. In some embodiments, in response to receiving reminder data indicating a reminder to join the real-time communication, the device displays the instructions (e.g., name, initials, photo, or avatar) of the participants in the real-time communication session that initiated the reminder. Generating a reminder including an audio output when the specified conditions are met allows users to quickly identify that the real-time communication session is available when they miss a visual notification (e.g., 614). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0280] Optionally, at blocks 722 and 724, in an embodiment where the notification includes a menu access affordance (e.g., 626B), the device receives a second user input (e.g., 650C) corresponding to a selection of the menu access affordance. The device, in response to receiving the second user input corresponding to the selection of the menu access affordance, displays a menu (e.g., 630) including a video join option (e.g., 634) (e.g., affordance, drop-down menu, checkbox) and an audio-only option (e.g., 642) (e.g., affordance, drop-down, checkbox). In response to receiving the selection of the menu access affordance, a menu including different options for joining the real-time communication session is displayed, providing additional control options without cluttering the UI with additional displayed controls. Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0281] Alternatively, at boxes 726 and 728, the device receives a third user input while displaying a menu (e.g., 630). Based on a third user input (e.g., 650F) corresponding to the selection of the video join option (e.g., 634), the device joins the real-time communication session using the audio and video transmitted from the electronic device. In some embodiments, the device prompts the user to check whether they want to join with only audio and / or only video (e.g., "Do you want to join with video?"). When the specified conditions are met (e.g., in response to the video join option being selected), joining the real-time communication session using the audio and video transmitted from the electronic device allows the user to join the real-time communication session using audio and video without the user selecting or modifying the additional input and output of the device (e.g., turning on the camera and / or turning on the microphone). Executing the optimization operation when a set of conditions has been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0282] Optionally, at block 730, based on a third user input corresponding to selection of the audio-only join option (e.g., 642), the device joins the real-time communication session using audio transmitted from the electronic device (e.g., audio recorded by one or more microphones of the device) without transmitting video data from the electronic device. In some embodiments, the device displays an indicator (e.g., Figure 6L658 in). This makes the user feel comfortable because the video is not shared). When the specified conditions are met, joining the real-time communication session using audio transmitted from the electronic device allows the user to join the real-time communication session using only audio without requiring the user to select or modify additional inputs and outputs of the device (e.g., turning off the camera and / or turning on the microphone). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0283] At block 732, in response to receiving second data indicating that the real-time communication session between the three or more participants is no longer active, the device stops displaying the notification (e.g., 614) or updates the notification to indicate that the real-time communication session is no longer active. In some embodiments, when the real-time communication session is active, the notification is updated to indicate the current duration of the real-time communication session. In some embodiments, when the real-time communication session is no longer active, the notification stops displaying the duration of the session and displays an indication that the session has ended (e.g., "Call Ended"). In response to receiving data indicating that the real-time communication session is no longer active, the notification is stopped from displaying, providing visual feedback to the user indicating that certain actions associated with the notification and / or the real-time communication session are no longer available. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, in response to receiving data indicating that the real-time communication session is no longer active, the notification is stopped from displaying, and an action is automatically performed when a set of conditions are met without further user input. Performing optimization operations when a set of conditions have been met and no further user input is required enhances the operability of the device and makes the user-device interface more efficient (e.g., by reducing unnecessary elements on the user interface and helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0284] Optionally, at blocks 736, 738, and 740, the device receives third data indicating that a second real-time communication session is available between two or more participants. Based on the second real-time communication session being available between the two participants, the device displays a first type of notification (e.g., 672) (e.g., a full-screen notification) indicating that the second real-time communication session is active. In some embodiments, the first type of notification (e.g., 672) includes a menu (e.g., 630) with options for video (e.g., 634), audio only (e.g., 643), and / or rejection (e.g., 622). Based on the second real-time communication session being available between three or more participants, the device displays a second type of notification (e.g., 676) that is different from the first type of notification and indicates that the second real-time communication session is active. In some embodiments, the second type of notification is a banner notification (e.g., 676) that is not full-screen or not persistent full-screen. Optionally, at block 742, the device initially displays the second type of notification in a first state (e.g., a full-screen state corresponding to full-screen notification 672). Optionally, at step 744, the device transitions the second type of notification (e.g., 672) to a second state (e.g., 676) (e.g., a persistent non-full screen state (e.g., a persistent banner located at the top edge of the display)). Displaying a specific type of notification that a real-time communication session is active based on when specified conditions are met allows the user to quickly identify that real-time communication is available based on the specified conditions while providing the user with more and / or less reduced distractions (e.g., displaying a full screen notification when a real-time communication session is available between two participants, and displaying a banner when a real-time communication session is available between two participants). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0285] Optionally, at blocks 746, 748, 750, and 752, the device receives fourth data indicating that a third real-time communication session is available. In response to receiving the fourth data indicating that the third real-time communication session is available, based on determining that an active application (e.g., an application (other than, for example, the lock screen (e.g., 668) or home screen (e.g., 1400) of the device), such as an instant messaging application or a third-party application provided by an application developer other than the developer of the operating system), the device displays a third type (e.g., a non-full screen state (e.g., a banner located at the top edge of the display)) notification indicating that the third real-time communication session is available (e.g., 614 or 676). In response to receiving the fourth data indicating that the third real-time communication session is available, based on determining that an active application (e.g., an application (other than, for example, the lock screen (e.g., 668) or home screen (e.g., 1400) of the device), such as an instant messaging application or a third-party application provided by an application developer other than the developer of the operating system) is available, the device displays a third type (e.g., a non-full screen state (e.g., a banner located at the top edge of the display)) notification indicating that the third real-time communication session is available. Figure 6W ), the device provides a fourth type of notification (e.g., 678 or a full-screen notification (e.g., 616)) indicating that a third real-time communication session is available. Displaying a specific type of notification for a real-time communication session based on when specified conditions are met allows the user to quickly identify that real-time communication is available based on the specified conditions while providing the user with more and / or less reduced distractions (e.g., displaying a full-screen notification when the device is displaying an application and displaying a banner when the device is not displaying an application). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] Optionally, at boxes 754, 756, 758, and 760, the device receives fifth data indicating that a fourth real-time communication session is available. In response to receiving the fifth data indicating that the fourth real-time communication session is available, based on determining that the current context of the electronic device (e.g., the current operating mode of the device (e.g., whether the device is in Do Not Disturb (DND) mode; whether certain applications or hardware are operationally valid; whether the device is locked or unlocked; whether one or more sensors of the device detect that the user is looking at the display) meets the alert issuance criteria, the device displays a notification that the fourth real-time communication session is available (e.g., 676) and provides a non-visual notification (e.g., 678) (e.g., audio or tactile output) that the fourth real-time communication session is available. In some embodiments, the output or alert corresponds to the notification (e.g., 614). In response to receiving the fifth data indicating that the fourth real-time communication session is available, based on determining that the current context of the electronic device does not meet the alert issuance criteria, the device displays the notification that the fourth real-time communication session is available (e.g., 614) without providing the notification that the fourth real-time communication session is available. Non-visual notification (e.g., 678). In some embodiments, when the first type of alarm is not issued, a second type of alarm is issued (e.g., a visual alarm, such as a banner notification (e.g., 676)). In some embodiments, the relevant context of the device is whether the device is in a locked state or an unlocked state. In some embodiments, when the device is locked, the first type of alarm is issued (e.g., audio or tactile output); when the device is unlocked (e.g., when the user actively operates the device / interacts with the device), the alarm is not issued (e.g., suppressed). In some embodiments, the relevant context of the device is whether the device detects (e.g., via one or more sensors (e.g., camera, infrared sensor)) whether the user is looking toward the display or not. In some embodiments, when the device is locked, the first type of alarm is issued (e.g., audio or tactile output). In some embodiments, when the device detects that the user is not looking toward the display (e.g., Figure 6AA ), a first type of alert is generated (e.g., 682). In some embodiments, when the user actively operates the device / interacts with the device (e.g., Figure 6AB ), an alarm of the first type is not generated (e.g., suppressed).

[0287] In some embodiments, the alarm issuance criteria include the device being locked in order to meet the requirements of the alarm issuance criteria. In some embodiments, the alarm issuance criteria include the device not detecting the user's attention to the device in order to meet the requirements of the alarm issuance criteria. (For example, based on detected user input with an input device of the electronic device, based on user gaze detected via gaze tracking, facial tracking, etc. (for example, 680)). Based on when the specified conditions are met, a notification is displayed (for example, 614) and a determination is made whether to provide a non-visual notification, allowing the user to quickly identify that real-time communication is available based on the specified conditions while providing the user with more and / or less interference. Performing optimization operations when a set of conditions have been met and no further user input is required enhances the operability of the device and makes the user-device interface more efficient (for example, by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0288] Note that the processes described above with respect to method 700 (e.g., 7A to 7F ) can also be applied in a similar manner to the methods described below. For example, method 700 optionally includes one or more features of the various methods described below with reference to methods 900, 1100, 1300, and 1500. For example, method 700 can use methods 1300 and 1500 to initiate a real-time communication session, and starting and running a real-time communication session can include methods 900 and 1100. For the sake of brevity, these details are not repeated below.

[0289] Figures 8A to 8BK An exemplary user interface for a real-time communication session according to some embodiments is shown. The user interface in these figures is used to illustrate the Figures 9A to 9K The process is described below in the process.

[0290] Figure 8A Device 600 is shown displaying an instant messaging user interface 604 for a messaging conversation between the group "Dream Team." The real-time communication session between the participants of the group is not active (e.g., as indicated by the lack of a real-time communication affordance 610). Figure 8A As shown, the header of the instant messaging user interface 604 is expanded to display the video join affordance 626B (described above). Figure 8A As shown, device 600 receives user input 850A (e.g., a tap on video join affordance 626B) corresponding to a request to display a user interface for a real-time communication session between participants of participant group "Dream Team." Figure 8AAs shown, the request includes a request to start a new real-time video communication session. In some embodiments, the request includes a request to join an existing real-time communication session (e.g., as described above with respect to Figure 6F described).

[0291] In response to user input 650A, device 600 displays a message 800 in message area 606 along with an indication 802 that a real-time communication session is being initiated. In some embodiments, message 800 includes some of the features of notification 614 described above. Once the real-time communication session is initiated, device 600 displays a real-time communication user interface 804 that simultaneously includes multiple representations of participants in the real-time communication session.

[0292] Based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold (e.g., more than five participants, including the user of device 600), device 600 simultaneously displays representations of a first plurality of participants other than the user of device 600 at one or more sizes (e.g., three different sizes) greater than the corresponding threshold size, while displaying representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of one or more other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size.

[0293] exist Figures 8A to 8BK In the illustrated embodiment, the threshold number of participants is five participants (including the user of device 600). Since the group "Dream Team" includes fifteen participants, device 600 determines that the number of participants in the real-time communication session includes a number of participants greater than the threshold. Figure 8C As shown, based on this determination, real-time communication user interface 804 includes a 4-to-1 layout of four primary representations 810A through 810D (collectively 810) of four respective participants in addition to the user of device 600 displayed in canvas area 811, a user representation 806 of the camera view of device 600 (corresponding to the user of device 600), and an overflow area 820 having overflow representations 822A through 822J (collectively 822) corresponding to participants of the real-time communication session in addition to the user of device 600 and the participants represented in canvas area 811, with a subset thereof (e.g., 822A through 822D) displayed at any time. In some embodiments, the layout is one of a plurality of predefined layouts in which the representations of the participants are anchored at different locations in user interface 804. Figure 8AW Another embodiment of user interface 804 is shown. Figure 8AWIn the illustrated embodiment, the overflow area 820 includes overflow representations of all participants in the real-time communication session other than the user of the device 600 , such that participants represented in the canvas area 811 are also represented in the overflow area 820 .

[0294] If the number of participants in the group is less than or equal to five, then based on determining that the number of participants in the real-time communication session is less than or equal to five, device 600 displays primary representations of the participants and representations of users of device 600 without displaying overflow representations in the overflow area.

[0295] Each of the primary representations 810 is displayed in one of three predetermined representation sizes (small, medium, and large). Primary representation 810A is displayed in a small size, primary representation 810B is displayed in a large size, and primary representation 810C and primary representation 810D are displayed in a medium size. Figure 8C 810. Each of the primary representations 810 overlaps at least one other primary representation. The overflow representations 822 are all displayed at the same size, which is smaller than the size of the small representations associated with the primary representations 810. In the illustrated embodiment, the respective threshold sizes mentioned above are larger than the size of the overflow representations 822 and smaller than the size of the small representations of the primary representations 810. The user representation 806 is the same size as the overflow representations 822 and is aligned with or displayed within the overflow region 820.

[0296] like Figure 8C As shown, the real-time communication session user interface 804 displays three full overflow representations and one partial overflow representation at a time. The remaining overflow representations are hidden and can be displayed by scrolling the overflow area 820 (e.g., in response to user input or detected events, such as a participant actively participating by talking or moving). Figure 8C, the overflow representation 820D is partially shaded to indicate that there are more overflow representations and / or that the overflow area 820 can be scrolled. In some embodiments, the order of the overflow representations 822 in the overflow area 820 depends on the order in which the participants join the real-time communication session. In some embodiments, the order of the overflow representations 822 in the overflow area 820 changes over time (e.g., due to, for example, user selection of overflow representations or active participation of participants represented in the overflow area, when participants join or leave the real-time communication session or when participants move into or out of the canvas area 811, as described in more detail below). In some embodiments, the order of the overflow representations in the overflow area 820 is fixed (e.g., the order in which the overflow representations are displayed does not change over time). In some embodiments, the order of the overflow representations is based on the order of the lists associated with the group (e.g., the order in which participants were added to the group when the group was created). In some embodiments, the display of the main representation 810 (e.g., the real-time video feed) is updated at a higher rate than the display of the overflow representations 822.

[0297] Figure 8C The real-time communication session interface 804 is shown shortly after the real-time communication session is initiated and before any participants (other than those associated with the device 600) join the real-time communication session. Figure 8C As shown, the main representation 810 and the overflow representation 822 include placeholders for participants who are not connected to the real-time communication session. Figure 8C As shown, the placeholder includes a representation with an indication of the participant's name (e.g., first name or first and last initials) and a colored object. In some embodiments, the color and / or pattern of each colored object is different or selected from a set of predefined colors and / or patterns. Optionally, the placeholder includes a status indicator indicating that the participant is waiting to be connected to the real-time communication session, and a reminder enable representation ("ring tone") for sending a reminder to the corresponding participant that the real-time communication session is available. In some embodiments, the reminder enable representation is not initially included on the placeholder and is displayed based on determining that the corresponding participant has not joined the real-time communication session after a predetermined time period.

[0298] like Figure 8C As shown, device 600 receives (e.g., detects) user input 850A (e.g., a tap) corresponding to selection of primary representation 810C. Figure 8D As shown, in response to receiving user input 850A, device 600 modifies primary representation 810C to include additional identifying information (e.g., the participant's name, Marcel) and an alert-enabling representation 808A for sending an alert to the corresponding participant that a real-time communication session is available.

[0299] Figure 8DAlso shown is a participant corresponding to primary representation 810B joining the real-time communication session. In some embodiments, device 600 receives data indicating that a participant corresponding to primary representation 822B has joined the real-time communication session. Figure 8D As shown, in response to a participant corresponding to primary representation 822B joining the real-time communication session, device 600 replaces the placeholder for primary representation 822B with real-time video data from the corresponding participant. Figure 8AX ), device 600 replaces overflow representation 822B corresponding to the participant with real-time video data from the participant. As used herein, the phrase "real-time video data from a participant" (or its equivalent) includes real-time video data transmitted by a device associated with the participant (e.g., a smartphone, laptop, desktop computer, or tablet) or by a device associated with an account associated with the participant (e.g., an email account or a cloud storage account).

[0300] like Figure 8D As shown, device 600 receives (e.g., detects) user input 850B (e.g., a tap) corresponding to selection of the reminder enable representation 808. In response to input 850B, device 600 sends an instruction to cause a new notification (e.g., a call, a ringtone, etc.) indicating that a real-time communication session is available at a device associated with the corresponding participant, wherein the new notification is different from a notification previously sent to the participant. In some embodiments, the new notification is more intrusive than the previous notification (e.g., an audio ringtone, a tactile output, or a full-screen notification compared to a banner notification without audio or tactile output). Figure 8D , device 600 also displays an alert-enabled representation 808B on primary representation 810D to indicate that the participant corresponding to primary representation 810D is currently being alerted that the real-time communication session is available (e.g., due to an alert sent by another participant currently connected to the real-time communication session, such as the participant represented by primary representation 810B or an alert previously sent by the user of device 600). Device 600 also modifies alert-enabled representation 808A to indicate that the corresponding participant is being notified (e.g., by replacing “ring” with “ring”, as shown in FIG. Figure 8E shown).

[0301] like Figure 8EAs shown, based on determining that a predetermined time period has passed since the start of the real-time communication session, device 600 stops displaying overflow representation 822 and increases the size of user representation 806. In some embodiments, when overflow representation 822 is removed, primary representation 810 is resized and / or moved. In the illustrated embodiment, primary representation 810D is moved downward. In some embodiments, device 600 stops displaying overflow representation 822 based on determining that the number of participants currently participating in the real-time communication session is less than or equal to a threshold number of participants (e.g., five participants, including the user of device 600).

[0302] In the corresponding Figure 8E At a time, two participants have joined the real-time communication session. In some embodiments, after a further amount of time, the device 600 stops displaying one of the primary representations 810 and changes the layout of the real-time communication user interface 804 to a 3-to-1 layout. In some embodiments, the 4-to-1 layout and the 3-to-1 layout are in a set of predetermined layouts that have similar relative placements of the primary representations 810. In some embodiments, having relatively similar placements of the primary representations includes placement of the primary representations in similar positions on the user display 602 when there is a transition between layouts within the set (e.g., a position of a representation in one layout overlaps a position of the representation in another layout of the same set).

[0303] In some embodiments, the device 600 determines which primary representation to remove based on factors such as whether the corresponding participant has joined the real-time communication session (e.g., the representation is not removed if the corresponding participant has joined), whether the participant has been alerted (e.g., the representation of the alerted participant will not be removed until a predetermined amount of time after the corresponding alert-enabled representation is selected), the placement of the participant in the participant list (e.g., representations of participants earlier in the list are removed before representations of participants later in the list), and the placement of the representation on the real-time communication user interface 804 (e.g., representations closer to the top of the display 602 are removed before representations lower on the display 602). Figure 8F As shown, device 600 stops displaying primary representation 810A and moves primary representation 810B upward (e.g., to reduce overlap between representations 810B and 810C), while maintaining the relative vertical order of the remaining representations (e.g., representation 810B is larger than and above representation 810C, and representation 810C is above and the same size as representation 810D). Figure 8FAlso shown is reminder affordance 808A modified to "Ring Again," indicating that the corresponding participant has been reminded at least once and is not currently being reminded. Reminder affordance 808B is also modified (e.g., from "Ring" to "Ring" and a different color) to indicate that the corresponding participant has been previously reminded, but not by the user of device 600.

[0304] In the corresponding Figure 8F In some embodiments, after a further amount of time, the device 600 stops displaying the other primary representation 810 and changes the layout of the real-time communication user interface 804 to a 2-to-1 layout (e.g., Figures 8E to 8F 1 and 3 to 1 layouts in the same set of predetermined layouts shown in FIG. Figure 8G As shown, device 600 stops displaying primary representation 810D and zooms in on primary representations 810B and 810C. Figure 8G A three participant (e.g., 2-to-1) layout is shown, where the primary representations of the two participants other than the user of device 600 are equal sized and vertically aligned one above the other. The vertical order of primary representations 810B and 810C is maintained, with primary representation 810B above primary representation 810C. Figure 8F In the layout of FIG. 8 , there is no overlap between the primary representations, and the user representation 806 overlaps with the primary representation 810C.

[0305] In the corresponding Figure 8G In some embodiments, after a further amount of time, the device 600 stops displaying the other primary representation and changes the layout of the real-time communication user interface 804 to a 1-to-1 layout. Figure 8H As shown, device 600 stops displaying primary representation 810C and enlarges primary representation 810B to full screen size. In some embodiments, device 600 stops displaying primary representation 810C and enlarges primary representation 810B to full screen size at a predetermined amount of time after selecting reminder enable representation 808A. Figure 8G The layout changes to Figure 8H layout. Figure 8H A two participant (e.g., 1-to-1) layout is shown, with the remaining primary representation 810B displayed full screen and overlapped by the user representation 806. Figure 8H As shown, respectively Figures 8E to 8G Compared to the square shapes of the primary representations 810 and user representations 806 in the 4-to-1 layout, the 3-to-1 layout, and the 2-to-1 layout shown in FIG, the remaining primary representations 810B and user representations 806 are vertically elongated rectangles (e.g., oriented along the outline).

[0306] Go to Figure 8I, the third participant Stephen joins the real-time communication session. Figure 8I As shown, in response to the participant Stephen joining the real-time communication session, the device 600 Figure 8H 1 layout is converted to a 2-to-1 layout with a primary representation 810E of the most recently joined participant over representations 810B of existing participants (other than the user of device 600). In some embodiments, device 600 displays a notification 812 and / or generates an audio or tactile output indicating that a participant has joined the real-time communication session, such as Figure 8I In some embodiments, because a new primary representation is displayed when participant Stephen joins the real-time communication session, device 600 avoids displaying notification 812 or generating audio or tactile output because primary representation 810E serves as an indication that the participant has joined the real-time communication session.

[0307] Participant Stephen joins the real-time communication session in audio-only mode and provides real-time audio data without video. The audio-only communication status of participant Stephen is indicated by the main representation 810E and the notification 812. The main representation 810E indicates the audio-only communication status by displaying the participant's name and an avatar associated with the participant. The notification 812 indicates the communication status in text ("Joining this call with audio"). Figure 8J As shown, device 600 stops displaying notification 812 (eg, after a predetermined amount of time after participant Stephen joins the real-time communication session).

[0308] In some embodiments, in response to receiving data indicating that a participant who has joined via audio only is actively participating (e.g., speaking), device 600 displays a visual indication on the representation of the participant who is speaking. Figure 8J , a representative image (eg, an avatar) within primary representation 810E is animated (eg, with a "speech circle") to indicate that participant Stephen is speaking.

[0309] Go to Figure 8K , the fourth participant Marcel joins the real-time communication session. Figure 8K As shown, in response to a fourth participant joining the real-time communication session, the device 600 switches from a 2-to-1 configuration to a 3-to-1 configuration (e.g., as shown in FIG. Figure 8F ), where the primary representation 810F of the most recently joined participant is above the primary representations 810B and 810E of the existing participants (except the user of device 600).

[0310] Participant Marcel joins the real-time communication session in video mode and provides real-time video and audio data. The video communication status of participant Marcel is indicated by the main representation 810F. The main representation 810F indicates the video communication status by displaying the real-time video stream. Figure 8K As shown, device 600 does not immediately display a notification indicating that participant Marcel has joined the real-time communication session.

[0311] Go to Figure 8L , the fifth participant Hoan joins the real-time communication session. Figure 8L As shown, in response to participant Hoan joining the real-time communication session, device 600 switches from a 3-to-1 layout to a 4-to-1 layout, with primary representation 810G of the most recently joined participant being the smallest primary representation and positioned above and to the right of primary representation 810F. In some embodiments, based on determining that both participants Marcel and Hoan have joined the real-time communication session within a predetermined time window, device 600 displays a combined notification 816 indicating that participants Marcel and Hoan have joined the real-time communication session, as shown in FIG. Figure 8L In some embodiments, the combined notification 816 is displayed based on determining that the participants have joined with the same communication status (e.g., video).

[0312] As indicated by the real-time video stream displayed in primary representation 810G and representation 816A of the video camera in notification 816 , participant Hoan joins the real-time communication session in video mode, causing device 600 to receive real-time video and audio data associated with participant Hoan.

[0313] like Figure 8L As shown, the device 600 receives (e.g., detects) user input 850C (e.g., a double click) at the location of the primary representation 810F on the display 602. In some embodiments, as shown in FIG. Figure 8M As shown, in response to receiving user input 850C, device 600 enlarges primary representation 810F to the maximum representation size and shrinks primary representation 810E to the medium size (while maintaining the 4 to 1 layout). Figure 8M As shown, device 600 stops displaying notification 816 (eg, after a predetermined amount of time after participant Hoan joins the real-time communication session).

[0314] In some embodiments, the configuration of the layout is dynamically changed in response to receiving data indicating that a participant is actively participating in a real-time communication session. Figure 8M In FIG, device 600 receives data indicating that participant Hoan corresponding to primary representation 810G is actively participating. Figure 8MAs shown, in response to receiving data indicating that the participant corresponding to primary representation 810G is actively participating, device 600 enlarges primary representation 810G from a small representation size to a medium representation size and shrinks primary representation 810B from a medium representation size to a small representation size. In this way, the 4-to-1 layout maintains a small representation, two medium representations, and one large representation.

[0315] like Figure 8M As shown, the device 600 receives (e.g., detects) user input 850D (e.g., a double click) at the location of the primary representation 810G on the display 602. In some embodiments, as shown in FIG. Figure 8N As shown, in response to receiving user input 850D, device 600 enlarges primary representation 810G and brings primary representation 810G to the foreground before other displayed primary representations. Except for user representation 806, the background behind the enlarged primary representation 810G is grayed out.

[0316] like Figure 8O As shown, device 600 receives (e.g., detects) user input 850E (e.g., a tap) on cancel affordance 818. Figure 8P As shown, in response to receiving user input 850E, device 600 reduces the size of primary representation 810G and returns the display to the 4 to 1 layout displayed prior to user input 850D, except that primary representation 810G occupies the large primary representation instead of primary representation 801F, which is reduced to the medium representation size.

[0317] Go to Figure 8Q , the sixth participant joins the real-time communication session. Figure 8Q As shown, in response to a sixth participant joining the real-time communication session, and based on the number of connected participants exceeding a predetermined threshold of five participants, device 600 displays an overflow representation 822A of the sixth participant in overflow area 820 and moves user representation 806 to overflow area 820, with overflow representation 822A moving to the left of user representation 806. Figure 8QAs shown, the overflow representation 822A and the user representation 806 are centered in the overflow area 820. In some embodiments, the device 600 replaces the smallest primary representation (e.g., 810B) with the representation of the newly joined participant (e.g., the sixth participant) and displays the overflow representation of the replaced participant in the overflow area 820 (e.g., the representation 810B is reduced in size and moved to the location of the overflow representation 822A in the overflow area 820). In some embodiments, in response to the sixth participant joining the real-time communication session, and based on the number of connected participants exceeding a predetermined threshold of five participants, the device 600 displays overflow representations of the currently joined participant or all participants (currently joined and invited) in the overflow area, such as, for example Figure 8AY shown.

[0318] like Figure 8Q As shown, overflow region 820 and primary representation 810 do not overlap. In some embodiments, to make room for the overflow region, device 600 reduces the size and / or vertically translates one or more of the primary representations. In some embodiments, device 600 reduces the size of the medium-sized primary representation and maintains the size of the small and large primary representations. In some embodiments, device 600 reduces the vertical spacing and / or increases the vertical overlap between primary representations.

[0319] like Figure 8R As shown, the seventh participant joins the real-time communication session. In response to the seventh participant joining the real-time communication session, the device 600 adds the overflow representation 822B of the seventh participant in the overflow area 820 to the left of the overflow representation 822A. Figure 8R As shown, the overflow representation 822A and the user representation 806 are shifted to the right (with Figure 8Q In some embodiments, the overflow representation 822B of the seventh participant is added to the right of the overflow representation 822A between the overflow representation 822A and the user representation 806. Figure 8AZ In the illustrated embodiment, in response to a seventh participant joining the real-time communication session, the device 600 replaces the minimal primary representation (e.g., Figure 8AY 810H in ), and changes the overflow representation 822F corresponding to participant Pablo from a placeholder to a real-time video stream.

[0320] like Figure 8S As shown, the eighth participant joins the real-time communication session. In response to the eighth participant joining the real-time communication session, the device 600 adds the overflow representation 822C of the eighth participant in the overflow area 820 to the left of the overflow representation 822B. Figure 8SAs shown, overflow representations 822A and 822B and user representation 806 are shifted to the right (with Figure 8Q ), so that the overflow representations 822A, 822B and 822C and the user representation 806 are centered in the overflow area 820.

[0321] like Figure 8T As shown, the ninth participant joins the real-time communication session. In response to the ninth participant joining the real-time communication session, the device 600 adds the overflow representation 822D of the ninth participant in the overflow area 820 to the left of the overflow representation 822C. Figure 8T As shown, overflow representations 822D, 822C, and 822B are fully displayed, along with user representation 806. Overflow representation 822A is partially displayed, with the right portion overlapping user representation 806. Overflow representation 822A is shaded (e.g., having a shadow effect), with overflow representation 822A meeting user representation 806. In the illustrated embodiment, the size of overflow representation 822 and user representation 806 remains the same as the seventh, eighth, and ninth participants join the real-time communication session. Figure 8T 824 . The device 600 is shown displaying a combined notification 824 indicating that participants Mary, Mick, John, and Tom have joined the real-time communication (represented by overflow representations 822A, 822B, 822C, and 822D, respectively). In some embodiments, when additional participants (e.g., beyond the ninth participant) join the real-time communication session, the overflow representation of the newly joined participant is added to the left end of the existing overflow representation, and the existing overflow representation is shifted to the right in the overflow area 820 to make room for the new overflow representation, and the user representation 806 remains fully displayed on the right side of the overflow area 820 (e.g., the existing overflow representation is scrolled to the right "below" the user representation 806). In some embodiments, the overflow representation of the newly joined participant is added to the right end of the existing overflow representation, wherein the user representation 806 remains fully displayed on the right side of the overflow area 820. In some embodiments, if the overflow area is completely occupied, the representation of the newly joined participant added to the right end of the existing overflow representation is not immediately displayed (e.g., the new overflow representation is displayed in response to scrolling the overflow representation to the left).

[0322] Go to Figure 8U, device 600 receives data indicating that the participant corresponding to overflow representation 822B is actively participating in the real-time communication session. In some embodiments, device 600 replaces the primary representation with a representation of the actively participating participant in response to determining that the activity level of the actively participating participant has increased above the activity level of the participant represented by the primary representation. In the illustrated embodiment, in response to receiving data indicating that the participant corresponding to overflow representation 822B is actively participating in the real-time communication session, device 600 replaces the minimal primary representation with a representation of the actively participating participant and displays the overflow representation of the participant corresponding to the replaced primary representation. Figure 8U As shown, the primary representation 810B (the smallest primary representation) is shaded or begins to fade out; the overflow representation 822B of the actively participating participant is shaded or begins to fade out and decreases in size; and when the new overflow representation 822E corresponding to the participant of the primary representation 810B (the replaced primary representation) enters the overflow area 820 from the left edge of the display 602 to the left of the overflow representation 822D, the overflow representations 822C and 822D shift to the right. In some embodiments, the primary representation 810B is slightly enlarged (e.g., to emphasize that it is being replaced). Figure 8V to Figure 8W As shown, primary representation 810B is replaced by primary representation 810H of the actively participating participant; overflow representation 822B continues to shrink until it is removed; and overflow representations 822C and 822D continue to shift to the right until overflow representation 822E is fully displayed and overflow representation 822C occupies the position in overflow area 820 previously occupied by removed overflow representation 822B. Figures 8U to 8W , the overflow representation 822A and the user representation 806 remain in the same position.

[0323] In some embodiments, in response to active engagement by a user of device 600, device 600 visually distinguishes user representation 806. Figure 8AZ In the illustrated embodiment, device 600 displays a wider border around user representation 806 and highlights the initials bar at the bottom of user representation 806.

[0324] Go to Figure 8X , the device 600 is rotated from a portrait (vertical) orientation to a landscape (horizontal) orientation. Figure 8XAs shown, in response to the device 600 being rotated, the device 600 maintains the representation in the corresponding area of ​​the display 602 that was occupied before the device 600 was rotated (e.g., the user representation and each primary representation and overflow representation occupy the same area on the display 602 before and after the device 600 is rotated). The images within the representation are rotated so that the participant appears upright relative to the landscape orientation. In some embodiments, the device 600 displays an animated rotation of the primary representation or an image within the primary representation (e.g., 90 degrees opposite to the rotation direction of the device 600). In some embodiments where the overflow representation is not a square, the participant's video data is cropped and / or translated to include (e.g., center) the user's face in the corresponding overflow representation when the device 600 is rotated. In some embodiments where the primary representation is not a square, the participant's video data is cropped and / or translated to center the user's face in the corresponding primary representation when the device 600 is rotated. In some embodiments, the participant's video data is cropped to include the user's face (e.g., regardless of whether the device 600 is rotated, such as when the received video data has an aspect ratio or format different from that of the representation in which it is displayed).

[0325] like Figure 8X As shown, the participant corresponding to the primary representation 810H begins to actively participate. In response, the device 600 enlarges the size of the primary representation 810H and reduces the size of the primary representation 810E, as shown in FIG. Figure 8Y As shown. Figures 8BA to 8BB In the illustrated embodiment, the device 600 visually distinguishes the overflow representation 822F corresponding to the active participant (e.g., an emphasized initials bar).

[0326] In some embodiments, in response to a participant joining the real-time communication session, the device 600 replaces the existing primary representation with the primary representation of the newly joined participant and moves the replaced participant to the overflow area 820. Figure 8Z In FIG, participant Pablo joins the real-time communication session. In response, device 600 replaces the smallest primary representation 810E (Stephen) with the primary representation 810I of the newly joined participant Pablo and displays overflow representation 822F (Stephen) in overflow area 820. To make room for overflow representation 822F, overflow representations 822A, 822C, 822D, and 822E are scrolled (e.g., according to Figure 8Z The device 600 is shown oriented upwards. Figure 8Z As shown, overflow representation 822A is no longer visible, and overflow representation 822C is partially hidden by user representation 806.

[0327] like Figure 8AAAs shown, device 600 is rotated from a landscape (horizontal) orientation to a portrait (vertical) orientation. In response, device 600 rotates the images of the participants in the representations while maintaining the area occupied by each representation on the display (e.g., device 600 reverses the rotation performed in response to the device being rotated from the portrait orientation to the landscape orientation).

[0328] exist Figure 8AB , the participant corresponding to the primary representation 810G leaves (disconnects) the real-time communication session. Figure 8BC In the embodiment shown, in response, the device 600 displays a notification 828 indicating that the participant has left the real-time communication session. Since the number of participants remains greater than five, the device 600 maintains a 4 to 1 layout with the display of the overflow area 820. Figure 8AC As shown, the primary representation 810G is replaced by the primary representation 810J of the participant Stephen having the next highest activity level, and the overflow representation 822F corresponding to the participant Stephen is removed from the overflow area 820. When the overflow representation 822F is removed, the remaining overflow representations 822A, 822C, 822D, and 822E scroll to the left. Figure 8BD In the illustrated embodiment, the overflow representation 822D of participant Hoan is updated to display a placeholder indicating that participant Hoan is not connected and a reminder affordance requesting participant Hoan to rejoin.

[0329] In some embodiments, if the number of participants in the real-time communication decreases from six to five when a participant leaves the real-time communication session, the device 600 stops displaying the overflow area 820 and, optionally, increases the size and / or vertical spacing of one or more of the primary representations.

[0330] exist Figure 8AC , device 600 receives (e.g., detects) user input 850F (e.g., a double click) corresponding to selection of overflow representation 822E by participant Allison. Figure 8AD As shown, in response to receiving user input 850F, device 600 displays an enlarged representation 830 of participant Allison before the 4-to-1 layout of the currently displayed primary representation 810. The display of the enlarged representation 830 is similar to Figure 8N 810G in the display, except that the enlarged representation 830 is not presented based on the main representation in the 4 to 1 layout.

[0331] like Figure 8AD As shown, device 600 receives (e.g., detects) user input 850G (e.g., a tap) corresponding to selection to turn off affordance 818. Figures 8AE to 8AFAs shown, in response to receiving user input 850F, device 600 displays a primary representation 810K of participant Allison in place of primary representation 810J of participant Stephen (e.g., the participant with the lowest activity status replaces the participant with the current primary representation). Primary representation 810K (at least initially) includes a band (or grid) 842A along the bottom edge that includes the participant's name and an affordance for enlarging the representation (e.g., as shown in FIG. 1 ). Figure 8AD shown). Figures 8AE to 8AF An exemplary animation of replacing the primary representation 810J with the primary representation 810K is shown. The primary representation 810K is initially displayed in a small representation size (similar to Figure 8AC The primary representation 810J in the display is then enlarged to a medium representation size, while the primary representation 810H is reduced from medium to small. The overflow representation 822E corresponding to the participant Allison is reduced in size and / or faded out, while the overflow representation 822F of the replaced participant is translated into the overflow area 820. In some embodiments, ceasing to display the primary representation 810J and displaying the primary representation 810K includes a crossfade from the primary representation 810J to the primary representation 810K and / or a tint to create a "shimmer" effect. Figures 8BE to 8BH An embodiment is shown in which the primary representation 810J is removed using a glitter effect.

[0332] In some embodiments, the overflow area 820 is scrollable. Figures 8BH to 8BI In the embodiment shown in , the device 600 scrolls the overflow representation 822 in the overflow area 820 in response to active participation of a participant for whom the corresponding overflow representation is not currently displayed or is not fully displayed. Figure 8BH As shown, the overflow representation 822F of the participant (corresponding to the main representation 810I) is partially hidden in the overflow area 820. Figure 8BH , the participant corresponding to the primary representation 810I actively participates in the real-time communication session. Figure 8BI As shown, in response to receiving data indicating that a participant corresponding to primary representation 810I is actively participating, device 600 zooms in on primary representation 810I, scrolls overflow representation 822 so that overflow representation 822F is fully displayed (e.g., near the middle of the displayed portion of overflow area 820), and visually indicates overflow representation 822F by highlighting the initials bar at the bottom of overflow representation 822F.

[0333] Go to Figure 8AG , device 600 detects contact 850H, which moves horizontally from right to left across overflow region 820 while maintaining contact with display 602. Figure 8AHAs shown, in response to receiving contact 850H, device 600 scrolls overflow representation 822 to the left on display 602 such that overflow representation 822F moves partially off the left edge of display 602 and overflow representation 822A no longer overlaps user representation 806. The ISE, contact 850H, and overflow representation 822 move the same horizontal amount.

[0334] When the overflow area 820 is scrolled, the primary representation 810 remains unchanged. Moreover, the user representation 806 remains in the overflow area 820.

[0335] exist Figures 8BJ to 8BK In the embodiment shown, contact 850H starts from the left side of overflow area 820 and moves horizontally to the right. In response, device 600 scrolls overflow representation 822 to the right on display 602. Figures 8BJ to 8BK , when the device 600 reaches the user representation 806, the device removes (eg, no longer displays) the representation from the display 602.

[0336] Go to Figure 8AH , the device 600 receives (e.g., detects) user input 850I (e.g., a tap) at a location on the display 602 where the primary representation 810 is not displayed. Figure 8AI As shown, in response to receiving user input 850I, the device 600 displays a call control menu 832, which includes a call enable indication 834 (e.g., to leave the real-time communication session), an effect enable indication 836, a menu enable indication 838, and a processing enable indication 840.

[0337] like Figure 8AI As shown, the call control menu 832 does not overlap with the overflow area 820 and the primary representations 810. To make room for the call control menu 832, the device 600 reduces the size and / or vertically translates one or more of the primary representations 810 (e.g., reduces the size of the canvas area 811) and moves the overflow area 820 upward on the display 602. In some embodiments, the device 600 reduces the size of the medium-sized primary representation and maintains the size of the small and large primary representations. In some embodiments, the device 600 reduces the vertical spacing and / or increases the vertical overlap between the primary representations.

[0338] In response to receiving user input 850I, device 600 also displays bands 842A through 842D on each primary representation. Bands 842A through 842D include the names of the corresponding participants and affordances of the enlarged representations (e.g., Figure 8O In some embodiments, the vertical height of bands 842A through 842D is the same as the amount of vertical overlap between the primary representations.

[0339] like Figure 8AJAs shown, device 600 receives (e.g., detects) user input 850J (e.g., a tap) corresponding to selection of menu enable representation 838. In some embodiments, user input 850J includes an upward swipe originating from call control menu 832 (e.g., on processing enable representation 840). Figure 8AK As shown, in response to user input 850J, device 600 expands call control menu 832 to display additional information and controls related to the real-time communication session, similar to the information about Figure 6H Additional information and controls described in call control menu 630.

[0340] like Figure 8AL As shown, device 600 receives (e.g., detects) user input 850K (e.g., a tap) corresponding to selection of effect affordance 836. Figure 8AM As shown, in response to receiving user input 850K, device 600 displays a magnified image 843 of the user of device 600 from camera 603 and modifies call control menu 832. In modified call control menu 832, effect enable representation 836 is highlighted, and effect option enable representations 844A to 844E are displayed. Figure 8AN , device 600 receives (e.g., detects) user input 850L (e.g., a tap) corresponding to selection of effect option enable representation 844A. Figure 8AO As shown, in response to receiving user input 850L, device 600 modifies call control menu 832 to include avatar option affordances 846A through 846C. Figure 8AO , device 600 receives (e.g., detects) user input 850M (e.g., a tap) corresponding to selection of avatar affordance 846C. Figure 8AP As shown, in response to receiving user input 850M, device 600 displays effect 843A on the user's face in image 843 and scrolls call control menu 832 to center affordance 846C of the currently selected avatar in call control menu 832. Figure 8AP , device 600 receives (e.g., detects) user input 850N (e.g., a tap) on cancel enable indication 848. Figure 8AQ As shown, in response to receiving user input 850N, the device 600 returns the call control menu 832 to Figure 8AN , and maintain the display of effect 843A on the enlarged image 843. Figure 8AQ , the device 600 receives (eg, detects) user input 8500 (eg, a tap) on the display 602 outside of the magnified image 843 and the call control menu 832. Figure 8ARAs shown, in response to receiving user input 850O, device 600 stops displaying magnified image 843 and displays effect 843A on the image of the user in user representation 806. When effect 843A is activated, as in Figure 8AR , the real-time video stream of the user of device 600 includes effect 843A.

[0341] like Figure 8AR As shown, call control menu 832 includes media affordance 844E. Device 600 receives (e.g., detects) user input 850P (e.g., a tap) corresponding to selection of media affordance 844E. Figure 8AS As shown, in response to receiving user input 850P, device 600 modifies call control menu 832 to include media option affordances 852A through 852D. Figure 8AS , device 600 receives (e.g., detects) user input 850Q (e.g., a tap) corresponding to selection of media option affordance 852A (sketch). In response to receiving user input 850Q, device 600 displays a representation of the media content associated with media option affordance 852A in canvas area 811.

[0342] like Figure 8AT As shown, in response to receiving user input 850Q, device 600 replaces primary representation 810K (e.g., a minimal primary representation) with representation 810L of the selected media item. Figure 8AT , device 600 receives (e.g., detects) user input 850R (e.g., a downward swipe on call control menu 832) corresponding to a request to dismiss call control menu 832. Figure 8AU As shown, in response to receiving user input 850R, device 600 ceases displaying call control menu 832 .

[0343] In some embodiments, selecting the media option affordance 852A causes the associated media content to be shared with other participants of the real-time communication session (e.g., displayed in a canvas area on a display of another participant's device). In some embodiments, the participants of the real-time communication session can interact with the shared media content (e.g., via a sequence of one or more inputs in a user interface of the real-time communication application). Figure 8AV An embodiment is shown in which additional feature 810L-1 has been added to the media content of representation 810L. In some embodiments, in response to a participant's interaction with the media content associated with representation 801L, device 600 zooms in on representation 801L (e.g., similar to how a primary representation is zoomed in and then selected or how a participant actively engages with that representation). Figure 8AV As shown, representation 810L is enlarged (with Figure 8AT compared to). Figure 8AV Also shown is a representation 810M of other content shared in the real-time communication session (eg, a game between two participants).

[0344] Figures 9A to 9K Flowchart showing a method according to some embodiments. Method 900 is performed at a device having a display (e.g., 100, 300, 500, 600). Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0345] As described below, method 900 provides, among other things, an intuitive way to provide an adaptive and intelligent real-time communication user interface. Furthermore, method 900 provides a user with an intuitive way to interact with the real-time communication user interface. This method reduces the user's cognitive burden by providing an adaptive and intelligent real-time communication interface based on the number of participants in the real-time communication, thereby creating a more efficient human-computer interface. For battery-powered computing devices, this enables users to more quickly and efficiently view the real-time communication user interface based on a predetermined number of participants in the real-time communication, thereby conserving power and increasing the time between battery charges.

[0346] At block 901 , a device receives a request (eg, 850A) to display a user interface for a real-time communication session between two or more participants.

[0347] At block 902, in response to receiving a request to display a user interface for a real-time communication session between two or more participants, the device displays a real-time communication user interface (e.g., 804) including simultaneously displaying multiple representations of participants in the real-time communication session (e.g., 810A through 810D), wherein displaying the multiple representations of participants in the real-time communication session includes blocks 902 and 903. In some embodiments, the real-time communication session user interface includes a placeholder associated with a first participant (e.g., Figure 8C 810A) (e.g., a graphical placeholder (e.g., an image or avatar), a text placeholder (e.g., a name or initials)) of a first participant who has been invited to join the real-time communication session but has not yet joined the real-time communication session (e.g., a participant other than a user of an electronic device; a participant who has yet to join the communication session and has been sent an invitation to join the communication session). In some embodiments, if the participant is sharing a real-time media stream that includes a real-time video stream, the representation includes an image of the real-time video stream (e.g., Figure 8D In some embodiments, if a participant is sharing a real-time media stream in audio-only format, the representation includes an avatar of the corresponding participant (e.g., Figure 8IIn some embodiments, the plurality of representations of participants in the real-time communication session includes a user communication state selected from an audio-only state (e.g., the participant is communicating using only audio), a video state (e.g., the participant is talking using both video and audio), a video paused state (e.g., the participant's video is paused), a video undecoded state (e.g., the participant's video stream cannot be processed due to latency issues, issues with the video stream format, etc.), a left state (e.g., the participant has left the real-time communication session), and a waiting-to-join state (e.g., the participant has been invited to join the real-time communication session but has not yet joined the real-time communication).

[0348] At block 903, based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, the device simultaneously displays representations of a first plurality of participants (e.g., 810A through 810D) other than the user of the electronic device at one or more sizes greater than the corresponding threshold sizes, while displaying representations of a second plurality of participants (e.g., 822) in an overflow area (e.g., 820), wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of the one or more other participants displayed in the overflow area are displayed at sizes smaller than the corresponding threshold sizes. In some embodiments, the currently active participant (e.g., Figure 8J 810E) (e.g., the most active participant, the participant who is speaking or moving in a manner that satisfies the set of active participant criteria) is displayed as part of the first plurality of participants. In some embodiments, the currently active participant is displayed in a primary area (e.g., 811) of the user interface (e.g., an area displaying the first plurality of participants). In some embodiments, if a participant that is not currently displayed in the primary area (e.g., corresponding to Figure 8U 822B) becomes the currently active participant, the representation of the participant is displayed in the main area (e.g., Figure 8V In some embodiments, representations of the first plurality of participants (e.g., 810) are displayed at two or more different sizes (e.g., where more active participants or more recently active participants are displayed at a larger size than less active participants or more recently active participants), and representations of the second plurality of participants in the overflow area (e.g., 822) are displayed at the same size.

[0349] In some embodiments, the overflow area (e.g., 820) also includes representations of the first plurality of participants (e.g., 810A through 810D) (e.g., representations of the participants displayed in the main area of ​​the user interface are also displayed in the overflow area) (e.g., Figure 8AXIn some embodiments, the overflow area (e.g., 820) also includes placeholder representations of real-time communication session invitees who are not currently connected to the real-time communication session (e.g., Figure 8AW 822A in the example) (e.g., avatar, name, initials). Displaying representations of participants differently based on whether a specified condition is met (e.g., based on whether the real-time communication session includes more than the number of participants) allows a user to more easily or efficiently view representations of participants in the real-time communication session. Performing optimization operations when a set of conditions have been met without requiring further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0350] Optionally, at blocks 904 and 905, the device updates a display of representations of a first plurality of participants (e.g., 810A through 810D) (e.g., the arrangement, position, size, positioning, visual indications, visual content, and / or video feeds of the representations) at a first rate (e.g., the representations of the first plurality of participants are a real-time video feed updated at a frame rate equal to or greater than a typical video refresh rate, such as 24, 30, or 60 frames per second), and updates a display of representations of a second plurality of participants (e.g., 822) in the overflow area (e.g., the arrangement, position, size, positioning, visual indications, visual content, and / or video feeds of the representations) at a second rate, wherein the second rate is lower than the first rate (e.g., the thumbnails in the overflow area are updated over time, but at a rate that is less than the video refresh rate of the representations of the plurality of participants in the canvas area 811, such as 1, 5, or 10 frames per second). Updating the display of representation groups at different rates allows the device to provide feedback to the user about the current state of representations in real-time communications, while conserving processing power to provide one group of representations at a faster rate than another group of representations, thereby enabling the user to view higher priority representation groups at a higher rate. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0351] Optionally, at block 906, the device displays a representation of the view of the electronic device's camera (e.g., a self-viewing camera view) in the overflow area (e.g., 806) while displaying representations of the second plurality of participants (e.g., 822) in the overflow area (e.g., 820). Displaying the representation of the view of the electronic device's camera provides the user with visual feedback about the current view state of the user's camera (e.g., what the user is displaying to other participants in the real-time communication session). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0352] Optionally, at blocks 907, 908, 909, and 910, the device detects an input corresponding to the overflow region (e.g., 850H) (e.g., a horizontal finger drag). In response to detecting the input corresponding to the overflow region, the device scrolls the overflow region (e.g., scrolls the representations of the second plurality of participants in the first direction according to the input) to stop displaying (e.g., hiding) a subset of the representations of the second plurality of participants (e.g., Figures 8BJ to 8BK While scrolling the overflow area, the device maintains display of a representation of the view of the electronic device's camera (e.g., 806) in the overflow area (e.g., Figures 8BJ to 8BK ). In some embodiments, scrolling the overflow area includes an animation that displays one or more representations of the second plurality of participants, which appear to scroll below the representation of the device's camera's view. Maintaining display of the representation of the electronic device's camera's view while scrolling the overflow area provides the user with continuous feedback about the current view state of the electronic device's camera (e.g., what the user is displaying to other participants in the real-time communication session) while also allowing the user to scroll through the representations of the participants in the overflow area. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0353] Optionally, at blocks 911, 912, and 913, a representation (e.g., Figure 8ACIn response to detecting the input corresponding to selection of a representation from the plurality of representations of the second plurality of participants in the overflow area, the device displays the representation of the participant corresponding to the selected representation in the overflow area (e.g., 822E). Figure 8AE 810K) replaces the representations of the first plurality of participants in the plurality of representations (e.g., Figure 8AC In some embodiments, replacing the display of a representation in the plurality of representations of the first plurality of participants with a representation of a participant corresponding to the selected representation in the overflow area includes ceasing to display the selected representation in the representations of the second plurality of participants in the overflow area. In some embodiments, ceasing to display the selected representation in the overflow area includes an animation of the selected representation shrinking. In some embodiments, other representations in the overflow area are translated to fill the gap in which the selected representation is displayed. In some embodiments, replacing the display of a representation in the plurality of representations of the first plurality of participants with a representation of a participant corresponding to the selected representation in the overflow area includes displaying the representation of the participant corresponding to the selected representation at a first size, and then enlarging the representation of the participant corresponding to the selected representation to a second size that is larger than the first size. In some embodiments, the representation of the selected participant replaces the smallest representation in the canvas area (corresponding to the participant with the lowest activity level in the canvas area) and then automatically increases in size (e.g., increases to a medium representation size), while the size of another representation (corresponding to the participant with the second lowest activity level in the canvas area) decreases (e.g., decreases to a small representation size).

[0354] Replacing the display of a representation corresponding to a selected representation in the overflow area and / or ceasing to display the selected representation in the overflow area provides additional control options without cluttering the UI and provides the user with more control over the device and the display of representations by helping the user quickly replace the representation with the user's selected representation. Providing additional control over the device without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0355] Optionally, at block 914, the device displays a first animation effect (e.g., Figures 8AA to 8AC) (e.g., the representation grows larger and then shrinks, optionally with a cross-fade and tint to create a "shimmer" effect), thereby affecting a representation in the representations of the first plurality of participants. Optionally, at block 915, the device displays a second animation effect affecting a selected representation in the representations of the second plurality of participants, the second animation effect sharing animation characteristics with the first animation effect (e.g., Figures 8BF to 8BH (e.g., the rate of fading / retracting from the primary display may correspond to a flickering rate in the overflow display). Displaying a second animation corresponding to the first animation provides visual feedback to quickly identify which representations are being switched and / or replaced. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0356] Optionally, at block 916, further in response to detecting an input corresponding to the overflow area (e.g., 850H), the device displays new (e.g., previously hidden) representations of the second plurality of participants in the overflow area (e.g., Figures 8AG to 8AH 822A in FIG. 822B ). Displaying new representations of participants in the overflow area in response to input allows the user to view the representations more easily and efficiently by increasing the accessibility of the displayed representations. Performing optimization operations when a set of conditions have been met without requiring further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0357] Optionally, at block 917, in response to receiving data indicating that a new participant has joined or has been added to the real-time communication session, the device displays a representation of the new participant in the overflow area at the end of the series of representations of the second plurality of participants in the overflow area (e.g., Figure 8R 822B in the .

[0358] Optionally, at blocks 918, 919, 920, and 921, the device receives second data related to an activity level (e.g., speaking, moving) of a first participant in the second plurality of participants. In response to receiving the second data, the device increases, emphasizes (e.g., adds or initiates) a visual feature (e.g., size, highlights the representation (e.g., displays a border), animates (e.g., pulses avatar), brings an image to the front) of a representation corresponding to a representation of the first participant in the plurality of representations of the second plurality of participants in the overflow area (e.g., displays a border), and animates (e.g., pulses avatar). Figure 8U 822B). In response to receiving the second data, the device reduces, de-emphasizes (e.g., increases or initiates) a visual feature of a representation (e.g., size, highlighting the representation (e.g., displaying a border), animation (e.g., pulsing avatar), bringing an image to the front) based on the activity level of a first participant in the second plurality of participants, the representation corresponding to the first participant in the representation of the second plurality of participants in the overflow area. Emphasizing and / or de-emphasizing visual indications when specified conditions are met allows a user to quickly identify the activity levels of participants (including the most active and / or least active participants). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In addition, emphasizing and / or de-emphasizing when specified conditions are met provides the user with visual feedback indicating the activity level of each participant. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0359] Optionally, at blocks 922 and 923, while displaying the real-time communication user interface, the device detects that a second participant in the second plurality of participants becomes a currently active participant (e.g., a most active participant, a participant who speaks or moves in a manner that satisfies a set of active participant criteria). In response to detecting that the second participant in the second plurality of participants becomes the currently active participant and based on determining that a representation of the second participant in the second plurality of participants is not currently displayed in the overflow area, the device scrolls the overflow area to display a representation of the second participant in the second plurality of participants in the overflow area (e.g., Figures 8BH to 8BI 822F). Dynamically scrolling the overflow area to display active participants in the overflow area allows viewing representations of active participants in the overflow area without selecting any additional inputs. Reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0360] At block 924, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than or equal to a threshold value, the device displays representations of participants other than the user of the electronic device at one or more sizes that are greater than the corresponding threshold sizes and does not display representations of participants other than the user of the electronic device at one or more sizes that are less than or equal to the corresponding threshold sizes (e.g., Figure 8P ). In some embodiments, the electronic device receives (e.g., before displaying a user interface for a real-time communication session) a plurality of real-time media streams associated with respective participants of a plurality of participants in a real-time video communication session. In some embodiments, the number of received real-time media streams corresponds to the number of participants currently joining the real-time video communication session, excluding participants using the device. Displaying the representations of participants differently based on whether a specified condition is met (e.g., based on whether the real-time communication session includes less than or equal to the number of participant display representations without an overflow area) allows the user to more easily or more efficiently view the representations of the participants in the real-time communication session. Performing optimization operations when a set of conditions has been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0361] In some embodiments, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than a second threshold, none of the multiple representations of remote participants (e.g., participants other than the user) overlap with each other (e.g., Figure 8J In some embodiments, based on determining that the number of participants in the real-time communication session includes a number of participants that is greater than or equal to a second threshold, at least some of the multiple representations of remote participants (e.g., participants other than the user) overlap with each other (e.g., Figure 8K ).

[0362] Optionally, at blocks 925 and 926, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than a second threshold (e.g., two participants, including the user of the electronic device (e.g., one other participant)), the device displays at least one of the representations of the participants other than the user of the device (e.g., a first aspect ratio (e.g., a rectangular aspect ratio)). Figure 8H Based on determining that the number of participants in the real-time communication session includes a number of participants greater than a second threshold (e.g., three or more participants), the device displays at least one of the representations of the participants other than the user of the device (e.g., a square aspect ratio) in a second aspect ratio that is different from the first aspect ratio (e.g., a square aspect ratio). Figure 8I810B). Based on satisfying specified conditions, representations of participants are displayed in different aspect ratios based on the number of participants in the real-time communication session, allowing the device to provide an intelligent real-time communication user interface by maximizing the size and / or organization of the representations to provide a better user experience based on the number of participants, thereby allowing the user to more easily and efficiently view the representations on the device. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0363] Optionally, at blocks 927 and 928, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than a second threshold (e.g., two participants), the device displays a representation of the view of the device's camera (e.g., a self-view from the device's camera) in a third aspect ratio (e.g., a rectangular aspect ratio) (e.g., Figure 8H Based on determining that the number of participants in the real-time communication session includes a number of participants greater than or equal to a second threshold (e.g., three or more participants), the device displays a representation of the view of the device's camera (e.g., a self-view from the device's camera) in a fourth aspect ratio (e.g., a square aspect ratio) that is different from the third aspect ratio (e.g., a square aspect ratio). Figure 8J 806). Displaying the device's camera view in different aspect ratios based on the number of participants in the real-time communication session based on specified conditions being met allows the device to provide a smart real-time communication user interface to the user by maximizing the size of the representation of the camera view and allowing the user to more easily and efficiently view the representation on the device. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0364] Optionally, at blocks 929 and 930, based on the number of participants in the real-time communication session being a first number of participants, the device displays representations of the participants (e.g., Figure 8JIn some embodiments, the first layout is one of a plurality of predefined layouts in which representations of users are anchored at different locations in the user interface (e.g., a randomly selected layout). In some embodiments, the layout (e.g., arrangement) of the representations of the first plurality of participants (e.g., excluding the user of the device) is determined based on the number of participants in the real-time communication session. Based on the number of participants in the real-time communication session being a second number of participants, the device displays the representations of the participants in a second layout that is different from the first layout (e.g., Figure 8K ) (e.g., the layout changes as participants join / leave the real-time communication session). In some embodiments, the first layout and the second layout are in a series of predetermined layouts that have similar relative placement of representations of users. (e.g., the series of predetermined layouts includes layouts that work well together). In some embodiments, the series of predetermined layouts includes layout members for multiple possible numbers of participants (e.g., a first member for one non-device user participant (e.g., 1 or more layouts), a second member for two non-device user participants (e.g., 2 or more layouts), and so on. In some embodiments, the electronic device includes multiple series of predetermined layouts, where the members of each series share common features). Displaying representations with similar relative placement of user representations in different layouts based on the number of users allows users to view representations more easily and efficiently through increased visibility. Performing optimization operations when a set of conditions have been met and no further user input is required enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and...

Claims

1. A method performed at an electronic device having a display, comprising: receiving a request to display a user interface for a video communication session between two or more participants; as well as In response to receiving the request to display the user interface for the video communications session between two or more participants, displaying a video communications user interface including simultaneously displaying a plurality of video representations of participants in the video communications session, wherein: Based on determining that the number of participants in the video communication session is greater than a first threshold number, at least some of the plurality of simultaneously displayed video representations of participants other than a user of the electronic device partially overlap one another; Based on determining that the number of participants in the video communication session is less than or equal to the first threshold number, none of the multiple simultaneously displayed video representations of participants other than the user of the electronic device overlap one another; Based on determining that the number of participants in the video communication session is greater than a second threshold number, displaying video representations of a first plurality of participants other than users of the electronic device at one or more sizes greater than corresponding threshold sizes, while displaying video representations of a second plurality of participants in an overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the video representations of the second plurality of participants displayed in the overflow area are displayed at one or more sizes less than or equal to the corresponding threshold sizes; updating a display of the video representations of the first plurality of participants at a first rate; and Display of the video representations of the second plurality of participants in the overflow area is updated at a second rate, wherein the second rate is lower than the first rate.

2. The method according to claim 1, comprising: detecting a change in relative activity levels of participants in the video communication session after displaying the video communication session; as well as In response to detecting a change in relative activity levels of participants in the video communication session: Based on determining that the number of participants in the video communication session is greater than the first threshold number, adjusting the sizes of multiple video representations of participants relative to each other while maintaining at least some overlap between the multiple video representations of multiple participants other than a user of the electronic device; as well as Based on determining that the number of participants in the video communications session is less than or equal to the first threshold number, maintaining sizes of the multiple video representations of participants relative to each other.

3. The method of claim 1 , wherein displaying the plurality of video representations of participants in the video communication session further comprises: Based on determining that the number of participants in the video communication session is less than or equal to the second threshold number, video representations of participants other than a user of the electronic device are displayed at one or more sizes greater than the corresponding threshold sizes.

4. The method of claim 3, wherein at least some of the video representations of the first plurality of participants other than the user of the electronic device overlap with each other, and wherein none of the video representations of the second plurality of participants in the overflow area overlap with each other.

5. The method according to claim 3, comprising: detecting an input corresponding to the overflow region; as well as In response to detecting the input corresponding to the overflow region: scrolling the video representations of the second plurality of participants displayed in the overflow area to hide display of a subset of the video representations of the second plurality of participants; as well as While scrolling the overflow area, display of a video representation of a view of a camera of the electronic device is maintained in the overflow area.

6. The method according to claim 3, comprising: receiving data indicating that the number of participants in the video communication session has increased while displaying video representations of the participants other than the user of the electronic device; as well as In response to receiving the data indicating that the number of participants in the video communications session has increased: reducing the size of one or more of the video representations of participants other than the user of the electronic device; and One or more video representations of new participants in the video communications session are displayed.

7. The method of claim 3, wherein the video communications session user interface further comprises a placeholder associated with a particular participant who has been invited to the video communications session but has not yet joined the video communications session.

8. The method of claim 7, wherein the placeholder comprises a first reminder affordance for the particular participant, the method comprising: receiving input corresponding to a selection of the first alert affordance; as well as In response to receiving the input corresponding to selection of the first reminder affordance, a new notification is sent to the particular participant that is different from a previous notification sent to the particular participant.

9. The method of claim 7, wherein the placeholders and the video representations of the first plurality of participants are arranged in a first layout, the method comprising: Based on determining that the particular participant has not joined the video communication session within a predetermined amount of time after being invited: stop displaying said placeholder; as well as Changing the arrangement of the video representations of the first plurality of participants to a second layout different from the first layout includes enlarging and / or moving one or more of the video representations of the first plurality of participants.

10. The method according to claim 7, comprising: Based on determining that the particular participant has joined the video communication session, replacing display of the placeholder with a video representation of the particular participant.

11. The method according to claim 3, comprising: detecting an input corresponding to a selection of one of the video representations of the second plurality of participants in the overflow area; as well as In response to detecting the input corresponding to a selection of one of the video representations of the second plurality of participants in the overflow area: Replace display of one of the first plurality of participants' video representations with the participant's video representation corresponding to the selected video representation in the overflow area.

12. The method according to claim 3, comprising: while displaying the video communication user interface, detecting that a second participant of the second plurality of participants becomes a currently active participant; as well as In response to detecting that the second participant of the second plurality of participants becomes the currently active participant and based on determining that a video representation of the second participant of the second plurality of participants is not currently displayed in the overflow area, scrolling the overflow area to display a video representation of the second participant of the second plurality of participants in the overflow area.

13. The method of claim 3, wherein the overflow area further includes placeholder video representations of invitees to the video communications session who are not currently connected to the video communications session.

14. The method according to claim 3, comprising: receiving a fourth input corresponding to a request to display a menu while displaying the plurality of video representations of participants in the video communications session; as well as In response to receiving the fourth input: displaying a menu in the user interface of the video communication session; moving at least a third video representation of the plurality of video representations of participants in the video communications session; as well as A size of at least a fourth video representation of the plurality of video representations of participants in the video communications session is changed.

15. The method according to claim 3, comprising: receiving input corresponding to a request to display a first menu while displaying the plurality of video representations of participants in the video communications session; In response to receiving the input corresponding to the request to display the first menu, displaying the first menu in the video communication user interface, wherein the first menu includes options for selecting a content item; receiving input corresponding to a selection of a content item; as well as In response to receiving the input corresponding to a selection of the content item, replacing the video representation of one participant displayed at a size greater than the corresponding threshold size with the video representation of the content item displayed at a size greater than the corresponding threshold size.

16. The method of claim 1, wherein displaying the plurality of video representations of participants in the video communication session further comprises: displaying at least one of the video representations of the participants other than the user of the electronic device at a first aspect ratio based on determining that the number of participants in the video communication session is less than a second threshold number; as well as Based on determining that the number of participants in the video communication session is greater than the second threshold number, at least one of the video representations of the participants other than the user of the electronic device is displayed at a second aspect ratio different from the first aspect ratio.

17. The method of claim 1, wherein displaying the plurality of video representations of participants in the video communication session further comprises: displaying a video representation of a view of a camera of the electronic device at a third aspect ratio based on determining that the number of participants in the video communication session is less than a third threshold number; as well as Based on determining that the number of participants in the video communication session is greater than or equal to the third threshold number, displaying the video representation of the view of the camera of the electronic device at a fourth aspect ratio different from the third aspect ratio.

18. The method of claim 1, wherein displaying the plurality of video representations of participants in the video communication session further comprises: Based on the number of participants in the video communication session being a first number, displaying the video representations of the participants in a first layout; as well as Based on the number of participants in the video communication session being a second number, displaying the video representations of the participants in a second layout different from the first layout.

19. The method according to claim 1, comprising: displaying, as part of the plurality of video representations of participants in the video communications session, a video representation of a view of a camera of the electronic device at a first size; receiving a first input corresponding to a selection to display a first menu while displaying the plurality of video representations of participants in the video communications session; in response to receiving the first input corresponding to a selection to display the first menu, displaying the first menu in the video communications user interface, wherein the first menu includes an option to display a visual effect on video representations of participants in the video communications session; as well as In response to receiving a second input corresponding to a selection of the option to display a visual effect, the video representation of the view of the camera of the electronic device is displayed at a second size, wherein the second size is larger than the first size.

20. The method according to claim 1, comprising: receiving a third input corresponding to a selection to zoom in on a first video representation of the plurality of video representations of the participants in the video communications session; in response to receiving the third input corresponding to a selection to zoom in on the first video representation of the plurality of video representations of the participants in the video communications session, zooming in on the first video representation; as well as A size of at least one video representation of the plurality of video representations of the participants in the video communications session that is different from the first video representation is reduced.

21. The method according to claim 1, comprising: In response to receiving first data indicating a change in orientation of the electronic device, changing the display of a first video representation of the multiple video representations of the participants in the video communication session based on the position of a face in the first video representation, wherein the change in orientation of the electronic device is a change from a portrait orientation of the electronic device to a landscape orientation of the electronic device or from a landscape orientation of the electronic device to a portrait orientation of the electronic device.

22. An electronic device, comprising: monitor; one or more processors; and A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-21.

23. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors of an electronic device having a display, cause the one or more processors to perform the method according to any one of claims 1-21.