Video application graphical effects

Abstract

The present disclosure generally relates to methods and user interfaces for displaying graphical effects in video applications. In some embodiments, methods and user interfaces are described for displaying graphical effects during a real-time communication session. In some embodiments, methods and user interfaces are described for sharing content during a real-time communication session. In some embodiments, methods and user interfaces are described for displaying graphical effects in multiple video applications. In some embodiments, methods and user interfaces are described for framing a subject in a representation of the field of view of one or more camera sensors.

Description

[Technical Field]

[0001] (Cross-Reference to Related Applications) This application claims priority from U.S. Provisional Patent Application No. 63 / 448,237, entitled "VIDEO APPLICATION GRAPHICAL EFFECTS", filed on February 24, 2023; U.S. Provisional Patent Application No. 63 / 465,093, entitled "VIDEO APPLICATION GRAPHICAL EFFECTS", filed on May 9, 2023; U.S. Provisional Patent Application No. 63 / 470,885, entitled "VIDEO APPLICATION GRAPHICAL EFFECTS", filed on June 3, 2023; and U.S. Patent Application No. 18 / 416,784, entitled "VIDEO APPLICATION GRAPHICAL EFFECTS", filed on January 18, 2024. The entire content of each of these applications is incorporated herein by reference.

[0002] (Technical Field) The present disclosure relates generally to computer user interfaces, and more specifically to displaying graphical effects in video applications. [Background Art]

[0003] Users often apply graphical effects to the representation of a camera field of view in video applications. Such techniques typically require a user to provide various inputs in order to apply a graphical effect. [Summary of the Invention]

[0004] However, some techniques for displaying graphical effects using electronic devices are generally cumbersome and inefficient. For example, some existing techniques use complex and time-consuming user interfaces that may involve multiple key presses or keystrokes. These existing techniques take more time than necessary, wasting both the user's time and the device's energy. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for displaying graphical effects. Such methods and interfaces optionally complement or replace other methods for displaying graphical effects. Such methods and interfaces reduce the cognitive burden on the user and create a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and increase the interval between battery charges.

[0006] According to some embodiments, methods are described for being performed in a computer system that communicates with a display generation component. The method includes, via a display generation component, displaying a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors; and, while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of the person, the method includes, displaying a first graphical effect at a first effect location within the representation of the field of view of one or more camera sensors according to a determination that the first gesture was detected at a first gesture location within the field of view of one or more camera sensors; and displaying a first graphical effect at a second effect location within the representation of the field of view of one or more camera sensors that is different from the first effect location, according to a determination that the first gesture is at a second gesture location within the field of view of one or more camera sensors that is different from the first gesture location.

[0007] According to some embodiments, a non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component, the one or more programs display, via the display generating component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is displayed, a first gesture performed by a first part of the person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of the person, A command to display a first graphical effect via a display generation component includes, displaying, according to a determination that a first gesture has been detected at a first gesture location within the field of view of one or more camera sensors, displaying the first graphical effect at a first effect location within a representation of the field of view of one or more camera sensors, according to a determination that the first gesture is at a second gesture location within the field of view of one or more camera sensors, different from the first effect location, displaying the first graphical effect at a second effect location within a representation of the field of view of one or more camera sensors, different from the first effect location.

[0008] According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, the one or more programs display, via the display generation component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is displayed, a first gesture performed by a first part of the person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of the person, the display A command to display a first graphical effect via a demonstration generation component includes, displaying, a command to display the first graphical effect at a first effect location in a representation of the field of view of one or more camera sensors, according to a determination that a first gesture has been detected at a first gesture location within the field of view of one or more camera sensors, and a command to display the first graphical effect at a second effect location in a representation of the field of view of one or more camera sensors, different from the first effect location, according to a determination that the first gesture is at a second gesture location within the field of view of one or more camera sensors, different from the first gesture location.

[0009] According to several embodiments, a computer system that communicates with a display generation component is described. The computer system comprises one or more processors and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs display a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of the person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of the person, the display generation component A command to display a first graphical effect via an element includes, displaying the first graphical effect at a first effect location in a representation of the field of view of one or more camera sensors, according to a determination that a first gesture has been detected at a first gesture location within the field of view of one or more camera sensors, and displaying the first graphical effect at a second effect location in a representation of the field of view of one or more camera sensors, different from the first effect location, according to a determination that the first gesture is at a second gesture location within the field of view of one or more camera sensors, different from the first gesture location.

[0010] According to several embodiments, a computer system communicating with a display generation component is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs having means to display a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, via the display generation component; and a first gesture performed by a first part of a person while the representation of the field of view of one or more camera sensors is being displayed, which is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of a person, the display generation component The system includes means for displaying a first graphical effect via a component, the display means for displaying a first graphical effect at a first effect location in a representation of the field of view of one or more camera sensors, according to a determination that a first gesture has been detected at a first gesture location within the field of view of one or more camera sensors, and means for displaying a first graphical effect at a second effect location in a representation of the field of view of one or more camera sensors, different from the first effect location, according to a determination that a first gesture is at a second gesture location within the field of view of one or more camera sensors, different from the first gesture location.

[0011] According to several embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component, the one or more programs display, via the display generating component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is displayed, a first gesture performed by a first part of the person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of the person, the display A command to display a first graphical effect via a generating component includes, displaying, a command to display a first graphical effect at a first effect location in a representation of the field of view of one or more camera sensors, according to a determination that a first gesture has been detected at a first gesture location within the field of view of one or more camera sensors, and a command to display a first graphical effect at a second effect location in a representation of the field of view of one or more camera sensors, different from the first effect location, according to a determination that the first gesture is at a second gesture location within the field of view of one or more camera sensors, different from the first gesture location.

[0012] According to some embodiments, a method is described that is performed in a computer system that communicates with a display generation component. The method includes: displaying a representation of the field of view of one or more camera sensors via the display generation component, wherein a person is visible within the representation of the field of view of one or more camera sensors; while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of a person, displaying a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors via the display generation component, in accordance with a determination that a first set of criteria is met; and ceasing to display the graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, in accordance with a determination that the first set of criteria is not met.

[0013] According to some embodiments, a non-temporary computer-readable storage medium is described. ru. A non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, the one or more programs include instructions to display, via the display generation component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors; while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of a person, the display generation component displays a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, according to a determination that a first set of criteria is met; and to discontinue displaying the graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, according to a determination that a first set of criteria is not met.

[0014] According to some embodiments, a temporary computer-readable storage medium is described. ru. A temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, the one or more programs include instructions to display, via the display generation component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of a person, the display generation component displays a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, according to a determination that a first set of criteria is met, and to discontinue displaying the graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, according to a determination that a first set of criteria is not met.

[0015] According to some embodiments, a computer system communicating with a display generation component is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to display, via the display generation component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors; a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors while the representation of the field of view of one or more camera sensors is being displayed; and, in response to the detection of the first gesture performed by the first part of a person, a graphical effect corresponding to the first gesture is displayed via the display generation component in accordance with a determination that a first set of criteria is met; and, in accordance with a determination that a first set of criteria is not met, the graphical effect corresponding to the first gesture is not displayed within the representation of the field of view of one or more camera sensors.

[0016] According to some embodiments, a computer system communicating with a display generation component and one or more input devices is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to: means for displaying a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors; means for detecting a first gesture performed by a first part of a person within the field of view of one or more camera sensors while the representation of the field of view of one or more camera sensors is being displayed; means for detecting a first gesture performed by a first part of a person within the field of view of one or more camera sensors; means for displaying a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors via the display generation component, in accordance with a determination that a first set of criteria is met; and means for ceasing to display the graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors, in accordance with a determination that a first set of criteria is not met.

[0017] According to several embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, the one or more programs including instructions to display, via the display generation component, a representation of the field of view of one or more camera sensors, wherein a person is visible within the representation of the field of view of one or more camera sensors, and while the representation of the field of view of one or more camera sensors is being displayed, a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, and in response to the detection of the first gesture performed by the first part of a person, the display generation component displays a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors according to a determination that a first set of criteria is met, and to discontinue displaying the graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors according to a determination that a first set of criteria is not met.

[0018] A method is described according to several embodiments. The method is described as being performed in a first computer system that communicates with a display generating component, one or more input devices, and one or more camera sensors, the first computer system being associated with a first user. The method includes displaying a user interface for a real-time communication session between the first computer system and one or more second computer systems via a display generating component, the user interface for the real-time communication session receiving an input requesting to share content associated with a first participant in the real-time communication session while the user interface for the real-time communication session is being displayed, and displaying a user interface element within the user interface for the real-time communication session via a display generating component in response to receiving a request to display content associated with a first participant in the real-time communication session, the user interface element being a composite of content associated with a first participant in the real-time communication session and a representation of the first participant within the user interface for the real-time communication session.

[0019] According to some embodiments, a non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a first computer system that communicates with a display generating component, one or more input devices, and one or more camera sensors, wherein the first computer system is associated with a first user, and the one or more programs include instructions to display a user interface for a real-time communication session between the first computer system and one or more second computer systems via the display generating component, and to display a user interface element within the user interface for a real-time communication session via the display generating component in response to receiving a request to display content associated with a first participant in the real-time communication session, wherein the user interface element is a composite of content associated with a first participant in the real-time communication session and a representation of the first participant within the user interface for the real-time communication session.

[0020] According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a first computer system that communicates with a display generating component, one or more input devices, and one or more camera sensors, wherein the first computer system is associated with a first user, and the one or more programs include instructions to display a user interface for a real-time communication session between the first computer system and one or more second computer systems via the display generating component, and to display a user interface element within the user interface for a real-time communication session via the display generating component in response to receiving a request to display content associated with a first participant in the real-time communication session, wherein the user interface element is a composite of content associated with a first participant in the real-time communication session and a representation of the first participant within the user interface for the real-time communication session.

[0021] According to several embodiments, a first computer system is described. The first computer system communicates with a display generation component, one or more input devices, and one or more camera sensors, and the first computer system is associated with a first user. The first computer system comprises one or more processors and a memory that stores one or more programs configured to be executed by one or more processors, the one or more programs including instructions to display a user interface for a real-time communication session between the first computer system and one or more second computer systems via the display generation component, the user interface for a real-time communication session

[0022] According to several embodiments, a first computer system is described. The first computer system communicates with a display generation component, one or more input devices, and one or more camera sensors, and the first computer system is associated with a first user. The first computer system comprises one or more processors and a memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions to display a user interface for a real-time communication session between the first computer system and one or more second computer systems via a display generation component, the user interface element being a composite of content associated with the first participant in the real-time communication session and a representation of the first participant in the real-time communication session.

[0023] According to several embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a first computer system that communicates with a display generation component, one or more input devices, and one or more camera sensors, the first computer system being associated with a first user. One or more programs include instructions to display a user interface for a real-time communication session between the first computer system and one or more second computer systems via the display generation component, the user interface for the real-time communication session being displayed, the input being received requesting to share content associated with a first participant in the real-time communication session, and in response to receiving a request to display content associated with a first participant in the real-time communication session, a user interface element being displayed within the user interface for the real-time communication session via the display generation component, the user interface element being a composite of content associated with a first participant in the real-time communication session and a representation of the first participant within the user interface for the real-time communication session.

[0024] According to several embodiments, a method is described that is performed in a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors. The method includes detecting a first set of one or more inputs via one or more input devices, which include a request to display a user interface of a first application; and, in response to detecting the first set of one or more inputs, displaying the user interface of the first application via the display generation component, which includes displaying a first representation of the field of view of one or more camera sensors, and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, wherein the display includes displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a first graphical enhancement having second characteristics different from the first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value, and the user interface of the first application Displaying a screen, detecting a second set of one or more inputs including a request to display a user interface for a second application different from a first application via one or more input devices, and displaying the user interface for the second application via a display generation component in response to the detection of the second set of one or more inputs, includes displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement, and displaying a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a graphical enhancement having a second characteristic according to a determination that a parameter associated with the graphical enhancement is set to a second value, and displays the user interface for the second application.

[0025] According to some embodiments, non-temporary computer-readable storage media are described.A non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors, and the one or more programs detect a first set of one or more inputs via one or more input devices, which includes a request to display the user interface of a first application, and in response to detecting the first set of one or more inputs, the display generation component displays the user interface of a first application, which includes displaying a first representation of the field of view of one or more camera sensors, and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, which includes displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value, The command includes displaying a user interface for a first application, which includes displaying a first graphical enhancement having a second characteristic different from that characteristic; detecting a second set of one or more inputs via one or more input devices, which includes a request to display a user interface for a second application different from that of the first application; and, in response to detecting the second set of one or more inputs, displaying a user interface for a second application via a display generation component, which includes displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement, which displays a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a first value; and displaying a graphical enhancement having a second characteristic according to a determination that a parameter associated with the graphical enhancement is set to a second value.

[0026] According to some embodiments, a temporary computer-readable storage medium is described.A temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors, wherein one or more programs detect a first set of one or more inputs via one or more input devices, including a request to display the user interface of a first application, and in response to detecting the first set of one or more inputs, the display generation component displays the user interface of the first application via the display generation component, which includes displaying a first representation of the field of view of one or more camera sensors and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, which includes displaying a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value The command includes displaying a user interface for a first application, which includes displaying a first graphical enhancement having a second characteristic different from that of the first application, detecting a second set of one or more inputs via one or more input devices, which includes a request to display a user interface for a second application different from that of the first application, and, in response to detecting a second set of one or more inputs, displaying a user interface for a second application via a display generation component, which includes displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement, which includes displaying a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a graphical enhancement having a second characteristic according to a determination that a parameter associated with the graphical enhancement is set to a second value, and displaying a user interface for a second application.

[0027] According to some embodiments, a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors is described.The computer system comprises one or more processors and memory for storing one or more programs configured to run by one or more processors, wherein one or more programs detect a first set of one or more inputs via one or more input devices, which include a request to display a user interface for a first application, and in response to detecting the first set of one or more inputs, displaying the user interface for a first application via a display generating component includes displaying a first representation of the field of view of one or more camera sensors and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, and displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and a first graphical enhancement having second characteristics different from the first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value The command includes displaying a user interface for a first application, which includes displaying a graphical enhancement; detecting a second set of one or more inputs via one or more input devices, which includes a request to display a user interface for a second application different from the first application; and, in response to detecting the second set of one or more inputs, displaying the user interface for the second application via a display generation component, which includes displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement, which displays a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value; and displaying a graphical enhancement having second characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value.

[0028] According to some embodiments, a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors is described.The computer system comprises one or more processors and memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include means for detecting a first set of one or more inputs including a request to display a user interface of a first application via one or more input devices, and means for displaying a user interface of a first application via a display generation component in response to detecting a first set of one or more inputs, the display means includes displaying a first representation of the field of view of one or more camera sensors and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, the display means includes displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a first graphical enhancement having second characteristics different from the first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value. The means for displaying a user interface of a first application, including means for displaying a graphical enhancement; means for detecting a second set of one or more inputs via one or more input devices, including a request to display a user interface of a second application different from the first application; and means for displaying a user interface of a second application via a display generation component, in response to the detection of the second set of one or more inputs, including displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement, wherein the display means includes means for displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and means for displaying a graphical enhancement having second characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value, and commands to the means for displaying a user interface of a second application.

[0029] According to several embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, one or more input devices, and one or more camera sensors. One or more programs detect a first set of one or more inputs via one or more input devices, which includes a request to display a user interface of a first application, and in response to detecting the first set of one or more inputs, displaying the user interface of the first application via the display generation component includes displaying a first representation of the field of view of one or more camera sensors, and a graphical enhancement applied to the representation of the field of view of one or more camera sensors, and displaying a graphical enhancement having first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a first value, and displaying a first graphical enhancement having second characteristics different from the first characteristics according to a determination that a parameter associated with the graphical enhancement is set to a second value different from the first value, the first application The command to display the user interface of the second application includes: displaying the user interface of the first application; detecting a second set of one or more inputs including a request to display the user interface of a second application different from the first application via one or more input devices; and, in response to the detection of the second set of one or more inputs, displaying the user interface of the second application via a display generation component, which includes displaying a second representation of the field of view of one or more camera sensors using a graphical enhancement; and displaying a graphical enhancement having a first characteristic according to a determination that a parameter associated with the graphical enhancement is set to a first value; and displaying a graphical enhancement having a second characteristic according to a determination that a parameter associated with the graphical enhancement is set to a second value.

[0030] According to some embodiments, a method executed in a computer system in communication with a display generation component and one or more camera sensors is described. The method includes: detecting a subject at a first position while displaying a representation of a field of view of the one or more camera sensors via the display generation component; detecting input corresponding to a request to frame the subject within the representation of the field of view of the one or more camera sensors while the subject is at the first position; in response to detecting the input, framing the subject in a representation of a first portion of the field of view of the one or more camera sensors; detecting movement of the subject from the first position to a second position; and in response to detecting the movement of the subject from the first position to the second position, maintaining display of the representation of the first portion of the field of view of the one or more camera sensors.

[0031] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more camera sensors. The one or more programs include instructions for: detecting a subject at a first position while displaying a representation of a field of view of the one or more camera sensors via the display generation component; detecting input corresponding to a request to frame the subject within the representation of the field of view of the one or more camera sensors while the subject is at the first position; in response to detecting the input, framing the subject in a representation of a first portion of the field of view of the one or more camera sensors; detecting movement of the subject from the first position to a second position; and in response to detecting the movement of the subject from the first position to the second position, maintaining display of the representation of the first portion of the field of view of the one or more camera sensors.

[0032] According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component and one or more camera sensors, the one or more programs including instructions that, while displaying a representation of the field of view of one or more camera sensors via the display generating component, detect a subject in a first position, detect an input corresponding to a request to frame the subject within a representation of the field of view of one or more camera sensors while the subject is in the first position, frame the subject in a representation of a first portion of the field of view of one or more camera sensors in response to the detection of the input, detect the movement of the subject from the first position to a second position, and maintain the display of the representation of the first portion of the field of view of one or more camera sensors in response to the detection of the movement of the subject from the first position to the second position.

[0033] According to some embodiments, a computer system communicating with a display generation component and one or more camera sensors is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions via the display generation component to detect a subject in a first position while displaying a representation of the field of view of one or more camera sensors, to detect an input corresponding to a request to frame the subject in a representation of the field of view of one or more camera sensors while the subject is in the first position, to frame the subject in a representation of a first portion of the field of view of one or more camera sensors in response to the detection of the input, to detect the movement of the subject from the first position to a second position, and to maintain the display of the representation of the first portion of the field of view of one or more camera sensors in response to the detection of the movement of the subject from the first position to the second position.

[0034] According to some embodiments, a computer system communicating with a display generation component and one or more camera sensors is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to means for detecting a subject in a first position while displaying a representation of the field of view of one or more camera sensors, detecting an input corresponding to a request to frame the subject in a representation of the field of view of one or more camera sensors while the subject is in the first position, framing the subject in a representation of a first portion of the field of view of one or more camera sensors in response to the detection of the input, detecting the movement of the subject from the first position to a second position, and maintaining the display of the representation of the first portion of the field of view of one or more camera sensors in response to the detection of the movement of the subject from the first position to the second position.

[0035] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more camera sensors, the one or more programs including instructions that, via the display generation component, detect a subject in a first position while displaying a representation of the field of view of one or more camera sensors, detect an input corresponding to a request to frame the subject in a representation of the field of view of one or more camera sensors while the subject is in the first position, frame the subject in a representation of a first portion of the field of view of one or more camera sensors in response to the detection of the input, detect the movement of the subject from the first position to a second position, and maintain the display of the representation of the first portion of the field of view of one or more camera sensors in response to the detection of the movement of the subject from the first position to the second position.

[0036] A method is described according to several embodiments. The method includes, in a computer system communicating with a display generating component and one or more input devices, detecting a request via one or more input devices to display a user interface menu associated with first content, and, in response to detecting a request to display a user interface menu, displaying the user interface menu via the display generating component, wherein the display includes, if selected according to a determination that a set of sharing option criteria is met and the content is shared during a real-time communication session, displaying a first set of one or more shared user interface elements that, if selected according to a determination that a set of sharing option criteria is met and the content is not shared during a real-time communication session, displaying a second set of one or more shared user interface elements that, if selected, includes the first set of one or more shared user interface elements, which is different from the second set of one or more shared user interface elements.

[0037] According to some embodiments, a non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component and one or more input devices. The one or more programs include instructions to detect a request to display a user interface menu associated with first content via one or more input devices, and to display the user interface menu via the display generating component in response to detecting the request to display the user interface menu, and to display a first set of one or more shared user interface elements that, if selected according to a determination that a set of sharing option criteria is satisfied and the content is shared during a real-time communication session, include the first set of one or more shared user interface elements that, if selected according to a determination that a set of sharing option criteria is satisfied and the content is not shared during a real-time communication session, include the first set of one or more shared user interface elements distinct from the second set of one or more shared user interface elements.

[0038] According to some embodiments, a temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component and one or more input devices. The one or more programs include instructions to detect a request to display a user interface menu associated with first content via one or more input devices, and to display the user interface menu via the display generating component in response to detecting the request to display the user interface menu, and displaying includes, if selected according to a determination that a set of sharing option criteria is met and the content is shared during a real-time communication session, displaying a first set of one or more shared user interface elements that include the first content in a real-time communication session, if selected according to a determination that a set of sharing option criteria is met and the content is not shared during a real-time communication session, wherein the first set of one or more shared user interface elements is different from the second set of one or more shared user interface elements.

[0039] According to several embodiments, a computer system communicating with a display generation component and one or more input devices is described. The computer system comprises one or more processors and a memory storing one or more programs configured to be executed by one or more processors. One or more processors detect a request to display a user interface menu associated with first content via one or more input devices, and in response to detecting a request to display a user interface menu, the display includes, through the display generation component, an instruction to display the user interface menu, which, if selected according to a determination that a set of sharing option criteria is met and the content is shared during a real-time communication session, includes displaying a first set of one or more shared user interface elements that include the first content in a real-time communication session, if selected according to a determination that a set of sharing option criteria is met and the content is not shared during a real-time communication session, wherein the first set of one or more shared user interface elements is different from the second set of one or more shared user interface elements.

[0040] According to some embodiments, a computer system communicating with a display generation component and one or more input devices is described. The computer system includes means for detecting a request to display a user interface menu associated with first content via one or more input devices, and means for displaying the user interface menu via the display generation component in response to detecting a request to display the user interface menu, wherein the display includes displaying a first set of one or more shared user interface elements that, when selected according to a determination that a set of sharing option criteria is satisfied and the content is shared during a real-time communication session, include the first set of one or more shared user interface elements that, when selected according to a determination that a set of sharing option criteria is satisfied and the content is not shared during a real-time communication session, include the first set of one or more shared user interface elements distinct from the second set of one or more shared user interface elements.

[0041] According to several embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices. One or more programs include instructions to detect a request to display a user interface menu associated with first content via one or more input devices, and to display the user interface menu via the display generation component in response to detecting a request to display a user interface menu, and to display a first set of one or more shared user interface elements that, if selected according to a determination that a set of shared option criteria is met and the content is shared during a real-time communication session, include displaying a first set of one or more shared user interface elements that, if selected according to a determination that a set of shared option criteria is met and the content is not shared during a real-time communication session, include displaying a second set of one or more shared user interface elements that include the first set of one or more shared user interface elements, which is different from the second set of one or more shared user interface elements.

[0042] The executable instructions that perform these functions are optionally contained within a non-temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0043] Therefore, the device is provided with a faster and more efficient method and interface for displaying graphical effects, thereby improving the effectiveness, efficiency, and user satisfaction of such a device. Such methods and interfaces may complement or replace other methods for displaying graphical effects. [Brief explanation of the drawing]

[0044] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0045] [Figure 1A] This is a block diagram showing a portable multifunctional device with a touch-sensitive display, according to several embodiments.

[0046] [Figure 1B] This is a block diagram showing exemplary components for event handling according to several embodiments.

[0047] [Figure 2] Several embodiments of a portable multifunctional device having a touchscreen are shown.

[0048] [Figure 3] This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to several embodiments.

[0049] [Figure 4A] This document illustrates an exemplary user interface for an application menu on a portable multifunction device, according to several embodiments.

[0050] [Figure 4B] This document illustrates exemplary user interfaces for multifunctional devices having a touch-sensitive surface separate from the display, according to several embodiments.

[0051] [Figure 5A] This document describes personal electronic devices according to several embodiments.

[0052] [Figure 5B] A Studio of Personal Electronic Devices in Several Embodiments.

[0053] [Figure 5C] The following are illustrative diagrams of communication sessions between electronic devices according to several embodiments.

[0054] [Figure 6A] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6B] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6C] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6D] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6E] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6F] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6G] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6H]This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6I] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6J] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6K] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6L] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6M] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6N] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6O] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6P] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6Q] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6R] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6S]This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6T1] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6T2] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6U] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6V] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6W] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6X] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 6Y] This document presents exemplary user interfaces for displaying graphical effects based on gesture detection, according to several embodiments.

[0055] [Figure 7A] This flowchart illustrates a method for displaying graphical effects based on gesture detection, according to several embodiments. [Figure 7B] This flowchart illustrates a method for displaying graphical effects based on gesture detection, according to several embodiments.

[0056] [Figure 8A] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8B] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8C] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8D] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8E] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8F] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8G] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8H] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8I] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8J] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8K] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8L]This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8M] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8N] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8O] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8P] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8Q] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8R] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8S] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8T] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8U] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8V]This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8W] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8X] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8Y] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments. [Figure 8Z] This document presents exemplary user interfaces for sharing content between computer systems during a real-time communication session, based on several embodiments.

[0057] [Figure 9] This flowchart illustrates several embodiments of methods for sharing content between computer systems during a real-time communication session.

[0058] [Figure 10A] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10B] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10C] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10D]This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10E] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10F] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10G] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10H] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10I] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10J] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10K] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10L1] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10L2] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10M1]This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10M2] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10N1] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10N2] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10O] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10P] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 10Q] This document presents exemplary user interfaces for displaying graphical enhancements in several different video applications, according to several embodiments.

[0059] [Figure 11A] This flowchart illustrates a method for displaying graphical enhancements in several different video applications, according to several embodiments. [Figure 11B] This flowchart illustrates a method for displaying graphical enhancements in several different video applications, according to several embodiments.

[0060] [Figure 12A]This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12B] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12C] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12D] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12E] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12F] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12G] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12H] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12I] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12J] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12K]This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12L] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments. [Figure 12M] This document illustrates exemplary user interfaces for framing representations of subjects within the field of view of one or more camera sensors, according to several embodiments.

[0061] [Figure 13] This flowchart illustrates a method for framing a subject in a representation of the field of view of one or more camera sensors, according to several embodiments.

[0062] [Figure 14A] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14B] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14C] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14D] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14E] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14F] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14G] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14H] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14I]This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14J] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14K] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14L] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14M] This document shows exemplary user interfaces for displaying menus according to several embodiments. [Figure 14N] This document shows exemplary user interfaces for displaying menus according to several embodiments.

[0063] [Figure 15] This is a flowchart illustrating methods for displaying a menu according to several embodiments. [Modes for carrying out the invention]

[0064] The following description includes exemplary methods, parameters, etc. However, it should be noted that such descriptions are not intended to limit the scope of this disclosure, but rather are provided to describe exemplary embodiments.

[0065] There is a need for electronic devices that provide efficient methods and interfaces for enhancing communication through graphical effects and / or shared content during real-time communication sessions. For example, there is a need to provide efficient methods and interfaces for displaying graphical effects. Another example is the need to provide efficient methods and interfaces for preventing unintended effects during real-time communication sessions. Another example is the need to provide efficient methods and interfaces for sharing content during real-time communication sessions. Another example is the need to provide efficient methods and interfaces for managing effects across multiple video applications. Such technologies can reduce the cognitive burden on users, improve communication through graphical effects and / or shared content, and thereby increase productivity. Furthermore, such technologies can reduce the power consumption of the processor and battery that would normally be wasted on redundant user input.

[0066] Figures 1A to 1B, 2, 3, 4A to 4B, and 5A to 5C below provide a description of exemplary devices for performing techniques for managing event notifications. Figures 6A to 6Y show exemplary user interfaces for displaying graphical effects in response to detected gestures. Figures 7A to 7B are flowcharts illustrating how to display graphical effects in response to detected gestures according to several embodiments. The user interfaces in Figures 6A to 6Y are used to illustrate processes described later, including the processes in Figures 7A to 7B. Figures 8A to 8Z show exemplary user interfaces for sharing content during a real-time communication session. Figure 9 is a flowchart illustrating how to share content during a real-time communication session according to several embodiments. The user interfaces in Figures 8A to 8Z are used to illustrate processes described later, including the processes in Figure 9. Figures 10A to 10Q show exemplary user interfaces for managing graphical enhancements across multiple different video applications according to several embodiments. Figures 11A to 11B are flowcharts illustrating methods for managing graphical enhancements across multiple different video applications. The user interfaces in Figures 10A to 10Q are used to illustrate the processes described below, including the processes in Figures 11A to 11B. Figures 12A to 12M show exemplary user interfaces for framing representations of subjects within representations of the field of view of one or more cameras. Figure 13 is a flowchart illustrating methods for framing representations of subjects within representations of the field of view of one or more cameras, according to several embodiments. Figures 12A to 12M are used to illustrate the processes described below, including the process in Figure 13. Figures 14A to 14N show exemplary user interfaces for displaying menus. Figure 15 is a flowchart illustrating methods for displaying menus, according to several embodiments. Figures 14A to 14N are used to illustrate the processes described below, including the process in Figure 14.

[0067] The processes described below enhance the usability of the device and streamline the user-device interface by various techniques, including providing users with improved visual feedback, reducing the number of inputs required to perform actions, offering additional control options without cluttering the user interface with additionally displayed controls, performing actions without requiring further user input when a set of conditions is met, and / or other techniques. These techniques also reduce power consumption and improve the device's battery life by enabling users to use the device more quickly and efficiently.

[0068] Furthermore, in any method described herein that is conditional on one or more conditions being met in one or more steps, it should be understood that the method described can be repeated in multiple iterations such that all the conditions that the steps of the method are conditional on are met in different iterations of the method. For example, if a method requires that a first step be performed if a condition is met, and a second step be performed if the condition is not met, a person skilled in the art will understand that the steps described in the claim are repeated in an unspecified order until the conditions are met and then not met. Thus, a method described in one or more steps that depends on one or more conditions being met can be rewritten as a method that is repeated until each of the conditions described in the method is met. However, this is not required for a claim of a system or computer-readable medium that includes instructions that perform a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency has been met without explicitly repeating the steps of the method until all the conditions that the steps of the method are conditional on are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0069] In the following description, terms such as “first,” “second,” etc., are used to describe various elements, but these elements should not be limited by these terms. In some embodiments, these terms are used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch may be called the second touch, and similarly, the second touch may be called the first touch. In some embodiments, the first touch and the second touch are two distinct references to the same touch. In some embodiments, both the first touch and the second touch are touches, but they are not the same touch.

[0070] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. In the descriptions of the various embodiments and the accompanying claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise explicitly stated in the context. Furthermore, it should be understood that, as used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the enumerated items relating to the invention. It will be further understood that, as used herein, the terms “includes,” “comprises,” and / or “comprising,” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0071] The term "if" can be interpreted, at will, depending on the context, as meaning "when" or "upon," or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined" or "[a ​​stated condition or event] is detected" can be interpreted, at will, depending on the context, as meaning "upon determining" or "in response to determining," or "[the stated condition or event] is detected" or "[the stated condition or event] is detected."

[0072] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Optionally, other portable electronic devices such as laptop computers or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad) are also used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). In some embodiments, the electronic device is a computer system communicating (e.g., via wired communication, via wireless communication) with a display-generating component. The display-generating component is configured to provide a visual output, such as a display via a CRT display, a display via an LED display, or a display via image projection. In some embodiments, the display-generating component is integrated with the computer system. In some embodiments, the display generation component is separate from the computer system. As used herein, "display" content includes displaying content (e.g., video data rendered or decoded by the display controller 156) by transmitting data (e.g., image data or video data) via a wired or wireless connection to an integrated or external display generation component in order to visually generate the content.

[0073] The following discussion describes electronic devices including displays and touch-sensitive surfaces. However, it should be understood that electronic devices optionally include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0074] The device typically supports a variety of applications, including drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0075] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or modified on an application-by-application basis and / or within individual applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications with intuitive and transparent user interfaces for the user.

[0076] Here, we turn our attention to embodiments of portable devices equipped with touch-sensitive displays. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112 according to several embodiments. The touch-sensitive display 112 may be referred to for convenience as a “touchscreen” and may be known or referred to as a “touch-sensitive display system”. Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact intensity sensors 165 (e.g., touch-sensitive surfaces such as the touch-sensitive display system 112 of Device 100) that detect the intensity of contact on Device 100. Device 100 optionally includes one or more tactile output generators 167 that generate tactile outputs on Device 100 (for example, on touch-sensitive surfaces such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0077] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (e.g., finger contact), or a proxy for the force or pressure of contact on the touch-sensitive surface. The strength of contact has a range of values, including at least four distinct values, and more typically, including several hundred (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine the estimated force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or modification of the contact area detected on the touch-sensing surface, the capacitance and / or modification of the touch-sensing surface adjacent to the contact, and / or the resistance and / or modification of the touch-sensing surface adjacent to the contact may optionally be used as a substitute for the force or pressure of the contact on the touch-sensing surface. In some implementations, the substitute measurement of the contact force or pressure is used directly to determine whether or not an intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of the contact force or pressure is converted into an estimate of the force or pressure, and this estimate is used to determine whether or not an intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of contact as an attribute of user input, it becomes possible for users to access additional device functions that may otherwise be inaccessible (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control such as a knob or button) on reduced-size devices where the implementation area for displaying affordances is limited.

[0078] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with the touch-sensitive surface of the user (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. In another embodiment, movement of a touch-sensitive surface may be optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.

[0079] It should be understood that device 100 is merely an example of a portable multifunction device, and that device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0080] Memory 102 optionally includes high-speed random-access memory and optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0081] The peripheral interface 118 can be used to connect the device's input and output peripherals to the CPU 120 and memory 102. One or more processors 120 operate or execute various software programs (such as computer programs (including instructions)) and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data. In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0082] The RF (radio frequency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. The RF circuit 108 optionally includes a well-known circuit for detecting a near-field communication (NFC) field using a short-range communication radio. Wireless communication is not limited to this, but optionally includes Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPADA), and long-term evolution.Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, Email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant messaging (e.g., Extensible Messaging and Presence Protocol) Using any of several communication standards, protocols, and technologies, including the XMPP protocol, the Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), the Instant Messaging and Presence Service (IMPS), and / or the Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this specification.

[0083] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 in Figure 2). The headset jack provides an interface between the audio circuit 110 and a detachable audio input / output peripheral device, such as an output-only headphone or a headset that has both an output (e.g., headphones for one or both ears) and an input (e.g., a microphone).

[0084] The I / O subsystem 106 connects input / output peripherals on device 100, such as the touchscreen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, a depth camera controller 169, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / transmit electrical signals to other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some embodiments, one or more input controllers 160 are optionally connected to (or not connected to) one of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in Figure 2) optionally include up / down buttons for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206 in Figure 2). In some embodiments, the electronic device is a computer system communicating with one or more input devices (e.g., via wireless communication over wired communication). In some embodiments, one or more input devices include a touch-sensitive surface (e.g., a trackpad as part of a touch-sensitive display). In some embodiments, one or more input devices include one or more camera sensors (e.g., one or more optical sensors 164 and / or one or more depth camera sensors 175), for example, to track user gestures as input (e.g., hand gestures and / or air gestures). In some embodiments, one or more input devices are integrated with the computer system. In some embodiments, one or more input devices are separate from the computer system.In some embodiments, an air gesture is a gesture detected without the user touching (or independently of) an input element that is part of the device, and is based on detected movement of a part of the user's body in the air, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0085] As described in U.S. Patent Application No. 11 / 322,549, “Unlocking a Device by Performing Gestures on an Unlock Image,” filed December 23, 2005, U.S. Patent No. 7,657,849, which is incorporated herein by reference in its entirety, a quick press of a push button optionally releases the lock on the touchscreen 112, or optionally initiates a process to unlock the device using gestures on the touchscreen. A longer press of a push button (e.g., 206) optionally turns power on or off the device 100. The functionality of one or more of the buttons is optionally customizable by the user. The touchscreen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0086] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively, “graphics”). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0087] The touchscreen 112 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on the touchscreen 112 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 112 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touchscreen 112 and the user corresponds to the user's finger.

[0088] The touchscreen 112 optionally uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies may also be used in other embodiments. The touchscreen 112 and the display controller 156 optionally, but not limited to, use any of a number of currently known or future-developed touch sensing technologies, including capacitive, resistive, infrared, and surface acoustic technologies, as well as other proximity sensor arrays or other elements that determine one or more points of contact with the touchscreen 112, to detect contact and any movement or interruption thereof. In exemplary embodiments, projected mutual capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.

[0089] The touch-sensitive displays in some embodiments of the touchscreen 112 are optionally similar to the multi-touch-sensitive touchpads described in U.S. Patent No. 6,323,846 (Westerman et al.), No. 6,570,557 (Westerman et al.), and / or No. 6,677,932 (Westerman), and / or U.S. Patent Application Publication 2002 / 0015024(A1), which are each incorporated herein by reference in their entirety. However, the touchscreen 112 displays visual output from device 100, whereas the touch-sensitive touchpad does not provide visual output.

[0090] The touch-sensitive displays in some embodiments of the touchscreen 112 are described in the following applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input These are described in (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface", (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface", (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard", and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

[0091] The touchscreen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally touches the touchscreen 112 using any suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily using finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0092] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0093] Device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with generating, managing, and distributing power within the portable device.

[0094] The device 100 also optionally includes one or more optical sensors 164. Figure 1A shows an optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts that light into data representing an image. The optical sensor 164 works in conjunction with an imaging module 143 (also called a camera module) to optionally capture still images or video. In some embodiments, the optical sensor is located on the back of the device 100 opposite the touchscreen display 112 on the front of the device, so that the touchscreen display can be used as a viewfinder for acquiring still images and / or video. In some embodiments, the optical sensor is located on the front of the device so that the user's image is optionally acquired for video conferencing while the user is viewing other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that a single optical sensor 164 can be used for both video conferencing and still image and / or video acquisition, together with the touchscreen display.

[0095] Device 100 also optionally includes one or more depth camera sensors 175. Figure 1A shows a depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object in the scene (e.g., a face) from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with an imaging module 143 (also called a camera module), the depth camera sensor 175 is optionally used to determine the depth map of different parts of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is positioned on the front of Device 100 to optionally acquire an image of the user with depth information for video conferencing while the user is viewing other video conferencing participants on a touchscreen display, and also to capture a selfie image with depth map data. In some embodiments, the depth camera sensor 175 is positioned on the back of the device, or on both the back and front of Device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that the depth camera sensor 175, together with the touchscreen display, can be used for both video conferencing and the acquisition of still and / or video images.

[0096] In some embodiments, the depth map (e.g., depth map image) includes information (e.g., values) about the distance of objects in the scene from a viewpoint (e.g., camera, light sensor, depth camera sensor). In one embodiment of the depth map, each depth pixel defines the position on the Z-axis of the viewpoint where its corresponding 2D pixel is located. In some embodiments, the depth map is composed of pixels, each pixel defined by a value (e.g., 0 to 255). For example, a value of "0" represents the furthest pixel located in the "3D" scene, and a value of "255" represents the pixel located closest to the viewpoint (e.g., camera, light sensor, depth camera sensor) in the "3D" scene. In other embodiments, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various feature parts of the target object as seen from the depth camera (e.g., the relative depth of the eyes, nose, mouth, and ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the contour of the target object in the z-direction.

[0097] Device 100 also optionally includes one or more contact intensity sensors 165. Figure 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in the I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112). In some embodiments, at least one contact intensity sensor is located on the back of Device 100, opposite the touchscreen display 112 located on the front of Device 100.

[0098] Furthermore, device 100 optionally includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is optionally coupled to an input controller 160 in the I / O subsystem 106. The proximity sensor 166 optionally functions as described in U.S. Patent Applications 11 / 241,839, “Proximity Detector In Handheld Device,” 11 / 240,788, “Proximity Detector In Handheld Device,” 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output,” 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices,” and 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals,” which are all incorporated herein by reference. In some embodiments, if the multifunction device is placed near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 112.

[0099] Device 100 also optionally includes one or more tactile output generators 167. Figure 1A shows a tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact intensity sensor 165 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on device 100 that can be sensed by the user of device 100. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of device 100) or horizontally (e.g., forward / backward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite the touchscreen display 112 located on the front of device 100.

[0100] The device 100 also optionally includes one or more accelerometers 168. Figure 1A shows an accelerometer 168 coupled to a peripheral interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally functions as described in U.S. Patent Application Publication 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and U.S. Patent Application Publication 20060017692, "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. Device 100 optionally includes, in addition to one or more accelerometers 168, a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information regarding the location and orientation of Device 100 (e.g., longitudinal or transverse).

[0101] In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) stores device / global internal state 157, as shown in Figures 1A and 3. The device / global internal state 157 includes one or more of the following: active application state, indicating which application is active if there is an application currently active; display state, indicating which application, view, or other information occupies various areas of the touchscreen display 112; sensor state, including information obtained from various sensors and input control devices 116 of the device; and location information relating to the device's location and / or orientation.

[0102] An operating system 126 (for example, an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitate communication between various hardware components and software components.

[0103] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors that are the same as and / or compatible with the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices.

[0104] The contact / motion module 130 optionally detects contact with the touchscreen 112 and other touch-sensitive devices (e.g., a touchpad or physical click wheel) (in cooperation with the display controller 156). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of a finger dragging), and determining whether contact has been terminated (e.g., detecting a finger up event or interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0105] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (for example, to determine whether a user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (for example, the intensity thresholds can be adjusted without modifying the physical hardware of device 100, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to one of a range of default thresholds without modifying the trackpad or touchscreen display hardware. In addition, some implementations provide the user of the device with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click "intensity" parameter).

[0106] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same position (or substantially the same position) as the finger down event (e.g., the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event.

[0107] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components that modify the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). In this specification, the term “graphics” includes, but is not limited to, any object that can be displayed to the user, including characters, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0108] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like, as needed, specifying the graphics to be displayed, along with coordinate data and other graphic property data, and then generates screen image data to output to the display controller 156.

[0109] The haptic feedback module 133 includes various software components for generating commands used by a tactile output generator(s) 167, and generates tactile outputs at one or more locations on the device 100 in response to the user's interaction with the device 100.

[0110] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., the contact module 137, the email client module 140, the IM module 141, the browser module 147, and any other applications that require text input).

[0111] The GPS module 135 determines the device's location and provides this information for use in various applications (for example, to the phone module 138 for use in location-based dialing, to the camera module 143 as picture / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0112] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof: ● Contact module 137 (sometimes called the address book or contact list), ●Telephone module 138, ●Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ●Training support module 142, ● Camera module 143 for still images and / or video, ●Image management module 144, ●Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, ●Optionally, a widget module 149 may include one or more of the following: weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, other widgets obtained by the user, and user-created widgets 149-6. ●Widget creator module 150 for creating user-created widget 149-6, ● Search module 151, ●Video and music player module 152, which integrates a video player module and a music player module. ●Memo Module 153, ● Map module 154 and / or, ● Online video module 155.

[0113] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0114] Together with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (stored, for example, in the application internal state 192 of the contact module 137 in memory 102 or memory 370), which includes adding names(s) to the address book, deleting names(s) from the address book, associating telephone numbers(s) to names, email addresses(s) to names, addresses(s) to names, or other information, associating images to names, categorizing and sorting names, and providing telephone numbers or email addresses to initiate and / or facilitate communication by the telephone module 138, video conferencing module 139, email client module 140, or IM module 141.

[0115] The telephone module 138 works in conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to optionally input character sequences corresponding to telephone numbers, access one or more telephone numbers in the contact module 137, modify entered telephone numbers, dial individual telephone numbers, make calls, and disconnect and terminate calls at the end of a call. As previously mentioned, wireless communication may optionally use any of several communication standards, protocols, and technologies.

[0116] The video conferencing module 139 works in conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138 to include executable commands for starting, running, and ending video conferences between the user and one or more other participants in accordance with the user's commands.

[0117] The email client module 140, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for creating, sending, receiving, and managing emails in response to user commands. In conjunction with the image management module 144, the email client module 140 makes it extremely easy to create and send emails containing still or video images captured by the camera module 143.

[0118] The instant messaging module 141, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for inputting character sequences corresponding to instant messages, modifying previously entered characters, sending individual instant messages (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). In this specification, “instant messaging” refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and internet-based messaging (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0119] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, the training support module 142 includes executable commands, which create training (e.g., having time, distance, and / or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, calibrate sensors used to monitor training, select and play music for training, and display, store, and transmit training data.

[0120] The camera module 143 works in conjunction with the touchscreen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144 to include executable commands for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0121] The image management module 144 works in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143 to include executable commands for arranging, modifying (e.g., editing), or otherwise manipulating still images and / or videos, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing them.

[0122] The browser module 147 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for browsing the internet according to user commands, including searching, linking, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages.

[0123] The calendar module 148 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147 to include executable commands for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do lists, etc.) in accordance with user commands.

[0124] The widget module 149 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to optionally download and use mini-applications (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or mini-applications created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0125] The widget creator module 150 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to be used by the user to optionally create widgets (for example, to turn a user-specified portion of a web page into a widget).

[0126] The search module 151 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for searching for characters, music, sounds, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0127] The video and music player module 152 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147 to include executable commands that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands for displaying, presenting, or otherwise playing videos (for example, on the touchscreen 112 or on an external display connected via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0128] The memo module 153 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for creating and managing memos, to-do lists, etc., according to user commands.

[0129] The map module 154 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147 to optionally receive, display, modify, and store maps and map-related data (e.g., driving directions, data on shops and other points of interest at a specific location or nearby, and other location-based data) in accordance with user commands.

[0130] The online video module 155, in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, includes instructions that enable the user to access, browse, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and perform other management of online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the entirety of which is incorporated herein by reference.

[0131] Each of the modules and applications identified above corresponds to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., the computer methods and other information processing methods described herein). These modules (e.g., instruction sets) do not need to be implemented as separate software programs (e.g., computer programs containing instructions), procedures, or modules, and therefore in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. For example, a video player module is optionally combined with a music player module to form a single module (e.g., the video and music player module 152 in Figure 1A). In some embodiments, memory 102 optionally stores subsets of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0132] In some embodiments, device 100 is a device in which the operation of a default set of functions in the device is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.

[0133] A default set of functions, which are performed only through the touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0134] Figure 1B is a block diagram showing exemplary components for event processing according to several embodiments. In some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) includes an event sorting unit 170 (e.g., within the operating system 126) and individual applications 136-1 (e.g., any of the aforementioned applications 137-151, 155, 380-390).

[0135] The event sorting unit 170 receives event information and determines the application 136-1 that distributes the event information, and the application view 191 of application 136-1. The event sorting unit 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, a device / global internal state 157 is used by the event sorting unit 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event sorting unit 170 to determine the application view(s) to which the event information is distributed.

[0136] In some embodiments, the application internal state 192 includes additional information such as resume information to be used when the application 136-1 resumes execution, user interface state information that indicates or is ready to display information displayed by the application 136-1, a state queue that allows the user to return to a previous state or view of the application 136-1, and one or more redo / undo queues of previous actions performed by the user.

[0137] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, user touch as part of a multi-touch gesture on the touch-sensitive display 112). The peripheral interface 118 transmits information received from the I / O subsystem 106, or from sensors such as the proximity sensor 166, one or more accelerometers 168, and / or the microphone 113 (via the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0138] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (e.g., reception of input exceeding a predetermined noise threshold and / or exceeding a predetermined duration).

[0139] In some embodiments, the event sorting unit 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0140] The hit view determination module 172 provides a software procedure for determining where in one or more views a sub-event occurred when the touch-sensitive display 112 is displaying two or more views. A view consists of control devices and other elements that the user can see on the display.

[0141] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of an individual application) in which a touch is detected optionally corresponds to the program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected optionally refers to a hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch gesture.

[0142] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy from which sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source identified as the hit view.

[0143] The active event recognition determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 173 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy still remain actively involved views.

[0144] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., an event recognition unit 180). In embodiments including an active event recognition unit determination module 173, the event dispatcher module 174 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores event information retrieved by individual event receiving units 182 in an event queue.

[0145] In some embodiments, the operating system 126 includes an event sorting unit 170. Alternatively, application 136-1 includes an event sorting unit 170. In yet another embodiment, the event sorting unit 170 is a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0146] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each containing instructions for handling touch events occurring within a separate view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, a separate application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a separate event processing unit 190 includes one or more event data 179 received from a data update unit 176, an object update unit 177, a GUI update unit 178, and / or an event sorting unit 170. The event processing unit 190 optionally uses or calls the data update unit 176, the object update unit 177, or the GUI update unit 178 to update the application's internal state 192. Alternatively, one or more application views 191 include one or more event processing units 190. In some embodiments, one or more of the data update unit 176, object update unit 177, and GUI update unit 178 are included in individual application views 191.

[0147] Each individual event recognition unit 180 receives event information (e.g., event data 179) from the event sorting unit 170 and identifies events from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution commands 188 (optionally including sub-event distribution commands).

[0148] The event receiving unit 182 receives event information from the event sorting unit 170. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, an event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the device's current orientation (also called the device's orientation).

[0149] The event comparison unit 184 compares event information with a definition of a default event or sub-event, and based on the comparison, determines the event or sub-event, or determines or updates the state of the event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a default sequence of default sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events within an event (187-1 and / or 187-2) include, for example, a touch start, a touch end, a touch movement, a touch cancellation, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object for a predetermined stage (touch start), a first lift-off for the predetermined stage (touch end), a second touch on the displayed object for the predetermined stage (touch start), and a second lift-off for the predetermined stage (touch end). In another example, the definition of event 2(187-2) is a drag on a displayed object. A drag includes, for example, a touch (or contact) on the displayed object to a predetermined stage, movement of the touch across the touch-sensitive display 112, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0150] In some embodiments, the event definition 186 includes event definitions for individual user interface objects. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with an individual event processing unit 190, the event comparison unit uses the results of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0151] In some embodiments, the definition of an individual event 187 also includes a delay action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0152] If an individual event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, the individual event recognition unit 180 enters an event impossible, event failed, or event terminated state, and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0153] In some embodiments, an individual event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0154] In some embodiments, an individual event recognition unit 180 activates an event processing unit 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, the individual event recognition unit 180 delivers event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and delaying the sending of) sub-events to individual hit views. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag captures the flag and executes a default process.

[0155] In some embodiments, the event distribution command 188 includes a sub-event distribution command that distributes event information about a sub-event without activating an event processing unit. Instead, the sub-event distribution command distributes event information to an event processing unit associated with a set of sub-events, or to an actively involved view. The event processing unit associated with the set of sub-events or the actively involved view receives the event information and executes a predetermined process.

[0156] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates telephone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates new user interface objects or updates the positions of user interface objects. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares display information and sends it to graphics module 132 for display on touch-sensitive display.

[0157] In some embodiments, the event processing unit(s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are contained in a single module of an individual application 136-1 or application view 191. In other embodiments, they are contained in two or more software modules.

[0158] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0159] Figure 2 shows a portable multifunction device 100 having a touchscreen 112 according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, and in other embodiments described below, the user can select one or more of the graphics by performing gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward) and / or rolling (from right to left, left to right, upward and / or downward) with a finger in contact with the device 100. In some implementations or situations, accidental contact with a graphic does not constitute a selection of that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon does not arbitrarily select the corresponding application.

[0160] Device 100 also optionally includes one or more physical buttons, such as a "Home" button or a menu button 204. As previously mentioned, the menu button 204 is optionally used to navigate to any application 136 within the set of applications running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touchscreen 112.

[0161] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, one or more volume buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external port 124 for docking / charging. The push button 206 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined period of time, to lock the device by pressing down and releasing the button before a predetermined period of time has elapsed, and / or to unlock the device or initiate an unlocking process. In alternative embodiments, device 100 also accepts verbal input via a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more tactile output generators 167 for generating tactile output to the user of device 100.

[0162] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to several embodiments. Device 300 does not need to be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), game system, or control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 that interconnect these components. The communication buses 320 optionally include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. Device 300 includes an input / output (I / O) interface 330, which includes a display 340, and the display 340 is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 that generates tactile output on device 300 (for example, similar to the tactile output generator(s) 167 described above with reference to Figure 1A), and a sensor 359 (for example, light, acceleration, proximity, touch sensing, and / or a contact intensity sensor similar to the contact intensity sensor(s) 165 described above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, that are stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, whereas memory 102 of the portable multifunction device 100 (Figure 1A) optionally does not store these modules.

[0163] Each of the elements identified above in Figure 3 is optionally stored in one or more of the memory devices described above. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or computer programs (e.g., instruction sets or instructions) identified above do not need to be implemented as separate software programs (e.g., computer programs (including instructions)), procedures, or modules, and therefore in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. In some embodiments, memory 370 optionally stores subsets of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0164] Next, we optionally turn our attention to an embodiment of a user interface implemented in, for example, a portable multi-functional device 100.

[0165] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to several embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular signals and Wi-Fi signals 402, ●Time 404, ●Bluetooth indicator 405, ●Battery status indicator 406, ●Tray 408 containing icons for frequently used applications, as shown below. ○Optionally including an indicator 414 for the number of missed calls or voicemail messages, an icon 416 of the telephone module 138 labeled "Telephone", ○Optionally includes an indicator 410 for the number of unread emails, an icon 418 labeled "Mail", for the email client module 140. ○ Icon 420 of browser module 147, labeled "Browser", and ○ Icon 422 for the video and music player module 152, also known as the iPod (trademark of Apple Inc.) module 152, which is labeled "iPod", and ● Icons of other applications, such as the following: ○ Icon 424 of IM module 141, labeled "Message", ○ Icon 426 of calendar module 148, labeled "Calendar", ○ Icon 428 of image management module 144, labeled "Photo" ○ Icon 430 of camera module 143, labeled "Camera" ○ Icon 432 of online video module 155, labeled "online video", ○ Icon 434 of stock price widget 149-2, labeled "Stock Price" ○ Icon 436 of map module 154, labeled "Map" ○ Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of the alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of training support module 142, labeled "Training Support" ○ Icon 444 of memo module 153, labeled as "Memo", and ○ An icon 446 labeled "Settings," which provides access to the settings of device 100 and its various applications 136, for a settings application or module.

[0166] Please note that the icon labels shown in Figure 4A are for illustrative purposes only. For example, the icon 422 for the video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label for an individual application icon includes the name of the application to which that individual application icon corresponds. In some embodiments, the label for a particular application icon is different from the name of the application to which that particular application icon corresponds.

[0167] Figure 4B shows an exemplary user interface on a device (e.g., device 300 in Figure 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355 in Figure 3) separate from the display 450 (e.g., touchscreen display 112). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contact on the touch-sensitive surface 451, and / or one or more tactile output generators 357 for generating tactile output to the user of device 300.

[0168] Some of the following embodiments are given by reference to input on a touchscreen display 112 (a combination of a touch-sensing surface and a display), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 4B. In some embodiments, the touch-sensing surface (e.g., 451 in Figure 4B) has a primary axis (e.g., 452 in Figure 4B) corresponding to a primary axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with the touch-sensing surface 451 (e.g., 460 and 462 in Figure 4B) at locations corresponding to each location on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in Figure 4B). In this way, user input (e.g., touches 460 and 462, and their movement) detected by the device on a touch-sensitive surface (e.g., 451 in Figure 4B) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B) when the touch-sensitive surface is separate from the display. It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0169] In addition, while the following examples are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of touch), followed by a mouse click with cursor movement along the swipe path (e.g., instead of touch movement). As another example, a tap gesture may optionally be replaced by a mouse click (e.g., instead of touch detection and subsequent cessation of touch detection) while the cursor is located over the tap gesture location. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0170] Figure 5A shows an exemplary personal electronic device 500. Device 500 includes a body 502. In some embodiments, device 500 may include some or all of the functions described with respect to devices 100 and 300 (e.g., Figures 1A to 4B). In some embodiments, device 500 has a touch-sensitive display screen 504, hereafter referred to as touchscreen 504. Alternatively, or in addition to touchscreen 504, device 500 may have a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors that detect the intensity of the applied contact (e.g., touch). One or more intensity sensors on touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of device 500 may respond to touches based on their intensity, meaning that touches of different intensity may invoke different user interface behaviors on device 500.

[0171] For example, see, for instance, International Patent Application PCT / US2013 / 040061, “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed 8 May 2013, published as International Publication WO / 2013 / 169849, and International Patent Application PCT / US2013 / 069483, “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed 11 November 2013, published as International Publication WO / 2014 / 105276.

[0172] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 may be physical, if included. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 500 to be attached to, for example, hats, eyeglasses, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0173] Figure 5B shows an exemplary personal electronic device 500. In some embodiments, the device 500 may include some or all of the components described with respect to Figures 1A, 1B, and 3. The device 500 has a bus 512 that operably connects an I / O section 514 to one or more computer processors 516 and memory 518. The I / O section 514 may be connected to a display screen 504, which may have a touch-sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). In addition, the I / O section 514 may be connected to a communication unit 530 that receives application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 500 may include input mechanisms 506 and / or 508. The input mechanism 506 may optionally be, for example, a rotatable input device or a pressable and rotatable input device. In some embodiments, the input mechanism 508 is optionally a button.

[0174] In some embodiments, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or a combination thereof, all of which can be operably connected to the I / O section 514.

[0175] The memory 518 of the personal electronic device 500 may include one or more non-temporary computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, can cause the computer processors to execute, for example, the techniques described later, including processes 700, 750, 900, 1100, 1300, and 1500 (Figures 7A, 7B, 9, 11A, 11B, 13, and 15). The computer-readable storage media may be any medium that can tangibly contain or store computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a temporary computer-readable storage medium. In some embodiments, the storage medium is a non-temporary computer-readable storage medium. The non-temporary computer-readable storage medium may include, but is not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray technology, as well as persistent solid-state memory such as flash and solid-state drives. The personal electronic device 500 is not limited to the components and configurations shown in Figure 5B, and may include other or additional components in multiple configurations.

[0176] As used herein, the term “affordance” refers to user interaction graphical user interface objects that are optionally displayed on the display screens of devices 100, 300, and / or 500 (Figures 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0177] As used herein, the term “focus selector” refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other location marker, the cursor acts as a “focus selector,” and therefore, when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in Figure 3 or touch-sensitive surface 451 in Figure 4B) while the cursor is positioned over a particular user interface element, the particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display that allows direct interaction with user interface elements on the touchscreen display (e.g., touch-sensitive display system 112 in Figure 1A or touchscreen 112 in Figure 4A), a detected contact on the touchscreen acts as a “focus selector,” and therefore, when an input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (for example, by using the tab key or arrow keys to move focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or touch on the touchscreen display) controlled by the user to communicate the user's intended interaction with the user interface (for example, by pointing to the device an element of the user interface through which the user intends to interact).For example, the location of a focus selector (e.g., cursor, touch, or selection box) over an individual button while pressure input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that individual button (rather than other user interface elements displayed on the device's display).

[0178] As used herein and in the claims, the term “characteristic intensity” of a contact refers to the characteristics of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a set of intensity samples collected over a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined event (e.g., after detection of contact, before detection of lift-off of contact, before or after detection of the start of movement of contact, before detection of the end of contact, before or after detection of an increase in contact intensity, and / or before or after detection of a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the top 10 percentile value of the contact intensity, the maximum half value of the contact intensity, the maximum 90 percent value of the contact intensity, and so on. In some embodiments, the duration of contact is used when determining characteristic intensity (for example, when characteristic intensity is the average intensity of contact over time). In some embodiments, characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this embodiment, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity above the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity above the second threshold results in a third action. In some embodiments, the comparison between characteristic intensity and one or more thresholds is not used to determine whether a first action should be performed or a second action should be performed, but rather to determine whether one or more actions should be performed at all (for example, whether individual actions should be performed or whether individual actions should be withheld).

[0179] Figure 5C shows an exemplary diagram of a communication session between electronic devices 500A, 500B, and 500C. Devices 500A, 500B, and 500C are similar to electronic device 500, and each shares one or more data connections 510, such as an Internet connection, Wi-Fi connection, cellular connection, short-range communication connection, and / or any other such data connection or network, with each to facilitate real-time communication of audio and / or video data between the devices over a period of time. In some embodiments, the exemplary communication session may include a shared data session in which data is communicated from one or more of the electronic devices to other electronic devices to enable simultaneous output of individual content in the electronic devices. In some embodiments, the exemplary communication session may include a video conferencing session in which audio and / or video data is communicated between devices 500A, 500B, and 500C so that users of each device can engage in real-time communication using the electronic devices.

[0180] In Figure 5C, device 500A represents an electronic device associated with user A. Device 500A communicates (via data connection 510) with devices 500B and 500C associated with users B and C, respectively. Device 500A includes a camera 501A used to capture video data of the communication session and a display 504A (e.g., a touchscreen) used to display content associated with the communication session. Device 500A also includes other components such as a microphone (e.g., 113) for recording audio of the communication session and a speaker (e.g., 111) for outputting audio of the communication session.

[0181] Device 500A displays a communication UI 520A via display 504A, which is a user interface that facilitates a communication session (e.g., a video conferencing session) between device 500B and device 500C. The communication UI 520A includes a video feed 525-1A and a video feed 525-2A. Video feed 525-1A is a representation of video data captured in device 500B (e.g., using camera 501B) and communicated from device 500B to devices 500A and 500C during the communication session. Video feed 525-2A is a representation of video data captured in device 500C (e.g., using camera 501C) and communicated from device 500C to devices 500A and 500B during the communication session.

[0182] The communication UI 520A includes a camera preview 550A, which is a representation of the video data captured by device 500A via camera 501A. The camera preview 550A represents the expected video feed of user A as it will be displayed on devices 500B and 500C, respectively.

[0183] The communication UI 520A includes one or more controls 555A that control one or more aspects of a communication session. For example, a control 555A may include controls for muting the audio of a communication session, changing the camera view of a communication session (e.g., changing which camera is used to capture video of a communication session, adjusting the zoom value), ending a communication session, applying visual effects to the camera view of a communication session, and activating one or more modes associated with a communication session. In some embodiments, one or more controls 555A are optionally displayed on the communication UI 520A. In some embodiments, one or more controls 555A are displayed separately from the camera preview 550A. In some embodiments, one or more controls 555A are displayed as an overlay on at least a portion of the camera preview 550A.

[0184] In Figure 5C, device 500B represents an electronic device associated with user B communicating with devices 500A and 500C (via data connection 510). Device 500B includes a camera 501B used to capture video data of the communication session and a display 504B (e.g., a touchscreen) used to display content associated with the communication session. Device 500B also includes other components such as a microphone (e.g., 113) for recording audio of the communication session and a speaker (e.g., 111) for outputting audio of the communication session.

[0185] Device 500B displays a communication UI 520B similar to the communication UI 520A of device 500A via the touchscreen 504B. The communication UI 520B includes a video feed 525-1B and a video feed 525-2B. Video feed 525-1B is a representation of video data captured in device 500A (e.g., using camera 501A) and communicated from device 500A to devices 500B and 500C during a communication session. Video feed 525-2B is a representation of video data captured in device 500C (e.g., using camera 501C) and communicated from device 500C to devices 500A and 500B during a communication session. The communication UI 520B also includes a camera preview 550B, which is a representation of video data captured in device 500B via camera 501B, and one or more controls 555B similar to control 555A for controlling one or more aspects of a communication session. Camera preview 550B represents the expected video feed for user B as it will appear on devices 500A and 500C, respectively.

[0186] In Figure 5C, device 500C represents an electronic device associated with user C communicating with devices 500A and 500B (via data connection 510). Device 500C includes a camera 501C used to capture video data of the communication session and a display 504C (e.g., a touchscreen) used to display content associated with the communication session. Device 500C also includes other components such as a microphone (e.g., 113) for recording audio of the communication session and a speaker (e.g., 111) for outputting audio of the communication session.

[0187] Device 500C displays a communication UI 520C similar to the communication UI 520A of device 500A and the communication UI 520B of device 500B via the touchscreen 504C. The communication UI 520C includes a video feed 525-1C and a video feed 525-2C. Video feed 525-1C is a representation of video data captured in device 500B (e.g., using camera 501B) and communicated from device 500B to devices 500A and 500C during a communication session. Video feed 525-2C is a representation of video data captured in device 500A (e.g., using camera 501A) and communicated from device 500A to devices 500B and 500C during a communication session. The communication UI 520C also includes a camera preview 550C, which is a representation of the video data captured in device 500C via camera 501C, and one or more controls 555C for controlling one or more aspects of the communication session, similar to controls 555A and 555B. The camera preview 550C represents to user C the expected video feed of user C as it will be displayed on devices 500A and 500B, respectively.

[0188] The diagram in Figure 5C illustrates a communication session between three electronic devices, but a communication session can be established between two or more electronic devices, and the number of devices participating in a communication session can change as electronic devices join or leave the session. For example, if one of the electronic devices leaves the communication session, the audio and video data from the device that has stopped participating in the session is no longer represented on the participating devices. For example, if device 500B stops participating in the communication session, the data connection 510 between devices 500A and 500C, and the data connection 510 between devices 500C and 500B, no longer exist. In addition, device 500A does not include video feed 525-1A, and device 500C does not include video feed 525-1C. Similarly, when a device joins a communication session, connections are established between the participating device and the existing devices, and video and audio data are shared among all devices so that each device can output data communicated from the other devices.

[0189] The embodiment shown in Figure 5C represents a diagram of a communication session between multiple electronic devices, including the exemplary communication sessions shown in Figures 6A-6Y, 8A-8G, 10A-10Q, and 14E-14N. In some embodiments, the communication sessions shown in Figures 6A-6Y, 8A-8G, 10A-10Q, and 14E-14N include two or more electronic devices, even if other electronic devices participating in the communication session are not shown in the figures.

[0190] In this specification, “installed application” refers to a software application that has been downloaded onto an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched on the device (e.g., opened). In some embodiments, a downloaded application becomes an installed application by an installation program that extracts the program portion from the downloaded package and integrates the extracted portion with the operating system of the computer system.

[0191] In this specification, the terms “open application” or “running application” refer to a software application that has retained state information (e.g., as part of the device / global internal state 157 and / or application internal state 192). An open or running application is optionally one of the following types of applications: ● The active application currently displayed on the display screen of the device on which the application is being used. ● Background applications (or background processes) that are not currently displayed but whose applications have one or more processes being handled by one or more processors, and ● An application that is not running but has state information stored in memory (volatile and non-volatile, respectively) that can be used to resume the execution of the application, either suspended or suspended.

[0192] In this specification, the term “closed application” refers to a software application that does not retain state information (for example, state information for a closed application is not stored in the device’s memory). Therefore, closing an application involves stopping and / or removing the application process for the application and removing the state information for the application from the device’s memory. Generally, opening a second application while a first application is running does not close the first application. When the second application is displayed and the first application is stopped from being displayed, the first application becomes a background application.

[0193] Next, we will focus on embodiments of user interfaces ("UI") and related processes implemented on electronic devices such as portable multifunction device 100, device 300, or device 500.

[0194] Figures 6A to 6Y show exemplary user interfaces for displaying graphical effects during a real-time communication session, according to several embodiments. These user interfaces are used to illustrate processes described later, including those shown in Figures 7A to 7B.

[0195] In Figure 6A, computer system 600a (e.g., Isaac's computer) and computer system 600b (e.g., Sam's computer) communicate via a real-time communication session. Computer system 600a displays a user interface 602a that includes participant representations 606a associated with computer system 600a and participant representations 604a associated with computer system 600b. For example, on Isaac's computer (e.g., computer system 600a), representation 606a is Isaac's representation (e.g., self-view), and representation 604a is Sam's representation (e.g., remote participant representation).

[0196] Computer system 600b displays a user interface 602b that includes a participant representation 606b associated with computer system 600b and a participant representation 604b associated with computer system 600a. For example, on Sam's computer (e.g., computer system 600b), representation 606b is Sam's representation (e.g., SelfView), and representation 604b is Isaac's representation (e.g., a remote participant's representation).

[0197] In some embodiments, representations 606a and 606b are displayed in a smaller size than representations 604a and 604b. In some embodiments, representations 604a, 604b, 606a, and / or 606b include an animated avatar, a photograph, or a video feed from one or more camera sensors. In some embodiments, representations 604a, 604b, 606a, and 606b include a background environment behind the participant. In some embodiments, representations 604a, 604b, 606a, and 606b include a simulated background environment behind the participant's representation.

[0198] In Figure 6B, a participant in a real-time communication session associated with computer system 600a (e.g., Isaac) waves their hand within the field of view of one or more camera sensors of computer system 600a. In response to detection of Isaac's hand, computer system 600a displays a hint 608a on user interface 602a, and computer system 600b displays a hint 608b on user interface 602b. On user interface 602a, the hint 608a is displayed around Isaac's hand in representation 606a. On user interface 602b, the hint 608b is displayed around Isaac's hand in representation 604b. In some embodiments, the hint 608b is not displayed on user interface 602b on computer system 600b, but instead is displayed only on computer system 600a (e.g., the hint is displayed only on the computer system associated with the participant performing the gesture, e.g., on Isaac's device and not on Sam's device). Hints 608a and 608b indicate that graphical effects can be displayed in response to the performance of a gesture. In some embodiments, hints 608a and / or hints 608b include a circle displayed around the participant's hand. In some embodiments, hints 608a and / or hints 608b move with the participant's hand. In some embodiments, hints 608a and / or hints 608b include text describing that graphical effects can be displayed in response to the detection of a gesture. In some embodiments, hints 608a and / or hints 608b include graphical effects such as illumination, lighting, and / or color effects displayed around the participant's hand.

[0199] In Figure 6C, a participant in a real-time communication session associated with computer system 600a (e.g., Isaac) performs a thumbs-up gesture within the view of one or more camera sensors of computer system 600a. Upon detection of the thumbs-up gesture, computer system 600a displays prompt 610a, which includes a prompt to enable the display of a graphical effect in response to the detection of the performed gesture. If computer system 600a detects user input corresponding to the selection of the "yes" option 610a1, a graphical effect is displayed in representation 606a and / or representation 604b (e.g., Isaac's representation) in response to the detection of the gesture performed by Isaac within the view of one or more camera sensors communicating with computer system 600a. If computer system 600a detects user input corresponding to Isaac selecting the "No" option 610a2 in response to prompt 610a, the graphical effects are not displayed in representations 604b and 606a, in accordance with the computer system 600a detecting a gesture performed by Isaac within the computer system 600a's field of view. In some embodiments, if no selection is made, computer system 600a stops displaying prompt 610a after a predetermined time. In some embodiments, if no selection is detected within a predetermined time, the gesture-based response is not turned on. In some embodiments, the gesture-based response is turned on or off via an application menu or operating system menu, such as menu 612a as described with reference to Figure 6D or menu 1008 as described with reference to Figure 10A. In some embodiments, the selection of the state of the gesture-based response (e.g., on or off) is persistent (e.g., the state is maintained for one or more applications so that the user does not need to enable or disable the gesture-based response when an application is launched or a new real-time communication session is started).

[0200] In Figure 6D, a participant in a real-time communication session associated with computer system 600a (e.g., Isaac) opens menu 612a on computer system 600a and selects a thumbs-up graphical effect via input 614a (e.g., input via mouse and / or keyboard, and / or touch gestures). In some embodiments, menu 612a includes options for displaying graphical effects, including representations of thumbs-up, thumbs-down, peace sign, fireworks, rain, and / or confetti effects. In some embodiments, menu 612a includes preview thumbnails 612ab that show preview representations of graphical effects that can be applied via menu 612a before the graphical effects are applied to representations 606a and / or 604b. In some embodiments, menu 612a is an operating system menu. In some embodiments, menu 612a is an application menu associated with the real-time communication session.

[0201] In Figure 6E, upon detecting input 614a, the computer system 600a (e.g., Isaac's computer) displays a graphical effect 616a within representation 606a. In some embodiments, the graphical effect 616a includes a representation of a thumbs-up gesture, such as a thumbs-up emoji. Upon detecting input 614a, the computer system 600a displays a hint 613a. In some embodiments, the hint 613a includes a video clip indicating that a graphical effect can be displayed in response to performing a gesture. In some embodiments, the hint 613a includes text explaining that a graphical effect can be displayed in response to performing a gesture. The computer system 600b (e.g., Sam's computer) displays a graphical effect 616b within representation 604b. In some embodiments, the graphical effect 616b may include any or all features of the graphical effect 616a.

[0202] In Figure 6F, a participant in a real-time communication session associated with computer system 600a (e.g., Isaac) performs a thumbs-up gesture while moving their hand back and forth. In response to the detection of a moving thumbs-up gesture, computer systems 600a and 600b cease displaying graphical effects. In some embodiments, computer systems 600a and 600b cease displaying graphical effects in response to a moving gesture (or a gesture moving beyond a threshold speed). In some embodiments, computer systems 600a and 600b cease displaying graphical effects in response to a gesture that is maintained for less than a predetermined duration (e.g., 0.5, 1, 3, 5, or 10 seconds). In some embodiments, computer systems 600a and 600b display graphical effects in response to detecting a gesture if the gesture is not moving, moving below a threshold speed, and / or maintained beyond a predetermined duration.

[0203] In Figure 6G, in response to detecting that a participant in a real-time communication session associated with the computer system 600a (e.g., Isaac) is performing a thumbs-up gesture with their right hand, the computer system 600a displays a graphical effect 618a. The graphical effect 618a is contained within representation 606a (e.g., Isaac's representation). In some embodiments, the graphical effect 618a includes a representation of the thumbs-up gesture, such as emojis. In some embodiments, the graphical effect 618a includes a sequence of effects. In some embodiments, the sequence of effects includes a number of emojis that begin to appear at the location of Isaac's hand in representation 606a and move to the top of representation 606a. In some embodiments, the sequence of effects continues until the gesture is no longer detected. The computer system 600b displays the graphical effect 618b in Isaac's representation 604b. In some embodiments, the graphical effect 618b and representation 604b may include any or all features of the graphical effect 618a and / or representation 606a of Isaac on the computer system 600a, as described with respect to Figure 6G.

[0204] In FIG. 6G, Isaac starts moving his hand from in front of his body (e.g., a first location) to the right of his torso (e.g., a second location). In some embodiments, the movement speed of Isaac's hand is below a threshold speed. In FIG. 6H, Isaac's hand continues moving to the right of his torso (e.g., toward the second location), and computer system 600a displays graphical effect 618a at the current location of the gesture within representation 606a. In some embodiments, only the most recently displayed portion of graphical effect 618a aligns with the current horizontal location of Isaac's hand within representation 606a. For example, more recent effects are displayed at the current location of Isaac's hand, less recent effects are horizontally aligned with the first location of Isaac's hand and are vertically higher than the most recent effect, creating a "trail" of effects that floats away from the gesture location within representation 606a. Similarly, computer system 600b displays graphical effect 618b at a second location within Isaac's representation 604b in user interface 602b. In some embodiments, graphical effect 618b and / or representation 604b can include any or all features of graphical effect 618a and / or representation 606a as described with respect to FIG. 6H.

[0205] In FIG. 6I, Isaac's hand is now to the right of his torso. Graphical effect 618a is displayed at the second location within representation 606a. In some embodiments, graphical effect 618a creates a vertical line of effects over the gesture that moves vertically out of the frame within representation 606a. In some embodiments, graphical effect 618a is no longer displayed at the first location of Isaac's hand. Similarly, computer system 600b displays graphical effect 618b at the second location within representation 604b. In some embodiments, graphical effect 618b and / or representation 604b can include any or all features of graphical effect 618a and / or representation 606a as described with respect to FIG. 6I.

[0206] In FIG. 6J, a participant (e.g., Isaac) associated with the computer system 600a performs a thumbs-up gesture with the left hand. In response to detecting the gesture performed by Isaac's left hand, the computer system 600a displays a graphical effect 618a with an orientation that matches the orientation of Isaac's hand. For example, the palm and fingers are no longer included in the graphical effect 618a, and the back of the hand is represented instead. In some embodiments, the graphical effect 618a includes fingers and / or the palm or the back of the hand. Which portion of the hand is represented varies based on which hand performs the gesture and the orientation of the gesture. For example, in FIG. 6I, the gesture is performed by Isaac's right hand with the hand placed on the right side of Isaac's body, and the graphical effect 618a includes the fingers and the palm. Alternatively, if Isaac performs the same gesture with his right hand on the left side of his body, the back of his hand becomes visible in representation 606a, and the graphical effect 618a includes a representation of the back of the hand instead of the fingers. In some embodiments, the graphical effect 618a includes a wrist having the same orientation as the participant's hand. In some embodiments, in the graphical effect 618a, the wrist and the thumb are represented in the same orientation as the participant's hand. In some embodiments, the orientation of the graphical effect 618a matches the rotation angle of the participant's hand (e.g., when Isaac's hand is at an angle, the graphical effect 618 is rotated to match the angle of Isaac's hand). In some embodiments, the angle of the gesture is rotated to become a thumbs-down gesture. In response to detecting that the participant associated with the computer 600a is performing a thumbs-up gesture with the left hand, the computer system 600b displays a graphical effect 618b that matches the orientation of the participant's hand. In some embodiments, the graphical effect 618b displayed on the computer system 600b can include any or all features of the graphical effect 618a as described with reference to FIG. 6J.

[0207] In Figure 6K, a participant associated with the computer system 600a (e.g., Isaac) performs a thumbs-up gesture, and in response to the detection of the thumbs-up gesture, the computer system 600a displays a prompt 620a within the user interface 602a. The prompt 620a prompts the user to cancel the gesture-based response. In some embodiments, the prompt includes a "yes" option 620a2 corresponding to the choice to cancel the display of the graphical effect in response to the gesture detection, and a "no" option 620a3 corresponding to the choice to continue displaying the graphical effect in response to the gesture detection. In some embodiments, the prompt 620a is displayed a predetermined number of times after the graphical effect has been displayed in response to the gesture detection, and the prompt 620a is not displayed after the graphical effect has been displayed in response to the gesture more than a predetermined number of times. In Figure 6K, user input 620a1 corresponding to the prompt 620a and the choice to stop displaying the graphical effect in response to the gesture are detected. In response to the computer system 600b detecting that Isaac is performing a thumbs-up gesture, the computer system 600b displays the graphical effect 618b without displaying the prompt 620a.

[0208] In Figure 6L, after user input 620a1 (selecting prompt 620a to stop displaying the graphical effect in response to the detection of a gesture), computer system 600a does not display the graphical effect 618a in response to Isaac's thumbs-up gesture. Similarly, computer system 600b does not display the graphical effect 618b in representation 604b when Isaac performs a thumbs-up gesture.

[0209] In Figure 6M, a feature that displays a graphical effect in response to gesture detection is reactivated on computer system 600a, and a participant associated with computer system 600a (e.g., Isaac) performs a double thumbs-up gesture. In response to the detection of the double thumbs-up gesture, computer system 600a displays graphical effect 622a within representation 606a. Graphical effect 622a includes representations of both hands performing the double thumbs-up gesture (e.g., left-hand emoji and right-hand emoji). Similarly, in response to the detection of the double thumbs-up gesture, computer system 600b displays graphical effect 622b within Isaac's representation 604b. Graphical effect 622a and / or graphical effect 622b may include any or all of the features of graphical effect 618a and / or graphical effect 618b described with reference to Figures 6G, 6H, 6I, and 6J.

[0210] In Figure 6N, a participant associated with the computer system 600a (e.g., Isaac) performs a double thumbs-up gesture, and in response to the detection of the double thumbs-up gesture, the computer system 600a displays a graphical effect 624a within representation 606a. The graphical effect 624a includes simulated fireworks displayed behind the participant performing the gesture (e.g., Isaac) in representation 606a. In some embodiments, the graphical effect 624a illuminates a portion of the participant performing the gesture (e.g., Isaac's body) with simulated light. In some embodiments, the graphical effect 624a includes displaying a portion of the participant performing the gesture (e.g., Isaac's body) covered in simulated shadows. In some embodiments, the graphical effect 624a is displayed for a predetermined duration. In some embodiments, the graphical effect 624a is displayed as long as the double thumbs-up gesture is detected. In some embodiments, the graphical effect 624a disappears, and the computer system 600a stops displaying the graphical effect 624a after a predetermined time has elapsed since the detection of the gesture (for example, after the gesture is first detected or after the end of the gesture is detected). In some embodiments, the graphical effect 622a is replaced by the graphical effect 624a if the double thumbs-up gesture is detected for longer than a predetermined period. In some embodiments, the graphical effect 624a (e.g., a special effect such as fireworks) is displayed in response to the double thumbs-up gesture, instead of the graphical effect 622a (e.g., a double thumbs-up emoji). In some embodiments, the computer system 600a displays the graphical effect 624a depending on the type of gesture. In the computer system 600b, in response to the detection of the double thumbs-up gesture, the computer system 600b displays the graphical effect 624b within the representation 604b. In some embodiments, the graphical effect 624b may include any and / or all features of the graphical effect 624a as described with respect to Figure 6N.

[0211] In Figure 6O, in response to the detection of a double thumbs-down gesture, the computer system 600a displays a graphical effect 626a within the representation 606a. In some embodiments, the graphical effect 626a includes displaying a simulated rain effect behind the participant performing the gesture (e.g., Isaac). In some embodiments, the simulated rain effect appears in front of the representation of the participant performing the gesture (e.g., Isaac). In some embodiments, the graphical effect 626a (e.g., the simulated rain effect) interacts with the representation of the participant performing the gesture. In some embodiments, the simulated rain effect includes raindrops that come into contact with (e.g., fall on) the participant performing the gesture. In some embodiments, the simulated lighting effect includes a simulated lighting effect that shows the participant performing the gesture in a reduced-light environment (e.g., as if in a storm). In some embodiments, the graphical effect 626a is displayed for a predetermined duration. In some embodiments, the graphical effect 626a is displayed as long as the double thumbs-down gesture is detected. In some embodiments, the graphical effect 626a disappears, and the computer system 600a stops displaying the graphical effect 626a after a predetermined time has elapsed following the detection of the gesture (for example, after the gesture is first detected or after the end of the gesture is detected). In some embodiments, the computer system 600a displays the graphical effect 626a depending on different gesture types. In computer system 600b, in response to the detection of a double thumbs-down gesture, the computer system displays the graphical effect 626b within representation 604b. In some embodiments, the graphical effect 626 may include any or all of the features of the graphical effect 626a.

[0212] In Figure 6P, in response to detection that a participant associated with the computer system 600a (e.g., Isaac) is performing a peace sign (or victory sign) gesture, the computer system 600a displays a graphical effect 628a within representation 606a. The graphical effect 628a includes simulated confetti. In some embodiments, the peace sign gesture is performed with both hands. In some embodiments, the peace sign is performed with one hand. In some embodiments, a portion of the simulated confetti is displayed as if fragments of confetti were falling in front of the participant performing the peace sign gesture (e.g., some fragments of confetti block part of Isaac's body and / or facial representation), and a second portion of the confetti is displayed behind the participant performing the peace sign gesture in representation 606a (e.g., fragments of confetti are blocked by Isaac and / or his surroundings). In some embodiments, the graphical effect 628a is displayed for a predetermined duration. In some embodiments, the graphical effect 628a is displayed as long as a peace sign (or victory sign) gesture is detected. In some embodiments, the graphical effect 628a disappears, and the computer system 600a stops displaying the graphical effect 628a after a predetermined time has elapsed following the detection of the peace sign gesture. In some embodiments, the computer system 600a displays the graphical effect 628a depending on different gesture types. In the computer system 600b, in response to the detection of a peace sign gesture, the computer system 600b displays the graphical effect 628b within representation 604b. In some embodiments, the graphical effect 628b may include any or all of the features of the graphical effect 628a as described with respect to Figure 6P.

[0213] In Figure 6Q, after detecting that a participant associated with the computer system 600a (e.g., Isaac) is performing a peace sign gesture, the computer system 600a detects that Isaac is performing a wave gesture. In response to the detection of the wave gesture, the computer system 600a stops displaying the graphical effect 628a at the location corresponding to the location of the wave gesture in representation 606a. In some embodiments, the graphical effect 628a fades out at the location corresponding to the wave gesture. In some embodiments, a portion of the graphical effect 628a moves "off-screen" in representation 606a, as if blown out of view by the wave gesture. In computer system 600b, in response to the detection of the wave gesture, the computer system 600b stops displaying the graphical effect 628b at the location corresponding to the wave gesture in representation 604b. In some embodiments, the graphical effect 628b may include any or all of the features of the graphical effect 628a as described with respect to Figure 6Q.

[0214] In some embodiments, the computer system 600a displays graphical effects (e.g., graphical effect 616a, graphical effect 618a, graphical effect 622a, graphical effect 624a, graphical effect 626a, and / or graphical effect 628a) at locations in representation 606a corresponding to the locations of gestures within the field of view of one or more camera sensors (for example, the graphical effects are displayed at the same locations in representation 606a as the physical locations of the gesture relative to the participant performing the gesture). For example, in response to the computer system 600a detecting that Isaac is performing a wave gesture near Isaac's head, the computer system 600a stops displaying graphical effect 628a near Isaac's head in representation 606a. In another example, in response to the computer system 600a detecting that Isaac is performing a wave gesture in front of Isaac's left shoulder, the computer system 600a stops displaying graphical effect 628a in front of Isaac's left shoulder in representation 606a.

[0215] In some embodiments, the computer system 600b displays graphical effects (e.g., graphical effect 616b, graphical effect 618b, graphical effect 622b, graphical effect 624b, graphical effect 626b, and / or graphical effect 628b) at locations in the representation 604b corresponding to the locations of the gestures within the field of view of one or more camera sensors (for example, the graphical effects are displayed at locations relative to the representation of the participant performing the gesture, which are the same as the physical locations of the gestures relative to the participant within the field of view of one or more camera sensors).

[0216] In Figure 6R, a participant associated with the computer system 600a (e.g., Isaac) performs a gesture (e.g., a two-handed gesture forming a rectangle with the index fingers and thumbs of both hands) in response to a request to capture a screenshot and / or screen recording. In response to the detection of the gesture in response to the request to capture a screenshot and / or screen recording, the computer system 600a displays a user interface element 630a within the user interface 602a and captures the screenshot and / or screen recording. In some embodiments, the screenshot and / or screen recording is captured after a delay (e.g., a countdown). The user interface element 630a includes a countdown timer 630a1 indicating the amount of time before the screenshot and / or screen recording is captured. In some embodiments, the user interface element 630a includes a preview of the content to be captured before the screenshot and / or screen recording is taken. In some embodiments, the user interface element 630a includes a representation of the captured content (e.g., a thumbnail of the screenshot and / or screen recording) after the screenshot and / or screen recording has been captured. In some embodiments, clean recordings are recorded over a predetermined duration. In some embodiments, screenshots and / or screen recordings include representations of content shared in a communication session (e.g., shared screens, presentations, and / or shared media). In some embodiments, screenshots and / or screen recordings include representations 604a and / or 604b (e.g., representations of Isaac and / or Sam).

[0217] In computer 600b, upon detection of a gesture corresponding to a request to capture a screenshot and / or screen recording, computer system 600b displays a user interface element 630b (for example, a representation of the screenshot and / or screen recording captured by computer system 600a) on user interface 602b. In some embodiments, the representation of the screenshot and / or screen recording displayed by computer system 600b may include any or all features of the screenshot and / or screen recording captured by computer system 600a. In some embodiments, user interface element 630b may include any or all features of user interface element 630a, which are described with respect to Figure 6R.

[0218] In some embodiments, if a gesture is within a predetermined distance from the participant's head (for example, if the gesture obscures part of the participant's head and / or face), the graphical effect is not displayed. For example, in Figure 6S, a participant associated with computer system 600a (e.g., Isaac) performs a thumbs-up gesture (e.g., the same gesture as in Figure 6G, but closer to the participant's face) within a predetermined distance from the participant's head, and in response to the detection of the thumbs-up gesture, computer system 600a refrains from displaying the graphical effect (e.g., computer system 600a does not display graphical effect 618a). In computer system 600b, in response to the detection of a gesture within a predetermined distance from the participant's own head, computer system 600b refrains from displaying the graphical effect (e.g., computer system 600b does not display graphical effect 618b).

[0219] In some embodiments, if the gesture is performed while the participant performing the gesture is not looking at the camera sensor and / or the display generation components of the computer system, the graphical effect is not displayed. For example, in Figures 6T1 and 6T2, a participant associated with the computer system 600a (e.g., Isaac) performs a thumbs-up gesture (e.g., the same gesture as in Figure 6G) while not looking at the computer system 600a (e.g., the computer system 600a does not detect Isaac's gaze 632a, or the computer system 600a detects that the gaze 632a is not directed at the computer system 600a and / or the user interface 602a). In response to the detection of a thumbs-up gesture while the gaze 632a of the participant performing the gesture is not detected, the computer system 600a refrains from displaying the graphical effect (e.g., the computer system 600a does not display the graphical effect 618a). In computer system 600b, if gaze 632a is not detected while Isaac is performing a thumbs-up gesture, computer system 600b stops displaying graphical effects (for example, computer system 600b does not display graphical effect 618b).

[0220] In some embodiments, if a gesture is performed while the user interface of a real-time communication session is not displayed on the display components of the computer system, the graphical effect is not displayed. For example, in Figure 6U, a participant associated with computer system 600a (e.g., Isaac) performs a thumbs-up gesture while the user interface 602a is not displayed on computer system 600a. For example, Isaac has minimized the user interface of the real-time communication session, Isaac is looking at another program, another window is on top of the real-time communication session interface, and / or Isaac is looking at the operating system window. In response to the detection of the thumbs-up gesture performed by the participant associated with computer system 600a while computer system 600a is not displaying the user interface 602a, computer systems 600a and 600b stop displaying the graphical effect.

[0221] In some embodiments, the graphical effect is not displayed if the computer system associated with the participant performing the gesture mutes the audio input. For example, in Figure 6V, a participant associated with computer system 600a (e.g., Isaac) performs a thumbs-up gesture while the microphone communicating with computer system 600a (and used in a real-time communication session) is muted (e.g., not sending audio to computer system 600b). In response to the detection of the thumbs-up gesture while no audio is being sent associated with computer system 600a, computer systems 600a and 600b stop displaying the graphical effect.

[0222] In Figure 6X, in response to detecting that a participant associated with the computer system 600a (e.g., Isaac) performs a peace sign gesture (e.g., with only one hand), the computer system 600a displays a graphical effect 636a within representation 606a. The graphical effect 636a includes a simulated callout. In some embodiments, the graphical effect 636a includes a three-dimensional and / or depth effect, where different parts of the graphical effect 634a appear to be at different depths. For example, in representation 606a, a portion of the simulated callout (e.g., 636a3) is displayed in front of the participant performing the peace sign gesture (e.g., a portion of the simulated callout blocks part of Isaac's body and / or facial representation), and a second portion of the simulated callout (e.g., 636a1) is displayed behind the participant performing the peace sign gesture (e.g., a portion of the simulated callout is blocked by Isaac and / or his surroundings). In some embodiments, one or more individual elements of the simulated callout (e.g., the callout itself) include a depth effect (e.g., the central portion of the callout appears rounded, protruding from the edges of the callout). In some embodiments, the representation 606a includes a background effect (e.g., a blurred background). In some embodiments, the background effect is applied to a portion of the graphical effect 636a that is in the background (e.g., behind the participant making the peace sign gesture). For example, in Figure 6X, the simulated callout 636a1 is behind the participant making the peace sign gesture, so the background effect is applied to the simulated callout 636a1 (e.g., the simulated callout 636a1 is blurred relative to the rest of the graphical effect 636a). In some embodiments, the graphical effect 636a is displayed in color based on the color temperature of the field of view representation 606a of one or more camera sensors (for example, the color temperature of the scene in the field of view of one or more camera sensors is applied to the graphical effect 636a and / or the color of the graphical effect 636a matches the color temperature of the field of view representation 606a of one or more camera sensors).In some embodiments, the graphical effect 636a is displayed for a predetermined duration. In some embodiments, the graphical effect 636a is displayed as long as a peace sign gesture is detected. In some embodiments, the graphical effect 636a disappears, and the computer system 600a stops displaying the graphical effect 636a after a predetermined time has elapsed following the detection of the peace sign gesture. In some embodiments, the computer system 600a displays the graphical effect 636a depending on different gesture types. In the computer system 600b, in response to the detection of a peace sign gesture, the computer system 600b displays the graphical effect 636b within the representation 604b. In some embodiments, the graphical effect 636b can include any or all features of the graphical effect 636a as described with respect to Figure 6Y (for example, a simulated callout 636b3 is in front of the representation 604b, and a simulated callout 636b1 is blurred behind the representation 604b).

[0223] In Figure 6Y, a participant associated with the computer system 600a (e.g., Isaac) performs a double horn-up gesture (e.g., extending the index and little fingers on both hands and folding the middle and ring fingers), and in response to the detection of the double horn-up gesture, the computer system 600a displays a graphical effect 640a within representation 606a. The graphical effect 640a includes a simulated laser beam radiated from behind the participant performing the gesture (e.g., Isaac) in representation 606a and directed forward and to the sides of the participant. In some embodiments, the graphical effect 640a interacts with a part of the participant performing the gesture (e.g., Isaac's body). For example, in Figure 6Y, the simulated light from the graphical effect 640a illuminates the edges of the participant performing the gesture and / or creates a simulated shadow on a part of the participant performing the gesture (e.g., Isaac's body). In some embodiments, the graphical effect 640a is displayed for a predetermined duration. In some embodiments, the graphical effect 640a is displayed as long as a double horn-up gesture is detected. In some embodiments, the graphical effect 640a disappears, and the computer system 600a stops displaying the graphical effect 640a after a predetermined time has elapsed since the detection of the gesture (for example, after the gesture is first detected or after the end of the gesture is detected). In some embodiments, the computer system 600a displays the graphical effect 640a depending on different gesture types. In the computer system 600b, in response to the detection of a double horn-up gesture, the computer system 600b displays the graphical effect 640b within the representation 604b. In some embodiments, the graphical effect 640b can include any and / or all features of the graphical effect 640a as described with respect to Figure 6Y (for example, simulated light from the graphical effect 640b illuminates the edges of the representation 604b and / or creates a simulated shadow on part of the representation 604b).

[0224] Figure 7A is a flowchart illustrating how to display graphical effects during a real-time communication session using a computer system (e.g., a smartphone, desktop computer, laptop, tablet, head-mounted device, wearable device, and / or smartwatch) that communicates with display generation components (e.g., a display controller, a touch-sensitive display system, a projector, a head-mounted display, and / or a holographic display), according to several embodiments. Method 700 is performed on a computer system (e.g., 100, 300, 500, 600a, 600b, 800a, 800b, 800c, 1000a, 1000b, and / or 1200a) that communicates with display generation components (e.g., 601a and / or 601b). Some operations in Method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0225] As described below, Method 700 provides an intuitive method for displaying graphical effects during a real-time communication session. This method reduces the cognitive burden on the user to display graphical effects during a real-time communication session, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to display graphical effects faster and more efficiently during a real-time communication session saves power and increases the time between battery charges.

[0226] In Method 700, a computer system displays a representation of the field of view of one or more camera sensors (e.g., 602a, 602b, 606a, and / or 606b) (e.g., an image, a video feed, and / or a live preview) via a display generation component (702), wherein a person is visible within the representation of the field of view of one or more camera sensors. While the representation of the field of view of one or more camera sensors is displayed, a first gesture (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) performed by a first part of the person (e.g., a part of the person's body, the user's hand, and / or both of the user's hands) is detected within the field of view of one or more camera sensors (704). In response to the detection of a first gesture performed by a first part of a person (and, in some embodiments, in accordance with the determination that a set of criteria has been met), the computer system displays a first graphical effect (e.g., 616a, 616b, 618a, 618b, 622a, 622b, 624a, 624b, 626a, 626b, 628a, and / or 628b) (e.g., one or more graphical elements, animations, and / or graphical representations of the first gesture, such as emojis and / or animations corresponding to the first gesture) via a display generation component (706). In some embodiments, the computer system displays the graphical effect simultaneously with (e.g., within) a representation of the field of view of one or more camera sensors (e.g., 602a, 602b, 606a, and / or 606b). In some embodiments, displaying a graphical effect includes displaying an animation corresponding to a first gesture (e.g., fireworks animation, confetti animation, rain falling, and / or cloud movement).In accordance with a determination (e.g., by a computer system and / or an external computer system such as a computer system of a remote participant in a video communication session) that a first gesture is detected at a first gesture location within the field of view of one or more camera sensors (e.g., a thumbs-up gesture is in front of the subject "Isaac" in Figure 6G) (e.g., a first location for the field of view of one or more camera sensors, a first location for a part of the user, and / or a first location for the physical environment or an object in the physical environment), the computer system displays a first graphical effect at a first effect location within the representation of the field of view of one or more camera sensors (e.g., a first location on a display generating component, a first location in the user interface, a first location within the representation of the field of view of one or more camera sensors, a first gesture location, and / or a first location for the displayed representation of the user) (708). In some embodiments, the first graphical effect (e.g., a graphical effect corresponding to a first gesture) is different from the movement of a first part of a person corresponding to the first gesture. Based on the determination that the first gesture is in a second gesture location in the field of view of one or more camera sensors different from the first gesture location (e.g., a second location relative to the field of view of one or more camera sensors, a second location relative to a part of the user, and / or a second location relative to the physical environment or an object in the physical environment) (for example, a thumbs-up gesture is next to the subject "Isaac" in Figure 6I), the computer system displays the first graphical effect in the second effect location in the field of view of one or more camera sensors different from the first effect location (e.g., a second location on a display generation component, a second location in the user interface, a second location in the representation of the field of view of one or more camera sensors, a second gesture location, and / or a second location relative to the displayed representation of the user) (710).Displaying graphical effects in response to detected gestures provides users with a simple and efficient way to display graphical effects without requiring input via a touch-sensitive display, mouse, and / or keyboard, thereby reducing the number of inputs required to perform the action and providing additional control options without cluttering the UI with additional displayed controls. In some embodiments, the graphical effects are displayed by one or more other computer systems in a real-time video communication session with the computer system (e.g., along with representations of the field of view of one or more camera sensors).

[0227] In some embodiments, while a representation of the field of view of one or more camera sensors (e.g., 604a, 606a, 604b, and / or 606b) is displayed, a second gesture (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) is detected by a computer system within the field of view of the one or more camera sensors, the second gesture being performed (e.g., includes) by a first portion of a person (e.g., a user's first hand) and a second portion of the person (e.g., the user's second hand) (in some embodiments, the first gesture does not include the second portion of the person). In response to detecting the second gesture performed by the first portion of the person and the second portion of the person, the computer system displays, via a display generation component, a second graphical effect that is different from the first graphical effect (e.g., 622a, 622b, 624a, 624b, 626a, 626b, 628a, and / or 628b) (e.g., one or more graphical elements, animations, and / or one or more graphical representations of the second gesture, such as an emoji corresponding to the second gesture, one or more emojis corresponding to the first portion of the user performing the gesture, one or more emojis corresponding to the second portion of the user performing the gesture, and / or an animation corresponding to the second gesture). Displaying the second graphical effect in response to detecting the second gesture provides the user with a simple and efficient way to display the graphical effect without requiring input to be provided via a touch-sensitive display, mouse, and / or keyboard, which reduces the number of inputs needed to perform an action and provides additional control options without cluttering the UI with additional displayed controls.

[0228] In some embodiments, according to a determination that the second gesture includes a first part and a second part of the person, the computer system displays the second graphical effect in a first visual appearance via a display generation component; and according to a determination that the second gesture includes the first part of the person but does not include the second part of the person, the computer system displays the second graphical effect in a second visual appearance via a display generation component (for example, the second graphical effect is different from the effect displayed when the gesture is performed by a single part). In some embodiments, the second graphical effect includes the graphical effect displayed when the gesture is performed by a single part of the person (for example, the first part of the person). In some embodiments, the computer system displays the second graphical effect simultaneously with (for example, within) a representation of the field of view of one or more camera sensors. In some embodiments, displaying the second graphical effect includes displaying an animation corresponding to the second gesture (for example, an animation of fireworks, an animation of confetti, rain falling, and / or clouds moving).

[0229] In some embodiments, a first gesture includes a thumbs-up gesture performed by a first part of a person (as shown, for example, by the thumbs-up gesture performed by one hand in Figure 6G), and a second gesture includes a thumbs-up gesture performed by a first part of a person (e.g., the user's first hand) and a thumbs-up gesture performed by a second part of a person (e.g., the user's second hand) (as shown, for example, by the thumbs-up gesture performed by both hands in Figure 6M). Displaying a first graphical effect in response to the detection of a thumbs-up gesture performed by the first part, and displaying a second graphical effect in response to the detection of thumbs-up gestures performed by both the first and second parts, provides the user with a simple and efficient way to display graphical effects without the need to provide input via a touch-sensitive display, mouse, and / or keyboard, which reduces the number of inputs required to perform the action and provides additional control options without cluttering the UI with additional displayed controls.

[0230] In some embodiments, displaying a first graphical effect (e.g., 618a, 618b, 622a, 622b, 624a, 624b, 626a, 626b, 628a, and / or 628b) includes, upon determination that the first gesture was performed by a first type of body part (e.g., a first hand of a person, a first arm of a person, and / or a first side of a person), the computer system displaying a first graphical effect having first characteristics (e.g., a first orientation, a first graphical object, a first animation, and / or an effect oriented to match a first part type of a person performing the gesture) via a display generation component. Upon determining that a first gesture was performed by a second type of body part (e.g., a person's second hand, a person's second arm, and / or a person's second side), the computer system displays a first graphical effect via a display generation component that has a second characteristic (e.g., as shown by the left-hand emoji and left-hand thumbs-up gesture in Figure 6J) that is different from the first characteristic (e.g., an effect oriented to match the body part performing the gesture) (e.g., a second orientation, a second graphical object, a second animation, and / or an effect oriented to match the second part type of the person performing the gesture). Displaying a graphical effect having the first or second characteristic in response to detecting a gesture performed by the first or second part type provides the user with a simple and efficient way to display different graphical effects and variations of graphical effects without requiring input via a touch-sensitive display, mouse, and / or keyboard, which reduces the number of inputs that need to be performed to execute an action and provides additional control options without cluttering the UI with additional displayed controls.

[0231] In some embodiments, the first type of body part is the right hand, and the first characteristic is a right-hand emoji (e.g., a graphical representation of a right hand) (as shown, for example, in Figure 6G), and the second type of body part is the left hand, and the second characteristic is a left-hand emoji (e.g., a graphical representation of a left hand) (as shown, for example, in Figure 6J).

[0232] In some embodiments, while a representation of the field of view of one or more camera sensors (e.g., 604a, 606a, 604b, and / or 606b) is displayed (e.g., while a first graphical effect is displayed and / or while a first gesture is detected), a second gesture (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) performed by a second part of a person (e.g., different from the first part of the person) is detected within the field of view of one or more camera sensors (e.g., a part of the person's body, the user's hand, and / or both of the user's hands). In response to the detection of a second gesture performed by a second part of a person, the computer system displays, via display generation components, a second graphical effect different from the first graphical effect (e.g., 622a and / or 622b) (e.g., one or more graphical elements, animations, and / or graphical representations of the first gesture, such as emojis and / or animations corresponding to the first gesture) (e.g., simultaneously with the first graphical effect). Displaying a graphical effect in response to the detection of a second gesture provides the user with a simple and efficient way to display different graphical effects and variations of graphical effects without requiring input via a touch-sensitive display, mouse, and / or keyboard, thereby reducing the number of inputs required to perform the action and providing additional control options without cluttering the UI with additional displayed controls.

[0233] In some embodiments, the computer system displays a second graphical effect simultaneously with (for example, within) a representation of the field of view of one or more camera sensors. In some embodiments, displaying the second graphical effect includes displaying an animation corresponding to the second gesture (for example, an animation of fireworks, an animation of confetti, rain falling, and / or clouds moving). In some embodiments, the second gesture is detected while the first gesture is detected. In some embodiments, the second graphical effect is displayed simultaneously with the first graphical effect (for example, while both gestures are detected).

[0234] In some embodiments, displaying a first graphical effect and displaying a second graphical effect includes starting to display the first graphical effect within a predetermined time interval (for example, displaying graphical effects 618a and / or 618b shown in Figure 6G and graphical effects 618a and / or 618b shown in Figure 6J within a predetermined time interval) (for example, substantially simultaneously) (for example, the first and second gestures are performed simultaneously). Starting to display the first graphical effect and the second graphical effect within a predetermined time interval provides the user with feedback that both gestures have been detected, even if both gestures are temporally close, thereby providing the user with improved visual feedback. In some embodiments, the first and second gestures finish simultaneously. In some embodiments, the first and second graphical effects finish simultaneously. In some embodiments, the first and second gestures do not finish simultaneously. In some embodiments, the first and second graphical effects do not finish simultaneously.

[0235] In some embodiments, displaying a second graphical effect includes starting to display the second graphical effect after starting to display the first graphical effect (for example, if the second gesture is detected after the first gesture is detected) (for example, starting to display graphical effects 618a and / or 618b as shown in Figure 6G after starting to display graphical effects 618a and / or 618b as shown in Figure 6J). Starting to display the second graphical effect after starting to display the first graphical effect provides the user with feedback that gestures have been detected successively (one after the other), thereby providing the user with improved visual feedback.

[0236] In some embodiments, displaying a first graphical effect includes, upon determination that the first gesture is a first gesture type (e.g., a first gesture, a thumbs-up gesture, a thumbs-down gesture, a gesture performed with the left hand, a gesture performed with the right hand, a gesture performed with one hand, and / or a gesture performed with both hands), the computer system displaying a first graphical effect having a first visual appearance (e.g., 618a and / or 618b) (e.g., one or more graphical elements, animations, and / or graphical representations of the first gesture, such as emojis and / or animations corresponding to the first gesture). In accordance with the determination that the first gesture is of a second gesture type (e.g., a second gesture, a thumbs-up gesture, a thumbs-down gesture, a gesture performed with one hand, and / or a gesture performed with both hands), the computer system displays the first graphical effect through a display generation component in a second visual appearance different from the first graphical appearance (e.g., 622a, 622b, 624a, 624b, 626a, 626b, 628a, and / or 628b). Displaying a first graphical effect with a first visual appearance in response to detecting that a first gesture is of a first gesture type, and displaying a first graphical effect with a second visual effect in response to detecting that a first gesture is of a second gesture type, provides the user with a simple and efficient way to display different graphical effects and variations of graphical effects without requiring input via a touch-sensitive display, mouse, and / or keyboard, thereby reducing the number of inputs that need to be performed to execute an action and providing additional control options without cluttering the UI with additional displayed controls. In some embodiments, displaying a second graphical effect includes displaying an animation corresponding to the first gesture (e.g., fireworks animation, confetti animation, rain falling, and / or cloud movement).

[0237] In some embodiments, the computer system displays a first graphical effect (e.g., 618a, 618b, 622a, and / or 622b) via a display generation component, which includes displaying a sequence of effects including a first sequence effect and a second sequence effect (e.g., two or more emojis, one or more first effects followed by one or more second effects, and / or effects appearing continuously). In some embodiments, the effect appears continuously at a location on the display corresponding to the location of the gesture and moves to a second location. In some embodiments, the effect moves to the edge of the display and disappears when it reaches the edge of the display. In some embodiments, the sequence of effects includes a sequence of graphical elements of the first gesture, a sequence of animations, and / or a sequence of graphical representations.

[0238] In some embodiments, while a first gesture performed by a first part of a person is detected within the field of view of one or more camera sensors, the computer system detects movement by the first part of the person from a first location to a second location within the field of view of one or more camera sensors (e.g., a part of the person's body, the user's hand, and / or both hands of the user) (e.g., as shown in Figures 6G to 6I, subject Isaac moves his hand from in front of his body to the right of his body while performing a thumbs-up gesture) (e.g., movement from a first gesture location to a second gesture location and / or forward and backward movement). In response to the detection of movement by the first part of the person, the computer system applies a first sequence effect to the first display location in the representation of the field of view of one or more camera sensors corresponding to the first location (e.g., Figure 6G) via a display generation component. ShownThe graphical effects 618b and / or 618a are displayed at the first location, and the second sequence effect is displayed at the second display location in the representation of the field of view of one or more camera sensors corresponding to the second location (for example, as shown in Figure 6H, the graphical effects 618b and / or 618a are displayed at the first location) (for example, as a person's hand moves, the location of the graphical effect on the display generating component also moves to match the movement and location of the person's hand). By displaying the first sequence effect at the first display location and the second sequence effect at the second display location in response to the detection of movement by the first part, the user is provided with feedback that the gesture is still being detected while the gesture is moving, thereby providing the user with improved visual feedback.

[0239] In some embodiments, displaying a first graphical effect (e.g., 624a, 624b, 626a, 626b, 628a, and / or 628b) includes displaying the first graphical effect such that a portion of the first graphical effect is obscured by at least a portion of the representation of a person (e.g., 604a, 604b, 606a, and / or 616b) within the representation of the field of view of one or more camera sensors (e.g., the effect is displayed as if it were behind at least a portion of the person performing the gesture, and / or the graphical effect does not obscur the portion of the person performing the gesture). By displaying the first graphical effect such that a portion of the first graphical effect is obscured by at least a portion of the representation of a person, the computer system enables the user to distinguish the representation of a person within the representation of the field of view of one or more camera sensors from the first graphical effect, thereby providing the user with improved visual feedback.

[0240] In some embodiments, displaying a first graphical effect includes, according to a determination that a first gesture is detected in a first orientation (e.g., the hand performing the gesture is fully upright and the back of the hand faces one or more camera sensors, as shown in Figure 6I) (e.g., angle of the hand performing the gesture and / or side of the hand), the computer system displays the first graphical effect via a display generation component in an orientation of the first effect corresponding to the first orientation. According to a determination that a first gesture is detected in a second orientation (e.g., the hand performing the gesture is angled to one side, as shown in Figure 6J, and / or the palm of the hand performing the gesture is facing one or more camera sensors, as shown in Figure 6H) (e.g., angle of the hand performing the gesture and / or side of the hand), the computer system displays the first graphical effect via a display generation component in an orientation of the second effect corresponding to the second orientation different from the orientation of the first effect. Displaying a first graphical effect with a first effect orientation in response to detection that a first gesture has a first orientation, and displaying a first graphical effect with a second effect orientation in response to detection that a first gesture has a second orientation, provides users with a simple and efficient way to display different graphical effects and variations of graphical effects without requiring input via a touch-sensitive display, mouse, and / or keyboard, provides users with feedback that a gesture is in a particular orientation, which reduces the number of inputs that need to be performed to execute the action, provides additional control options without cluttering the UI with additional displayed controls, and provides improved visual feedback.

[0241] In some embodiments, the orientation of the first gesture is based on the position of the fingers and / or palm in space relative to the field of view of one or more camera sensors, for example, whether the thumb is pointing directly up, slightly to the side, forward, and / or backward while performing a thumbs-up gesture. In some embodiments, the orientation of the first gesture is based on which side of the person's body is detected within the field of view of one or more camera sensors (for example, the palm or back of the hand facing the camera while performing the first gesture).

[0242] In some embodiments, displaying a first graphical effect includes displaying an animation based on the movement of a first part of a person (for example, as shown in Figures 6G to 6I, subject Isaac moves his hand from in front of him to the right of his body while performing a thumbs-up gesture). Displaying an animation based on the movement of a first part of a person provides the user with feedback that the gesture is still being detected while it is in motion, thereby providing the user with improved visual feedback. In some embodiments, the movement of a first part of a person is detected. In some embodiments, displaying a first graphical effect includes displaying a first animation according to a determination that the movement has first characteristics (e.g., speed, direction, and / or pattern), and displaying a second animation different from the first animation according to a determination that the movement has second characteristics different from the first characteristics. In some embodiments, the speed, direction, magnitude, and / or amount of the animation (e.g., the number of graphical objects included in the animation, and / or the speed of one or more graphical objects included in the animation) is based on one or more characteristics of the person's movement.

[0243] In some embodiments, displaying an animation includes displaying an animation based on the speed of movement of a first part of a person (for example, subject Isaac moves his hand from in front of his body to the right of his body while performing a thumbs-up gesture, as shown in Figures 6G to 6I) (for example, the first graphical effect includes movement based on the speed of movement of a first part of a person). Displaying an animation based on the speed of movement of a first part of a person provides the user with feedback that the gesture is still being detected while the gesture is in motion, thereby providing the user with improved feedback. In some embodiments, according to a determination that the movement of a first part of the user has a first speed, the first graphical effect has a first movement, and according to a determination that the movement of a first part of the user has a second speed different from the first speed, the first graphical effect has a second movement different from the first movement.

[0244] In some embodiments, while the first graphical effect is being displayed, a second gesture (e.g., a waving gesture as shown in Figure 6Q) (e.g., moving, waving gesture, talking gesture, air gesture, thumbs-up gesture, thumbs-down gesture, victory sign gesture, heart gesture, and / or clapping gesture) performed by a third part of a person (e.g., a first part, a second part, or a part different from the first and second parts) is detected by a computer system within the field of view of one or more camera sensors. In response to the detection of the second gesture, the computer system changes the state of the first graphical effect (e.g., the confetti portion is no longer displayed near the waving gesture, as shown in Figure 6Q) (e.g., changing the location of at least a portion of the first graphical effect and / or discontinuing the display of at least a portion of the first graphical effect). Changing the state of the first graphical effect in response to the detection of a second gesture provides the user with feedback that a second gesture has been detected, offers an efficient way to change the graphical effect after it has been displayed, thereby providing the user with improved feedback and offering additional control options without cluttering the UI with additional displayed controls.

[0245] In some embodiments, changing the state of the first graphical effect includes moving at least a portion of the first graphical effect to a different location within a representation of the field of view of one or more camera sensors (e.g., without moving and / or changing the orientation of at least a portion of the first graphical effect). In some embodiments, changing the state of the first graphical effect includes moving the first graphical effect to the edge of the display, and then moving at least a portion of the first graphical effect outside the display. In some embodiments, changing the state of the first graphical effect includes moving at least a portion of the first graphical effect that is within a threshold distance of a second gesture (e.g., at that location).

[0246] In some embodiments, representations of the field of view of one or more camera sensors (e.g., 604a, 604b, 606a, and / or 606b) are included in a real-time (e.g., real-time video and / or audio) communication session (e.g., in a real-time communication session application) (e.g., transmitted and / or shared). Displaying graphical effects in response to detected gestures provides users with a simple and efficient way to display graphical effects during a real-time communication session without the need to provide input via a touch-sensitive display, mouse, and / or keyboard, or to interact with a menu interface that may obscure participants in the communication session, thereby reducing the number of inputs required to perform the action and providing additional control options without cluttering the UI with additional displayed controls.

[0247] In some embodiments, the representation of the field of view of one or more camera sensors includes a representation (e.g., a badge) of a first participant in a real-time communication session (e.g., 604a, 604b, 606a, and / or 606b) (e.g., within a view of a participant in a real-time communication session associated with a computer system, and / or within a view of a participant in a real-time communication session associated with a remote computer system). Displaying a representation of a first participant in a real-time communication session provides the user with feedback that they are in a real-time communication session, provides feedback that a graphical effect has been displayed, and thereby provides the user with improved feedback.

[0248] In some embodiments, while a representation of the field of view of one or more camera sensors is displayed, according to a determination that a first part of a person is being tracked (e.g., the location, position, and / or current shape of the part is detected and / or monitored by the computer system), the computer system displays hint user interface elements (e.g., 608a and / or 608b) via a display generation component indicating that a graphical effect can be displayed in response to a gesture performed by the first part of the person. According to a determination that a first part of a person is not being tracked (e.g., the location, position, and / or current shape of the part is not detected and / or monitored by the computer system), the computer system discontinues displaying hint user interface elements indicating that a graphical effect can be displayed in response to a gesture performed by the first part of the person. Displaying a hint user interface element in response to a gesture performed by a first part of a person, based on the determination that a first part of a person is being tracked, and deleting the display of the hint user interface element, based on the determination that a first part of a person is not being tracked, provides the user with feedback that a particular body part is being tracked and can be used to display a graphical effect without requiring interaction with the menu interface, thereby providing improved feedback and performing an action when a set of conditions is met without requiring further user input. In some embodiments, the hint user interface element includes highlighting a part of a person that can be used to trigger a graphical effect. In some embodiments, the hint user interface element includes text indicating that a graphical effect can be displayed in response to a gesture performed by a first part of a person.

[0249] In some embodiments, displaying a hint user interface element (e.g., 608a and / or 608b) includes displaying the hint user interface element at a first hint location (e.g., within the user interface and / or within a representation of the field of view of one or more camera sensors) (e.g., as shown in Figure 6B). In some embodiments, the movement of a first part of a person is detected by a computer system. In response to the detection of movement of the first part of a person, the computer system displays the hint user interface element at a second position different from the first position (e.g., as shown in Figure 6C) (based on the movement and / or change in position of the first part of the person) via a display generation component. Displaying the hint user interface element at a first hint location and at a second hint location in response to the detection of movement of the first part provides the user with feedback that a particular body part is being tracked and can be used to display graphical effects, thereby providing improved feedback. In some embodiments, the hint user interface element moves along with the first part of the user.

[0250] In some embodiments, while displaying a representation of the field of view of one or more camera sensors, the hint user interface element (e.g., 608b) is not displayed within a representation of a person displayed on a remote computer system (e.g., 600b) (e.g., a representation of the field of view of one or more camera sensors, or other representations of a person in a real-time communication session interface and / or application) (e.g., the hint user interface element is displayed by the computer system of the person who is detected to be performing the first gesture, but not by the computer systems of other participants in the real-time communication session). Because the user associated with the remote computer system has not initiated the display of the hint user interface element, ceasing to display the hint user interface element within a representation of a person displayed on the remote computer system provides the user associated with the remote computer system with a tidy user interface, thereby providing the user with additional control options without cluttering the user interface with additional displayed controls. In some embodiments, according to the determination that the first gesture was performed by a first participant in a real-time communication session, the hint user interface element is displayed on a display generating component that communicates with the computer system associated with the first participant, without being displayed on a computer system that is not associated with the first participant.

[0251] In some embodiments, the hint user interface element (e.g., 608a and / or 608b) is displayed on one or more display-generating components communicating with one or more computer systems (e.g., 600a and / or 600b) communicating via a real-time communication session (e.g., with computer systems) (e.g., the hint user interface element is displayed on two or more (e.g., all) of the computer systems participating in the real-time communication session, and for example, the hint is shown to a user who has not been detected to be performing a first gesture). Displaying the hint user interface element on one or more display-generating components communicating with one or more computer systems can be used to provide the user with feedback that a particular body part is being tracked and to display a graphical effect even when a particular user is not performing any action or gesture, thereby performing an action when a set of conditions is met without requiring further user input.

[0252] In some embodiments, in response to the detection of a first gesture performed by a first part of a person, and in accordance with the determination that a set of criteria is met, the computer system displays a cancel option (e.g., a selectable user interface object, button, icon, affordance) (e.g., 620a) via a display generation component (and, in some embodiments, simultaneously with displaying the first graphical effect) that, when selected, disables the display of the graphical effect in response to the detected gesture (e.g., prevents the computer system from displaying the first graphical effect in response to the detection of the first gesture performed by the first part of a person) (e.g., 620a). Displaying a cancel option to disable the display of the graphical effect in response to the detection of a first gesture performed by a first part of a person, and in accordance with the determination that a set of criteria is met, provides the user with a quick and efficient ability to disable the graphical effect without the need to access additional menu items when the display of the graphical effect is not a desired feature, thereby reducing the number of inputs required to perform the action. In some embodiments, the computer system detects the selection of the cancel option (e.g., a tap or other selection input directed thereto), and in response to the detection of the selection of the cancel option, the computer system disables the display of the graphical effect in response to the detected gesture. In some embodiments, the cancel option is displayed to all users. In some embodiments, the cancel option is displayed only to the user who triggered the effect. In some embodiments, selecting the cancel option stops the display of the effect in response to the detection of a gesture for all participants. In some embodiments, selecting the cancel option stops the display of the effect in response to the detection of a gesture only for the user who selected the cancel option. In some embodiments, if no selection is made (e.g., no input corresponding to yes or no is detected), the display generation component stops displaying the cancel option after a predetermined period.

[0253] In some embodiments, a set of criteria is met when the number of times a first graphical effect is displayed (e.g., in response to the detection of a first gesture performed by a first part of a person) meets a predetermined threshold number (e.g., is less than or equal to that number) (e.g., the cancel option is displayed the first N times the first gesture is detected). According to a determination that the set of criteria is not met (e.g., the number of times the first graphical effect is displayed does not meet the threshold number (e.g., is equal to or greater than the threshold number)), the computer system stops displaying the cancel option (e.g., 620a).

[0254] In some embodiments, the computer system detects an input (e.g., 614a) (e.g., an input different from the first gesture) that corresponds to a request to display a first graphical effect (e.g., 616a and / or 616b) (e.g., a selection of a user interface element corresponding to the first graphical effect, such as an option in a menu containing one or more selectable options corresponding to each graphical effect). In response to detecting an input that corresponds to a request to display a first graphical effect, the computer system displays the first graphical effect via a display generation component, and the computer system displays a graphical indication (e.g., 613a) via the display generation component (and, in some embodiments, simultaneously with the first graphical effect) that shows how the first graphical effect can be displayed in response to a gesture (e.g., a video showing how the gesture can be performed, and / or a diagram showing how the gesture can be performed). In some embodiments, the graphical indication is overlaid on a representation of the field of view of one or more camera sensors. Displaying a graphical indication showing how to display the first graphical effect in response to detecting input corresponding to a request to display the first graphical effect provides the user with improved feedback by allowing them to trigger the first graphical effect using a gesture without anticipating or requiring further user input to display the graphical indication, thereby reducing the number of inputs required to perform the action and performing the action when a set of conditions is met without requiring further user input.

[0255] In some embodiments, while a representation of the field of view of one or more camera sensors is displayed, a second gesture (e.g., a square gesture as shown in Figure 6R) (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) performed by a second part of a person (e.g., a part of the person's body, the user's hand, and / or both of the user's hands) is detected within the field of view of one or more camera sensors (e.g., a part of the person's body, the user's hand, and / or both of the user's hands). In response to the detection of a second gesture performed by a second part of a person (and, in some embodiments, according to the determination that the second gesture is of a distinct gesture type), the computer system initiates a process to save an image (e.g., thumbnails 630a and / or thumbnails 630b as shown in Figure 6R) (e.g., a representation of the field of view of one or more camera sensors, a screen capture, and / or a screen recording) (e.g., to memory communicating with the computer system, such as local or network memory storage). Initiating a process to save an image in response to the detection of a second gesture performed by a second part of a person provides the user with an efficient way to save an image without requiring additional user interface elements, thereby providing additional control options without cluttering the UI with additional displayed controls.

[0256] In some embodiments, while displaying the first graphical effect (or, in some embodiments, including displaying the first graphical effect), the computer system displays a simulated lighting effect (for example, as shown in Figure 6N, where the back and edges of the subject performing the gesture (Isaac) are illuminated, and a portion of the front of the subject is covered in shadow) on the representation of the person (e.g., 604a, 606a, 604b, and / or 606b) (e.g., an image of the person within the field of view of one or more camera sensors) via a display generation component. Displaying a simulated lighting effect on the representation of the person while displaying the first graphical effect provides the user with a more visually distinctive graphical effect and provides the user with feedback that the first gesture has been detected, thereby providing the user with improved feedback. In some embodiments, displaying the simulated lighting effect includes applying the lighting effect to a third portion of the person and applying the shadow effect to a fourth portion of the person. In some embodiments, the simulated lighting effect includes an effect that illuminates a part of the figure's representation (e.g., the head, shoulders, sides of the figure's body, and / or edges of the figure's body) while casting a shadow on one or more other parts of the figure's representation (e.g., a fireworks effect). In some embodiments, the simulated lighting effect includes a rain effect that casts a shadow on the figure's representation, and / or a confetti effect that casts a small shadow across the figure's representation.

[0257] In some embodiments, while a representation of the field of view of one or more camera sensors is displayed, a second gesture (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) performed by a second part of a person (e.g., a part of the person's body, the user's hand, and / or both hands of the user) is detected within the field of view of one or more camera sensors (e.g., a part of the person's body, the user's hand, and / or both hands of the user). In response to the detection of the second gesture performed by the second part of the person, and according to a determination that it is invalid to display a graphical effect in response to the gesture detection, the computer system displays a prompt (e.g., 610) (e.g., an instruction and / or selectable option) via a display generation component to enable the display of a graphical effect in response to the detected gesture, and discontinues the display of the second graphical effect. According to a determination that it is valid to display a graphical effect in response to the gesture detection, the computer system displays a second graphical effect corresponding to the second gesture. Displaying a prompt to enable the display of a graphical effect in response to a detected gesture, and then discontinuing the display of a second graphical effect in response to the detection of a second gesture and in accordance with the determination that the display of a graphical effect in response to the gesture detection is disabled, provides an efficient technique for displaying graphical effects in response to detected gestures without requiring the opening of additional menus or options, thereby reducing the number of inputs required to perform the action.

[0258] In some embodiments, displaying a first graphical effect includes displaying a first graphical effect having a depth effect (as described, for example, with respect to 634a in Figure 6W and / or 636a in Figure 6X). In some embodiments, displaying a first graphical effect having a depth effect includes displaying a first portion of the first graphical effect at a first depth (e.g., in front of a person or representation of a person) and displaying a second portion of the first graphical effect at a second depth different from the first depth (e.g., behind a person or representation of a person). In some embodiments, displaying a first graphical effect having a depth effect includes displaying the first graphical effect at a first depth at a first time and displaying the first graphical effect at a second depth different from the first depth at a second time different from the first time. In some embodiments, displaying a representation of the field of view of one or more camera sensors (e.g., 606a and / or 602b) includes displaying a representation of a person (e.g., 606a and / or 604b) and a representation of the background behind the representation of the person, and includes applying visual effects (e.g., blurring, portrait effect, lighting effect, low resolution, and / or out-of-focus) to a portion of the first graphical effect behind the background and the representation of the person (e.g., as described for 636a in Figure 6X). In some embodiments, the first graphical effect moves from the foreground (e.g., in front of the representation of the person) to the background (e.g., behind the representation of the person), and the computer system applies visual effects to the first graphical effect when the first graphical effect is displayed in the background (e.g., moves to the background) (e.g., blurring the first graphical effect, reducing the definition of the first graphical effect, and / or displaying the first graphical effect out of focus).

[0259] In some embodiments, displaying a first graphical effect includes displaying a first graphical effect having colors based on the color temperature of the representation of the field of view of one or more camera sensors (e.g., a computer system applies the color temperature of the scene in the field of view of one or more camera sensors to the first graphical effect, and / or the computer system matches the colors of the first graphical effect to the color temperature of the representation of the field of view of one or more camera sensors) (for example, as described with respect to 636a of Figure 6X).

[0260] It should be noted that the details of the process described above with respect to Method 700 (for example, Figure 7A) are similarly applicable to the methods described below. For example, Method 750 optionally includes one or more characteristics of the various methods described above with reference to Method 700. For example, Method 750 optionally displays a hint that a graphical effect can be displayed in response to a gesture when a first set of criteria is not met. As another example, Method 900 optionally includes one or more characteristics of the various methods described above with reference to Method 700. For example, Method 900 optionally displays a graphical effect in response to the computer system detecting a gesture while sharing content associated with a participant. As yet another example, Method 1100 optionally includes one or more characteristics of the various methods described above with reference to Method 700. For example, Method 1100 optionally displays a hint for displaying a graphical effect in both the first and second applications. As yet another example, Method 1300 optionally includes one or more characteristics of the various methods described above with reference to Method 700. For example, Method 1300 optionally displays a graphical effect in response to the detection of a gesture while the representation of the subject is being framed by the computer system. For brevity, these details will not be repeated below.

[0261] Figure 7B is a flowchart illustrating a method for displaying graphical effects during a real-time communication session using a computer system, according to several embodiments. Method 750 is performed on a computer system (e.g., 100, 300, 500, 600a, 600b, 800a, 800b, 800c, 1000a, 1000b, and / or 1200a) communicating with display generation components (e.g., 601a and / or 601b). Some operations in Method 750 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0262] As described below, Method 750 provides an intuitive method for displaying graphical effects during a real-time communication session. This method reduces the cognitive burden on the user to display graphical effects during a real-time communication session, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to display graphical effects faster and more efficiently during a real-time communication session saves power and increases the time between battery charges.

[0263] In method 750, the computer system displays a representation of the field of view of one or more camera sensors (e.g., 602a, 602b, 606a, and / or 606b) (e.g., an image, a video feed, and / or a live preview) via a display generation component (752), wherein the person is visible within the representation of the field of view of one or more camera sensors. While the representation of the field of view of one or more camera sensors is displayed, a first gesture (e.g., a body gesture, a hand gesture, an air gesture, a thumbs-up gesture, a thumbs-down gesture, a victory sign gesture, a heart gesture, and / or a clapping gesture) performed by a first part of the person (e.g., a part of the user's body, the user's hand, and / or both of the user's hands) is detected within the field of view of one or more camera sensors (754). In response to the detection of a first gesture performed by a first part of a person (756), and in accordance with the determination that a first set of criteria is met, the computer system displays a graphical effect corresponding to the first gesture within a representation of the field of view of one or more camera sensors (e.g., 618a, 618b, 622a, 622b, 624a, 624b, 626a, 626b, 628a, and / or 628b) via a display generating component (758). In some embodiments, the computer system displays the graphical effect simultaneously with (e.g., within) a representation of the field of view of one or more camera sensors. In some embodiments, displaying a graphical effect includes displaying an animation corresponding to the first gesture (e.g., an animation of fireworks, an animation of confetti, rain falling, and / or clouds moving). In some embodiments, the graphical effect is displayed by (e.g., together with a representation of the field of view of one or more camera sensors) by one or more other computer systems in a real-time video communication session with the computer system.In accordance with the determination that the first set of criteria is not met, the computer system refrains from displaying a graphical effect corresponding to the first gesture within the representation of the field of view of one or more camera sensors (760) (for example, as shown in Figures 6F and 6S-6V, no graphical effect is displayed in response to the detection of the gesture) (for example, the representation of the field of view of one or more camera sensors is displayed without displaying a graphical effect). Displaying a graphical effect in response to the detection of the first gesture in accordance with the determination that the first set of criteria is met, and refraining from displaying a graphical effect in response to the detection of the first gesture in accordance with the determination that the first set of criteria is not met, indicates to the user that the criteria required for the graphical effect to be displayed have been successfully met, enabling the user to initiate the displayed effect without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing the unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform the action.

[0264] In some embodiments, the first set of criteria includes location criteria that are satisfied when the location of the first part of a person relative to the location of a second part of a person (e.g., a second part of the user's body, the user's head, the user's face, the user's hands, and / or both hands of the user) satisfies a set of location criteria (e.g., the criterion that the hand performing the gesture is not in front of the face of the person performing the gesture, as shown in Figure 6S) (e.g., the first gesture triggers a graphical effect and / or the first gesture is displayed). Displaying a graphical effect in response to the detection of a first gesture according to the determination that the first set of criteria, including the satisfied location criteria, is satisfied provides the ability to initiate the display of a graphical effect to be displayed without navigating the user interface, and requiring the satisfaction of location criteria reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action. In some embodiments, the first set of criteria is based on the location of the first part of a person relative to a second part of a person. In some embodiments, the location criterion is met when a first part of a person is at a first location relative to a second part of the person (and the computer system displays a graphical effect), and the location criterion is not met (and the computer system stops displaying the graphical effect) if the first part of the person is at a second location (different from the first location) relative to the second part of the person. In some embodiments, the location criterion is not met if the first part of the person is within a threshold distance of a second part, such as the user's face, body, and / or other hands.

[0265] In some embodiments, the first part of the person is the person's hands and the second part is the person's face, and the first set of criteria is not met when the person's hands are over the person's face (as shown, for example, in Figure 6S). Declining to display a graphical effect in response to the detection of a first gesture, according to a determination that the first set of criteria is not met, including when the person's hands are over the person's face, allows the user to initiate the displayed effect without having to navigate the user interface, and requiring the location criterion to be met reduces errors by preventing unintended display of the effect, thereby providing additional control options without cluttering the UI with additional displayed controls and reducing the number of inputs required to perform the action.

[0266] In some embodiments, the first set of criteria includes a duration criterion that is met when the duration of a first gesture (e.g., continuous duration or cumulative duration) meets a threshold duration (e.g., greater than or equal to the threshold duration) (for example, as shown in Figure 6F, if the threshold duration is not met, the graphical effect is not displayed). In some embodiments, the computer system does not display the graphical effect if the duration of the first gesture is less than the threshold duration (e.g., the first gesture is ignored). Displaying the graphical effect in response to the detection of a first gesture according to the determination that the first set of criteria, including the duration criterion, is met provides the user with a way to initiate the display of the graphical effect without navigating the user interface, and requiring the duration criterion to be met reduces errors by preventing unintended display of the effect, thereby providing additional control options without cluttering the UI with additional displayed controls and reducing the number of inputs required to perform the action.

[0267] In some embodiments, a first set of criteria includes criteria that are met when the detected movement of a first gesture (e.g., lateral movement, vertical movement, and / or shaking movement) satisfies a movement threshold (e.g., as shown in Figure 6F, if the movement threshold is not met, the graphical effect is not displayed) (e.g., a velocity threshold, speed threshold, direction threshold, and / or distance threshold) (e.g., it is less than or equal to it) (e.g., the first gesture does not trigger a graphical effect, and / or the first gesture is ignored). Displaying a graphical effect in response to the detection of a first gesture according to a determination that a first set of criteria is met, including a determination that the detected movement of the gesture meets a movement threshold, and discontinuing the display of a graphical effect in response to the detection of a first gesture according to a determination that a first set of criteria is not met, including a determination that the detected movement of the gesture does not meet a movement threshold, provides the user with an indication that the criteria required to display a graphical effect have been met, allowing the user to initiate the displayed effect without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing the unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action.

[0268] In some embodiments, a first set of criteria is based on the movement of a first part of a person. In some embodiments, the computer system does not display a graphical effect (e.g., the first gesture is ignored) if the movement of the first gesture exceeds a movement threshold (e.g., exceeds a velocity threshold).

[0269] In some embodiments, a first set of criteria includes criteria that are met when it is determined that an indicator of attention of a person performing a first gesture is directed towards a representation of the field of view of one or more camera sensors (and / or, in some embodiments, a display generating component, one or more camera sensors, and / or a computer system). Displaying a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria is met, including the determination that the indicator of attention is directed towards the representational field of view of one or more camera sensors, and refraining from displaying a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria is not met, including the determination that the indicator of attention is not directed towards the representational field of view of one or more camera sensors, provides the user with an indication that the criteria required to display a graphical effect are met, enabling the user to initiate the effect to be displayed without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action. In some embodiments, if the attention indicator of the person performing the first gesture is not determined to be directed towards a field of view representation of one or more camera sensors, the first set of criteria is not met (for example, the graphical effect corresponding to the first gesture is not displayed).

[0270] In some embodiments, the attention indicator includes the gaze of a person performing the first gesture (e.g., 632a) (as shown, for example, in Figures 6T1 and 6T2) (for example, the attention indicator is determined to be directed towards a field of view representation of one or more camera sensors when the person performing the gesture is looking in the direction of a field of view representation of one or more camera sensors and / or looking at a display generating component). Discontinuing the display of a graphical effect in response to the detection of a first gesture based on whether a first set of criteria is met, including the determination that a person's gaze is directed towards the field of view of one or more camera sensors, and discontinuing the display of a graphical effect in response to the detection of a first gesture based on whether a first set of criteria is not met, including the determination that a person n's gaze is not directed towards the field of view of one or more camera sensors, provides the user with an indication that the criteria required to display a graphical effect are met, allowing the user to initiate the displayed effect without having to navigate the user interface. Requiring the satisfaction of a first set of criteria reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action.

[0271] In some embodiments, the attention indicator includes the focus state (or, in some embodiments, the activity state) of the window displayed by the display generation component (e.g., the subject (Isaac) performing the gesture is not looking at the user interface 602a, as shown in Figure 6U) (e.g., the attention indicator is determined to be directed towards a field of view of one or more camera sensors when the application window or user interface includes a field of view representation of one or more camera sensors). Displaying a graphical effect in response to the detection of a first gesture according to the determination that a first set of criteria including the focus state is met, and refraining from displaying a graphical effect in response to the detection of a first gesture according to the determination that a first set of criteria including the focus state is not met, provides the user with an indication that the criteria required to display a graphical effect are met, allowing the user to initiate the displayed effect without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform the action. In some embodiments, if the application window is minimized or closed (while the application is still running and a person and gesture are detected within the field of view of one or more camera sensors), the attention indicator is determined to be directed towards a representation of the field of view of one or more camera sensors.

[0272] In some embodiments, a first set of criteria includes criteria that are met when an audio setting associated with a representation of the field of view of one or more camera sensors (e.g., microphone setting) (e.g., microphone setting for a real-time communication session) is active (e.g., no graphical effect is displayed when audio is inactive, as shown in Figure 6V) (e.g., enabled and / or not muted). Displaying a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria, including criteria that are met when the audio setting is active, is met, and discontinuing to display a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria is not met, provides the user with an indication that the criteria required to display a graphical effect are met, allowing the user to initiate the effect to be displayed without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action. In some embodiments, the criteria are not met (e.g., no graphical effect is displayed) when the audio setting is disabled or muted.

[0273] In some embodiments, a first set of criteria includes criteria that are met when an audio setting associated with a representation of the field of view of one or more camera sensors is not muted within a predetermined time period (e.g., a non-zero time period, 1 second, 2 seconds, 3 seconds, or 5 seconds) (e.g., no graphical effect is displayed when the audio is muted, as shown in Figure 6V). Displaying a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria, including the criteria that are met when the audio setting is not muted within a predetermined time period, is met, and discontinuing to display a graphical effect in response to the detection of a first gesture according to a determination that the first set of criteria is not met, provides the user with an indication that the criteria required to display a graphical effect are met, enabling the user to initiate the displayed effect without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform an action. In some embodiments, if the audio settings are muted within a predetermined time period, the first set of criteria is not met (for example, graphical effects are not displayed).

[0274] In some embodiments, a computer system (e.g., 600a) communicates with one or more external computer systems (e.g., 600b) (e.g., smartphones, desktop computers, laptops, tablets, head-mounted devices, wearable devices, and / or smartwatches) via a real-time communication session (e.g., a live audio and / or video communication session). In some embodiments, a representation of the field of view of one or more camera sensors is included in the real-time communication session (e.g., transmitted, shared, and / or displayed) (e.g., displayed in the user interface of a real-time communication application). Discontinuing the display of graphical effects in response to the detection of a first gesture based on whether a first set of criteria is met, and discontinuing the display of graphical effects in response to the detection of a first gesture based on whether a first set of criteria is not met, while a computer system is communicating with one or more external computer systems via a real-time communication session, indicates to the user that the criteria required to display the graphical effect have been successfully met, allowing the user to initiate the displayed effect without having to navigate the user interface. Requiring the fulfillment of a first set of criteria reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform actions.

[0275] In some embodiments, the graphical effect corresponding to the first gesture is different from the movement of a part of a person corresponding to the first gesture (for example, the graphical effect includes one or more graphical elements, animations, and / or graphical representations of the first gesture, such as emojis and / or animations corresponding to the first gesture). Displaying the graphical effect in response to the detection of the first gesture according to the determination that a first set of criteria is met, and the graphical effect is different from the movement of a part of a person, and discontinuing to display the graphical effect in response to the detection of the first gesture according to the determination that a first set of criteria is not met, provides the user with an indication that the criteria required to display the graphical effect have been met, allowing the user to initiate the effect to be displayed without having to navigate the user interface, and requiring the first set of criteria to be met reduces errors by preventing unintended display of effects, thereby providing improved visual feedback and reducing the number of inputs required to perform the action.

[0276] It should be noted that the details of the process described above with respect to Method 750 (e.g., Figure 7B) are similarly applicable to the methods described below / above. For example, Method 700 optionally includes one or more characteristics of the various methods described above with reference to Method 750. For example, Method 700 optionally refrains from displaying graphical effects if the criteria are not met. As another example, Method 900 optionally includes one or more characteristics of the various methods described above with reference to Method 750. For example, Method 900 optionally refrains from displaying graphical effects in response to the computer system detecting a gesture while sharing content associated with a participant, if the criteria are not met. As another example, Method 1100 optionally includes one or more characteristics of the various methods described above with reference to Method 750. For example, Method 1100 optionally refrains from displaying graphical enhancements if a set of criteria is not met. As another example, Method 1300 optionally includes one or more characteristics of the various methods described above with reference to Method 750. For example, Method 1300 optionally discontinues displaying graphical effects if the criteria are not met, in response to the computer detecting a gesture while framing the representation of the subject in the field of view of one or more camera sensors. For brevity, these details will not be repeated below.

[0277] Figures 8A to 8Z show exemplary user interfaces for sharing content during a real-time communication session, according to several embodiments. These user interfaces are used to illustrate the processes described below, including the process shown in Figure 9.

[0278] In Figure 8A, computer system 800a (e.g., Abby's computer) displays the user interface 802 for a real-time communication session between computer system 800a, computer system 800b (e.g., Nate's computer), and computer system 800c (e.g., Jose's computer). In some embodiments, computer system 800a, computer system 800b, and / or computer system 800c are desktop computers, laptop computers, tablets, smartphones, and / or other mobile devices. The user interface 802 includes representations 802a, 802b, and 802c (e.g., representations of participants in the real-time communication session, i.e., Abby, Nate, and Jose). Representation 802a (e.g., video feeds, still images, and / or avatars) represents a participant (e.g., Abby) associated with computer system 800a (e.g., Abby's computer). Representation 802b represents a participant (e.g., Nate) associated with computer system 800b (e.g., Nate's computer). Representation 802c represents a participant (e.g., Jose) associated with computer system 800c (e.g., Jose's computer). Computer system 800b displays user interface 804 containing representations of participants in a real-time communication session (e.g., Abby, Nate, and Jose), and computer system 800c displays user interface 806 containing representations of participants in a real-time communication session (e.g., Abby, Nate, and Jose).

[0279] In Figure 8B, the computer system 800a (e.g., Abby's computer) detects a request to share content. In response to the detection of the request to share content, the computer system 800a begins sharing content in a first mode and displays the user interface 808. The user interface 808 includes the shared content 808a, representation 808b1, and representation 808b2. Representations 808b1 and 808b2 are representations of other participants in the real-time communication session. In some embodiments, the shared content 808a is shown in an expanded format (e.g., fullscreen) or a reduced format (e.g., part of the user interface 808). In some embodiments, the representations of participant 808b1 and participant 808b2 obscure parts of the shared content 808a. In some embodiments, the user interface 808 does not include a representation of the participant associated with the shared content 808a (e.g., Abby).

[0280] In response to detecting a request to share content, computer system 800b displays user interface 810, and computer system 800c displays user interface 812. User interface 810 on computer system 800b includes complex 810a, which is a composite of a representation 810b of the shared content 808a and a representation 810c of a participant associated with the shared content 808a (e.g., Abby). The representation 810c of the participant associated with the shared content 808a is not displayed separately from the representation 810b of the shared content 808a. In some embodiments, the representation 810b of the shared content 808a is displayed as if it were behind (and over the shoulders of) the representation 810c of the participant associated with the shared content 808a (e.g., Abby). In some embodiments, the representation 810c of the participant associated with the shared content 808a obscures part of the shared content 810b. In some embodiments, the representation 810c of the shared content 808a is of a different size than that shown in Figure 8B (e.g., larger and / or occupies a larger portion of the user interface 810). In some embodiments, the representation 810c of the participant associated with the shared content 808a is of a different size than that shown in Figure 8B (e.g., smaller and / or occupies a smaller portion of the user interface 810). In some embodiments, the representation 810c of the shared content 808a is positioned closer to the center of the user interface 810 than that shown in Figure 8B. In some embodiments, the representation 810c of the participant associated with the shared content 808a is positioned further to the side (e.g., to the right) of the user interface 810 than that shown in Figure 8B (e.g., offset). The user interface 810 includes representations of non-shared participant 810d1 and non-shared participant 810d2 (e.g., participants not associated with the shared content, such as Nate and Jose) displayed in front of the complex 810a. In some embodiments, the complex 810a is displayed in an expanded format (e.g., full screen).In some embodiments, the representations of non-shared participant 810d1 and non-shared participant 810d2 are displayed in a different order depending on which representation is associated with a particular computer system. For example, on Nate's computer, Nate's representation is displayed on the left side of the display and Jose's representation is displayed on the right side of the display. Similarly, on Jose's computer, Jose's representation is displayed on the left side of the display and Nate's representation is displayed on the right side of the display. In some embodiments, participants can choose where their representation is displayed relative to other representations. User interface 812 on computer system 800c may include any or all of the features of user interface 810, as described with reference to Figure 8B.

[0281] In Figure 8C, computer system 800a (e.g., Abby's computer) detects a request (e.g., via menu 808c) to change the configuration in which shared content 808a is displayed on computer system 800b (e.g., Nate's computer) and computer system 800c (e.g., Jose's computer). In some embodiments, menu 808c is a configuration menu associated with a real-time communication session application used for real-time communication sessions between computer systems 800a, 800b, and 800c. In some embodiments, menu 808c is a configuration menu associated with the operating system of computer system 800a.

[0282] In Figure 8D, in response to a request to change the configuration in which the shared content 808a is displayed, the computer system 800a provides the shared content 808a to the real-time communication session in a second mode and displays the user interface 814. The user interface 814 includes representations of non-shared participants 814c1, non-shared participants 814c2 (e.g., Nate and Jose), and a shared participant (e.g., Abby) 814b. In some embodiments, the representation of shared participant 814b includes a representation of the participant's neck and head (e.g., a representation of only the participant's neck and head). In some embodiments, the representation of shared participant 814b is a video feed of the shared participant segmented from a physical background (e.g., a background within the field of view of one or more camera sensors). In some embodiments, the representation of shared participant 814b includes an animated avatar animated to match the movement of the shared participant's mouth (e.g., Abby) when the shared participant is speaking. In some embodiments, the representation of shared participant 814b includes a circular colored background. In some embodiments, the top of the representation of shared participant 814b (e.g., the top of the participant's head) is displayed outside the border of a colored circular background (e.g., interrupting the frame). In some embodiments, the representation of shared participant 814b casts a simulated shadow on the shared content 808a. In some embodiments, the shared content 808a and the representation of shared participant 814b are combined together and delivered to computer systems 800b and 800c as a single feed (e.g., a combined feed). In response to a request to change the configuration in which the shared content 808a is displayed, computer system 800b displays user interface 816 and computer system 800c displays user interface 818. User interface 816 includes representation 816a of shared content 808a, representation of shared participant 816b, representation of non-shared participant 816c1, and representation of non-shared participant 816c2.In some embodiments, the representations of non-shared participant 816c1 and non-shared participant 816c2 are displayed in a different order (e.g., from left to right) depending on which representation is associated with which participant associated with a part...

Claims

1. It is a method, In a computer system having at least a first camera sensor that communicates with a display generation component, Through the aforementioned display generation component, A representation of the field of view of the first camera sensor, wherein the representation of the field of view of the first camera sensor includes a first representation of a participant associated with the computer system, A second representation of a participant associated with an external computer system, The participant associated with the aforementioned computer system differs from the participant associated with the aforementioned external computer system, The aforementioned external computer system is different from the aforementioned computer system. A second representation of the participant associated with an external computer system, Displaying them simultaneously, While the representation of the field of view of the first camera sensor, including the first representation, is displayed simultaneously with the second representation, a first gesture performed by the first part of the participant associated with the computer system is detected within the field of view of the first camera sensor, and the representation of the first part of the participant is included in the first representation. In response to the detection of the first gesture performed by the first part of the participant associated with the computer system, In accordance with the determination that the first gesture is maintained for a predetermined duration and that the first gesture is detected at a first gesture location within the field of view of the first camera sensor, the first graphical effect is displayed at a first effect location within the representation of the field of view of the first camera sensor via the display generation component. If the first gesture is maintained beyond the predetermined duration, and the first gesture is located at a second gesture location within the field of view of the first camera sensor that is different from the first gesture location, the first graphical effect is displayed via the display generation component at the second effect location within the representation of the field of view of the first camera sensor that is different from the first effect location. In accordance with the determination that the first gesture is maintained for less than the predetermined duration, the display of the first graphical effect is discontinued while the representation of the first part of the participant performing the first gesture is displayed. Methods that include...

2. While the representation of the field of view of the first camera sensor is displayed, a second gesture is detected within the field of view of the first camera sensor, and the second gesture is performed by the first portion of the participant associated with the computer system and the second portion of the participant associated with the computer system. The method according to claim 1, further comprising displaying a second graphical effect different from the first graphical effect via the display generation component in response to the detection of a second gesture performed by the first portion of the participant associated with the computer system and the second portion of the participant associated with the computer system.

3. The method according to claim 2, wherein the first gesture includes a thumbs-up gesture performed by the first portion of the participant associated with the computer system, and the second gesture includes a thumbs-up gesture performed by the first portion of the participant associated with the computer system and a thumbs-up gesture performed by the second portion of the participant associated with the computer system.

4. Displaying the first graphical effect is In accordance with the determination that the first gesture was performed by a first type of body part, the first graphical effect having the first characteristics is displayed via the display generation component, The method according to claim 1, comprising displaying the first graphical effect having second characteristics different from the first characteristics via the display generating component, in accordance with the determination that the first gesture was performed by a second type of body part.

5. The first type of body part is the right hand, and the first characteristic is the emoji of the right hand. The method according to claim 4, wherein the second type of body part is a left hand, and the second characteristic is a left hand emoji.

6. While the representation of the field of view of the first camera sensor is displayed, a second gesture performed by the second part of the participant associated with the computer system is detected within the field of view of the first camera sensor. The method according to claim 1, further comprising displaying a second graphical effect different from the first graphical effect via the display generation component in response to the detection of the second gesture performed by the second portion of the participant associated with the computer system.

7. The method according to claim 6, wherein displaying the first graphical effect and displaying the second graphical effect includes starting to display the first graphical effect within a predetermined time interval for starting to display the second graphical effect.

8. The method according to claim 6, wherein displaying the second graphical effect includes starting to display the second graphical effect after starting to display the first graphical effect.

9. Displaying the first graphical effect is In accordance with the determination that the first gesture is of the first gesture type, the first graphical effect is displayed with a first visual appearance via the display generation component, The method according to claim 1, comprising: displaying the first graphical effect in a second visual appearance different from the first visual appearance via the display generation component, in accordance with the determination that the first gesture is of a second gesture type.

10. The method according to claim 1, wherein displaying the first graphical effect via the display generation component includes displaying a sequence of effects including a first sequence effect and a second sequence effect.

11. While the first gesture performed by the first portion of the participant associated with the computer system is detected within the field of view of the first camera sensor, To detect the movement of the first portion of the participant associated with the computer system from a first location to a second location within the field of view of the first camera sensor, The method according to claim 10, further comprising, in response to detection of movement by the first portion of the participant associated with the computer system, displaying the first sequence effect at a first display location in the representation of the field of view of the first camera sensor corresponding to the first location, and the second sequence effect at a second display location in the representation of the field of view of the first camera sensor corresponding to the second location, via the display generation component.

12. The method according to claim 1, wherein displaying the first graphical effect includes displaying the first graphical effect such that a portion of the first graphical effect is obscured by at least a portion of the representation of the participant associated with the computer system within the representation of the field of view of the first camera sensor.

13. Displaying the first graphical effect is In accordance with the determination that the first gesture is detected in a first orientation within the field of view of the first camera sensor, the first graphical effect is displayed via the display generation component in the orientation of the first effect corresponding to the first orientation. The method according to claim 1, comprising: displaying the first graphical effect via the display generation component in a second effect orientation corresponding to the second orientation different from the orientation of the first effect, based on the determination that the first gesture was detected in a second orientation within the field of view of the first camera sensor, which is different from the first orientation.

14. The method according to claim 1, wherein displaying the first graphical effect includes displaying an animation based on the movement of the first portion of the participant associated with the computer system.

15. The method according to claim 14, wherein displaying the animation includes displaying the animation based on the speed of movement of the first portion of the participant associated with the computer system.

16. While the first graphical effect is being displayed, a second gesture performed by a third part of the participant associated with the computer system is detected within the field of view of the first camera sensor. The method according to claim 1, further comprising changing the state of the first graphical effect in response to the detection of the second gesture.

17. The method according to claim 1, wherein the representation of the field of view of the first camera sensor is included in a real-time communication session.

18. The method according to claim 17, wherein the representation of the field of view of the first camera sensor includes a representation of a first participant in the real-time communication session.

19. While the representation of the field of view of the first camera sensor is displayed, In accordance with the determination that the first portion of the participant associated with the computer system is being tracked, a hint user interface element is displayed indicating that graphical effects can be displayed via the display generation component in response to gestures performed by the first portion of the participant associated with the computer system. The method according to claim 1, further comprising: discontinuing the display of the hint user interface element indicating that a graphical effect can be displayed in response to a gesture performed by the first part of the participant associated with the computer system, in accordance with the determination that the first part of the participant associated with the computer system is not being tracked.

20. Displaying the hint user interface element includes displaying the hint user interface element at a first hint location, and the method is The movement of the first part of the participant associated with the computer system is detected. The method according to claim 19, further comprising displaying the hint user interface element at a second hint location different from the first hint location via the display generation component in response to the detection of the movement of the first portion of the participant associated with the computer system.

21. The method according to claim 19, further comprising the fact that while the representation of the field of view of the first camera sensor is being displayed, the hint user interface element is not displayed in the representation of the participant associated with the computer system displayed on the remote computer system.

22. The method according to claim 19, wherein the hint user interface element is displayed on one or more display generation components that communicate with one or more computer systems communicating via a real-time communication session.

23. In response to the detection of the first gesture performed by the first part of the participant associated with the computer system, The method according to claim 1, further comprising, in accordance with the determination that a set of criteria is met, displaying a cancel option via the display generation component, which, when selected, disables the display of graphical effects in response to a detected gesture.

24. The set of criteria is satisfied when the number of times the first graphical effect has been displayed satisfies the threshold number of times, and the method is The method according to claim 23, further comprising deactivating the display of the cancellation option in accordance with the determination that the set of criteria is not met.

25. To detect an input corresponding to a request to display the first graphical effect, In response to detecting the input corresponding to the request to display the first graphical effect, The first graphical effect is displayed via the display generation component, The method according to claim 1, further comprising displaying a graphical indication that shows a method for displaying the first graphical effect in response to a gesture via the display generation component.

26. While the representation of the field of view of the first camera sensor is displayed, a second gesture performed by a second part of the participant associated with the computer system is detected within the field of view of the first camera sensor. The method according to claim 1, further comprising initiating a process for saving an image in response to the detection of the second gesture performed by the second portion of the participant associated with the computer system.

27. The method according to claim 1, further comprising displaying a simulated lighting effect on the representation of the participant associated with the computer system via the display generation component while the first graphical effect is being displayed.

28. While the representation of the field of view of the first camera sensor is displayed, a second gesture performed by the second part of the participant associated with the computer system is detected within the field of view of the first camera sensor. In response to the detection of a second gesture performed by the second part of the participant associated with the computer system, According to the determination that displaying graphical effects in response to gesture detection is disabled, The display generation component displays a prompt to enable the display of graphical effects in response to the detected gesture, The second graphical effect will be discontinued, The method according to claim 1, further comprising displaying a second graphical effect corresponding to the second gesture, in accordance with a determination that it is effective to display a graphical effect in response to the detection of a gesture.

29. The method according to claim 1, wherein displaying the first graphical effect includes displaying the first graphical effect having a depth effect.

30. The method according to claim 29, wherein displaying the representation of the field of view of the first camera sensor includes displaying the representation of the participant associated with the computer system and the representation of the background behind the representation of the participant associated with the computer system, and includes applying a visual effect to a portion of the first graphical effect behind the representation of the background and the representation of the participant associated with the computer system.

31. The method according to claim 1, wherein displaying the first graphical effect includes displaying the first graphical effect having a color based on the color temperature of the representation of the field of view of the first camera sensor.

32. Displaying the first graphical effect at the first effect location includes moving the first graphical effect from the first effect location to a first separate location within the representation of the field of view that is different from the first effect location, while the first gesture is maintained at the first gesture location. Displaying the first graphical effect at the second effect location includes moving the first graphical effect from the second effect location to a second separate location within the representation of the field of view that is different from the second effect location, while the first gesture is maintained at the second gesture location. The method according to claim 1.

33. A computer program that causes a computer to perform the method described in any one of claims 1 to 32.

34. At least a first camera sensor and A memory storing the computer program described in claim 33, One or more processors capable of executing the computer program stored in the memory, A computer system comprising, The computer system is configured to communicate with the display generation component. Computer system.

35. Means for carrying out the method described in any one of claims 1 to 32 A computer system equipped with the following features.

Images