Creative Camera

Abstract

To provide faster and more efficient methods and interfaces for displaying visual effects on electronic devices. [Solution] This disclosure generally relates to displaying visual effects on image data. In some examples, the visual effect includes an avatar displayed on the user's face. In some examples, the visual effect includes a sticker applied to the image data. In some examples, the visual effect includes a screen effect. In some examples, the visual effect is modified based on depth data in the image data.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is related to U.S. Provisional Patent Application No. 62 / 668,227, entitled "Creative Camera", filed on May 7, 2018. The content of this application is hereby incorporated by reference in its entirety into this specification.

[0002] The present disclosure generally relates to computer user interfaces, and more specifically, to techniques for displaying visual effects.

Background Art

[0003] Visual effects are used to enhance the user experience when capturing and viewing media using an electronic device. Visual effects can change the appearance of image data or represent an idealized or completely fictional representation of the environment captured within the image.

Summary of the Invention

[0004] However, some techniques for displaying visual effects using an electronic device are generally cumbersome and inefficient. For example, in some existing techniques, complex and time - consuming user interfaces that may involve multiple key presses or keystrokes are used. Existing techniques require more time than necessary, wasting the user's time and the device's power. The latter problem is particularly significant in battery - operated devices.

[0005] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for displaying visual effects. Such methods and interfaces optionally complement or replace other methods for displaying visual effects. Such methods and interfaces reduce the cognitive burden on the user and generate a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces save power and increase the interval between battery charging.

[0006] A method is described below. The method is performed in an electronic device having a camera, a display device, and one or more input devices. The method includes: displaying a messaging user interface including a camera affordance for a message conversation including at least a first participant via the display device; detecting a first input directed to a camera affordance via one or more input devices; displaying a camera user interface including a capture affordance in response to the detection of the first input; detecting a second input directed to a capture affordance via one or more input devices; capturing image data using the camera in response to the detection of the second input; stopping the display of the capture affordance; displaying a transmit affordance at a location in the camera user interface previously occupied by the capture affordance; detecting a third input directed to a transmit affordance via one or more input devices; and initiating a process to transmit the captured image data to the first participant in response to the detection of the third input.

[0007] A non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera, a display device, and one or more input devices, the one or more programs including instructions to display a messaging user interface including camera affordances for a message conversation including at least a first participant via the display device, to detect a first input directed to camera affordances via one or more input devices, and in response to the detection of the first input, to display a camera user interface including capture affordances, to detect a second input directed to capture affordances via one or more input devices, and in response to the detection of the second input, to capture image data using the camera, to stop displaying the capture affordances, to display transmit affordances at a location in the camera user interface previously occupied by the capture affordances, and to detect a third input directed to transmit affordances via one or more input devices, and in response to the detection of the third input, to initiate a process to transmit the captured image data to the first participant.

[0008] A temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera, a display device, and one or more input devices, the one or more programs including instructions to display a messaging user interface including camera affordances for a message conversation including at least a first participant via the display device, to detect a first input directed to camera affordances via one or more input devices, and in response to the detection of the first input, to display a camera user interface including capture affordances, to detect a second input directed to capture affordances via one or more input devices, and in response to the detection of the second input, to capture image data using the camera, to stop displaying the capture affordances, to display transmit affordances at a location in the camera user interface previously occupied by the capture affordances, and to detect a third input directed to transmit affordances via one or more input devices, and in response to the detection of the third input, to initiate a process to transmit the captured image data to the first participant.

[0009] An electronic device is described below. The electronic device includes a camera, a display device, one or more input devices, one or more processors, and a memory for storing one or more programs configured to be executed by the one or more processors, wherein one or more programs include instructions for: displaying a messaging user interface including camera affordances for a message conversation including at least a first participant via the display device; detecting a first input directed to camera affordances via one or more input devices, and in response to detecting the first input, displaying a camera user interface including capture affordances; detecting a second input directed to capture affordances via one or more input devices, and in response to detecting the second input, capturing image data using the camera, stopping the display of capture affordances, displaying transmit affordances at locations in the camera user interface previously occupied by capture affordances; and detecting a third input directed to transmit affordances via one or more input devices, and in response to detecting the third input, initiating a process to transmit the captured image data to the first participant.

[0010] The following describes an electronic device. The electronic device includes a camera, a display device, one or more input devices, means for displaying a messaging user interface including camera affordances for a message conversation including at least a first participant via the display device, means for detecting a first input directed to camera affordances via one or more input devices, means for displaying a camera user interface including capture affordances in response to the detection of the first input, means for detecting a second input directed to capture affordances via one or more input devices, means for capturing image data using the camera, stopping the display of capture affordances, and displaying transmit affordances in a location within the camera user interface previously occupied by the capture affordances in response to the detection of the second input, means for detecting a third input directed to transmit affordances via one or more input devices, and means for initiating a process to transmit the captured image data to the first participant in response to the detection of the third input.

[0011] A method is described below. The method is performed in an electronic device having a camera and a display device. The method is to display a camera user interface via the display device, the camera user interface comprising a camera display area including a representation of image data captured via the camera, and a first affordance associated with a first camera display mode; detecting a gesture directed to the first affordance while a subject is located within the camera's field of view and representations of the subject and background are displayed in the camera display area, and activating a first camera display mode in response to the detection of a gesture directed to the first affordance, the activation of the first camera display mode comprising displaying an avatar selection area including one selected from a plurality of avatar options, and displaying a representation of the selected avatar option on the representation of the subject in the camera display area; detecting a change in the subject's posture while the first camera display mode is active, and, in response to the detection of a change in the subject's posture, changing the appearance of the displayed representation of the selected avatar option based on the detected change in the subject's posture while maintaining the display of the background.

[0012] This document describes non-temporary computer-readable storage media. A non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera and a display device, the one or more programs including instructions to display a camera user interface via the display device, which includes a camera display area containing a representation of image data captured via the camera, and a first affordance associated with a first camera display mode; to detect a gesture directed towards the first affordance while a subject is located within the camera's field of view and representations of the subject and background are displayed in the camera display area, and to activate a first camera display mode in response to the detection of a gesture directed towards the first affordance, the activation of the first camera display mode includes displaying an avatar selection area containing one selected from a plurality of avatar options, and displaying a representation of the selected avatar option on the representation of the subject in the camera display area, the detection of a change in the subject's posture while the first camera display mode is active, and to change the appearance of the displayed representation of the selected avatar option based on the detected change in the subject's posture, while maintaining the display of the background.

[0013] This section describes temporary computer-readable storage media. A temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera and a display device, the one or more programs including, via the display device, a camera user interface including a camera display area containing a representation of image data captured via the camera, and a first affordance associated with a first camera display mode, the detection of a gesture directed toward the first affordance while a subject is located within the camera's field of view and representations of the subject and background are displayed in the camera display area, and the activation of the first camera display mode includes displaying an avatar selection area containing one selected from a plurality of avatar options, and displaying a representation of the selected avatar option on the representation of the subject in the camera display area, the detection of a change in the subject's posture while the first camera display mode is active, and the instruction to change the appearance of the displayed representation of the selected avatar option based on the detected change in the subject's posture, while maintaining the display of the background.

[0014] An electronic device is described below. The electronic device includes a camera, a display device, one or more processors, and a memory for storing one or more programs configured to be executed by the one or more processors, wherein one or more programs display a camera user interface via the display device, which includes a camera display area containing a representation of image data captured via the camera, and a first affordance associated with a first camera display mode, and while a subject is located within the camera's field of view and representations of the subject and background are displayed in the camera display area, the program detects a gesture directed towards the first affordance, and in response to the detection of a gesture directed towards the first affordance, activates a first camera display mode, which includes displaying an avatar selection area containing one selected from a plurality of avatar options, and displaying a representation of the selected avatar option on the representation of the subject in the camera display area, and while the first camera display mode is active, detects a change in the subject's posture, and in response to the detection of a change in the subject's posture, modifies the appearance of the displayed representation of the selected avatar option based on the detected change in the subject's posture, while maintaining the display of the background, and includes instructions.

[0015] The following describes an electronic device. The electronic device includes a camera, a display device, one or more input devices, and means for displaying a camera user interface via the display device, the camera user interface including a camera display area containing a representation of image data captured via the camera, and a first affordance associated with a first camera display mode; means for detecting a gesture directed to the first affordance while a subject is located within the camera's field of view and representations of the subject and background are displayed in the camera display area; and means for activating a first camera display mode in response to the detection of a gesture directed to the first affordance, the activation of the first camera display mode including displaying an avatar selection area containing one selected from a plurality of avatar options, and displaying a representation of the selected avatar option on the representation of the subject in the camera display area; and means for detecting a change in the subject's posture while the first camera display mode is active, and means for changing the appearance of the displayed representation of the selected avatar option based on the detected change in the subject's posture, while maintaining the display of the background, in response to the detection of a change in the subject's posture.

[0016] A method is described below. The method is performed on an electronic device having a display device. The method includes, via the display device, displaying a media user interface including a media display area including a representation of a media item and an effect affordance; detecting a gesture directed towards the effect affordance; and, in response to the detection of a gesture directed towards the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, wherein, according to the determination that the media item is associated with corresponding depth data, the plurality of effect options include each effect option for applying an effect based on the depth data, and according to the determination that the image data does not include depth data, each effect option is unavailable for activation in the plurality of effect options.

[0017] A non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device equipped with a display device, the one or more programs include instructions to display a media user interface via the display device, which includes a media display area including a representation of a media item and effect affordances, to detect a gesture directed toward an effect affordance, and in response to the detection of a gesture directed toward an effect affordance, to display a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, the plurality of effect options including each effect option for applying an effect based on depth data according to the determination that the media item is associated with corresponding depth data, and the respective effect options not available for activation in the plurality of effect options according to the determination that the image data does not include depth data.

[0018] A temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device equipped with a display device, the one or more programs include instructions to display a media user interface via the display device, which includes a media display area including a representation of a media item and effect affordances, to detect a gesture directed toward an effect affordance, and in response to the detection of a gesture directed toward an effect affordance, to display a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, the plurality of effect options include each effect option for applying an effect based on the depth data according to the determination that the media item is associated with corresponding depth data, and each effect option is unavailable for activation in the plurality of effect options according to the determination that the image data does not include depth data.

[0019] An electronic device is described below. The electronic device includes a display device, one or more processors, and a memory that stores one or more programs configured to be executed by the one or more processors, wherein the one or more programs display a media user interface via the display device, which includes a media display area including a representation of a media item and effect affordances, detects a gesture directed toward an effect affordance, and, in response to the detection of a gesture directed toward an effect affordance, includes instructions that display a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, wherein, according to the determination that the media item is associated with corresponding depth data, the plurality of effect options include each effect option for applying an effect based on the depth data, and according to the determination that the image data does not include depth data, each effect option is unavailable for activation in the plurality of effect options.

[0020] The following describes an electronic device. The electronic device includes a display device, one or more input devices, means for displaying a media user interface via the display device, which includes a media display area including a representation of a media item, and effect affordances, means for detecting a gesture directed towards an effect affordance, and means for displaying a plurality of effect options for applying an effect to a media item simultaneously with the representation of the media item in response to the detection of a gesture directed towards an effect affordance, wherein, according to the determination that the media item is associated with corresponding depth data, the plurality of effect options include each effect option for applying an effect based on the depth data, and according to the determination that the image data does not include depth data, each effect option is unavailable for activation in the plurality of effect options.

[0021] A method is described below. The method is performed on an electronic device having a display device. The method is to display a live video communication user interface of a live video communication application via the display device, wherein the live video communication user interface includes a representation of a subject participating in a live video communication session and a first affordance; to detect a gesture directed toward the first affordance; and to activate a camera effect mode and increase the size of the representation of the subject participating in the live video communication session in response to the detection of a gesture directed toward the first affordance.

[0022] A non-temporary computer-readable storage medium is described. The non-temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs include instructions for displaying a live video communication user interface via the display device, which includes a representation of a subject participating in a live video communication session and a first affordance, detecting a gesture directed toward the first affordance, and in response to the detection of a gesture directed toward the first affordance, activating a camera effect mode and increasing the size of the representation of the subject participating in the live video communication session.

[0023] A temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs include instructions for displaying a live video communication user interface via the display device, which includes a representation of a subject participating in a live video communication session and a first affordance, detecting a gesture directed toward the first affordance, and in response to the detection of a gesture directed toward the first affordance, activating a camera effect mode and increasing the size of the representation of the subject participating in the live video communication session.

[0024] An electronic device is described below. The electronic device includes a display device, one or more processors, and a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for a live video communication application, a live video communication user interface including a representation of a subject participating in a live video communication session and a first affordance, detecting a gesture directed toward the first affordance, and in response to detecting a gesture directed toward the first affordance, activating a camera effect mode and increasing the size of the representation of the subject participating in the live video communication session.

[0025] The following describes an electronic device. The electronic device includes a display device, one or more input devices, and means for displaying a live video communication user interface of a live video communication application via the display device, wherein the live video communication user interface includes a representation of a subject participating in a live video communication session and a first affordance, means for detecting a gesture directed toward the first affordance, and means for activating a camera effect mode and increasing the size of the representation of the subject participating in the live video communication session in response to the detection of a gesture directed toward the first affordance.

[0026] A method is described below. The method is performed in an electronic device having a camera and a display device. The method includes displaying a representation of image data captured through one or more cameras via the display device, wherein the representation includes a representation of a subject and the image data corresponds to depth data including depth data of the subject; and displaying a representation of a virtual avatar via the display device that is displayed in place of at least a portion of the representation of the subject, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the subject, determined based on the depth data of the subject, wherein the representation of the virtual avatar is based on depth data, and the first portion of the virtual avatar corresponds to the first This includes, in accordance with a determination that a set of depth-based display criteria is met, which includes the requirement that the depth data of the subject indicates that it has a simulated depth in front of the subject, including a representation of the first part of the virtual avatar that is displayed in place of the first part of the subject as part of the representation of the virtual avatar so that the depth-based display criteria are met, and, in accordance with a determination that the first part of the virtual avatar does not meet the set of depth-based display criteria for the first part of the subject based on the depth data, excluding the representation of the first part of the virtual avatar from the representation of the virtual avatar and displaying the first part of the subject in the area that would have been occupied by the first part of the virtual avatar.

[0027] A non-transitory computer-readable storage medium will be described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device including a camera and a display device. The one or more programs cause, via the display device, a representation of image data captured via one or more cameras to be displayed, the representation including a representation of a subject, and the image data corresponding to depth data including depth data of the subject. The one or more programs include instructions to display, via the display device, a representation of a virtual avatar to be displayed in place of at least a portion of the representation of the subject. The virtual avatar is disposed at a simulated depth relative to the representation of the subject determined based on the depth data of the subject. Displaying the representation of the virtual avatar includes, based on the depth data, including in the representation of the virtual avatar, as a part of the representation of the virtual avatar, a representation of a first portion of the virtual avatar that has a simulated depth that is a depth-based display criterion and that is in front of a corresponding first portion of the subject, according to a determination that the depth data of the subject indicates that a set of depth-based display criteria is satisfied. Displaying the representation of the virtual avatar includes, based on the depth data, excluding from the representation of the virtual avatar a representation of a first portion of the virtual avatar according to a determination that the first portion of the virtual avatar does not satisfy a set of depth-based display criteria for the first portion of the subject, and displaying the first portion of the subject in an area that would have been occupied by the first portion of the virtual avatar.

[0028] A temporary computer-readable storage medium is described. The temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera and a display device, the one or more programs including instructions to display a representation of image data captured via one or more cameras via the display device, the representation including a representation of a subject, and the image data corresponding to depth data including depth data of the subject, and to display a representation of a virtual avatar which is displayed via the display device in place of at least a portion of the representation of the subject, the virtual avatar being positioned at a simulated depth relative to the representation of the subject determined based on the depth data of the subject, and displaying the representation of the virtual avatar means that, based on the depth data, a first part of the virtual avatar is a depth-based table The display includes, in accordance with a determination that satisfies a set of depth-based display criteria, which includes the requirement that depth data of a subject indicates that a first part of a virtual avatar has a simulated depth in front of a corresponding first part of a subject, including, as part of the representation of the virtual avatar, a representation of the first part of the virtual avatar that is displayed in place of the first part of the subject, and, in accordance with a determination that, based on depth data, the first part of the virtual avatar does not satisfy the set of depth-based display criteria for the first part of the subject, the representation of the first part of the virtual avatar and the representation of the first part of the subject in the area that would have been occupied by the first part of the virtual avatar.

[0029] The following describes an electronic device. The electronic device includes a camera, a display device, one or more processors, and a memory for storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to display, via the display device, a representation of image data captured via one or more cameras, the representation including a representation of a subject, and the image data corresponding to depth data including depth data of the subject, and to display, via the display device, a representation of a virtual avatar which is displayed in place of at least a portion of the representation of the subject, the virtual avatar which is positioned at a simulated depth relative to the representation of the subject determined based on the depth data of the subject, and the display of the representation of the virtual avatar which, based on the depth data, a first portion of the virtual avatar is depth-based The display includes, in accordance with a determination that satisfies a set of depth-based display criteria, which includes the requirement that depth data of a subject indicates that a first part of a virtual avatar has a simulated depth in front of a corresponding first part of a subject, including, as part of the representation of the virtual avatar, a representation of the first part of the virtual avatar displayed in place of the first part of the subject, and, in accordance with a determination that, based on depth data, the first part of the virtual avatar does not satisfy the set of depth-based display criteria for the first part of the subject, and, displaying the first part of the subject in the area that would have been occupied by the first part of the virtual avatar.

[0030] An electronic device will be described. The electronic device includes a camera, a display device, and means for displaying, via the display device, a representation of image data captured via one or more cameras, where the representation includes a representation of a subject and the image data corresponds to depth data that includes depth data of the subject, and means for displaying, via the display device, a representation of a virtual avatar that is displayed in place of at least a portion of the representation of the subject, where the virtual avatar is disposed at a simulated depth determined based on the depth data of the subject. Displaying the representation of the virtual avatar includes, based on the depth data, including, as part of the representation of the virtual avatar, a representation of a first portion of the virtual avatar that has a simulated depth that is a depth-based display criterion and that is in front of a corresponding first portion of the subject, as indicated by the depth data of the subject, in accordance with a determination that a set of depth-based display criteria is satisfied, and excluding, from the representation of the virtual avatar, a representation of a first portion of the virtual avatar, and displaying a first portion of the subject in an area that would have been occupied by the first portion of the virtual avatar, in accordance with a determination that the first portion of the virtual avatar does not satisfy a set of depth-based display criteria for the first portion of the subject.

[0031] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. The executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0032] Therefore, devices will be provided with faster and more efficient methods and interfaces for displaying visual effects, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can complement or replace other methods for displaying visual effects.

[0033] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings. Here, similar reference numerals refer to the corresponding parts throughout those drawings. [Brief explanation of the drawing]

[0034] [Figure 1A] This is a block diagram showing a portable multifunctional device equipped with a touch-sensitive display, according to one embodiment.

[0035] [Figure 1B] This is a block diagram showing exemplary components for event processing according to some embodiments.

[0036] [Figure 2] A portable multifunctional device having a touchscreen, according to one embodiment, is shown.

[0037] [Figure 3] This is a block diagram of an exemplary multifunctional device comprising a display and a touch-sensitive surface, according to one embodiment.

[0038] [Figure 4A] This figure shows an exemplary user interface for an application menu on a portable multifunction device, according to one embodiment.

[0039] [Figure 4B] This figure shows an exemplary user interface for a multifunctional device having a touch-sensitive surface separated from a display, according to one embodiment.

[0040] [Figure 5A] This figure shows a personal electronic device according to one embodiment.

[0041] [Figure 5B] This is a block diagram of a personal electronic device according to one embodiment.

[0042] [Figure 6A] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6B] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6C] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6D] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6E] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6F] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6G] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6H] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6I] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6J] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6K] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6L] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6M] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6N] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6O] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6P] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6Q] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6R] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6S] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6T] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6U]This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6V] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6W] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6X] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6Y] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6Z] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AA] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AB] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AC] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AD] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AE] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AF] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AG] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AH] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AI] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AJ] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AK] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AL] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AM] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AN] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AO] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AP] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AQ] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AR]This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AS] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AT] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AU] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AV] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AW] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AX] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AY] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6AZ] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BA] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BB] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BC] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BD] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BE] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BF] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BG] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BH] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BI] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BJ] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BK] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BL] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BM] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BN] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BO]This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BP] This figure shows an exemplary user interface for displaying visual effects in a messaging application. [Figure 6BQ] This figure shows an exemplary user interface for displaying visual effects in a messaging application.

[0043] [Figure 7A] This flowchart illustrates a method for displaying visual effects in a messaging application. It shows the process described below, including the processing shown in Figure 7A, using the user interface shown in Figures 6A to 6BQ. [Figure 7B] This flowchart illustrates a method for displaying visual effects in a messaging application. It shows the process described below, including the processing shown in Figure 7B, using the user interface shown in Figures 6A to 6BQ.

[0044] [Figure 8A] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8B] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8C] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8D] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8E] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8F]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8G] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8H] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8I] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8J] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8K] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8L] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8M] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8N] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8O] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8P] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8Q] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8R]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8S] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8T] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8U] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8V] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8W] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8X] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8Y] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8Z] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AA] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AB] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AC] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AD]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AE] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AF] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AG] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AH] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AI] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AJ] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AK] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AL] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AM] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AN] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AO] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AP]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AQ] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AR] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AS] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AT] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AU] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AV] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AW] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AX] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AY] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8AZ] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BA] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BB]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BC] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BD] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BE] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BF] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BG] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BH] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BI] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BJ] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BK] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BL] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BM] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BN]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BO] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BP] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 8BQ] This figure shows an exemplary user interface for displaying visual effects in a camera application.

[0045] [Figure 9A] This is a flowchart illustrating how to display visual effects in a camera application. [Figure 9B] This is a flowchart illustrating how to display visual effects in a camera application.

[0046] [Figure 10A] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10B] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10C] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10D] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10E] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10F]This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10G] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10H] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10I] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10J] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10K] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10L] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10M] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10N] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10O] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10P] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10Q] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10R]This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10S] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10T] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10U] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10V] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10W] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10X] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10Y] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10Z] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AA] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AB] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AC] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AD]This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AE] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AF] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AG] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AH] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AI] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AJ] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AK] This figure shows an example user interface for displaying visual effects in media item viewing mode. [Figure 10AL] This figure shows an example user interface for displaying visual effects in media item viewing mode.

[0047] [Figure 11A] This flowchart shows how to display visual effects in media item viewing mode. [Figure 11B] This flowchart shows how to display visual effects in media item viewing mode.

[0048] [Figure 12A]This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12B] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12C] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12D] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12E] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12F] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12G] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12H] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12I] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12J] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12K] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12L] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12M]This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12N] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12O] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12P] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12Q] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12R] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12S] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12T] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12U] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12V] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12W] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12X] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12Y]This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12Z] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AA] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AB] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AC] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AD] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AE] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AF] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AG] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AH] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AI] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AJ] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AK]This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AL] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AM] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AN] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AO] This figure shows an exemplary user interface for displaying visual effects in a live video communication session. [Figure 12AP] This figure shows an exemplary user interface for displaying visual effects in a live video communication session.

[0049] [Figure 13A] This flowchart illustrates how to display visual effects during a live video communication session. [Figure 13B] This flowchart illustrates how to display visual effects during a live video communication session.

[0050] [Figure 14A] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14B] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14C] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14D]This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14E] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14F] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14G] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14H] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14I] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14J] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14K] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14L] This figure shows an exemplary user interface for displaying visual effects in a camera application. [Figure 14M] This figure shows an exemplary user interface for displaying visual effects in a camera application.

[0051] [Figure 15A] This is a flowchart illustrating how to display visual effects in a camera application. [Figure 15B] This is a flowchart illustrating how to display visual effects in a camera application. [Modes for carrying out the invention]

[0052] The following description includes exemplary methods, parameters, etc. However, it should be understood that the purpose of such description is not to limit the scope of this disclosure, but to provide an example of the embodiments.

[0053] There is a need for electronic devices that provide efficient methods and interfaces for displaying visual effects. For example, while programs for displaying visual effects already exist, these programs are inefficient and difficult to use compared to the following technologies that allow users to display visual effects in various applications. Such technologies can reduce the cognitive burden on users displaying visual effects in applications, thereby improving productivity. Furthermore, such technologies can reduce the power consumption of the processor and battery that would normally be wasted on redundant user input.

[0054] The following Figures 1A-1B, 2, 3, 4A-4B, and 5A-5B provide a description of an exemplary device for performing technology to manage event notifications.

[0055] Figures 6A to 6BQ show exemplary user interfaces for displaying visual effects in a messaging application. Figures 7A and 7B are flowcharts illustrating a method for displaying visual effects in a messaging application according to one embodiment. The following describes the process, including the processing shown in Figures 7A and 7B, using the user interfaces of Figures 6A to 6BQ.

[0056] Figures 8A to 8BQ show exemplary user interfaces for displaying visual effects in a camera application. Figures 9A and 9B are flowcharts illustrating a method for displaying visual effects in a camera application according to some embodiments. The following describes the process, including the processing shown in Figures 9A and 9B, using the user interfaces of Figures 8A to 8BQ.

[0057] Figures 10A to 10AL show exemplary user interfaces for displaying visual effects in a camera application. Figures 11A and 11B are flowcharts illustrating a method for displaying visual effects in a media item viewing mode according to one embodiment. The following processes, including those shown in Figures 11A and 11B, are shown using the user interfaces of Figures 10A to 10AL.

[0058] Figures 12A to 12AP show exemplary user interfaces for displaying visual effects in a live video communication session. Figures 13A and 13B are flowcharts illustrating a method for displaying visual effects in a live video communication session according to one embodiment. The process described below, including the process in Figures 13A and 13B, is illustrated using the user interfaces of Figures 12A to 12AP.

[0059] Figures 14A to 14M show exemplary user interfaces for displaying visual effects in a camera application. Figures 15A and 15B are flowcharts illustrating a method for displaying visual effects in a camera application according to one embodiment. The process described below, including the process in Figures 15A and 15B, is illustrated using the user interfaces of Figures 14A to 14M.

[0060] In the following description, terms such as “first,” “second,” etc., are used to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the various embodiments described, a first touch may be referred to as a second touch, and similarly, a second touch may be referred to as a first touch. Both the first touch and the second touch are touches, but they are not the same touch.

[0061] The terms used in the descriptions of the various embodiments described herein are intended solely to describe specific embodiments and not to limit them. As used in the descriptions of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural form unless the context explicitly indicates otherwise. As used herein, the terms “and / or” should also be understood to refer to and include all possible combinations of one or more of the enumerated items relating to the description. It will be further understood that, as used herein, the terms “includes,” “including,” “comprises,” and / or “comprising,” when used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0062] The term "if" can be interpreted, at will, depending on the context, as meaning "when," "upon," "in response to determining," or "in response to detecting." Similarly, the phrases "if it is determined" or "if (a stated condition or event) is detected" can be interpreted, at will, depending on the context, as meaning "upon determining," "in response to determining," "upon detecting (the stated condition or event)," or "in response to detecting (the stated condition or event)."

[0063] Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices of Apple Inc., Cupertino, California. Other portable electronic devices, such as laptop or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads), are also used optionally. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0064] The following discussion describes electronic devices including displays and touch-sensitive surfaces. However, please understand that electronic devices may optionally include one or more other physical user interface devices, such as physical keyboards, mice, and / or joysticks.

[0065] This device generally supports a variety of applications, including drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0066] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, can be optionally adjusted and / or modified on an application-by-application basis and / or within each application. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications with intuitive and transparent user interfaces for the user.

[0067] Now, we turn our attention to embodiments of portable devices having a touch-sensitive display. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112, according to some embodiments. The touch-sensitive display 112 may be referred to as a “touchscreen” for convenience, and may be known or referred to as a “touch-sensitive display system.” Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on Device 100 (e.g., a touch-sensitive surface such as the touch-sensitive display system 112 of Device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on Device 100 (for example, generating tactile output on a touch-sensitive surface such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0068] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact (e.g., finger contact) on the touch-sensitive surface, or a substitute (substitute) for the force or pressure of contact on the touch-sensitive surface. The strength of contact includes at least four distinct values ​​and, more typically, has a range of values ​​including hundreds (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine an estimate of the force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensing surface, the capacitance and / or change in the touch-sensing surface adjacent to the contact, and / or the resistance and / or change in the touch-sensing surface adjacent to the contact may be optionally used as a substitute for the force or pressure of the contact on the touch-sensing surface. In some implementations, the substitute measurement for the force or pressure of the contact is used directly to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of contact as an attribute of user input, it becomes possible to enable user access to additional device functions that may not be otherwise accessible to the user on a device with limited area and reduced size, for displaying affordances (e.g., on a touch-sensitive display) and / or for receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control unit such as a knob or button).

[0069] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive user’s surface (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may optionally be interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. In another embodiment, movement of a touch-sensitive surface may be optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. Such interpretations of touch by the user depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.

[0070] Device 100 is merely one embodiment of a portable multifunction device, and it should be understood that Device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of components. The various components shown in Figure 1A may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0071] Memory 102 optionally includes high-speed random-access memory and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0072] The peripheral interface 118 allows the input and output peripherals of this device to be connected to the CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and to process data. In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0073] The RF (radio requency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals, or electromagnetic signals to electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes, but is not limited to, well-known circuits for performing these functions, including, but not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identification module (SIM) card, and memory. The RF circuit 108 optionally communicates wirelessly with other devices with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). The RF circuit 108 optionally includes well-known circuits for detecting near-field communication (NFC) fields, such as by near-field communication radio. Wireless communication uses one of several communication standards, protocols, and technologies at will. These communication standards, protocols, and technologies include Global System for Mobile Communications (GSM®), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), near-field communication (NFC), and wideband code division multiple access.Bluetooth®, Bluetooth® Low Energy® (BTLE), Wireless Fidelity® (Wi-Fi®)® (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant Messaging (e.g., Extensible Messaging and Presence Protocol (XMPP)), Session Initiation Protocol for Instant Messaging and Presence Leveraging Examples of suitable communication protocols include, but are not limited to, Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS), or any other communication protocols not yet developed as of the filing date of this document.

[0074] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved by the peripheral interface 118 from memory 102 and / or RF circuit 108 and / or transmitted to memory 102 and / or RF circuit 108. In some embodiments, the audio circuit 110 further includes a headset jack (e.g., 212 in Figure 2). The headset jack provides an interface between the audio circuit 110 and a detachable audio input / output peripheral, such as output-only headphones or a headset having both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0075] The I / O subsystem 106 connects the input / output peripherals of device 100, such as the touchscreen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive / transmit electrical signals to / from the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally connected to (or not connected to) any of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in Figure 2) optionally include up / down buttons for adjusting the volume of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206 in Figure 2).

[0076] As described in U.S. Patent Application No. 11 / 322,549, “Unlocking a Device by Performing Gestures on an Unlock Image,” filed December 23, 2005, U.S. Patent No. 7,657,849 (this patent is incorporated herein by reference in its entirety), the touchscreen 112 may be optionally unlocked by a quick press of a push button, or the process of using gestures on the touchscreen to unlock the device may be optionally initiated. The device 100 may be optionally powered on or off by a longer press of a push button (e.g., 206). One or more functions of the buttons may be optionally user-customizable. The touchscreen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0077] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives and / or transmits electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as “graphics”). In some embodiments, optionally, some or all of the visual output corresponds to user interface objects.

[0078] The touchscreen 112 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on the touchscreen 112 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 112 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the contact point between the touchscreen 112 and the user corresponds to the user's finger.

[0079] The touchscreen 112 optionally uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies are used in other embodiments. The touchscreen 112 and display controller 156 optionally detect touch and any movement or interruption using any of several currently known or future-developed touch sensing technologies, including but not limited to capacitive technology, resistive technology, infrared technology, and surface ultrasonic technology, as well as other proximity sensor arrays or other elements for determining one or more contact points with the touchscreen 112. In exemplary embodiments, projected mutual capacitive sensing technology is used, as seen in the iPhone® and iPod Touch® by Apple Inc. of Cupertino, California.

[0080] The touch-sensitive displays in some embodiments of the touchscreen 112 are optionally similar to the multi-touch-sensitive touchpads described in the following U.S. Patents: No. 6,323,846 (Westerman et al.), No. 6,570,557 (Westerman et al.), and / or No. 6,677,932 (Westerman), and / or U.S. Patent Application Publication 2002 / 0015024(A1). These documents are each incorporated herein by reference in their entirety. However, the touchscreen 112 displays visual output from device 100, whereas the touch-sensitive touchpad does not provide visual output.

[0081] Touch-sensitive displays in some embodiments of the touchscreen 112 are described in the following applications. (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller", (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen", (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices", (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices", (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices", (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User (7) "Operation Of A Computer With A Touch Screen Interface," filed September 16, 2005, No. 11 / 228,700, and (8) "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed September 16, 2005, No. 11 / 228,737, and (9) "Multi-Functional Hand-Held Device," filed March 3, 2006. All of these applications are incorporated herein by reference in their entirety.

[0082] The touchscreen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally touches the touchscreen 112 using any suitable object or accessory, such as a stylus or finger. In some embodiments, the user interface is designed to work primarily with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform user-desired actions.

[0083] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0084] Device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within the portable device.

[0085] Device 100 also optionally includes one or more optical sensors 164. Figure 1A shows optical sensors connected to an optical sensor controller 158 in the I / O subsystem 106. The optical sensors 164 optionally include charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) phototransistors. The optical sensors 164 receive light from the environment projected through one or more lenses and convert that light into data that represents an image. In use with an imaging module 143 (also referred to as a camera module), the optical sensors 164 optionally capture still images or video. In some embodiments, the optical sensors are located on the back of Device 100, opposite the touchscreen display 112 on the front of the device, to enable the touchscreen display as a viewfinder for acquiring still and / or video images. In some embodiments, the optical sensors are located on the front of the device, optionally so that the user can view other video conference participants on the touchscreen display while simultaneously obtaining an image of the user for video conferencing. In some embodiments, the position of the optical sensor 164 can be changed by the user (for example, by rotating the lens and sensor within the device housing), thereby allowing a single optical sensor 164 to be used with a touchscreen display for both video conferencing and the acquisition of still and / or video images.

[0086] Device 100 also optionally includes one or more depth camera sensors 175. Figure 1A shows a depth camera sensor coupled to a camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data from the environment to generate a three-dimensional model of an object in the scene (e.g., a face) from a viewpoint (e.g., the depth camera sensor). In some embodiments, the depth camera sensor 175 is optionally used in conjunction with an imaging module 143 (also called a camera module) to determine the depth map of different parts of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located on the front of Device 100, and an image of the user with depth information can be optionally acquired for video conferencing so that the user can view other video conference participants on a touchscreen display and capture a selfie using the depth map data. In some embodiments, the depth camera sensor 175 is located on the back of the device, or on both the back and front of Device 100. In some embodiments, the user can change the position of the depth camera 175 (for example, by rotating the lens and sensor within the device housing), and the depth camera sensor 175 can be used in conjunction with a touchscreen display for both video conferencing and the acquisition of still and / or video images.

[0087] In some embodiments, the depth map (e.g., depth map image) includes information (e.g., values) about the distance of objects in the scene from a viewpoint (e.g., camera, light sensor, depth camera sensor). In one embodiment of the depth map, each depth pixel defines the position on the Z-axis of the viewpoint where its corresponding 2D pixel is located. In some embodiments, the depth map consists of pixels, each defined by a value (e.g., 0 to 255). For example, a value of "0" represents the furthest pixel in the "3D" scene, and a value of "255" represents the pixel closest to the viewpoint (e.g., camera, light sensor, depth camera sensor) in the "3D" scene. In other embodiments, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depth of various features of the object of interest at the viewpoint of the depth camera (e.g., the relative depth of the eyes, nose, mouth, and ears of the user's face). In some embodiments, the depth map includes information that allows the device to determine the contour of the object of interest in the z-direction.

[0088] Device 100 also optionally includes one or more contact strength sensors 165. Figure 1A shows a contact strength sensor coupled to a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other strength sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the surrounding environment. In some embodiments, at least one contact strength sensor is located on or near the touch-sensing surface (e.g., touch-sensing display system 112). In some embodiments, at least one contact strength sensor is located on the back of Device 100, opposite the touchscreen display 112 located on the front of Device 100.

[0089] Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is optionally connected to an input controller 160 in the I / O subsystem 106. The proximity sensor 166 may optionally perform actions as described in U.S. Patent Application No. 11 / 241,839, “Proximity Detector In Handheld Device”, No. 11 / 240,788, “Proximity Detector In Handheld Device”, No. 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output”, No. 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices”, and No. 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals”, all of which are incorporated herein by reference. In some embodiments, if the multifunction device is positioned near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 112.

[0090] Device 100 also optionally includes one or more tactile output generators 167. Figure 1A shows a tactile output generator connected to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact intensity sensor 165 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on device 100 that can be sensed by the user of device 100. In some embodiments, at least one tactile output generator is located on or near a touch-sensitive surface (e.g., a touch-sensitive display system 112) and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward from the surface of device 100) or laterally (e.g., forward / backward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite the touchscreen display 112 located on the front of device 100.

[0091] Device 100 also optionally includes one or more accelerometers 168. Figure 1A shows an accelerometer 168 coupled to a peripheral interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally functions as described in U.S. Patent Application Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Application Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. In addition to one or more accelerometers 168, device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information regarding the position and orientation of device 100 (e.g., vertical or horizontal).

[0092] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, as shown in Figures 1A and 3, memory 102 (Figure 1A) or memory 370 (Figure 3) stores device / global internal state 157. The device / global internal state 157 includes one or more of the following: active application state, indicating which application is active, if there is an application currently active; display state, indicating which applications, views, or other information occupy various areas of the touchscreen display 112; sensor state, including information obtained from various sensors and input control devices 116 of the device; and position information relating to the location and / or orientation of the device.

[0093] An operating system 126 (for example, an embedded operating system such as Darwin®, RTXC®, LINUX®, UNIX®, OS X®, iOS®, WINDOWS®, or VxWorks®) includes various software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0094] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire®, etc.) are adapted to connect to other devices directly or indirectly through a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are the same as, or similar to, the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices, and / or compatible multi-pin (e.g., 30-pin) connectors.

[0095] The contact / motion module 130 (in cooperation with the display controller 156) optionally detects contact with the touchscreen 112 and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger being lowered), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of a finger being dragged), and determining whether contact has stopped (e.g., detecting a finger being lifted or an interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by a series of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0096] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by the user (for example, whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (for example, the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse "click" threshold for a trackpad or touchscreen display may be set to one of a range of predefined thresholds without changing the hardware of the trackpad or touchscreen display. In addition, in some implementations, the user of the device is provided with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0097] The contact / motion module 130 selectively detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movement, timing, and / or intensity of the detected contact). Thus, gestures are detected selectively by detecting specific contact patterns. For example, detecting a finger tap gesture involves detecting a finger-down event (e.g., at the icon's position) followed by a finger-lift-off event at the same (or substantially the same) location as the finger-down event. In another embodiment, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger-down event followed by one or more finger-drag events followed by a finger-lift-off event.

[0098] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for modifying the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). As used herein, the term “graphics” includes, but is not limited to, any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, and animations.

[0099] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like, as needed, specifying the graphics to be displayed along with coordinate data and other graphic characteristic data, and then generates screen image data to be output to the display controller 156.

[0100] The haptic feedback module 133 includes various software components that generate commands used by a haptic output generator(s) 167 to generate haptic outputs at one or more locations on the device 100 in response to user interaction with the device 100.

[0101] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input).

[0102] The GPS module 135 determines the device's location and provides this information for use in various applications (for example, to the phone 138 for use in location-based phone calls, to the camera 143 as photo / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0103] Application 136 optionally includes the following modules (or instruction sets), or subsets or supersets thereof: ● Contact module 137 (sometimes also called the address book or contact list), ●Telephone module 138, ●Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ●Training support module 142, ● Camera module 143 for still images and / or video images, ●Image management module 144, ●Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, ● A widget module 149 that optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5 and other widgets obtained by the user, as well as user-created widgets 149-6. ●Widget creator module 150 for creating user-created widget 149-6, ●Search module 151, ● A video and music player module 152 that integrates a video player module and a music player module. ●Memo Module 153, ●Map module 154 and / or ● Online video module 155.

[0104] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java®-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0105] Together with the touchscreen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (stored, for example, in the application internal state 192 of the contact module 137 in memory 102 or memory 370), which includes adding names(s) to the address book, removing names(s) from the address book, associating telephone numbers(s) to names, email addresses(s) to names, addresses(s) to names, or other information, associating images to names, categorizing and sorting names, providing telephone numbers or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing module 139, email 140, or IM 141, and so on.

[0106] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the telephone module 138 is used to optionally input a series of characters corresponding to a telephone number, access one or more telephone numbers in the contact module 137, modify the entered telephone number, dial the corresponding telephone number, conduct a conversation, and disconnect or hang up the phone when the conversation is complete. As described above, wireless communication optionally uses one of several communication standards, protocols, and technologies.

[0107] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, the video conferencing module 139 includes executable commands for initiating, conducting, and ending a video conference between the user and one or more other participants, according to user instructions.

[0108] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the email client module 140 includes executable commands for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails containing still or video images captured by the camera module 143.

[0109] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the instant messaging module 141 includes executable commands for entering strings corresponding to instant messages, modifying entered characters, sending corresponding instant messages (for example, using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for telephone-based instant messaging, or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and internet-based messaging (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0110] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, the training support module 142 communicates with training sensors (sports devices) to create training (e.g., with time, distance, and / or calorie consumption targets), receives training sensor data, calibrates sensors used to monitor training, selects and plays music for training, and displays, stores, and transmits training data.

[0111] In conjunction with the touchscreen 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0112] In conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable commands for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still and / or video images.

[0113] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the browser module 147 includes executable commands for browsing the Internet in accordance with user instructions, including searching for, linking, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0114] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, the calendar module 148 includes executable commands that, in accordance with user instructions, create, display, modify, and store calendars and data associated with the calendars (e.g., calendar items, to-do lists, etc.).

[0115] Together with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, the widget module 149 is a mini-application that can be optionally downloaded and used by the user (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or a mini-application created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets®).

[0116] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, the widget creator module 150 is used by the user to optionally create widgets (for example, to convert a user-specified area of ​​a web page into a widget).

[0117] In conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching for text, music, sound, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0118] In conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, the video and music player module 152 includes executable commands that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands for displaying, presenting, or otherwise playing video (on the touchscreen 112 or on an external display connected via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0119] In conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the memo module 153 includes executable commands for creating and managing memos, to-do lists, etc., according to user instructions.

[0120] In conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data on shops and other points of interest at a specific location or nearby, and other location-based data) in accordance with user instructions.

[0121] In conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions that enable the user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen or on an external display connected via external port 124), send emails containing links to specific online videos, and otherwise manage them. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of online video applications can be found in U.S. Patent Provisional Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed on 20 June 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed on 31 December 2007, the contents of which are incorporated herein by reference in their entirety.

[0122] Each of the modules and applications identified above corresponds to an executable set of instructions that perform one or more of the above functions and to the methods described in this application (e.g., methods implemented by a computer and other information processing methods described herein). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, optionally, various subsets of these modules may be combined or otherwise reorganized. For example, the video player module may optionally be combined with the music player module to form a single module (e.g., the video and music player module 152 of FIG. 1A). In some embodiments, the memory 102 optionally stores a subset of the modules and data structures identified above. Further, the memory 102 optionally stores additional modules and data structures not described above.

[0123] In some embodiments, the device 100 is a device in which the operation of a predefined set of functions on the device is exclusively performed through a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the main input control device for the operation of the device 100, the number of physical input control devices (push buttons, dials, etc.) on the device 100 is optionally reduced.

[0124] The set of predefined functions that are exclusively performed through the touch screen and / or the touch pad optionally includes navigation between user interfaces. In some embodiments, when touched by a user, the touch pad navigates the device 100 from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, the "menu button" is implemented using the touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of the touch pad.

[0125] FIG. 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or memory 370 (FIG. 3) includes an event sorter 170 (e.g., within operating system 126) and a corresponding application 136-1 (e.g., any one of the applications 137-151, 155, 380-390 described above).

[0126] The event sorting unit 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. The event sorting unit 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 indicating the current application view(s) displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, the device / global internal state 157 is used by the event sorter 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 to which the event information is to be delivered.

[0127] In some embodiments, the application internal state 192 includes additional information such as one or more of resume information used when the application 136-1 resumes execution, user interface state information indicating information that is being or is ready to be displayed by the application 136-1, a state queue for waiting to allow the user to return to the previous state or view of the application 136-1, and a redo / undo queue for the previous actions performed by the user.

[0128] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, user touch on the touch-sensitive display 112 as part of a multi-touch gesture). The peripheral interface 118 transmits information received from the I / O subsystem 106, or from sensors such as the proximity sensor 166, one or more accelerometers 168, and / or the microphone 113 (through the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0129] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when a significant event occurs (e.g., receiving an input that exceeds a predetermined noise threshold and / or for a longer period than predetermined).

[0130] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0131] The hit view determination module 172 provides a software procedure for determining where in one or more views a sub-event occurred when the touch-sensitive display 112 displays two or more views. A view consists of a control unit and other elements that the user can see on the display.

[0132] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of each application) in which a touch is detected optionally corresponds to a program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0133] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest-level view where the first sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module 172, the hit view generally receives all sub-events related to the same touch or input source that identified it as a hit view.

[0134] The active event recognition determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 173 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to a region associated with a particular view, higher-level views in the hierarchy still remain actively involved views.

[0135] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., an event recognition unit 180). In embodiments including an active event recognition unit determination module 173, the event dispatcher module 174 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information acquired by the corresponding event receiving unit 182 in an event queue.

[0136] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, application 136-1 includes an event sorter 170. In yet other embodiments, the event sorter 170 is either a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0137] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each containing instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Generally, a corresponding application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event sorter 170. The event handler 190 optionally uses or calls the data updater 176, object updater 177, or GUI updater 178 to update the application's internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. In some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0138] The corresponding event recognition unit 180 receives event information (e.g., event data 179) from the event sorter 170 and identifies events from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution commands 188 (optionally including sub-event distribution commands).

[0139] The event receiver 182 receives event information from the event sorter 170. The event information includes information about sub-events, such as touches or touch motions. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch motion, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, an event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the device's current orientation (also called the device's orientation).

[0140] The event comparison unit 184 compares event information with a predefined event or sub-event definition, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes, for example, definitions of events such as event 1 (187-1), event 2 (187-2) (for example, a predefined series of sub-events). In some embodiments, the sub-events in event 187 include, for example, the start of a touch, the end of a touch, movement of a touch, stopping of a touch, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object during a predetermined stage (start of touch), a first lift-off during a predetermined stage (end of touch), a second touch on the displayed object during a predetermined stage (start of touch), and a second lift-off during a predetermined stage (end of touch). In another embodiment, the definition of event 2(187-2) is a drag operation on a displayed object. This drag operation includes, for example, touching (or contacting) the displayed object during a predetermined stage, moving the touch across the touch-sensitive display 112, and lifting off the touch (ending the touch). In some embodiments, the event also includes information about one or more associated event handlers 190.

[0141] In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparison unit uses the results of the hit test to determine which event handler 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event handler associated with the object that triggers the hit test.

[0142] In some embodiments, the definition of a corresponding event (187) also includes a delay action that delays the delivery of event information until it is determined whether a series of sub-events correspond to an event type in the event recognition unit.

[0143] When the corresponding event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, the corresponding event recognition unit 180 enters an event impossible, event failed, or event terminated state, and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are any other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0144] In some embodiments, the corresponding event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how or are made capable of interacting with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0145] In some embodiments, each event recognition unit 180 activates an event handler 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, each event recognition unit 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and delaying the sending of) sub-events to the respective hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event handler 190 associated with the flag captures the flag and performs a predefined process.

[0146] In some embodiments, the event distribution command 188 includes a sub-event distribution command that distributes event information about a sub-event without activating an event handler. Instead, the sub-event distribution command distributes event information to an event handler associated with a set of sub-events, or to a view that is actively involved. The event handler associated with the set of sub-events or the view that is actively involved receives the event information and performs a predefined process.

[0147] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates the phone numbers used in contact module 137 or stores video files used in the video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphic module 132 for display on the touch-sensitive display.

[0148] In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of their respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0149] The preceding discussion regarding the event processing of user touches on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 with input devices, but it should be understood that not all of this begins on a touchscreen. For example, mouse movement and mouse button presses optionally linked to pressing or holding one or more keyboards, touch movements such as tapping, dragging, and scrolling on a touchpad, pen stylus input, device movement, verbal instructions, detected eye movements, biometric input, and / or any combination thereof may optionally be used as inputs corresponding to sub-events that define the events to be recognized.

[0150] Figure 2 shows a portable multifunctional device 100 having a touchscreen 112 according to one embodiment. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, and in embodiments described later, the user can select one or more graphics by making gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, left to right, upward and / or downward). In some implementations or situations, unintended contact with a graphic does not result in the selection of a graphic. For example, when the gesture corresponding to selection is a tap, a swipe gesture (swiping over an application icon) is an optional selection and does not select the corresponding application.

[0151] Device 100 also optionally includes one or more physical buttons, such as a "Home" or menu button 204. As previously stated, the menu button 204 is optionally used on device 100 to navigate to any application 136 within the set of applications to be executed. Alternatively, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on the touchscreen 112.

[0152] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, one or more volume buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. The push button 206 is optionally used to turn the device on / off by pressing and holding the button down for a predetermined time, to lock the device by pressing and releasing the button before the predetermined time has elapsed, and / or to unlock the device or initiate an unlocking process. In alternative embodiments, device 100 also receives verbal input through a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more tactile output generators 167 for generating tactile output to the user of device 100.

[0153] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia playback device, navigation device, educational device (such as a children's learning toy), game system, or control device (e.g., a home or commercial controller). Device 300 generally includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication buses 320 optionally include circuits (sometimes called chipsets) that interconnect and control communication between system components. Device 300 generally includes an input / output (I / O) interface 330, which includes a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 (similar to the tactile output generator(s) 167 described above with reference to Figure 1A) for generating tactile output on device 300, and a sensor 359 (e.g., an optical sensor, an accelerometer, a proximity sensor, a touch sensor and / or a contact intensity sensor similar to the contact intensity sensor(s) 165 described above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices, and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, the programs, modules, and data structures stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while memory 102 of the portable multifunction device 100 (Figure 1A) does not optionally store these modules.

[0154] Each of the elements of Figure 3 identified above is optionally stored in one or more of the aforementioned memory devices. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) identified above do not necessarily have to be implemented as separate software programs, procedures, or modules; therefore, in various embodiments, various subsets of these modules are combined or otherwise reorganized. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0155] Here, we focus on an embodiment of a user interface that is optionally implemented on a portable multi-functional device 100, for example.

[0156] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to one embodiment. A similar user interface may be optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular and Wi-Fi signals 402, ●Time 404, ●Bluetooth® indicator 405, ●Battery status indicator 406, ●Tray 408 containing icons for frequently used applications such as the following: ○Optionally including an indicator 414 for the number of missed calls or voicemail messages, an icon 416 for the telephone module 138 labeled "Telephone", ○Optionally including an indicator 410 for the number of unread emails, an icon 418 labeled "Mail" for the email client module 140, ○ Icon 420 for browser module 147, labeled as "Browser", and ○ Icon 422 for the video and music player module 152, also known as iPod® (registered trademark) (trademark of Apple Inc.) module 152, which is labeled as "iPod", and ● Icons for other applications, such as the ones listed below. ○ Icon 424 for IM module 141 labeled "Message", ○ Icon 426 for calendar module 148, labeled "Calendar", ○ Icon 428 for image management module 144, labeled as "Photo" ○ Icon 430 for camera module 143, labeled "Camera" ○ Icon 432 for online video module 155, labeled "online video" ○ Icon 434 for stock price widget 149-2, labeled "Stock Price" ○ Icon 436 for map module 154, labeled as "Map" ○ Icon 438 for weather widget 149-1, labeled "Weather" ○ Icon 440 for alarm clock widget 149-4, labeled as "Clock" ○ Icon 442 for training support module 142, labeled "Training Support" ○ Icon 444 for memo module 153, which is labeled as "Memo", and ○ An icon 446 for a settings application or module, labeled "Settings," which provides access to settings for device 100 and its various applications 136.

[0157] Note that the icon labels shown in Figure 4A are for illustrative purposes only. For example, the icon 422 for the video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application to which that application icon corresponds. In some embodiments, the label for a particular application icon is different from the name of the application to which that particular application icon corresponds.

[0158] Figure 4B shows an exemplary user interface on a device (e.g., device 300 in Figure 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355 in Figure 3) separated from a display 450 (e.g., touchscreen display 112). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contact on the touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for the user of device 300.

[0159] Some of the following embodiments will be described with reference to input on a touchscreen display 112 (when the touch-sensing surface and the display are combined), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 4B. In some embodiments, this touch-sensing surface (e.g., 451 in Figure 4B) has a principal axis (e.g., 452 in Figure 4B) corresponding to a principal axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with the touch-sensing surface 451 (e.g., 460 and 462 in Figure 4B) at positions corresponding to their respective positions on the display (e.g., in Figure 4B, 460 corresponds to 468 and 462 corresponds to 470). Thus, when the touch-sensitive surface is separate from the display, user input detected by the device on the touch-sensitive surface (e.g., 451 in Figure 4B) (e.g., touches 460 and 462, and their movement) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B). It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0160] Furthermore, while the following embodiments primarily refer to and describe finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced with a mouse click (e.g., instead of touch), and then the cursor moves along the swipe path (e.g., instead of touch movement). In another embodiment, a tap gesture may optionally be replaced with a mouse click while the cursor is positioned over the tap gesture (e.g., instead of touch detection, followed by cessation of touch detection). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or a mouse and finger touch may optionally be used simultaneously.

[0161] Figure 5A shows an exemplary personal electronic device 500. Device 500 includes a body 502. In some embodiments, device 500 may include some or all of the features described with respect to devices 100 and 300 (e.g., Figures 1A to 4B). In some embodiments, device 500 has a touch-sensitive display screen 504, hereafter referred to as touchscreen 504. Instead of, or in addition to, touchscreen 504, device 500 has a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally has one or more intensity sensors for detecting the intensity of the applied contact (e.g., touch). One or more intensity sensors on touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of device 500 may respond to touches based on the intensity of the touch, meaning that touches of different intensity may invoke different user interface behaviors on device 500.

[0162] Exemplary techniques for detecting and processing touch intensity can be found, for example, in the related applications, International Patent Application PCT / US2013 / 040061, filed 8 May 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application" (published as International Publication 2013 / 169849), and PCT / US2013 / 069483, filed 11 November 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships" (published as International Publication 2014 / 105276), each of which is incorporated herein by reference in its entirety.

[0163] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508, if included, may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, may allow the device 500 to be attached to, for example, a hat, eyewear, earrings, necklace, shirt, jacket, bracelet, watch band, chain, trousers, belt, shoes, wallet, backpack, etc. These attachment mechanisms may allow the device 500 to be worn by a user.

[0164] Figure 5B shows an exemplary personal electronic device 500. In some embodiments, the device 500 may include some or all of the components described with respect to Figures 1A, 1B, and 3. The device 500 has a bus 512 that operably connects an I / O unit 514 to one or more computer processors 516 and memory 518. The I / O unit 514 may be connected to a display 504, which may have a touch-sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). Furthermore, the I / O unit 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth®, Near Field Communication (NFC), cellular, and / or other wireless communication technologies. The device 500 may include input mechanisms 506 and / or input mechanisms 508. The input mechanism 506 may optionally be, for example, a rotatable input device or a pressable and rotatable input device. The input mechanism 508 is optionally a button in some embodiments.

[0165] The input mechanism 508 is optionally a microphone in some embodiments. The personal electronic device 500 optionally includes various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or a combination thereof, all of which can be operably connected to the I / O unit 514.

[0166] The memory 518 of the personal electronic device 500 may include one or more non-temporary computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, can cause the computer processors to execute techniques described below, including processes 700, 900, 1100, 1300, and 1500 (Figures 7A-7B, 9A-9B, 11A-11B, 13A-13B, and 15A-15B). The computer-readable storage media can be any medium capable of tangibly accommodating or storing computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage medium is a temporary computer-readable storage medium. In some examples, the storage medium is a non-temporary computer-readable storage medium. Examples of non-temporary computer-readable storage media include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray® technology, and persistent solid-state memory such as flash and solid-state drives. The personal electronic device 500 is not limited to the components and configurations shown in Figure 5B, and may include other or additional components in multiple configurations.

[0167] As used herein, the term “affordance” refers to user-interactive graphic user interface objects that are optionally displayed on the display screens of devices 100, 300, and / or 500 (Figures 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0168] As used herein, the term “focus selector” refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations, including a cursor or other position marker, when input (e.g., press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in Figure 3, or touch-sensitive surface 451 in Figure 4B), and the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor functions as a “focus selector,” and the particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display (e.g., touch-sensitive display system 112 in Figure 1A, or touchscreen 112 in Figure 4A) that enables direct interaction with user interface elements on the touchscreen display, the contact detected on the touchscreen acts as a “focus selector,” so that when input (e.g., press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another (for example, by using the tab key or arrow keys to move focus from one button to another) without the movement of a corresponding cursor or touch on the touchscreen display, and in those implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is, in general, a user interface element (or touch on the touchscreen display) controlled by the user to communicate the user's intended interaction with the user interface (for example, by indicating to the device the user interface element that the user intends to interact with).For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., cursor, touch, or selection box) above the corresponding button indicates that the user is attempting to activate the corresponding button (rather than other user interface elements displayed on the device's display).

[0169] As used herein and in the claims, the term “characteristic intensity” of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predetermined number of intensity samples, or on a set of intensity samples collected over predetermined time periods (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) for predetermined events (e.g., after detection of contact, before detection of lift-off of contact, before or after detection of the start of movement of contact, before detection of the end of contact, before or after detection of an increase in contact intensity, and / or before or after detection of a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum intensity of the contact, the mean value of the contact intensity, the average value of the contact intensity, the top 10% value of the contact intensity, half the maximum intensity of the contact intensity, 90% of the maximum intensity of the contact intensity, etc. In some embodiments, the duration of contact is used to determine the characteristic intensity (for example, when the characteristic intensity is the average intensity of contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action was performed by a user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this embodiment, a first action is performed as a result of contact with a characteristic intensity that does not exceed the first threshold, a second action is performed as a result of contact with a characteristic intensity that exceeds the first intensity threshold but does not exceed the second intensity threshold, and a third action is performed as a result of contact with a characteristic intensity that exceeds the second threshold. In some embodiments, the comparison between the characteristic intensity and one or more thresholds is not used to determine whether to perform the first or second action, but rather to determine whether to perform one or more actions at all (for example, whether to perform each action or refrain from performing each action).

[0170] In some embodiments, a portion of the gesture is identified for the purpose of determining characteristic intensity. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position to an ending position, where the intensity of the contact increases. In this embodiment, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the continuous swipe contact (e.g., only the portion of the swipe contact at the ending position), rather than the entire continuous swipe contact. In some embodiments, optionally, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms remove small increases or decreases in the intensity of the swipe contact for the purpose of determining characteristic intensity.

[0171] The intensity of contact on the touch-sensitive surface is optionally characterized to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs an operation commonly associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device performs an operation different from an operation typically associated with clicking a physical mouse button or trackpad. In some embodiments, if contact is detected at a characteristic intensity below the light press intensity threshold (for example, above a slight contact detection intensity threshold below which contact is no longer detected), the device moves the focus selector in accordance with the movement of contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface diagrams.

[0172] An increase in the characteristic intensity of contact from an intensity below a light pressure intensity threshold to an intensity between the light and deep pressure intensity thresholds may be referred to as a "light pressure" input. An increase in the characteristic intensity of contact from an intensity below a deep pressure intensity threshold to an intensity above a deep pressure intensity threshold may be referred to as a "deep pressure" input. An increase in the characteristic intensity of contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and a light pressure intensity threshold may be referred to as detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above a contact detection intensity threshold to an intensity below a contact detection intensity threshold may be referred to detection of contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0173] In some embodiments described herein, one or more operations are performed in response to the detection of a gesture including each press input, or in response to the detection of each press input performed at each (or more) contact, each press input being detected at least in part on the detection of an increase in the intensity of the contact (or more) above a press input intensity threshold. In some embodiments, each operation is performed in response to the detection of an increase in the intensity of each contact (e.g., a "downstroke" of each press input) above a press input intensity threshold. In some embodiments, a press input includes an increase in the intensity of each contact above a press input intensity threshold and a decrease in the intensity of the contact below a subsequent press input intensity threshold, each operation is performed in response to the detection of a decrease in the intensity of each contact below a subsequent press input threshold (e.g., an "upstroke" of each press input).

[0174] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, which may be referred to as "jitter," and the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (for example, the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable ratio of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of each contact above the press input intensity threshold, and a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the subsequent press input intensity threshold, and each action is performed in response to the detection of a subsequent decrease in the intensity of each contact below that hysteresis intensity threshold (for example, an "upstroke" of each press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from an intensity below a hysteresis intensity threshold to an intensity above a press input intensity threshold, and optionally a decrease in subsequent contact intensity to an intensity below the hysteresis intensity, and each action is performed in response to the detection of that press input (e.g., depending on the situation, an increase in contact intensity or a decrease in contact intensity).

[0175] For the sake of clarity, the description of an action performed in response to a press input associated with a press input intensity threshold, or in response to a gesture involving a press input, is optionally triggered in response to the detection of any of the following: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below the hysteresis intensity threshold corresponding to the press input intensity threshold. In addition, in embodiments described as being performed in response to the detection of a decrease in contact intensity below the press input intensity threshold, the action is optionally performed in response to the detection of a decrease in contact intensity below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold.

[0176] Here, we focus on embodiments of user interfaces ("UI") and related processes implemented on electronic devices such as portable multifunction device 100, device 300, or device 500.

[0177] Figures 6A to 6BQ show exemplary user interfaces for displaying visual effects in a messaging application according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes shown in Figures 7A to 7B.

[0178] Figure 6A depicts a device 600 having a display 601, which in some cases is a touch-sensitive display. In some embodiments, the device 600 also includes a camera 602, which includes at least an image sensor capable of capturing data representing a portion of the light spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, the camera 602 includes multiple image sensors and / or other types of sensors. In some embodiments, in addition to capturing data representing detected light, the camera 602 may capture other types of data, such as depth data. For example, in some embodiments, the camera 602 also captures depth data using speckle, time-of-flight, parallax, or focus-based techniques. The image data that the device 600 captures using the camera 602 includes data corresponding to a portion of the light spectrum for a scene within the camera's field of view. In addition, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes enough data to determine or generate depth data for a portion of the light spectrum. In some embodiments, device 600 includes one or more features of devices 100, 300, or 500.

[0179] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermographic camera, or a combination thereof. In some examples, the device further includes a light-emitting device such as an IR flood light (e.g., an optical projector), a structured optical projector, or a combination thereof. The light-emitting device is optionally used to illuminate the subject while images are being captured by the visible light camera and the depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine a depth map of different parts of the subject captured by the visible light camera. In some embodiments, the depth map (e.g., a depth map image) includes information (e.g., values) about the distance of objects in the scene from the viewpoint (e.g., a camera). In one embodiment of the depth map, each depth pixel defines the position in the Z-axis of the viewpoint where its corresponding two-dimensional pixel is located. In some examples, the depth map consists of pixels, each defined by a value (e.g., 0 to 255). For example, a value of "0" represents the furthest pixel in a "3D" scene, and a value of "255" represents the pixel closest to the viewpoint (e.g., camera) in a "3D" scene. In other examples, the depth map represents the distance between objects in the scene and the viewpoint plane. In some embodiments, the depth map includes information about the relative depth of various features of an object of interest at the viewpoint of the depth camera (e.g., the relative depth of the eyes, nose, mouth, and ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the contour of an object of interest in the z direction. In some embodiments, the lighting effects described herein are displayed using parallax information from two cameras for the rear image (e.g., two visible light cameras) and depth information from a depth camera combined with image data from a visible light camera for the front image (e.g., a selfie image). In some embodiments, when depth information is determined using two visible light cameras and when depth information is determined using a depth camera, the same user interface is used to provide the user with a consistent experience, even when significantly different techniques are used to determine the information used when generating the lighting effects.In some embodiments, while displaying the camera user interface with one of the applicable lighting effects, the device detects a camera switching affordance selection and switches from the front camera (e.g., depth camera and visible light camera) to the rear camera (e.g., two spaced-apart visible light cameras) (e.g., two spaced-apart visible light cameras) (e.g., two spaced-apart visible light cameras) while displaying the camera user interface with one of the applicable lighting effects, while maintaining the display of user interface controls for applying the lighting effect and replacing the display of the front camera's field of view with the field of view of the rear camera (or vice versa).

[0180] In Figure 6A, device 600 displays a messaging user interface 603 for a messaging application. The messaging user interface 603 includes a message display area 604 containing a message 605 that is transmitted to a participant (represented by a recipient identifier 606) in a message conversation. The messaging user interface 603 also includes a message creation field 608 for displaying inputs (e.g., text input, multimedia input, etc.) to send to a participant in a message conversation. The messaging user interface 603 also includes a camera application affordance 609, an application dock affordance 610, a keyboard display area 612, and a text suggestion area 614.

[0181] In Figure 6B, device 600 detects input 616 (e.g., touch input on display 601) at a location corresponding to camera application affordance 609.

[0182] In Figure 6C, upon detecting input 616, device 600 launches the camera application associated with camera application affordance 609 and displays the camera application user interface 615. The camera application user interface 615 includes an image display area 620 that displays a representation of image data, such as streamed image data representing objects located within the field of view of a camera (e.g., a rear camera or camera 602) (e.g., a live camera preview, live camera recording, or live video communication session), or media items such as photographs or video recordings. In the embodiment shown in Figure 6C, the image display area 620 shows a live camera preview 620-1' from the rear camera of device 600.

[0183] The camera application user interface 615 also includes an area above the image display area 620, which includes camera-specific affordances 617 and a completion affordance 618 for exiting the camera application user interface 615. The camera-specific affordances include affordance 617-1 associated with the camera flash function, affordance 617-2 associated with the camera mode function, affordance 617-3 associated with the timer function, and affordance 617-4 associated with the filter function.

[0184] The camera application user interface 615 also includes a camera option area 625 located below the image display area 620. The camera option area 625 includes a camera selector affordance 627 for switching between cameras (e.g., the rear camera and camera 602), and camera option affordances 619 associated with different capture modes in which the camera can record image data. For example, video affordance 619-1 is associated with a function for activating the camera's video recording capture mode, and photo affordance 619-2 is associated with a function for activating the camera's still image capture mode. In the embodiments discussed below with respect to Figures 6C to 6AX, device 600 is in the still image capture mode for the operation associated with photo affordance 619-2. However, unless otherwise specified, these embodiments also apply to the video recording mode associated with video affordance 619-1.

[0185] The camera options area 625 further includes an effect affordance 622 for enabling or disabling a mode of device 600, which enables or disables device 600 in order to display visual effects in the image display area 620. This mode of device 600 is often referred to herein as an effect mode.

[0186] The camera options area 625 also includes a selectable capture affordance 621 for capturing image data represented in the image display area 620. In some embodiments, the device 600 captures the image data in a manner based on the currently enabled capture mode (e.g., video recording capture mode or image capture mode). In some embodiments, the device 600 captures the image data depending on the type of gesture detected on the capture affordance 621. For example, if the device 600 detects a tap gesture on the capture affordance 621, the device 600 captures a still image of the image data represented in the image display area 620 at the time the tap gesture occurred. If the device 600 detects a tap-and-hold gesture on the capture affordance 621, the device 600 captures a video recording of the image data represented in the image display area 620 for the duration of the tap-and-hold gesture. In some embodiments, the video recording stops when the finger is lifted from the affordance. In some embodiments, video recording continues until a subsequent input (e.g., a tap input) is detected at a position corresponding to an affordance. In some embodiments, the captured image (e.g., a still image or video recording) is inserted into the message creation field 608 and subsequently sent to the participants of the message conversation. In some embodiments, the captured image is sent directly to the participants of the message conversation without being inserted into the message creation field 608.

[0187] In Figure 6D, device 600 detects input 623 on effect affordance 622 (for example, a tap gesture on display 601).

[0188] In Figure 6E, in response to detecting input 623, device 600 activates camera 602 (e.g., switching from the rear camera), updates the image display area 620 to display a live camera preview 620-1 from camera 602, showing a representation of a subject 632 located within the field of view of camera 602, and a background 636 displayed behind the subject 632. As discussed herein, in some embodiments, the image data captured using camera 602 includes depth data that can be used to determine the depth of objects within the field of view of camera 602. In some embodiments, device 600 analyzes objects (e.g., in the image data) based on the depth of these detected objects and uses this determination to apply the visual effects discussed herein. For example, device 600 may classify the subject 632 as being in the foreground of the live camera preview 620-1, and objects located behind the user as being in the background of the live camera preview 620-1. These background objects are collectively referred to herein as background 636.

[0189] Device 600 also highlights an effect affordance 622 indicating that visual effects are enabled for display, and updates the camera option area 625 by replacing the camera option affordance 619 with a visual effect option affordance 624. The visual effect option affordances include avatar effect affordance 624-1 and sticker effect affordance 624-2. The visual effect option affordances 624 correspond to different visual effects that can be applied to the image displayed in the image display area 620. By selecting one of the visual effect option affordances (e.g., 624-1 or 624-2), a menu appears along with the visual effect options corresponding to the selected visual effect option affordance.

[0190] The user can enable or disable the effect mode of device 600 by selecting an effect affordance 622. When an effect affordance 622 is highlighted, the effect mode of device 600 is enabled, and a visual effect is displayed in the image display area 620. When the user taps on the highlighted affordance 622, the effect affordance 622 is no longer highlighted, and the effect mode is disabled so that the visual effect is not enabled for display in the image display area 620. In some embodiments, when the effect mode is enabled, device 600 updates the image shown in the image display area 620 to display one or more visual effects applied to the image (including visual effects applied to a live image stream), and when the effect mode is disabled, device 600 removes or hides the visual effects from the image shown in the image display area 620.

[0191] In Figure 6F, device 600 detects input 626 on avatar effect affordance 624-1 (for example, a tap gesture on display 601).

[0192] In Figure 6G, in response to detecting input 626, device 600 displays the avatar options menu 628 along with a scrollable list of avatar options 630. The avatar options menu 628 also includes a selection area 629 for indicating one of the selected avatar options 630. As shown in Figure 6G, robot avatar option 630-3 is placed in the selection area 629, indicating that robot avatar option 630-1 is selected. Other avatar options 630 shown in Figure 6G include customized female avatar option 630-1, null avatar option 630-2, rabbit avatar option 630-4, and alien avatar option 630-5. As will be discussed in detail below, null avatar option 630-2 corresponds to an avatar option for which the avatar is not displayed on the subject in the image display area 620. In some embodiments, null avatar option 630-2 is the avatar option that is selected by default when the avatar options menu 628 is displayed.

[0193] The avatar options 630 correspond to virtual avatar visual effects applied to the representation of a subject within the image display area 620. Specifically, each avatar option 630, when selected, corresponds to a virtual avatar that is superimposed on the face of the subject within the image display area, while other parts of the image within the image display area (such as the background or other parts of the user, such as the user's body) remain visible. A user (e.g., subject 632) located within the field of view of camera 602 can control the visual appearance of the virtual avatar by changing the facial posture (e.g., rotation or orientation), including the movement of various facial features (e.g., winking, tongue sticking out, smiling). Details for controlling the display and movement of the virtual avatar are provided in U.S. Patent Application No. 15 / 870,195, which is incorporated herein by reference for all purposes.

[0194] In some embodiments, the virtual avatar is a graphical representation of the user (e.g., a graphic representation of the user). In some embodiments, the virtual avatar is non-photorealistic (e.g., cartoonish). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar face features). In some embodiments, the avatar features correspond to (e.g., are mapped to) one or more physical features of the user's face such that detected movement of the user's physical features (e.g., determined based on a camera such as a depth-sensing camera) influences the avatar features (e.g., influences the graphic representation of the features).

[0195] In some embodiments, the user can manipulate the characteristics or features of the virtual avatar using camera sensors (e.g., camera module 143, light sensor 164) and optionally depth sensors (e.g., depth camera sensor 175). When the user's physical characteristics (such as facial features) and position (such as head position, head rotation, or head tilt) change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical characteristics and position. In some embodiments, changes in the user's physical characteristics and position represent various expressions, emotions, contexts, tones, or other nonverbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, contexts, tones, or other nonverbal communication.

[0196] In some embodiments, the virtual avatar is a customizable avatar (e.g., customizable avatar 835). A customizable avatar is a virtual avatar that can be selected and customized by the user to achieve a desired appearance (e.g., to look like the user). Customizable avatars generally have the appearance of a human character, rather than non-human characters such as anthropomorphic constructs of animals or other non-human objects. In addition, avatar features can be created or modified as needed using an avatar editing user interface (e.g., the avatar editing user interface discussed below with respect to Figures 8AA to 8AB). In some embodiments, a customizable avatar can be generated and configured to achieve a customized physical appearance, physical construct, or modeled behavior.

[0197] In some embodiments, the virtual avatars are non-customizable avatars. Non-customizable avatars are virtual avatars that can be selected by the user, but whose appearance can be modified via face tracking, as will be described in more detail below, but are generally not configurable. Instead, non-customizable avatars are pre-configured and generally do not have feature components that can be modified by the user. In some embodiments, non-customizable avatars have the appearance of non-human characters, such as anthropomorphic constructs of animals or other non-human objects (see, for example, robot avatar 633, rabbit avatar 634). Non-customizable avatars cannot be created by the user, or modified to achieve significant changes in their physical appearance, physical constructs, or modeled behavior of the non-customizable avatar.

[0198] In Figure 6G, since robot avatar 630-3 is selected, device 600 displays robot avatar 633 positioned on the face of subject 632 (e.g., user) displayed in image display area 620. Device 600 displays the avatar so that it remains fixed on the subject's face as the avatar moves within the field of view of camera 602. Device 600 also continuously modifies the avatar based on detected changes in the user's face, including changes in facial posture or expression, while maintaining the displayed subject's body and background 636 within the image shown in image display area 620.

[0199] As shown in Figure 6H, if device 600 detects that the user's head has moved out of the field of view of camera 602, device 600 displays a prompt 638, which includes, for example, a representation 640 of a selected avatar (e.g., a robot avatar), and a message 642 instructing the user to reposition their head within the field of view of camera 602. In some embodiments, prompt 638 is displayed with a blurred background 644 (including blurring the user 632 and background 636). In some embodiments, displaying prompt 638 includes displaying an animation of the selected avatar returning to the center position of the image display area 620, and showing slowed movement of the avatar and its features that appears to stop based on a physical model of the avatar features.

[0200] Once device 600 detects the user's head returning to the camera 602's field of view, the device continues to update the selected avatar (e.g., robot avatar 633) based on the changes detected in the user's face, as shown in Figure 6I. In some embodiments, when device 600 detects the user's face returning to the camera 602's field of view, device 600 displays avatar 633 moving from the center of the image display area 620 to the position of the user's face, and resumes updating the avatar based on the detected changes in the user's face.

[0201] In Figures 6I to 6K, device 600 detects input 644 (e.g., a scroll gesture on display 601) on a displayed list of avatar options 630. As input 644 moves left across display 601, device 600 displays avatar options 630, scrolling left to reveal additional avatar options (e.g., poop avatar options and fox avatar options). In some embodiments, when one of the avatar options enters the selection area 629, the device generates haptic feedback (e.g., haptic output with or without audible output) to indicate to the user that the end of input 644 at that point could result in a selection of the avatar option currently located within the selection area 629. For example, in Figure 6K, device 600 generates haptic feedback 646 when alien avatar option 630-5 is located within the selection area 629.

[0202] In Figure 6J, device 600 detects the end of input 644 and determines that alien avatar option 630-5 is located within the selection area 629. As a result, device 600 replaces robot avatar 633 with alien avatar 635, which corresponds to the selected alien avatar option 630-5, while maintaining the display of the subject's body and background 636. In some embodiments, replacing a previously selected avatar (e.g., robot avatar 633) with the currently selected avatar (e.g., alien avatar 635) includes displaying the animation of the previously selected avatar moving off-screen (e.g., outside the image display area 620), and the animation of the currently selected avatar moving from the avatar options menu 628 to the user's face within the image display area 620. In some embodiments, face tracking of the currently selected avatar begins when the avatar moves from the avatar options menu 628 to the user's face. In some embodiments, the background is blurred while the avatar switching operation is occurring.

[0203] In Figures 6M to 6N, device 600 detects a vertical gesture 646 on the avatar options menu 628 (e.g., a vertical swipe or touch-and-drag on the display 601) and, accordingly, expands the avatar options menu 628 to an enlarged view 628-1, as shown in Figure 6N. In Figure 6N, the avatar options 630 are arranged in a matrix and, as shown in Figure 6O, can be selected (e.g., gesture 648 selects robot avatar option 630-3) to display different avatars in the image display area 620.

[0204] As shown in Figure 6P, in response to the gesture 648, the device 600 displays an avatar options menu 628 containing the selected robot avatar options 630-3 (located in the selection area 629), and a robot avatar 633 displayed on the user's head within the image display area 620 (for example, by zooming in on the menu).

[0205] In Figure 6Q, device 600 detects input 649 on the close icon 650 (e.g., a tap gesture on the display 601), closes the application options menu 628, and returns to the camera user interface 615 shown in Figure 6R, which shows the camera options area 625, while maintaining the display of the robot avatar 633 of the user's face in the image stream (e.g., live camera preview) on the image display area 620. The camera options area 625 shows the capture affordance 621, the visual effect option affordance 624, and the highlight effect affordance 622. As shown below in Figures 6S and 6T, the user can select the avatar menu affordance 624-1 to display the avatar options menu 628 again.

[0206] For example, in Figure 6S, device 600 detects input 652 on avatar effect affordance 624-1 (e.g., a tap gesture on display 601). In Figure 6T, in response to detecting input 652, device 600 displays an avatar options menu 628 having avatar options 630 which replaces the camera options area 625, which includes capture affordance 621, effect affordance 622, and visual effect option affordance 624. The avatar options menu 628 can be closed again by selecting close affordance 650, as described above, and the user can return to the camera application user interface 615, as shown in Figure 6U.

[0207] In Figure 6U, device 600 detects input 654 on sticker effect affordance 624-2 (e.g., a tap gesture on display 601). In Figure 6V, in response to detecting input 654, device 600 displays a sticker options menu 656 having a scrollable list of stickers 658. Stickers are static graphical objects that can be selected by the user and applied to an image in the image display area 620. In some embodiments, stickers can be selected by a touch-and-drag gesture, which involves starting on a selected sticker, dragging the sticker to the desired position, and then ending the drag (e.g., lifting the finger) to position the sticker in the desired location. In some embodiments, stickers can be selected by a tap gesture, and the corresponding sticker is then displayed in the position on the image display area 620. An example of such an embodiment is shown in Figures 6W to 6X, which then show a tap gesture 660 on a helmet sticker 658-1 displayed in the center of the image display area 620.

[0208] Figures 6Y to 6Z show that a placed sticker (e.g., sticker 658-1) can be repositioned on the image display area 620 by dragging the sticker to a new position (see gesture 662 moving sticker 658-1 to a new position in Figure 6Z).

[0209] Figure 6AA shows that the placed sticker (e.g., sticker 658-1) can be rotated using a two-finger gesture 664.

[0210] Figure 6AB shows that a placed sticker (e.g., sticker 658-1) can be resized using a pinch or de-pinch gesture. In Figure 6AB, helmet sticker 658-1 is enlarged in response to a de-pinch gesture 666. The resized, rotated, and repositioned helmet sticker 658-1 is shown placed on the head of the robot avatar in Figure 6AC.

[0211] In Figure 6AD, device 600 detects input 668 on the close icon 670 (e.g., a tap gesture on the display 601), closes the sticker options menu 656, and returns to the camera application user interface 615, which includes a camera options area 625 showing a capture affordance 621, a visual effects option affordance 624, and a highlighting effect affordance 622 indicating that a visual effect is available, as shown in Figure 6AE. In Figure 6AE, device 600 displays the user's representation in an image on the image display area 620, but modifies it with visual effects, including a robot avatar 633 (e.g., selected from the avatar options menu 628) and a helmet sticker 658-1 (e.g., selected from the sticker options menu 656). The display of the visual effects is dynamic. For example, device 600 continuously modifies the robot avatar 633 in response to detected changes in the user's face. In some embodiments, the positioning of the sticker 658-1 is also dynamic and changes based on detected user movements, as will be discussed in more detail below.

[0212] When a visual effect is enabled for device 600 (for example, effect affordance 622 is highlighted), the applied visual effect (e.g., avatar and stickers) can be removed or hidden from the image display area 620 by deselecting the highlighted effect affordance 622 (for example, by selecting affordance 622 when it is highlighted to disable the visual effect). For example, in Figure 6AF, the device detects input 672 on the highlighted effect affordance 622 (e.g., a tap gesture on display 601). In response, device 600 removes the displayed visual effect (e.g., helmet sticker 658-1 and robot avatar 633 in Figure 6AF) from the image display area 620, as shown in Figure 6AG, while maintaining the display of an image stream (e.g., live camera preview 620-1) containing the subject 632 and background 636 within the field of view of camera 602.

[0213] In some embodiments, after the visual effects mode is disabled (for example, by deselecting the highlighting effect affordance 622), the removed visual effects can be restored, for example, by re-selecting the effect affordance 622 within a predetermined time. For example, in Figure 6AH, the device detects input 674 on the non-highlighting effect affordance 622 (for example, a tap gesture on the display 601). In response to detecting input 674, the device 600 highlights the effect affordance 622 while continuing to display an image stream (for example, a live camera preview 620-1) including the body of the subject 632 and the background 636, as shown in Figure 6AI, and redisplays the helmet sticker 658-1 and the robot avatar 633.

[0214] In Figure 6AJ, device 600 detects input 676 on the capture affordance 621 (e.g., a tap gesture on the display 601). Upon detecting input 676, device 600 captures an image of the live camera preview 620-1 in Figure 6AJ and displays media item 620-2 in Figure 6AK (e.g., a still image of the state of the live camera preview 620-1 at the time device 600 detected input 676 on the capture affordance 621).

[0215] In some embodiments, once an image is captured (e.g., stored as a media item), device 600 encodes depth data within the media item. Storing depth data in media makes it possible to later apply depth-based effects (e.g., effects based on the position of an object (e.g., a user's face) in the z-direction). In some embodiments, once an image is captured while an effect is being applied, the effect is encoded directly into visual (e.g., RGB) information to improve compatibility with other devices.

[0216] In Figure 6AK, device 600 displays a camera application user interface 615 showing a captured image (e.g., media item 620-2) in the image display area 620 (for example, media item 620-2 replaces the live camera preview 620-1 shown in the image display area 620). Media item 620-2 is a representation of the live camera preview 620-1 at the time the capture affordance 621 is selected. Thus, media item 620-2 includes the face of the subject 632, and a helmet sticker 658-1 and a robot avatar 633 displayed on the background 636.

[0217] Device 600 also replaces camera-specific affordances (e.g., affordance 617 shown in Figure 6AJ) with re-shooting affordance 679 and recipient identifier 606, the recipient identifier 606 indicating the currently selected recipient of media item 620-2 to which the user should subsequently send the media item, as discussed below.

[0218] Device 600 displays a camera options area 625 containing visual effect option affordances 624. By selecting a visual effect option affordance 624, the corresponding option menu can be displayed, and the captured media item 620-2 (and the recorded video media item 620-4, described later) can be modified using the menu.

[0219] Device 600 also updates the camera options area 625 to replace the capture affordance 621 and camera selector affordance 627 with the markup affordance 677, the edit affordance 678, and the transmit affordance 680. The mark affordance 677 allows the user to mark up the media item 620-2. The edit affordance 678 allows the user to edit the media item 620-2, for example, by cropping the image or adjusting other properties of the media item 620-2. As shown in Figure 6AK, the transmit affordance 680 appears in the same location / area where the capture affordance 621 was displayed, so that the transmit affordance 680 replaces the capture affordance 621 on the display.

[0220] The transmit affordance 680 allows the user to immediately send media item 620-2 to the recipient indicated by the recipient identifier 606. For example, in Figure 6AL, device 600 detects input 682 on transmit affordance 680 (e.g., a tap gesture on display 601) and immediately sends media item 620-2 to John, the contact corresponding to recipient identifier 606, as shown in the messaging user interface 603 in Figure 6AM. In some embodiments, device 600 sends media item 620-2 without displaying the messaging user interface 603. In some embodiments, such as shown in Figure 6AM, the device sends media item 620-2 and displays the messaging user interface 603 with media item 620-2 in the message display area 604 to indicate that media item 620-2 has been sent to a participant in the message conversation (e.g., John) and that it is displaying the application dock 690 located above the keyboard display area 612. The application dock 690 includes application affordances 692 that can be selected to access different applications.

[0221] In some embodiments, media item 620-2 is not immediately sent to participants in the messaging conversation. For example, in Figure 6AN, instead of selecting the send affordance 680, the user selects the complete affordance 618. In response to detecting input 684 on the complete affordance 618 (e.g., a tap gesture on the display 601), the device 600 displays a representation of media item 685 in the message creation field 608 of the messaging user interface 603, as shown in Figure 6AO. In this embodiment (shown in Figures 6AO to 6AR), instead of displaying the application dock 690, the device 600 displays a text suggestion area 614 along with the keyboard display area 612 before sending the message, allowing the user to compose text (e.g., 687) or optionally add other message content (e.g., multimedia content). Upon detecting input 686 on the send icon 688 (for example, a tap gesture on the display 601), device 600 sends a representation of media item 685 to the participants of the message conversation, as shown in Figures 6AQ to 6AR.

[0222] In some embodiments, transmitting media item 685 includes transmitting a media item having depth data encoded in the media item. Storing the media item on the media along with the depth data makes it possible to later apply depth-based effects (e.g., effects based on the position of an object (e.g., a user's face) in the z-direction) (e.g., by the recipient later). In some embodiments, once the media item is transmitted, its effects are directly encoded into visual (e.g., RGB) information to improve compatibility with other devices.

[0223] In some embodiments, after transmitting a representation of media item 685, device 600 displays a messaging user interface 603 having a text suggestion area 614 displayed above the keyboard display area 612, as shown in Figure 6AR. Upon receiving input 694 on the application dock affordance 610 (e.g., a tap gesture on the display 601), device 600 replaces the text suggestion area 614 with the application dock 690, as shown in Figure 6AT.

[0224] In some embodiments, the application dock 690 remains visible in the messaging user interface 603 until the user selects either the message creation field 608 or the keyboard area 612 (an action associated with creating the text of a message). For example, as shown in Figure 6AU, device 600 detects input 696 on the keyboard display area 612 (e.g., a tap gesture on device 601). In response, as shown in Figure 6AV, device 600 removes the application dock 690 and displays the text suggestion area 614 above the keyboard display area 612. In some embodiments, if input 696 is input on the message creation field (instead of the keyboard display area 612), device 600 also removes the application dock 690 and displays the text suggestion area 614, as shown in Figure 6AV.

[0225] Figure 6AW shows the messaging user interface 603 after the application dock affordance 610 has been selected (for example, in response to input 694 in Figure 6AS). The application dock 690 is displayed above the keyboard selection area 612 (for example, replacing the text suggestion area 614). In response to detecting input 698 on application affordance 692-1 (for example, a tap gesture on the display 601), the device 600 displays the application display area 699 in the position previously occupied by the text suggestion area 614 and the keyboard display area 612. The application display area 699 optionally includes the application dock 690, which includes a display 6100 of the selected application affordance (for example, application affordance 692-1), as shown in Figure 6AX.

[0226] In some embodiments, the application display area 699 includes graphical objects 6102 that can be selected for use in the messaging user interface 603. In some embodiments, the type of graphical object displayed in the application display area 699 depends on the application affordance selected to invoke the display of the application display area. In the embodiment shown in Figure 6AX, the selected application affordance 692-1 corresponds to an avatar application. Therefore, the graphical objects 6102-1 and 6102-2 displayed in the application display area 699 correspond to avatars that can be selected for use in the messaging user interface. If the application affordance selected from the application dock 610 corresponds to a sticker application, the graphical object 6102 corresponds to a sticker that can be selected for use in the messaging user interface.

[0227] As previously mentioned, the embodiments described with respect to Figures 6C to 6AX relate to device 600 operating in still image capture mode of operation associated with photographic affordance 619-2. However, unless otherwise specified, these embodiments also apply to the operation of device 600 when the device is operating in video recording mode associated with video affordance 619-1. Thus, the following descriptions of Figures 6AY to 6BQ, which show many of the embodiments described above that apply to device 600 when operating in video recording mode associated with video affordance 619-1, have been omitted where necessary to avoid redundancy.

[0228] Figures 6AY to 6AZ show that device 600 switches to video recording mode in response to the detection of input 6104 on video affordance 619-1 of the camera application user interface 615. When switching to video recording mode, device 600 excludes camera-specific affordances 617-2, 617-3, and 617-4 and displays timer 6122.

[0229] Figures 6AZ to 6BB show a device 600 that enables an effect mode in response to an input 6106 on the effect affordance 622, and then displays a visual effect option affordance 624 in the camera option area 625 while the video recording mode is enabled.

[0230] Figures 6BC to 6BE show device 600 displaying the avatar options menu 628 in response to detecting input 6108 on avatar effect affordance 624-1. The avatar options menu 628 includes avatar options 630, which can be selected to display an avatar corresponding to the subject's face during video mode. When robot avatar option 630-3 is selected, device 600 displays the robot avatar 633 on the subject's face while maintaining the display of the subject's body and background 636. The close icon 650 is selected by input 6110, closing the avatar options menu 628 and returning to the camera options area 625 with highlighted effect affordance 622 and displayed visual effect option affordance 624.

[0231] Figures 6BF to 6BH show a device 600 that generates a recorded video of image data represented in the image display area 620. The device 600 detects input 6112 on the capture affordance 621 and begins capturing the image data represented in the image display area 620. In some embodiments, the device 600 generates a video recording of the image data based on the determination that the video recording mode of the device 600 is enabled. In some embodiments, the device 600 generates a video recording or, optionally, a still image capture based on input 6112 detected on the capture affordance 621. For example, if input 6112 is detected as a tap gesture, the device 600 captures a still image of the image data represented in the image display area 620. If input 6112 is detected as a tap-and-hold gesture, the device 600 captures a video recording of the image data as described in more detail above.

[0232] In Figure 6BG, upon detecting input 6112 on capture affordance 621, device 600 begins processing to record image data represented in image display area 620, including the subject, background, and visual effects. Device 600 also modifies the camera application user interface 615 by removing camera effects area 625, expanding image display area 620, and displaying stop affordance 6114, and optionally image capture affordance 6116 for capturing still images of the live camera recording. In Figures 6BG and 6BH, image display area 620 represents a live camera recording 620-3 of an object within the field of view of camera 602, with the addition of visual effects (e.g., robot avatar 633) which are enabled according to the various selections made above. These visual effects, which may also include stickers and fullscreen effects, are displayed in image display area 620 as part of the live image data and recorded to form part of the resulting media item. As discussed herein, these visual effects can be optionally shown or hidden on a media item after the media item has been recorded.

[0233] In Figure 6BH, device 600 detects changes in the user's face and modifies the robot avatar 633 based on the detected changes in the user's face while recording a live image stream. Upon detecting input 6118 on stop affordance 6114, device 600 stops recording the image data represented in the image display area 620 and displays the playback interface shown in Figure 6BI.

[0234] In Figure 6BI, device 600 displays video media item 620-4 recorded in image display area 620, and displays reshoot affordance 679 and recipient identifier 606, the recipient identifier indicating the currently selected recipient of video item 620-4 to whom the user should subsequently send the media item, as discussed herein.

[0235] Device 600 also removes the stop affordance 6114 and the image capture affordance 6116 and displays a camera options area 625 with a video scrubber 6120, an effect affordance 622, an edit affordance 678, a markup affordance 677, and a transmit affordance 680 for the recorded video media items 620-4. The camera options area 625 also includes a visual effects option affordance 624. Selecting the visual effects option affordance 624 can display the respective option menus that can be used to modify the captured media items 620-4. Figures 6BJ to 6BN show an example of modifying the visual effects on the captured video media items 620-4. As shown in Figure 6BI, the transmit affordance 680 is displayed in the same location / area where the capture affordance 6116 was displayed, so that the transmit affordance 680 replaces the capture affordance 6116 on the display.

[0236] Figures 6BJ to 6BN show device 600 displaying the avatar options menu 628 in response to detecting input 6124 on avatar effect affordance 624-1. As shown in Figure 6BK, the avatar options menu 628 displays avatar options 630, including the selected robot avatar option 630-3. As shown in Figure 6BL, device 600 detects input 6126 and selects the rabbit avatar option 630-4, updating video media item 620-4 to display the rabbit avatar 634 positioned on the subject's head, while still displaying the subject's body and background 636, as shown in Figure 6BM. As shown in Figure 6BN, in response to detecting input 6128 on the close icon 650, device 600 closes the avatar options menu 628 and displays the camera options area 625 in Figure 6BO. The camera options area 625, including the video scrubber 6120, is updated from displaying the robot avatar 633 in Figure 6BJ to displaying the rabbit avatar 634. In some embodiments, in response to input directed to the video scrubber 6120 (e.g., a swipe input or drag input including a touch movement that starts at the position of the video scrubber 6120), the device scrubs through the recorded video (e.g., changing the appearance of the video preview and moving the scrubber or the playhead within the scrubber according to the input). In some embodiments, scrubbing through the recorded video demonstrates how effects applied to the video affect it over time (e.g., how stickers and / or avatars move and / or change their appearance as the user's head / face moves or changes their appearance within the field of view of one or more cameras). In some embodiments, the video scrubber 6120 includes selectable affordances for cropping the video (e.g., changing the start or end time of the video scrubber by dragging the editing affordances at both ends of the video scrubber).For example, in response to user input directed to the video scrubber 8120 in Figure 8AW, the device may scrub through video using avatars and stickers, as shown in Figures 8AW-8AZ, but at a rate determined based on changes in the input (e.g., magnitude and direction of touch movement on a touch-sensitive display). Interactions with the video scrubber 6120 (and the video scrubber 8120 described later) are applied selectively in the same manner as interactions with other scrubbers described herein.

[0237] As described above with respect to Figures 6AK to 6AO, video media items 620-4 can be immediately sent to participants in a message conversation in response to the detection of an input on the transmit affordance 680 (e.g., input 6130), as shown in Figures 6BP and 6BQ. The representation of video media item 620-4 can be displayed in the message creation field 608 in response to the input of the completion affordance 618.

[0238] Figures 7A and 7B are flowcharts illustrating a method for displaying visual effects in a messaging application using an electronic device, according to some embodiments. Method 700 is performed on a device (e.g., 100, 300, 500, 600) that includes a camera, a display device, and one or more input devices. Some operations in Method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0239] As described below, Method 700 provides an intuitive way to display visual effects in messaging applications. This method reduces the cognitive burden on users to apply visual effects to images sent in messaging applications, thereby creating a more efficient human-machine interface. In battery-powered computing devices, power is saved and battery life is increased by allowing users to display visual effects on images more quickly and efficiently.

[0240] An electronic device (e.g., 600) displays a messaging user interface (e.g., 603) via a display device (e.g., 601) that includes camera affordances (e.g., 609, selectable icons associated with a function for activating a camera application) for a message conversation involving at least a first participant (702).

[0241] An electronic device (e.g., 600) detects a first input directed towards a camera affordance (e.g., 616, a touch gesture on a touchscreen display at a location corresponding to a camera affordance) via one or more input devices (704).

[0242] Upon detecting a first input (e.g., 616), the electronic device (e.g., 600) displays a camera user interface (e.g., 615) (706). The camera user interface includes capture affordances (e.g., selectable icons associated with the ability to capture image data using the camera of the electronic device, 621) (708).

[0243] In some embodiments, the camera user interface (e.g., 615) includes effect mode affordances (e.g., 622, selectable icons associated with a function for activating modes in which various visual effects are available to modify the image data) associated with modes in which visual effects are enabled for display on the captured image data (710). Including effect mode affordances in the camera user interface makes it possible for the user to recognize that a particular effect (e.g., a visual effect) can be applied to an image via the camera user interface. Providing additional control of the device improves the usability of the device, makes the user device interface more efficient (e.g., by helping the user to provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, reduces the power consumption of the device and improves battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, the visual effect can be added to the representation of image data within the camera's field of view. In some embodiments, the visual effect can be added to the captured image data. In some embodiments, the visual effect is based on depth data. In some embodiments, the electronic device detects the selection of an effect mode affordance (e.g., 623) via one or more input devices. In some embodiments, upon detecting the selection of an effect mode affordance, the electronic device transitions from a first camera mode (e.g., standard camera mode) to a second camera mode different from the first camera mode (e.g., an effect camera mode, a mode in which various visual effects can be used to modify image data). In some embodiments, while the device is in the second camera mode, a visual indicator is displayed that the second camera mode is in operation (e.g., the effect mode affordance is highlighted).

[0244] In some embodiments, in response to detecting the selection of an effect mode affordance (e.g., 622) (e.g., 623), the electronic device (e.g., 600) ceases displaying one or more camera mode affordances. In some embodiments, in response to detecting the selection of an effect mode affordance, the electronic device displays a plurality of effect option affordances (e.g., 624, selectable icons, each associated with a function for generating a visual effect). In some embodiments, the effect option affordances include sticker affordances (e.g., 624-2) and / or avatar affordances (e.g., 624-1) at locations within the camera user interface (e.g., 615) previously occupied by one or more camera mode affordances. In some embodiments, the location where the effect option affordances are displayed is any location within a particular region (e.g., camera effect region 625) where camera mode affordances were previously displayed. In some embodiments, the location where each effect option affordance is displayed is the same location in which each camera mode affordance was displayed in that region. In some embodiments, displaying effect option affordances includes replacing camera mode affordances with effect option affordances.

[0245] In some embodiments, an electronic device (e.g., 600) detects the selection (e.g., 654) of a first effect option affordance (e.g., sticker affordance 624-2) via one or more input devices. In some embodiments, upon detection of the selection of a first effect option affordance, the electronic device stops displaying multiple effect option affordances (e.g., 624) and displays multiple selectable graphic icons (e.g., 658, sticker). By stopping the display of multiple effect option affordances and displaying multiple selectable graphic icons upon detection of the selection of a first effect option affordance, the user can quickly and easily recognize that the first effect option affordance is related to a graphic icon (e.g., sticker) option, thereby improving the usability of the device, making the user device interface more efficient (e.g., by assisting the user in providing appropriate input when operating / interacting with the device and reducing user errors), and in addition, reducing the power consumption of the device and improving battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, displaying a sticker involves displaying an area (e.g., a sticker menu 656) above the affordances of the effect options, which includes a number of sticker options that can be selected to be displayed on the image represented in the camera user interface. In some embodiments, the user selects a sticker (e.g., 658-1) by tapping on it (e.g., 660), and the sticker is automatically displayed on the image (e.g., in a default position such as the center of the image). In some embodiments, the user selects a sticker by touching it and dragging it onto the image from the sticker menu.In some embodiments, while displaying the capture affordance, the electronic device further detects the selection of a first of the effect option affordances, and in response, stops displaying the capture affordance.

[0246] In some embodiments, an electronic device (e.g., 600) detects the selection (e.g., 626) of a second effect option affordance (e.g., avatar affordance 624-1) via one or more input devices. In some embodiments, upon detection of the selection of a second effect option affordance, the electronic device stops displaying a plurality of effect option affordances (e.g., 624) and displays an avatar selection area (e.g., avatar menu 628) having a plurality of avatar affordances (e.g., 630 displayed in a linear arrangement) (e.g., affordances representing avatars). By stopping the display of a plurality of effect option affordances and displaying an avatar selection area upon detection of the selection of a second effect option affordance, the user can quickly and easily recognize that the second effect option affordance is related to avatar selection, thereby improving the usability of the device, making the user device interface more efficient (e.g., by assisting the user in providing appropriate input when operating / interacting with the device and reducing user errors), and in addition, reducing the power consumption of the device and improving battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, avatar affordances correspond to avatars that are customizable, non-customizable, or a combination thereof. In some embodiments, displaying an avatar selection area includes displaying an area (e.g., avatar menu 628) above effect option affordances, which includes a plurality of avatar affordances that can be selected to display the corresponding avatar on an image represented in the camera user interface.

[0247] In some embodiments, an electronic device (e.g., 600) detects a swipe input (e.g., 646, a vertical swipe gesture) on an avatar selection area (e.g., 628) via one or more input devices. In some embodiments, in response to detecting a swipe input on the avatar selection area, the electronic device enlarges the size of the avatar selection area (e.g., 628-1) to display a matrix of avatar affordances (e.g., 630). By enlarging the size of the avatar selection area and displaying the avatar affordances in a matrix in response to detecting a swipe input on the avatar selection area, the user can (simultaneously) see one or more additional selectable avatars that are not (simultaneously) visible in the avatar selection area. By providing additional control options without cluttering the user interface with the additionally displayed controls, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently. In some embodiments, increasing the size of the avatar display area includes vertically expanding the avatar display area using avatar affordances displayed in a matrix within the avatar display area in order to present a full-screen display of the avatar display area.

[0248] In some embodiments, the camera user interface (e.g., 615) further includes a first representation of the image data (e.g., a live camera preview 620-1). By providing the first representation of the image data (e.g., a live camera preview), visual feedback is provided for one or more changes (to the image) made by the user before saving / confirming the changes. By providing the user with improved visual feedback, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and by reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently. In some embodiments, in response to detecting the selection of an effect mode affordance (e.g., 624-1, 624-2), the electronic device (e.g., 600) determines that the first representation of the image data corresponds to image data acquired from a second camera (e.g., a rear camera), and stops displaying the first representation of the image data and displays a second representation of the image data (e.g., a live camera preview), which corresponds to image data acquired from a camera (e.g., a front camera). In some embodiments, the representation of the image data corresponding to the front camera includes a representation of the user located within the field of view of the front camera.

[0249] In some embodiments, while an electronic device (e.g., 600) is in a second camera mode (e.g., an effects camera mode, a mode in which various visual effects can be used to modify image data), the electronic device receives a transition request to a first camera mode (e.g., a normal mode, a non-effects mode) (e.g., a selection of an active visual effect affordance). In some embodiments, in response to receiving a transition request to the first camera mode, the electronic device transitions from the second camera mode to the first camera mode. In some embodiments, the electronic device deactivates a first visual effect (e.g., disables it, stops displaying the displayed first visual effect) depending on whether the first visual effect is active (e.g., actively applied to captured image data or a preview of image data for capture).

[0250] In some embodiments, after deactivating the first visual effect, the electronic device (e.g., 600) detects a subsequent selection of effect mode affordances (e.g., 624-1, 624-2) via one or more input devices. In some embodiments, in response to detecting a subsequent selection of effect mode affordances, the electronic device reactivates the first visual effect within a predetermined time after deactivating the first visual effect, according to the determination that a subsequent selection of effect mode affordances has occurred. By reactivating the first visual effect within a predetermined time after deactivating the first visual effect, according to the determination that a subsequent selection of effect mode affordances has occurred, the user can quickly and easily return to the previous visual effect (e.g., without having to reselect / regenerate the effect). The usability of the device is improved by providing additional control options without cluttering the user interface with additionally displayed controls, making the user device interface more efficient (e.g., by assisting the user in providing appropriate input when operating / interacting with the device and reducing user errors), and in addition, reducing the power consumption of the device and improving battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, after selecting an effect mode affordance to remove a visual effect from an image, if the effect mode affordance is selected again within a predetermined time, the removed visual effect is restored to the image.

[0251] An electronic device (e.g., 600) detects a second input (e.g., 676, a touch gesture on a touchscreen display at a location corresponding to a capture affordance) directed towards a capture affordance (e.g., 621) via one or more input devices (712).

[0252] An electronic device (e.g., 600) captures image data (716) using a camera (e.g., 602) in response to detecting a second input (714). In some embodiments, capturing image data includes capturing image data in a first image capture mode (e.g., photo capture mode, still image capture mode) according to a value of a second input characteristic (e.g., duration of contact) that satisfies a first capture mode criterion (e.g., less than a threshold duration). In some embodiments, capturing image data includes capturing image data in a second image capture mode (e.g., video capture mode, continuous capture mode) according to a value of a second input characteristic that satisfies a second capture mode criterion (e.g., greater than a second threshold duration). In some embodiments, the capture affordance (e.g., 621) is a multifunctional capture affordance. In some embodiments, when a tap is detected on the capture affordance, the electronic device captures a photograph (e.g., a still image). In some embodiments, when a press-and-hold gesture is detected on the capture affordance, the electronic device captures video (e.g., a series of images).

[0253] In response to detecting a second input (e.g., 676) (714), the electronic device (e.g., 600) stops displaying the capture affordance (e.g., 621) (718).

[0254] In some embodiments, an electronic device (e.g., 600) captures image data using a camera (e.g., 602) and then displays markup affordances (e.g., 677), editing affordances (e.g., 678), and reshoot affordances (e.g., 679). In some embodiments, while displaying the markup affordances, editing affordances, and reshoot affordances, the electronic device receives a fourth user input via one or more input devices. In some embodiments, upon detecting the fourth user input, the electronic device initiates processing for editing the captured image data according to the fourth user input corresponding to the editing affordance. In some embodiments, the editing process includes displaying one or more affordances for editing the captured image data. In some embodiments, upon detecting the fourth user input, the electronic device initiates processing for marking up the captured image data according to the fourth user input corresponding to the markup affordance. In some embodiments, the editing process includes displaying one or more affordances for marking up the captured image data. In some embodiments, upon detecting a fourth user input, the electronic device initiates a process to recapture the captured image data in accordance with the fourth user input corresponding to the recapture affordance. In some embodiments, initiating the process to recapture the captured image data includes capturing new image data and replacing the captured image data with the new image data.

[0255] In response to detecting a second input (e.g., 676) (714), the electronic device (e.g., 600) displays a transmit affordance (e.g., 680, a selectable icon associated with a function for transmitting the captured image data to a participant in a conversation, or for presenting the captured image data to a creation area before subsequent transmission) (720) at a location in the camera user interface (e.g., 615) previously occupied by a capture affordance (e.g., 621). By displaying the transmit affordance at a location in the camera user interface previously occupied by a capture affordance, visual feedback is provided that the captured image is ready to be transmitted to the intended recipient. By providing visual feedback to the user without cluttering the user interface, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user to provide appropriate input when operating / interacting with the device and by reducing user errors), and in addition, power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently. In some embodiments, the location where the transmit affordance (e.g., 680) is displayed is any location within a specific region (e.g., camera effect region 625) where the capture affordance was previously displayed. In some embodiments, the location where the transmit affordance is displayed is the same location within that region where the capture affordance was displayed. In some embodiments, displaying the transmit affordance includes replacing the capture affordance with the transmit affordance.

[0256] In some embodiments, while displaying a transmit affordance (e.g., 680), the electronic device (e.g., 600) displays a representation of the first participant (e.g., an icon, a photograph, an avatar, or other identifier associated with the first participant) (722). By displaying a representation of the first participant while displaying a transmit affordance, the user can quickly and easily recognize the intended recipient, thereby improving the usability of the device, making the user device interface more efficient (e.g., by assisting the user in providing appropriate input when operating / interacting with the device and reducing user errors), and in addition, reducing the device's power consumption and improving battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, the representation of the first participant acts as an indication to the user that the captured photograph is to be sent to the first participant. In some embodiments, the representation of the first participant is not displayed before image data is captured using a camera (e.g., 602) (724). In some embodiments, the camera user interface (e.g., 615) includes camera-specific affordances (e.g., corresponding to filters, lighting options, timer options, etc., 619) that are displayed before image data is captured. In some embodiments, the displayed camera-specific affordances are replaced with representations of the first participant by displaying representations of the first participant.

[0257] An electronic device (e.g., 600) detects a third input directed to a transmit affordance (e.g., a touch gesture on a touchscreen display at a location corresponding to a transmit affordance 680) via one or more input devices (726).

[0258] Upon detecting a third input, the electronic device (e.g., 600) initiates processing (e.g., immediately transmits or presents the image data captured within the creation area before subsequent transmissions) (728) and transmits the captured image data to the first participant.

[0259] In some embodiments, an electronic device (e.g., 600) displays a completion affordance (e.g., a selectable icon associated with a function to close the camera user interface and display the messaging user interface) before detecting a third input directed to a transmit affordance (e.g., 680). In some embodiments, the electronic device detects the selection of a completion affordance via one or more input devices. In some embodiments, in response to detecting the selection of a completion affordance, the electronic device displays a messaging user interface (e.g., 603) having a message creation area (e.g., 608). In some embodiments, in response to detecting the selection of a completion affordance, the electronic device displays a representation of the captured image data in the message creation area. In some embodiments, upon selecting a completion affordance, the camera user interface is closed and the captured image data is displayed in the message creation field of the messaging user interface without transmitting the captured image data.

[0260] In some embodiments, the electronic device (e.g., 600) is in a first image capture mode (e.g., photo capture mode, video capture mode) before detecting a third user input. In some embodiments, the electronic device maintains the first image capture mode in response to detecting a third user input. In some embodiments, the electronic device can be configured (e.g., user-configured) to capture image data according to multiple modes (e.g., photo capture mode, video capture mode). In some embodiments, the selection of the image capture mode is persistent even when the electronic device transitions from a first camera mode (e.g., standard camera mode, non-effect camera mode) to a second camera mode (e.g., effect camera mode).

[0261] In some embodiments, initiating the process of sending captured image data to a first participant includes sending the captured image data to the first participant (e.g., without displaying the messaging user interface 615). In some embodiments, selecting a send affordance (e.g., 680) from the camera user interface immediately sends the captured image data to other participants in the message conversation without displaying an intermediate user interface or requesting further input from the user.

[0262] In some embodiments, initiating the process of sending the captured image data to a first participant includes redisplaying a messaging user interface (e.g., 615), which further includes a keyboard area (e.g., 612) and application menu affordances (e.g., 610, selectable icons associated with a function for displaying the application menu user interface). Redisplaying the messaging user interface as part of initiating the process of sending the captured image data to a first participant provides visual feedback that the captured image data is being transmitted via a message conversation. Providing the user with improved visual feedback improves the usability of the device, makes the user device interface more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and reducing user errors), and in addition reduces the power usage of the device and improves battery life by enabling the user to use the device more quickly and efficiently. In some embodiments, the application menu affordance is displayed adjacent to the message creation field in the messaging user interface. In some embodiments, an electronic device (e.g., 600) detects the selection of an application menu affordance via one or more input devices. In some embodiments, upon detecting the selection of an application menu affordance, the electronic device displays an application menu area adjacent to (e.g., above) the keyboard area, which has a plurality of application affordances (e.g., selectable icons, each associated with a function for starting the application associated with each application affordance). In some embodiments, the application affordances include sticker affordances and avatar affordances.

[0263] In some embodiments, an electronic device (e.g., 600) detects a fourth input on a messaging user interface (e.g., 615). In some embodiments, upon determining that the fourth input corresponds to a location in the keyboard area (e.g., 612) or a location in the message creation area (e.g., 608) within the messaging user interface, the electronic device stops displaying the application menu area and displays a text suggestion area (e.g., an area with a list of suggested words for the user to conveniently select) at the location in the messaging user interface previously occupied by the application menu area.

[0264] In some embodiments, an electronic device (e.g., 600) detects the selection of one of several application affordances (e.g., sticker affordances or avatar affordances) within an application menu area. In some embodiments, upon detecting the selection of an application affordance (e.g., 610), the electronic device stops displaying the keyboard area (e.g., 612) and displays an application display area in the location previously occupied by the keyboard area within the messaging user interface (e.g., 615), the application display area containing several graphical objects (e.g., avatars or stickers) corresponding to the selected application affordance. In some embodiments, the selected application affordance is a sticker affordance, and the graphical object displayed in the application display area is a sticker. In some embodiments, the selected application affordance is an avatar affordance, and the graphical object displayed in the application display area is an avatar (e.g., a customizable avatar and / or a non-customizable avatar).

[0265] It should be noted that the details of the processing described above with respect to Method 700 (for example, Figures 7A to 7B) can also be applied in a similar manner to the methods described below. For example, Methods 900, 1100, 1300, and 1500 optionally include one or more of the characteristics of the various methods described above with respect to Method 700. For example, visual effects such as stickers and virtual avatars are displayed in image data in the camera application user interface, the media user interface, and the user interface for live video communication sessions. For brevity, these details will not be repeated below.

[0266] Figures 8A to 8BQ show exemplary user interfaces for displaying visual effects in a camera application according to some embodiments. These user interfaces are used to illustrate the processes described below, including the processes shown in Figures 9A and 9B.

[0267] In Figure 8A, device 600 displays the home screen 800 and detects input 801 on camera application affordance 802.

[0268] In Figure 8B, upon detecting input 801, device 600 launches the camera application associated with camera application affordance 802 and displays the camera application user interface 815. The camera application user interface 815 is similar to the camera application user interface 615 described above with respect to the embodiments shown in Figures 6A to 6BQ.

[0269] The camera application user interface 815 includes an image display area 820 that displays a representation of image data, such as streaming image data representing objects located within the field of view of a camera (e.g., a rear camera or camera 602) (e.g., a live camera preview, a live camera recording, or a live video communication session), or media items such as photographs or video recordings. In the embodiment shown in Figure 8B, the image display area 820 shows a live camera preview 820-1' from the rear camera of device 600.

[0270] The camera application user interface 815 also includes an area above the image display area 820 that includes camera-specific affordances 817. The camera-specific affordances include affordance 817-1 associated with the camera flash function, affordance 817-2 associated with the camera mode function, affordance 817-3 associated with the timer function, and affordance 817-4 associated with the filter function.

[0271] The camera application user interface 815 also includes a camera option area 825 (similar to the camera option area 625) located below the image display area 820. The camera option area 825 includes a camera selector affordance 827 for switching between cameras (e.g., the rear camera and camera 602), and camera option affordances 819 associated with different capture modes in which the camera can record image data. For example, video affordance 819-1 is associated with a function for activating the camera's video recording capture mode, and photo affordance 819-2 is associated with a function for activating the camera's still image capture mode. In the embodiments discussed below with respect to Figures 8B to 8AQ, device 600 is in the still image capture mode for the operation associated with photo affordance 819-2. However, unless otherwise specified, these embodiments also apply to the video recording mode associated with video affordance 819-1.

[0272] The camera options area 825 further includes an effect affordance 822 for enabling or disabling a mode (visual effect mode, effect mode) of device 600, which enables or disables device 600 in order to display visual effects in the image display area 820. The effect affordance 822 is similar to the effect affordance 622 and therefore has the same function as the effect affordance 622 unless otherwise specified. Thus, the effect affordance 822 is selected to enable the display of visual effects and deselected to disable the display of visual effects.

[0273] The camera option area 825 also includes a capture affordance 821 that functions similarly to the capture affordance 621 described above. The capture affordance 821 can be selected to capture image data represented in the image display area 820. In some embodiments, the device 600 captures the image data in a manner based on the currently enabled capture option (e.g., video recording capture mode or image capture mode). In some embodiments, the device 600 captures image data depending on the type of gesture detected on the capture affordance 821. For example, if the device 600 detects a tap gesture on the capture affordance 821, the device 600 captures a still image of the image data represented in the image display area 820 at the time the tap gesture occurred. If the device 600 detects a tap-and-hold gesture on the capture affordance 821, the device 600 captures a video recording of the image data represented in the image display area 820 for the duration of the tap-and-hold gesture. In some embodiments, the video recording stops when the finger is lifted from the affordance. In some embodiments, video recording continues until a subsequent input (e.g., a tap input) is detected at a location corresponding to an affordance. In some embodiments, the captured image (e.g., a still image or video recording) can be shared with other devices, for example, using a messaging application.

[0274] In Figure 8C, device 600 detects input 823 on effect affordance 822.

[0275] In Figure 8D, in response to detecting input 823, device 600 activates camera 602 (e.g., switching from the rear camera), updates the image display area 820 to display a live camera preview 820-1 from camera 602, showing a representation of a subject 832 located within the field of view of camera 602, and a background 836 displayed behind the subject 832. As discussed herein, in some embodiments, image data captured using camera 602 includes depth data that can be used to determine the depth of objects within the field of view of camera 602. In some embodiments, device 600 analyzes objects (e.g., in the image data) based on the depth of these detected objects and uses this determination to apply the visual effects discussed herein. For example, device 600 may classify subject 832 as being in the foreground of live camera preview 820-1, and objects located behind the user as being in the background of live camera preview 820-1. These background objects are collectively referred to herein as background 836.

[0276] Device 600 also highlights effect affordance 822, indicating that visual effects are enabled for display, and updates the camera option area 825 by replacing camera option affordance 819 with visual effect option affordance 824. The visual effect option affordances include avatar effect affordance 824-1 and sticker effect affordance 824-2. Visual effect option affordance 824 is similar to the visual effect option affordance 624 described above. Visual effect option affordance 824 corresponds to different visual effects that can be applied to the image displayed in the image display area 820. By selecting one of the visual effect option affordances (e.g., 824-1 or 824-2), a menu is displayed along with the visual effect options corresponding to the selected visual effect option affordance.

[0277] In Figures 8E to 8F, device 600 detects input 826 on avatar effect affordance 824-1 and displays avatar options menu 828. Avatar options menu 828 is similar to avatar options menu 628. Avatar options menu 828 includes a scrollable list of avatar options 830 and a selection area 829 for showing one of the selected avatar options 830. The displayed avatar options include customizable female avatar 830-1, null avatar option 830-2, robot avatar option 830-3, rabbit avatar option 830-4, and a second customizable female avatar 830-5. As shown in Figure 8F, null avatar option 830-2 is selected as the default avatar option 830. As a result, the representation of the subject 832 displayed in the image display area 820 is shown without the avatar appearing on the subject's face. Avatar option 830 can be pre-configured as avatar option 830-3 or 830-4, or it can be a customizable avatar as avatar option 830-1 or 830-5.

[0278] Avatar option 830 has the same functionality as avatar option 630. Thus, avatar option 830 corresponds to a virtual avatar visual effect applied to the representation of a subject within the image display area 820. Specifically, each avatar option 830, when selected, corresponds to a virtual avatar that is transposed onto the face of the subject within the image display area, while other parts of the image within the image display area (such as the background or other parts of the user, such as the user's body) remain displayed. A user (e.g., subject 832) located within the field of view of camera 602 can control the visual appearance of the virtual avatar by changing the pose of their face (e.g., rotation or orientation), as described above.

[0279] The avatar options 830 can be scrolled by gestures on the avatar options menu 828. For example, as shown in Figure 8H, selecting robot avatar option 830-3 then displays the robot avatar 833 on the subject's face in the image display area 820, as shown by horizontal gesture 844a in Figure 8G. Additional avatar options 830 are scrolled on the displayed avatar options menu 828 by continuously scrolling via horizontal gesture 844b. In some embodiments, when the user scrolls through the avatar options 830, the image display area 820 displays an avatar (e.g., the robot avatar 833 in Figure 8J) that includes a blur effect 803 applied to the image data displayed behind the avatar.

[0280] In Figure 8K, device 600 displays a customizable female avatar option 830-5 placed within the selection area 829, and an editing affordance 804 displayed below the customizable female avatar option 830-5, just before selection (for example, before gesture 844b has finished). Device 600 continues to display a robot avatar 833 with a blur effect 803. In some embodiments, once a customizable avatar option is placed in the selection area 829, device 600 displays an editing affordance 804 in the selection area 829. The editing affordance 804 can be selected to display an avatar editing user interface, as discussed below with respect to Figure 8AA.

[0281] Figure 8L shows the state after avatar option 830-5 has been selected (e.g., gesture 844b has finished). Device 600 removes the robot avatar 833 from the image display area 820 (while continuing the blur effect 803) and displays an animation of a customizable female avatar 835 that emerges from avatar option 830-5 in the selection area 829 and moves to the image display area 820 (e.g., the center of the image display area 820, or a position corresponding to the subject's face), where the avatar 835 is modified based on detected changes to the subject's face. In some embodiments, the animation includes an avatar (e.g., 835) that starts with a static appearance (e.g., a sticker matching the displayed avatar option) and then transitions to a dynamic state (shown in Figure 8L) where the avatar (e.g., 835) is modified based on device 600 detecting changes to the user's face. In some embodiments, the transition of the avatar from static to dynamic occurs before the avatar moves out of the selection area. In some embodiments, the transition of the avatar from static to dynamic occurs when the avatar moves from the selection area 829 to the image display area 820.

[0282] As shown in Figures 8M and 8N, once the avatar 835 is applied to the subject's face, the device 600 modifies the avatar 835 based on detected changes in the subject's face, including changes in the position of the face within the field of view of the camera 602.

[0283] In some embodiments, when device 600 no longer detects the subject's face within the camera 602's field of view, device 600 reapplies a blur effect 803 (similar to blur effect 644) to the background and displays a prompt 838 instructing the user to bring the face back into the camera 602's field of view. In the embodiment shown in Figure 8P, device 600 displays a prompt 838 including a representation 840 of the selected avatar (e.g., a customizable female avatar) and a message 842 instructing the user to reposition the head within the camera 602's field of view. In some embodiments, displaying prompt 838 includes displaying an animation of the selected avatar (e.g., 835) returning to the center position of the image display area 820, and showing a slowed-down movement of the avatar and its features that appears to stop based on a physical model of the avatar features.

[0284] In some embodiments, when device 600 detects the user's face returning to the camera 602's field of view, device 600 displays an avatar 835 that moves from the center of the image display area 820 to the position of the user's face, and resumes modifying the avatar based on the detected change in the user's face.

[0285] In some embodiments, the avatar options menu 828 can be enlarged with a vertical gesture (e.g., 805) to display an enlarged version of the avatar options menu shown in Figures 8Q-8T. The enlarged version of the avatar options menu displays an even greater number of avatar options 830, including different types of avatars (e.g., customizable and non-customizable). A new avatar option (e.g., robot avatar option 830-3) can be selected in the enlarged avatar display menu. The device 600 then displays the corresponding avatar (e.g., robot avatar 833) on the user's face when the enlarged version of the avatar options menu returns to its original (e.g., condensed) state, as shown in Figure 8U.

[0286] In some embodiments, device 600 displays different avatars above the user's head in response to detecting a swipe gesture on the image display area 820. For example, in Figure 8V, device 600 detects a swipe gesture 806 that moves to the right across the image display area 820. In response to detecting the swipe gesture 806, device 600 scrolls through the avatar options 830 in the avatar options menu 828. In some embodiments, the size of the scroll is determined by the size of the swipe gesture. This allows the user to scroll through multiple avatars based on the size of the swipe gesture. In addition, the direction of the scroll and the corresponding movement of the avatars on the image display area 820 are determined based on the direction of the swipe gesture 806. Therefore, the scrolling of the avatar options and the corresponding movement of the transitioning avatars can be to the left if the swipe is to the left, or to the right if the swipe is to the right (as shown in Figures 8V and 8W).

[0287] As you begin scrolling through the avatar options 830, the currently selected avatar option (e.g., robot avatar option 830-3 in Figure 8V) leaves the selection area 829, and as scrolling continues, a new avatar option (e.g., avatar option 830-1 in Figure 8W) enters the selection area 829. As you scroll through the avatar options, device 600 also displays in the image display area 820 the animation of the currently selected avatar (e.g., robot avatar 833), where the next avatar (e.g., customizable avatar 831) enters the image display area 820 from the left and moves towards the user's face, while the next avatar exits the image display area 820 to the right side of the image display area 820, away from the user's face. In some embodiments, device 600 applies a blur effect 803 to the image display area 820 while the avatars are being scrolled across the image display area 820.

[0288] In the embodiments shown in Figures 8V and 8W, the null avatar option 830-2 is not selected, which better illustrates the transition from the currently selected avatar to the new avatar appearing in the image display area 820. However, it should be understood that the null avatar option 830-2 can be selected by a swipe gesture on the image display area 820. In such cases, the currently selected avatar is moved off-screen while the background is blurred, and then the blur effect is removed to display the user's face without an avatar.

[0289] Figures 8X and 8Y show the selection of different customizable avatar options (e.g., customizable avatar options 830-5) and the corresponding displayed customizable avatar 835. When customizable avatar option 830-5 is displayed in the selection area 829, an editing affordance 804 is displayed. The editing affordance 804 can be selected (e.g., via input 807) to display the avatar editing user interface 808 shown in Figure 8AA. The avatar editing user interface 808 provides an interface for editing the avatar (e.g., customizable avatar 835) corresponding to the customizable avatar option (e.g., 830-5) associated with the selected editing affordance 804.

[0290] In some embodiments, the applied visual effects may include lighting effects such as a shadow 851 cast on the neck of a subject below the applied custom avatar 835, or light reflections from glasses. As device 600 modifies the avatar 835 to reflect the user's real-time actions, device 600 also modifies the lighting effects projected onto the avatar 835 and the subject, including moving the display position of the reflected shadows based on the relative positions of the modeled light source and the avatar 835.

[0291] As shown in Figure 8AA, the avatar editing user interface 808 includes various avatar feature options 810 (e.g., skin tone option 810-1 and face shape option 810-2) that represent the representation of the customizable avatar 835, as well as the currently selected avatar feature options and available functional options that can be selected to modify avatar 835. The avatar feature options correspond to values ​​for aspects of the currently selected avatar feature, such as a specific avatar feature, specifically an avatar head feature indicated by the highlighted avatar head affordance 809-1. The avatar editing user interface 808 shows the selected skin tone option 810-1a and the selected face shape option 810-2a represented in avatar 835. The displayed avatar feature options can be changed by selecting different avatar features. For example, in Figure 8AB, when the avatar hair affordance 809-2 is selected, device 600 updates the avatar editing user interface 808 to display different avatar hair feature options (e.g., hair texture option 811-1 and hair color option 811-2).

[0292] Device 600 modifies avatar 835 when different avatar feature options are selected. For example, in Figure 8AB, device 600 detects the selection of straight hair texture option 811-2a and updates avatar 835 to have the selected straight hair texture option. In some embodiments, device 600 modifies the representation of avatar 835 shown in the avatar editing user interface 808 in response to detected changes in the user's face.

[0293] Upon detecting the selection of the completed affordance 812, device 600 exits the avatar editing user interface 808 and returns to the camera application user interface 815, where it displays the selected avatar options 830-5 and the corresponding avatar 835 based on the hair texture option selected in the avatar editing user interface.

[0294] Figures 8AD to 8AF illustrate an embodiment in which a newly customized avatar can be created and added to the avatar options menu 828. Upon detecting the selection of a new avatar affordance 813, device 600 displays the avatar editing user interface 808 having a representation of the default avatar that does not include the selected avatar feature options, as shown in Figure 8AE. Device 600 can modify the default avatar based on the user's selection of avatar feature options and save the avatar as a new avatar selectable from the avatar options menu 828. Therefore, in Figure 8AF, device 600 detects the selection of a cancel affordance and, as shown in Figure 8AG, refrains from saving the modified avatar to the avatar options menu 828.

[0295] Upon detecting the selection of the cancel icon 850 in Figure 8AG, device 600 closes the avatar options menu 828 and displays the camera options area 825 as shown in Figure 8AH.

[0296] In response to detecting the selection of sticker effect affordance 824-2, device 600 displays a sticker options menu 856 having a scrollable list of stickers 858 in Figure 8AI. The sticker options menu 856 and stickers 858 are similar to the sticker options menu 656 and stickers 658 described above. Stickers are static graphical objects that can be selected by the user and applied to an image in the image display area 820. In some embodiments, stickers can be selected by a tap gesture, and the corresponding sticker is then displayed in the position on the image display area 820. In some embodiments, stickers can be selected by a touch-and-drag gesture, in which the sticker is placed in the desired position by starting on the selected sticker, dragging the sticker to the desired position, and then ending the drag (e.g., lifting the finger). An example of such an embodiment is shown in Figures 8AI to 8AK, where helmet sticker 858-1 is selected (e.g., input 853) and dragged to a position above (e.g., far from) the custom avatar 835. In Figure 8AK, device 600 detects that the subject 832 moves laterally from the position shown in Figure 8AJ. In response, device 600 displays both the helmet sticker 858-1 and the avatar 835, which move laterally based on the lateral movement (e.g., direction and size) of the subject's representation.

[0297] In Figure 8AK, device 600 detects input 854 on close affordance 855. In response, device 600 closes the sticker options menu 856 to display the camera options area 825 in Figure 8AL. In Figure 8AL, device 600 indicates that subject 832 tilts its head and shifts back to the center of the screen. In response to detecting this movement of subject 832, device 600 displays helmet sticker 858-1 above the customizable avatar 835 in Figure 8AL, shifting it laterally (from its position shown in Figure 8AK) based on the movement of the subject's head towards the center of the screen. Device 600 displays helmet sticker 858-1 while maintaining a relative distance from avatar 835. However, as shown in Figure 8AL, helmet sticker 858-1 does not rotate (e.g., tilt) in response to the rotation (e.g., tilt) of the subject's head, while avatar 835 does rotate (e.g., tilt).

[0298] In Figure 8AL, device 600 detects the selection of a capture affordance 821 that causes device 600 to capture the image of the live camera preview 820-1 displayed in the image display area 820, including visual effects (e.g., a customized avatar 835 and helmet sticker 858-1), as well as other image data including the subject 832 and background 836.

[0299] In Figure 8AM, device 600 displays a camera application user interface 815 showing a captured image (e.g., media item 820-2) in the image display area 820 (for example, media item 820-2 replaces the live camera preview 820-1 shown in the image display area 820). Media item 820-2 is a representation of the live camera preview 820-1 at the time the capture affordance 821 is selected. Thus, media item 820-2 includes a helmet sticker 858-1 and an avatar 835 displayed on the face of the subject 832 and the background 836.

[0300] Device 600 also replaces camera-specific affordances (e.g., affordance 817 shown in Figure 8AL) with re-capture affordance 879 and saves affordance 818 for saving the captured media item 820-2.

[0301] Device 600 also updates the camera options area 825 to replace the capture affordance 821 and camera selector affordance 827 with the markup affordance 877, the edit affordance 878, and the share affordance 880. The markup affordance 877 allows the user to mark up the media item 820-2. The edit affordance 878 allows the user to edit the media item 820-2, for example, by cropping the image or adjusting other properties of the media item 820-2. The share affordance 880 allows the user to send the media item 820-2 to another device, for example, a messaging application or an email application.

[0302] Device 600 displays a camera options area 825, which includes visual effect option affordances 824. By selecting a visual effect option affordance 824, the corresponding option menu can be displayed, and the captured media item 820-2 (and the recorded video media item 820-4, described later) can be modified using the option menu. For example, Figures 8AN to 8AP show device 600 adding a sticker to media item 820-2. Figure 8AN shows the sticker effect affordance 824-2 being selected, which causes device 600 to display the sticker option menu 856 in Figure 8AO, and also shows the selection of a rabbit sticker 858-2 placed on the subject 832 in media item 820-2, as shown in Figure 8AP. The sticker option menu 856 is closed in Figure 8AP, and the camera options area 825 is displayed in Figure 8AQ.

[0303] In Figure 8AQ, device 600 also displays media item 820-2, which includes a rabbit sticker 858-2 placed on subject 832, along with other visual effects (e.g., avatar 835 and helmet sticker 858-1).

[0304] Stickers can be added to recorded media items in a video format in the same manner as described above for media item 820-2 (still image). For example, Figure 8AR shows an embodiment in which video media item 820-4 is a recorded video. Media item 820-4 can be created in the same manner as described above for media item 620-4.

[0305] In Figure 8AR, device 600 displays the recorded video media item 820-4 in the image display area 820. Device 600 also displays a camera options area 825 with a video scrubber 8120, effect affordance 822, editing affordance 878, markup affordance 877, and sharing affordance 880 for the recorded video media item 820-4. The camera options area 825 also includes a visual effects option affordance 824. Selecting the visual effects option affordance 824 can display the respective option menus that can be used to modify the captured media item 820-4. For example, Figures 8AR to 8AV show device 600 adding stickers 858-2 and 858-3 to the media item 820-4.

[0306] In some embodiments, the displayed stickers may have differently modeled behavior in relation to the video media item (or live video stream). For example, some stickers may have an appearance applied to a display (e.g., 601) and remain static as objects in the image data move. An example of such a sticker is shown by the heart sticker 858-3 in Figures 8AX-8BC. In some embodiments, if device 600 does not detect the presence of all or part of the representation of a subject (e.g., head or face) in the media item, live camera preview, or camera 602's field of view, the sticker will have this static behavior once it is placed on the media item (e.g., 820-4) or live camera preview (e.g., 820-1). For example, if the subject's face is completely off-screen (e.g., not detected in the camera 602's field of view) and the sticker is placed on the image display area 820, once the face is detected in the camera 602's field of view and displayed on the image display area 820 along with the previously placed sticker, the sticker will remain stationary and will not track any movement of the subject's face.

[0307] Other stickers, when applied to a display, have the appearance of moving in accordance with objects in the image (e.g., items within the camera's field of view, including representations of avatars or subjects). In some embodiments, the sticker is positioned away from the object it follows. An example of such a sticker is shown by the helmet sticker 858-1 in Figures 8AK and 8AX-8BC. The helmet sticker 858-1 follows the lateral movement of the subject 832 but does not follow the rotational movement of the subject. Another example of such a sticker is shown by the starfish sticker 858-5 in Figures 8BE-8BQ. When positioned away from the subject's face, the starfish sticker 858-5 follows the movement of the subject's face while maintaining its relative position to the face (e.g., the starfish sticker 858-5 follows the forward, backward, and lateral movement of the subject's face but does not follow the rotational movement of the subject's face).

[0308] However, other stickers have the appearance of being applied to an object within the field of view of camera 602 and move to follow the object in the field of view (for example, while the sticker has the appearance of having depth matched to the object in the image). An example of such a sticker is shown by the rabbit sticker 858-2 in Figures 8AX-8BC. The rabbit sticker 858-2 moves with the lateral and rotational movement of the subject (specifically, the subject's shoulders). Another example of such a sticker is shown by the glasses sticker 858-4 in Figures 8BG-8BM. When placed on the face of the subject in Figures 8BG-8BM, the glasses sticker 858-4 follows the lateral movement of the subject's face (e.g., left and right, up / down), the forward and backward movement of the subject's face, and the rotational movement of the subject's face. The glasses sticker 858-4 has the appearance of being physically attached (e.g., glued) to the subject's face.

[0309] In some embodiments, the behavior of a sticker changes based on its position relative to an object within a media item (or a live camera preview or camera field of view). In some embodiments, the behavior of a sticker changes in response to a detection of a change in the sticker's position relative to an object. Examples of such stickers are shown in Figures 8BE to 8BQ and described in more detail below.

[0310] Figures 8AW to 8BC show video playback and editing of media item 820-4, which includes avatar 835, helmet sticker 858-1, rabbit sticker 858-2, and heart sticker 858-3. As media item 820-4 progresses during playback, the heart sticker 858-3 remains stationary, while having the appearance of being attached to the display 601 and not moving. The helmet sticker 858-1 has the appearance of being attached to the display but moves (e.g., deforms) in response to changes in the position of avatar 835. The rabbit sticker 858-2 has the appearance of being attached to the shoulder of subject 832 in media item 820-4. As the shoulder moves, the rabbit sticker 858-2 changes its position, including orientation and display size, to give the appearance of following the subject's shoulder during playback of media item 820-4.

[0311] In some embodiments, stickers or other virtual objects applied to the position of a representation of the field of view of one or more cameras, including each object (e.g., the face, hand, or other body part of the device user), which is tracked in three dimensions (e.g., via depth information from a depth sensor), are attached to each object such that, in addition to moving laterally (e.g., left and right, and / or up and down), the size and / or orientation of the virtual object changes as the distance of each object from one or more cameras and / or the orientation of each object relative to one or more cameras changes. For example, as shown in Figures 8AS to 8AT, a rabbit sticker 858-2 is placed on a representation of the subject's shoulder. In Figures 8AX to 8AY, as the shoulder moves above and toward the camera, the rabbit sticker 858-2 moves laterally (up) with the shoulder and expands as the shoulder moves toward the camera. In some embodiments, stickers or other virtual objects applied to the position of a representation in the field of view of one or more cameras, which is tracked in three dimensions (e.g., via depth information from a depth sensor) and is at a distance (e.g., remote) from each object (e.g., the face, hand, or other body part of the device user), move laterally (e.g., left and right, and / or up and down) within the field of view of one or more cameras, without adhering to the object, so that the size and / or orientation of the virtual objects do not change even if the distance of each object from one or more cameras and / or the orientation of each object relative to one or more cameras changes. For example, as shown in Figures 8AX and 8AY, the helmet sticker 858-1 is positioned at a distance from the representation of the subject. In Figures 8AX and 8AY, as the representation of the subject moves laterally towards the camera, the helmet sticker 858-1 moves laterally based on the movement of the user's representation, but its orientation or size does not change.

[0312] In Figure 8AY, device 600 detects input 882 (e.g., a swipe or drag gesture) on the scrubber 8120 and scrubs media item 820-4 based on the input. As shown in Figure 8AZ, in response to input 882, device 600 inverts through media item 820-4 to display the portion of media item 820-4 previously shown in Figure 8AX. Device 600 detects input 884 on the editing affordance 878.

[0313] In Figure 8BA, device 600 enters editing mode (to edit media item 820-4) and changes the end time of the video scrubber by dragging the affordance 885 to the end of the video scrubber 8120 (via input 883) to change the end time of the video scrubber.

[0314] As shown in Figure 8BB, the video scrubber 8120 is trimmed according to the input 883. Device 600 detects input 886 on completion affordance 887 and exits editing mode.

[0315] In Figure 8BC, the video scrubber 8120 is shown trimmed (for example, shorter than before entering editing mode in Figure 8AZ). Device 600 detects input 888 on effect affordance 822 and exits effect mode.

[0316] As shown in Figure 8BD, the visual effect displayed on media item 820-4 is hidden from view when an input 888 is detected that deselects the effect affordance 822.

[0317] In some embodiments, the sticker may have behavior determined based on the conditions of its placement (e.g., the sticker's position relative to other objects, the presence (or absence) of objects when the sticker is placed, etc.). For example, in some embodiments, the sticker may exhibit a first type of behavior when placed away from an object or area, and a second type of behavior when placed on an object or area. In some embodiments, the sticker may exhibit a third type of behavior if no object is present when the sticker is placed. In some embodiments, the sticker's behavior may change based on changes in its placement (e.g., relative to an object).

[0318] For example, Figures 8BE to 8BQ show exemplary embodiments illustrating the behavior of various stickers displayed in the live camera preview 820-1, where the sticker behavior changes based on changes in the placement of the sticker relative to the subject's face. In the embodiments shown in Figures 8BE to 8BQ, the sticker behavior is determined based on the position of the sticker relative to the subject's face.

[0319] In Figure 8BE, device 600 displays a camera application user interface 815, which has a similar appearance to that shown in Figure 8D, but with stickers 858 displayed in an image display area 820. The image display area 820 shows a live camera preview 820-1 from camera 602, showing a representation of a subject 832 located within the field of view of camera 602, and a background 836 displayed behind the subject 832. The glasses sticker 858-4, the starfish sticker 858-5, and the baseball sticker 858-6 are positioned and displayed away from (e.g., not on) the face of subject 832. The stickers shown in Figures 8BE to 8BQ can be positioned and moved within the image display area according to various embodiments discussed herein.

[0320] If sticker 858 is positioned away from the subject's face (e.g., not on it), sticker 858 exhibits a first type of behavior. For example, the sticker follows the lateral (e.g., side to side and up / down), forward (e.g., towards camera 602), and backward (e.g., away from camera 602) movements of the subject's face, but does not follow the rotational movement of the subject's face (e.g., does not follow the pitch and yaw of the subject's face).

[0321] In Figure 8BF, device 600 detects an input 889 on the glasses sticker 858-4 (e.g., a touch-and-drag gesture on the display 601). In response to detecting the input 889, device 600 displays the glasses sticker 858-4 moving with the input (e.g., in accordance with the direction and magnitude of the movement of the input 889), as shown in Figures 8BG and 8BH.

[0322] In some embodiments, device 600 provides an indication when the sticker moves to a position corresponding to an object the sticker is following. For example, in Figure 8BG, device 600 indicates that the glasses sticker 858-4 is moving from a position away from the subject's face to the position of the subject's face. As the glasses sticker 858-4 moves to the subject's face, device 600 displays a bracket 890 around the subject's face and optionally generates haptic feedback 892 (e.g., haptic output with or without audible output). In some embodiments, the bracket 890 provides a visual representation of an area where the glasses sticker 858-4 can be positioned so that it will be displayed on the subject's face (resulting in the modified behavior of the glasses sticker 858-4). Thus, the bracket 890 (and haptic feedback 892) can be shown to the user while the user is dragging the sticker into the positioning area (e.g., inside the bracket) where the sticker will be placed on the subject's face. In some embodiments, haptic feedback 892 indicates that the glasses sticker 858-4 has moved from an area outside the subject's face (e.g., an area where the sticker exhibits a first type of behavior) to an area within the subject's face (e.g., an area where the sticker exhibits a second type of behavior).

[0323] In some embodiments, when device 600 detects that the sticker (or other visual effect) has moved to a position corresponding to the object it is following, device 600 modifies the appearance and behavior of the sticker based on the object's position (in some embodiments, the sticker's behavior is modified after detecting the end of input 889). Device 600 also modifies the appearance and behavior of the sticker in the reverse manner when the sticker moves from a position corresponding to the object it is following to a position away from the object.

[0324] In Figure 8BH, once device 600 detects the position of the glasses sticker 858-4 on the subject's face (e.g., located within bracket 890), device 600 modifies the sticker based on the subject's face position (e.g., size and orientation). For example, the glasses sticker 858-4 increases in size (e.g., to fit proportionally to the size of the subject's face) and rotates from its static position in Figure 8BG to match the position of the subject's face in Figure 8BH. In some embodiments, the sticker is positioned on a plane corresponding to the subject's nose, and its coordinates are transformed as the face moves. In some embodiments, a face mesh (e.g., a depth mask) is used to determine the nose point. In some embodiments, the sticker is positioned on a plane but does not conform to the shape of the subject's face.

[0325] Device 600 also changes the behavior of the stickers to a second type of behavior (e.g., a behavior different from the first type of behavior). As shown in Figure 8BH, the second type of behavior may include, in addition to lateral and forward / backward movement, behavior in which the stickers follow the rotational movement of the subject's face (e.g., follow the pitch and yaw of the face) to maintain the 3D position of the face. The behavior of the other stickers does not change. Thus, the baseball sticker 858-6 and the starfish sticker 858-5 continue to have the first type of behavior because they are not placed on the subject's face. Instead, the starfish sticker 858-5 and the baseball sticker 858-6 continue to maintain their relative distance from the face.

[0326] In some embodiments, the bracket 890 persists with the input 889, which is positioned on the subject's face (for example, within the bracket). Thus, when device 600 detects the end of the input 889, device 600 stops displaying the bracket 890, as shown in Figure 8BI.

[0327] Figures 8BI and 8BJ show device 600 rotating the glasses sticker 858-4 along with the subject's face, while the starfish sticker 858-5 and baseball sticker 858-6 remain stationary (unlike the glasses sticker 858-4, they do not follow the rotational movement of the subject's face).

[0328] In Figure 8BK, the subject moves closer to camera 602. The glasses sticker 858-4 moves with the forward movement of the subject's face, maintaining its rotational position relative to the subject's face. Similarly, the baseball sticker 858-6 and the starfish sticker 858-5 move with the forward movement of the subject's face while maintaining their relative distance from the subject's face. The baseball sticker 858-6 and the starfish sticker 858-5 do not rotate with the subject's face.

[0329] In Figure 8BK, device 600 detects input 893 on capture affordance 821 and, accordingly, captures an image of live camera preview 820-1.

[0330] In Figure 8BL, device 600 displays a camera application user interface 815 showing a captured image (e.g., media item 820-5) in the image display area 820 (e.g., media item 820-5 replaces the live camera preview 820-1 shown in the image display area 820). Media item 820-5 is a representation of the live camera preview 820-1 at the time the capture affordance 821 is selected. The embodiment shown in Figure 8BL is similar to that shown in Figure 8AM, except that it has media item 820-5 (instead of media item 820-2), and media item 820-5 includes glasses sticker 858-4, starfish sticker 858-5, and baseball sticker 858-6 which are displayed against the face of the subject 832 and the background 836.

[0331] As discussed herein, the display of visual effects is similar across different embodiments. For example, unless otherwise specified, visual effects can be displayed and manipulated in a similar manner in camera applications, messaging applications, avatar editing applications, live video messaging applications, or any other applications discussed herein. In addition, visual effects can be displayed and manipulated in a similar manner across different types of image data. For example, unless otherwise specified, visual effects can be displayed and manipulated in a similar manner in live camera previews, media items, streamed image data, or any other image data discussed herein. For example, Figures 8BL to 8BQ show how to display stickers on media item 820-5 in a similar manner to that described above for live camera preview 820-1. Specifically, Figures 8BL to 8BQ show how stickers within a media item (e.g., 820-5) can be rearranged and their respective behaviors modified accordingly in a similar manner to that described above for live image preview 820-1.

[0332] In Figure 8BM, device 600 detects input 894 on the glasses sticker 858-4 and displays bracket 890. In Figure 8BN, input 894 causes the glasses sticker 858-4 to move away from the subject's face. When the glasses sticker is moved away from the face, device 600 generates haptic feedback 892, stops displaying bracket 890, and returns the glasses sticker 858-4 to its original shape (for example, a shape slightly tilted from when the glasses sticker was placed away from the subject's face in Figure 8BE). Device 600 also returns the behavior of the glasses sticker 858-4 to a first type of behavior (the behavior associated with a sticker placed away from the object the sticker is tracking) and modifies the appearance of the sticker based on the modified behavior. Thus, the eyeglasses sticker 858-4 is displayed having a larger size in Figures 8BN to 8BQ (compared to the size of the eyeglasses sticker 858-4 in Figures 8BE to 8BG), and, similar to the size changes of the starfish sticker 858-5 and the baseball sticker 858-6, maintains the relative spacing of the eyeglasses sticker 858-4 to the position of the subject's face within media item 820-5 based on the forward movement of the subject's face in Figures 8BJ to 8BK.

[0333] Figures 8BO to 8BQ illustrate the placement of different stickers on a subject's face within media item 820-5, and how the different stickers change based on their different behaviors as they are moved onto the subject's face. For example, in Figure 8BO, device 600 detects input 895 on baseball sticker 858-6. When baseball sticker 858-6 is dragged onto the subject's face in Figure 8BP, device 600 displays bracket 890, generates haptic feedback 892 (e.g., haptic output), and repositions baseball sticker 858-6 based on the angle of the subject's face. In other words, the behavior of baseball sticker 858-6 changes to follow the rotational movement of the subject's face, similar to how glasses sticker 858-4 follows the rotational movement of the subject's face when placed on the subject's face.

[0334] In Figure 8BQ, input 895 ends, and device 600 stops displaying bracket 890.

[0335] Figures 9A and 9B are flowcharts illustrating a method for displaying visual effects in a camera application using an electronic device, according to some embodiments. Method 900 is performed on a device (e.g., 100, 300, 500, 600) equipped with a camera and a display device. Some operations in Method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0336] As described below, Method 900 provides an intuitive way to display visual effects in a camera application. Such a method reduces the cognitive burden on the user to apply visual effects to images viewed in a camera application, thereby creating a more efficient human-machine interface. In battery-powered computing devices, power is saved and battery intervals are increased by allowing users to display visual effects on images more quickly and efficiently.

[0337] An electronic device (e.g., 600) displays a camera user interface (e.g., 815) via a display device (e.g., 601) (902). The camera user interface includes a camera display area (e.g., 820) (904) which includes a representation (e.g., 835) of image data captured via a camera (e.g., 602).

[0338] In some embodiments, the image data includes depth data (image data including the depth aspect of the captured image or video (e.g., depth data independent of RGB data)) (906). In some embodiments, the image data includes at least two components: RGB components that encode the visual characteristics of the captured image, and depth data that encodes information about the relative spatial relationships of elements in the captured image (e.g., depth data encodes that the user is in the foreground, and background elements such as trees located behind the user are in the background). In some embodiments, the depth data is a depth map. In some embodiments, the depth map (e.g., a depth map image) includes information (e.g., values) about the distance of objects in the scene from a viewpoint (e.g., a camera). In one embodiment of the depth map, each depth pixel defines the position on the Z-axis of the viewpoint where its corresponding two-dimensional pixel is located. In some examples, the depth map consists of pixels where each pixel is defined by a value (e.g., 0 to 255). For example, a value of "0" represents the furthest pixel in the "3D" scene, and a value of "255" represents the pixel closest to the viewpoint (e.g., the camera) in the "3D" scene. In other examples, the depth map represents the distance between objects in the scene and the viewpoint plane. In some embodiments, the depth map includes information about the relative depth of various features of the object of interest at the viewpoint of the depth camera (e.g., the relative depth of the eyes, nose, mouth, and ears of the user's face). In some embodiments, the depth map includes information that allows the device to determine the contour of the object of interest in the z direction.

[0339] In some embodiments, the depth data has a first depth component (e.g., a first portion of the depth data encoding the spatial position of the subject in the camera display area, a plurality of depth pixels forming a discrete portion of the depth map, such as the foreground or a specific object) which includes a representation of the subject within the camera display area (e.g., 820). In some embodiments, the depth data has a second depth component (e.g., a second portion of the depth data encoding the spatial position of the background within the camera display area, a plurality of depth pixels forming a discrete portion of the depth map, such as the background) which is separate from the first depth component, and the second depth aspect includes a representation of the background in the camera display area. In some embodiments, the first and second depth aspects are used to determine the spatial relationship between the subject in the camera display area and the background within the camera display area. This spatial relationship can be used to distinguish the subject from the background. This identification can be used, for example, to apply different visual effects (e.g., visual effects with depth components) to the subject and the background. In some embodiments, all areas of image data that do not correspond to the first depth component (e.g., areas of image data that are outside the range of the depth camera) are segmented (e.g., excluded) from the depth map.

[0340] In some embodiments, the representation (e.g., 835) of image data captured via a camera (e.g., 602) is a live camera preview (e.g., a stream of image data representing what is in the camera's field of view).

[0341] In some embodiments, while the first camera display mode is active, an electronic device (e.g., 600) detects a swipe gesture on the camera display area (e.g., 820). In some embodiments, in response to the detection of a swipe gesture on the camera display area, the electronic device (e.g., 600) changes the appearance of the displayed representation of the selected avatar option in the camera display area from a first appearance (e.g., an appearance based on the currently selected avatar option) to a second appearance (e.g., an appearance based on a different avatar option (e.g., a null avatar option, or an avatar option corresponding to a different avatar, including a different type of avatar (e.g., a customizable, non-customizable avatar)) where the second appearance corresponds to one of several avatar options (e.g., a different avatar option included in the avatar selection area). By changing the appearance of the displayed representation of the selected avatar option in response to the detection of a swipe gesture on the camera display area, a quick and easy way is provided for the user to change the representation of the selected avatar. The additional controls improve device usability by providing extra control options without cluttering the user interface (UI), making the user device interface more efficient (for example, by helping users provide appropriate input when operating / interacting with the device and reducing user errors), and also reduce device power consumption and improve battery life by enabling users to use the device more quickly and efficiently.

[0342] In some embodiments, if one of several avatar options is a null avatar option, the device (e.g., 600) stops displaying the avatar representation on the representation of the subject (e.g., the device stops replacing the user's head image data with a virtual avatar). In some embodiments, if one of several avatar options is an avatar option of a different avatar character (including customizable or non-customizable avatar characters), the device replaces the selected avatar character with the different avatar character (e.g., the device replaces the avatar representation with the representation of the different avatar). In some embodiments, replacing the selected avatar character with a different avatar character includes displaying an animation of the different avatar character moving to the center of the screen. In some embodiments, replacing the selected avatar character with a different avatar character includes displaying an animation of the different avatar character moving to the user's head. In some embodiments, replacing the selected avatar character with a different avatar character includes blurring the background while the selected avatar is being replaced. When a selected avatar character is replaced by a different avatar character, visual feedback is provided that the avatar character is changing by displaying animations (e.g., the different avatar character moves to the center of the screen, and the different avatar character moves to the user's head while blurring the background). By providing the user with improved visual feedback, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently.In some embodiments, the currently selected avatar option corresponds to a first type of avatar (e.g., a customizable avatar), and a different avatar option corresponds to a second type of avatar (e.g., a non-customizable avatar).

[0343] In some embodiments, changing the appearance of the displayed representation of a selected avatar option within the camera display area (e.g., 820) from a first appearance to a second appearance includes moving the first version of the representation of the selected avatar option, having the first appearance, out of the display. In some embodiments, changing the appearance of the displayed representation of a selected avatar option within the camera display area from a first appearance to a second appearance includes substantially moving the second version of the representation of the selected avatar option, having the second appearance, to the center of the display. By moving the first version of the representation of the selected avatar option out of the display and substantially moving the second version of the representation of the selected avatar option to the center of the display, visual feedback is provided that the first version has been replaced by the second version. By providing the user with improved visual feedback, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and by reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently.

[0344] In some embodiments, changing the appearance of the displayed representation of a selected avatar option from a first appearance to a second appearance includes moving the first version of the selected avatar representation, which has the first appearance, out of the display. In some embodiments, changing the appearance of the displayed representation of a selected avatar option from a first appearance to a second appearance includes moving the second version of the selected avatar representation, which has the second appearance, to a substantially different position from the representation of the subject displayed in the camera display area (e.g., 820). By moving the first version of the selected avatar representation out of the display and substantially moving the second version of the selected avatar representation to the position of the representation of the subject displayed in the camera display area, visual feedback is provided that the first version has been replaced by the second version. By providing the user with improved visual feedback, the usability of the device is improved, the user device interface is made more efficient (e.g., by helping the user provide appropriate input when operating / interacting with the device and by reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently.

[0345] In some embodiments, changing the appearance of the displayed representation of a selected avatar option from a first appearance to a second appearance includes changing the visual appearance of the background displayed in the camera display area (e.g., 820) (e.g., blurring the background, reducing the saturation of the background).

[0346] The camera user interface (908) also includes a first affordance (e.g., an affordance corresponding to a virtual avatar) associated with a first camera display mode (e.g., a mode that replaces image data of the user's head with a virtual avatar).

[0347] In some embodiments, the camera user interface (e.g., 815) further includes sticker affordances (e.g., 824-2, affordances corresponding to functions for enabling the display of stickers) associated with sticker display modes (e.g., modes that allow stickers to be applied to image data). In some embodiments, while displaying image data (and optionally, representations of selected avatar options) in the camera display area (e.g., 820), an electronic device (e.g., 600) detects a gesture directed towards a sticker affordance (e.g., Figure 8AH). In some embodiments, upon detection of a gesture directed towards a sticker affordance, the electronic device activates a sticker display mode, and activating the sticker display mode includes displaying a sticker selection area (e.g., 856) containing a plurality of sticker options (e.g., stickers of 858), detecting a selection of one of the plurality of sticker options (e.g., stickers of 858-2) in the sticker selection area, and, upon detection of the selection, displaying a representation of the selected sticker option (e.g., 858-2) on the image data in the camera display area. In some embodiments, the selected sticker options displayed on the image data have an appearance similar to that of a user placing a physical sticker on a display screen (e.g., 601), and do not move or interact with objects represented in the image data. In some embodiments, the selected sticker options displayed on the image data have an appearance similar to that of a user placing a physical sticker on a display screen (e.g., a user placing a physical sticker on a device screen), but move based on the movement of objects displayed in the image data. For example, the sticker appears to be placed on the display and interacts with objects represented in the image data (e.g., a person), but the sticker's movement is limited to movement along the x and y axes.In other words, the sticker appears to be applied to the device screen but can be moved across the screen by an object represented in the image data (for example, it appears as if a person in the image data is touching the sticker or dragging the sticker across the screen). In some embodiments, the selected sticker option displayed on the image data has the appearance of being inserted into the image data as an interactive object that forms part of the image data. By providing the appearance of being inserted into the image data as an interactive object that forms part of the image data, visual feedback is provided that the selected sticker can act as an interactive object in the image. By providing the user with improved visual feedback, the usability of the device is improved, the user device interface is made more efficient (for example, by helping the user to provide appropriate input when operating / interacting with the device and by reducing user errors), and in addition, the power consumption of the device is reduced and battery life is improved by enabling the user to use the device more quickly and efficiently. In such embodiments, the sticker appears to move based on interaction with an object represented in the image data. For example, the sticker may appear as if the object is sitting on a person's shoulder. As the person moves, the sticker moves with the person's shoulder, maintaining the appearance of sitting on the person's shoulder. This includes movement along the x and y axes, as well as movement along the z axis.

[0348] In some embodiments, while displaying representations of selected sticker options (e.g., 858-1, 858-2, 858-3, 858-4, 858-5, 858-6) on image data in a camera display area (e.g., 820), the device (e.g., 600) detects lateral movement of a subject (e.g., 832) within the field of view of one or more cameras (e.g., 602). In response to detecting lateral movement of a subject within the field of view of one or more cameras, the device moves the representation of the selected sticker option laterally (e.g., regardless of the sticker's relationship to the subject) according to the movement of the subject within the field of view of one or more cameras (see, for example, helmet sticker 858-1 in Figures 8AV to 8AY). By moving the representation of the selected sticker option laterally according to the lateral movement of a subject within the field of view of one or more cameras, visual feedback is provided that the selected sticker can act as an interactive object in the image. By providing users with improved visual feedback, device usability is enhanced, the user device interface becomes more efficient (for example, by helping users provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, power consumption of the device is reduced and battery life is improved by enabling users to use the device more quickly and efficiently.

[0349] In some embodiments, while displaying a representation of a selected sticker option (e.g., 858-1, 858-2, 858-3, 858-4, 858-5, 858-6) on image data in a camera display area (e.g., 820), the device (e.g., 600) detects the rotation of a subject (e.g., 832) within the field of view of one or more cameras (e.g., 602) (e.g., rotation around an axis perpendicular to the display, e.g., a subject rotating its head). Depending on the detection of the rotation of a subject within the field of view of one or more cameras, the device performs one or more of the following steps: Upon determining that the representation of the selected sticker option has a first relationship with respect to the subject (i.e., was positioned in a first relationship) (e.g., the sticker was initially (or currently) positioned on the display corresponding to the subject, e.g., the sticker is positioned on the representation of the subject's face or on another designated area (e.g., a bracket (e.g., 890))), the device rotates the representation of the selected sticker option according to the magnitude and orientation of the subject's rotation (e.g., the sticker rotates and twists to follow the pitch and yaw of the subject's face) (e.g., see the glasses sticker 858-4 in Figures 8BH-8BM). If the device determines that the representation of the selected sticker option does not have a first relationship with the subject (e.g., the sticker was initially (or is currently) placed in a position on the display away from the subject, e.g., the sticker is placed outside the representation of the subject's face or other designated area), the device stops rotating the representation of the selected sticker option according to the magnitude and orientation of the subject's rotation (see, for example, the starfish sticker 858-5 and the baseball sticker 858-6 in Figures 8BE-8BN). By selectively rotating the representation of the selected sticker option based on whether the representation of the selected st...

Claims

1. It is a method, In an electronic device having a camera, a display device, and one or more input devices, The display device shows a messaging user interface, including camera affordances, for a message conversation involving at least a first participant. To detect a first input directed to the camera affordance via one or more input devices, In response to detecting the first input, a camera user interface including capture affordances is displayed, Detecting a second input directed to the capture affordance via one or more input devices, In response to detecting the second input, The camera is used to capture image data, To stop displaying the aforementioned capture affordances, Displaying the transmit affordance at the location in the camera user interface previously occupied by the capture affordance, Detecting a third input directed to the transmission affordance via one or more input devices, Upon detecting the third input, the process of transmitting the captured image data to the first participant is initiated. Methods that include...

2. While the transmission affordance is being displayed, the representation of the first participant is displayed. The method according to claim 1, further comprising:

3. The method according to claim 2, wherein the representation of the first participant is not displayed before the image data is captured using the camera.

4. The camera user interface further includes effect mode affordances associated with modes in which visual effects are enabled for display on the captured image data, and the method The selection of the effect mode affordance is detected via one or more input devices, In response to detecting the selection of the effect mode affordance, the electronic device is transitioned from a first camera mode to a second camera mode different from the first camera mode. The method according to any one of claims 1 to 3, further comprising:

5. The aforementioned method, In response to detecting the selection of the aforementioned effect mode affordance, To stop the display of one or more camera mode affordances, Displaying multiple effect options at a location in the camera user interface previously occupied by one or more camera mode affordances, The method according to claim 4, further comprising:

6. The selection of a first of the effect option affordances is detected via one or more input devices. In response to detecting the selection of the first of the aforementioned effect option affordances, To stop displaying the aforementioned multiple effect option affordances, Displaying multiple selectable graphic icons, The method according to claim 5, further comprising:

7. The selection of the second of the effect option affordances is detected via one or more input devices. In response to detecting the selection of the second of the aforementioned effect option affordances, To stop displaying the aforementioned multiple effect option affordances, Displaying an avatar selection area with multiple avatar affordances, The method according to claim 5 or 6, further comprising:

8. Detecting a swipe input on the avatar selection area via one or more of the aforementioned input devices, In response to detecting the swipe input on the avatar selection area, The size of the avatar selection area is increased, and the plurality of avatar affordances arranged in a matrix are displayed. The method according to claim 7, further comprising:

9. The camera user interface further includes a first representation of image data, and the method is In response to detecting the selection of the aforementioned effect mode affordance, In accordance with the determination that the first representation of the image data corresponds to the image data acquired from the second camera, To stop displaying the first representation of the aforementioned image data, To display a second representation of the image data corresponding to the image data acquired from the aforementioned camera, The method according to claim 4, further comprising:

10. After capturing image data using the aforementioned camera, the markup affordances, editing affordances, and reshoot affordances are displayed. While the markup affordance, editing affordance, and reshoot affordance are being displayed, a fourth user input is received via one or more input devices. In response to detecting the fourth user input, The process for editing the captured image data is initiated according to the fourth user input corresponding to the editing affordance. The process for marking up the captured image data is initiated according to the fourth user input corresponding to the markup affordance. The process for re-capturing the captured image data is initiated according to the fourth user input corresponding to the re-capture affordance. The method according to any one of claims 1 to 9, further comprising:

11. The aforementioned method, The electronic device receives a request to transition to the first camera mode while it is in the second camera mode, Upon receiving the request to transition to the first camera mode, Transitioning the electronic device from the second camera mode to the first camera mode, In accordance with the fact that the first visual effect is active, the first visual effect is deactivated, The method according to claim 4, further comprising:

12. After deactivating the first visual effect, To detect the subsequent selection of the effect mode affordance via one or more input devices, In response to detecting the subsequent selection of the effect mode affordance, The first visual effect is reactivated in accordance with the determination that a subsequent selection of the effect mode affordance occurs within a predetermined time after the first visual effect has been deactivated. The method according to claim 11, further comprising:

13. Initiating the process of transmitting the captured image data to the first participant includes redisplaying the messaging user interface, further including the keyboard area and application menu affordances, and the method The selection of the application menu affordance is detected via one or more input devices, In response to detecting the selection of the aforementioned application menu affordance, an application menu area having multiple application affordances adjacent to the keyboard area is displayed. The method according to any one of claims 1 to 12, further comprising:

14. To detect a fourth input on the messaging user interface, In accordance with the determination that the fourth input corresponds to the position of the keyboard area or the position of the message creation area in the messaging user interface, To stop the display of the aforementioned application menu area, Display a text suggestion area in the location within the messaging user interface that was previously occupied by the application menu area. The method according to claim 13, further comprising:

15. To detect the selection of one of the multiple application affordances in the application menu area, In response to detecting the selection of the aforementioned application affordance, To stop the display of the aforementioned keyboard area, Display an application display area containing multiple graphical objects corresponding to the selected application affordance in the location within the messaging user interface previously occupied by the keyboard area, The method according to claim 13, further comprising:

16. The method according to any one of claims 1 to 15, wherein initiating the process of transmitting the captured image data to the first participant includes transmitting the captured image data to the first participant.

17. Before detecting the third input directed to the transmission affordance, Displaying completion affordances, The selection of the completed affordance is detected via one or more input devices, In response to detecting the selection of the completed affordance, Displaying the messaging user interface having a message creation area, The captured image data is displayed within the message creation area. The method according to any one of claims 1 to 16, further comprising:

18. Before detecting the third user input, the electronic device is in a first image capture mode, and the method is In response to detecting the third user input, the first image capture mode is maintained. The method according to claim 4, further comprising:

19. Capturing the aforementioned image data is The image data is captured in a first image capture mode according to the value of the second input characteristic that satisfies the first capture mode criterion, The image data is captured in a second image capture mode according to the value of the characteristic of the second input that satisfies the second capture mode criterion, The method according to any one of claims 1 to 18, including the method described in any one of claims 1 to 18.

20. The camera user interface further includes a camera display area that includes a representation of a live preview of the camera's field of view while the capture affordance is being displayed, and the method The selected avatar's representation is displayed on the representation of the subject within the camera's field of view while the subject's representation and background are displayed within the camera's display area. While the representation of the selected avatar is being displayed on the representation of the subject within the camera display area, a request to display the avatar selection area is received. Upon receiving the request to display the avatar selection area, To stop displaying the aforementioned capture affordances, Displaying an avatar selection area with multiple avatar affordances, The method according to any one of claims 1 to 19, further comprising:

21. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device comprising a camera, a display device, and one or more input devices, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 20.

22. It is an electronic device, Camera and, Display device and One or more input devices, One or more processors, An electronic device comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 1 to 20.

23. It is an electronic device, Camera and, Display device and One or more input devices, Means for carrying out the method described in any one of claims 1 to 20, An electronic device equipped with the following features.

24. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device comprising a camera, a display device, and one or more input devices, wherein the one or more programs are: The display device shows a messaging user interface, including camera affordances, for a message conversation involving at least a first participant. The first input directed to the camera affordance is detected via one or more input devices. Upon detecting the first input, a camera user interface including capture affordances is displayed. A second input directed to the capture affordance is detected via one or more of the aforementioned input devices. In response to detecting the second input, Using the aforementioned camera, image data is captured, The display of the aforementioned capture affordance is stopped, The transmission affordance is displayed at the location in the camera user interface that was previously occupied by the capture affordance. A third input directed to the transmission affordance is detected via one or more of the aforementioned input devices. A non-temporary computer-readable storage medium including an instruction that, upon detection of the third input, initiates a process of transmitting the captured image data to the first participant.

25. It is an electronic device, Camera and, Display device and One or more input devices, One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The display device shows a messaging user interface, including camera affordances, for a message conversation involving at least a first participant. The first input directed to the camera affordance is detected via one or more input devices. Upon detecting the first input, a camera user interface including capture affordances is displayed. A second input directed to the capture affordance is detected via one or more of the aforementioned input devices. In response to detecting the second input, Using the aforementioned camera, image data is captured, The display of the aforementioned capture affordance is stopped, The transmission affordance is displayed at the location in the camera user interface that was previously occupied by the capture affordance. A third input directed to the transmission affordance is detected via one or more of the aforementioned input devices. An electronic device including a command that, upon detecting the third input, initiates a process of transmitting the captured image data to the first participant.

26. It is an electronic device, Camera and, Display device and One or more input devices, A means for displaying a messaging user interface, including camera affordances, of a message conversation including at least a first participant, via the display device, means for detecting a first input directed to the camera affordance via one or more input devices, In response to the detection of the first input, means for displaying a camera user interface including capture affordances, means for detecting a second input directed to the capture affordance via one or more input devices, In response to detecting the second input, Using the aforementioned camera, image data is captured, The display of the aforementioned capture affordance is stopped, means for displaying a transmission affordance at a location in the camera user interface previously occupied by the capture affordance, means for detecting a third input directed to the transmission affordance via one or more input devices, In response to detecting the third input, means for initiating a process to transmit the captured image data to the first participant, An electronic device equipped with the following features.

27. It is a method, In an electronic device having a camera and a display device, The camera user interface is displayed via the aforementioned display device, and the camera user interface is A camera display area including a representation of image data captured via the aforementioned camera, Including a first affordance associated with a first camera display mode, The process involves detecting a gesture directed towards the first affordance while the subject is located within the camera's field of view and the subject's appearance and background are displayed in the camera's display area, Activating the first camera display mode in response to detecting the gesture directed toward the first affordance, wherein activating the first camera display mode means To display an avatar selection area containing one of the selected avatar options, This includes displaying the representation of the selected avatar option on the representation of the subject within the camera display area. While the first camera display mode is active, the change in the posture of the subject is detected, In response to detecting the change in the posture of the subject, the display appearance of the selected avatar option is modified based on the detected change in the posture of the subject, while maintaining the display of the background. Methods that include...

28. The aforementioned image data includes depth data, The method according to claim 27, wherein the depth data has a first depth including the representation of the subject within the camera display area.

29. Activating the first camera display mode means Before displaying the selected avatar option on the representation of the subject in the camera display area, Displaying the selected avatar option on the subject's representation in the camera display area, The method according to any one of claims 27 or 28, further comprising:

30. The avatar selection area further includes an option to stop displaying the representation of the selected avatar option on the representation of the subject in the camera display, and the method Receiving user input corresponding to the selection of an option to stop displaying the representation of the selected avatar option on the representation of the subject in the camera display area, In response to receiving user input corresponding to the selection of an option to stop displaying the representation of the selected avatar option on the representation of the subject in the camera display area, the display of the representation of the selected avatar option on the representation of the subject in the camera display area is stopped. The method according to any one of claims 27 to 29, further comprising:

31. Detecting a horizontal swipe gesture on the avatar selection area, In response to detecting the aforementioned horizontal swipe gesture, the system displays avatar creation affordances associated with the function for adding a new avatar option to the multiple avatar options. The method according to any one of claims 27 to 30, further comprising:

32. While the first camera display mode is active, a swipe gesture on the avatar selection area is detected, In response to detecting the swipe gesture on the avatar selection area, the appearance of the displayed representation of the selected avatar option in the camera display area is changed from a first appearance to a second appearance corresponding to one of the multiple avatar options. The method according to any one of claims 27 to 31, further comprising:

33. The method according to any one of claims 27 to 32, wherein the representation of the image data captured via the camera is a live camera preview.

34. While the first camera display mode is active, a swipe gesture on the camera display area is detected, In response to detecting the swipe gesture on the camera display area, the appearance of the displayed representation of the selected avatar option within the camera display area is changed from a first appearance to a second appearance corresponding to one of the multiple avatar options. The method according to claim 33, further comprising:

35. Changing the appearance of the displayed representation of the selected avatar option within the camera display area from the first appearance to the second appearance is: To move and remove from the display a first version having the first appearance of the representation of the selected avatar option, To substantially move the second version having the second appearance of the representation of the selected avatar option to the center of the display, The method according to claim 34, which includes the method described in claim 34.

36. Changing the appearance of the displayed representation of the selected avatar option from the first appearance to the second appearance is: Moving and removing the first version having the first appearance of the selected avatar's representation from the display, Substantially moving the second version having the second appearance of the representation of the selected avatar option to the position of the representation of the subject displayed in the camera display area, The method according to claim 34, which includes the method described in claim 34.

37. The method according to claim 34, wherein changing the appearance of the displayed representation of the selected avatar option from the first appearance to the second appearance includes changing the visual appearance of the background displayed in the camera display area.

38. Activating the first camera display mode means The selected avatar options are displayed in the avatar selection area with a static appearance. The selected avatar option is updated to have a dynamic appearance that changes based on the detected change in the subject's posture. Displaying an animation of the selected avatar having the dynamic appearance that moves from the avatar selection area to the representation of the subject in the camera display area, The method according to any one of claims 27 to 37, further comprising:

39. The method of claim 38, wherein updating the selected avatar option to have a dynamic appearance that changes based on the detected change in the posture of the subject includes first displaying the avatar option having a dynamic appearance together with an initial posture corresponding to the posture of the avatar option having a static appearance, before changing the appearance of the avatar option based on the detected change in the posture of the subject.

40. While the first camera display mode is active, In response to the determination that the subject is no longer located within the camera's field of view, the animation of the selected avatar option's representation is displayed as it moves to the center position within the camera's display area. The method according to any one of claims 27 to 39, further comprising:

41. While the first camera display mode is active, The method according to any one of claims 27 to 40, further comprising changing the visual appearance of the background displayed in the camera display area in response to a determination that the subject is no longer located within the field of view of the camera.

42. The camera user interface further includes a second affordance associated with a second camera display mode, and the method is The process involves detecting a gesture directed towards the second affordance while the subject is located within the camera's field of view and the representation of the subject and its background are displayed in the camera's display area. Activating the second camera display mode in response to the detection of the gesture directed toward the second affordance, wherein activating the second camera display mode means This includes displaying a visual effect selection area containing multiple graphical objects, The method according to any one of claims 27 to 41, further comprising:

43. While the second camera display mode is active, the selection of one of the plurality of graphical objects within the visual effect selection area is detected, The method according to claim 42, further comprising: displaying a representation of the selected graphical object within the camera display area in response to the detection of the selection.

44. The representation of the image data captured via the camera is a media item, the camera user interface further includes a third affordance associated with a third camera display mode, and the method is To detect the gesture directed towards the third affordance, Activating the third camera display mode in response to detecting the gesture directed toward the third affordance, wherein activating the third camera display mode means This includes displaying a visual effect selection area containing multiple graphical objects, The method according to any one of claims 27 to 43, further comprising:

45. While the third camera display mode is active, the selection of one of the plurality of graphical objects within the visual effect selection area is detected, The method according to claim 44, further comprising: detecting the selection, displaying a representation of the selected graphical object on the media item in the camera display area.

46. The camera user interface further includes sticker affordances associated with a sticker display mode, and the method is While the image data is displayed in the camera display area, a gesture directed towards the sticker affordance is detected. The process includes activating the sticker display mode in response to detecting the gesture directed toward the sticker affordance, and activating the sticker display mode means Display a sticker selection area that includes multiple sticker options. To detect the selection of one of the plurality of sticker options within the sticker selection area, In response to detecting the selection, the representation of the selected sticker option is displayed on the image data within the camera display area. The method according to any one of claims 27 to 45, further comprising:

47. While the representation of the selected sticker option is displayed on the image data within the camera display area, To detect the lateral movement of the subject within the field of view of one or more cameras, In response to detecting the lateral movement of the subject within the field of view of one or more cameras, the representation of the selected sticker option is moved laterally in accordance with the movement of the subject within the field of view of one or more cameras. The method according to claim 46, further comprising:

48. While the representation of the selected sticker option is displayed on the image data within the camera display area, To detect the rotation of the subject within the field of view of one or more cameras, In response to detecting the rotation of the subject within the field of view of one or more cameras, and in accordance with the determination that the representation of the selected sticker option has a first relationship with the subject, the representation of the selected sticker option is rotated according to the magnitude and direction of the rotation of the subject. In accordance with the determination that the representation of the selected sticker option does not have the first relationship with the subject, the rotation of the representation of the selected sticker option is stopped according to the magnitude and direction of the rotation of the subject. The method according to claim 46 or 47, further comprising:

49. While the representation of the selected sticker option is displayed on the image data within the camera display area, To detect the movement of the subject toward one or more of the cameras, In response to detecting the movement of the subject toward one or more cameras, In accordance with the determination that the representation of the selected sticker option has the first relationship with the subject, the representation of the selected sticker option is enlarged according to the magnitude of the movement of the subject toward one or more cameras, In accordance with the determination that the representation of the selected sticker option does not have the first relationship with the subject, the enlargement of the representation of the selected sticker option is stopped according to the magnitude of the movement of the subject toward one or more cameras, The method according to any one of claims 46 to 48, further comprising:

50. While the first camera display mode is active and the representation of the selected avatar option is displayed on the representation of the subject within the camera display area, To detect a touch gesture on the selected avatar option within the avatar selection area, In response to detecting the aforementioned touch gesture, an avatar editing user interface having multiple options for editing the selected avatar option is displayed. The method according to any one of claims 27 to 49, further comprising:

51. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device comprising a camera and a display device, wherein the one or more programs include instructions for performing the method according to any one of claims 27 to 50.

52. It is an electronic device, Camera and, Display device and One or more processors, An electronic device comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 27 to 50.

53. It is an electronic device, Camera and, Display device and One or more input devices, Means for carrying out the method described in any one of claims 27 to 50, An electronic device equipped with the following features.

54. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera and a display device, wherein the one or more programs are The camera user interface, via the display device, A camera display area including a representation of image data captured via the aforementioned camera, A camera user interface is displayed, including a first affordance associated with a first camera display mode. While the subject is located within the camera's field of view and the subject's appearance and background are displayed within the camera's display area, a gesture directed towards the first affordance is detected. Activating the first camera display mode in response to detecting the gesture directed towards the first affordance is: To display an avatar selection area containing one of the selected avatar options, Activating the first camera display mode, which includes displaying the representation of the selected avatar option on the representation of the subject within the camera display area; detecting changes in the posture of the subject while the first camera display mode is active; A non-temporary computer-readable storage medium containing instructions for changing the appearance of the displayed representation of the selected avatar option based on the detected change in the posture of the subject, while maintaining the display of the background, in response to the detection of the change in the posture of the subject.

55. It is an electronic device, Camera and, Display device and One or more processors, An electronic device comprising: a memory that stores one or more programs configured to be executed by one or more processors, wherein the one or more programs are The camera user interface, via the display device, A camera display area including a representation of image data captured via the aforementioned camera, A camera user interface is displayed, including a first affordance associated with a first camera display mode. While the subject is located within the camera's field of view and the subject's appearance and background are displayed within the camera's display area, a gesture directed towards the first affordance is detected. Activating the first camera display mode in response to detecting the gesture directed towards the first affordance is: To display an avatar selection area containing one of the selected avatar options, Activating the first camera display mode, which includes displaying the representation of the selected avatar option on the representation of the subject within the camera display area; detecting changes in the posture of the subject while the first camera display mode is active; An electronic device that includes a command to change the appearance of the displayed representation of the selected avatar option based on the detected change in the posture of the subject, while maintaining the display of the background, in response to the detection of the change in the posture of the subject.

56. It is an electronic device, Camera and, Display device and One or more input devices, A means for displaying a camera user interface via the aforementioned display device, wherein the camera user interface is A camera display area including a representation of image data captured via the aforementioned camera, A means including a first affordance associated with a first camera display mode, Means for detecting a gesture directed towards the first affordance while the subject is located within the camera's field of view and the subject's representation and background are displayed within the camera's display area, Means for activating the first camera display mode in response to the detection of the gesture directed toward the first affordance, wherein activating the first camera display mode means To display an avatar selection area containing one of the selected avatar options, A means including displaying the representation of the selected avatar option on the representation of the subject within the camera display area, While the first camera display mode is active, means for detecting changes in the posture of the subject, A means for changing the appearance of the displayed representation of the selected avatar option based on the detected change in the posture of the subject, while maintaining the display of the background, in response to detecting the change in the posture of the subject, An electronic device equipped with the following features.

57. It is a method, In an electronic device having a display device, Displaying a media user interface via the aforementioned display device, wherein the media user interface is A media display area including the representation of media items, Including effect affordances, Detecting a gesture directed towards the aforementioned effect affordance, In response to detecting the gesture directed towards the effect affordance, the representation of the media item and the display of multiple effect options for applying the effect to the media item are provided. In accordance with the determination that the media item is associated with the corresponding depth data, the plurality of effect options include each effect option for applying the effect based on the depth data, This includes the determination that the image data does not contain the depth data, which means that each of the effect options cannot be used for activation in the multiple effect options, Methods that include...

58. The method according to claim 57, wherein each of the aforementioned effect options corresponds to an effect for displaying an avatar in the media item.

59. The method according to claim 57 or 58, wherein each of the effect options corresponds to an effect for displaying a plurality of virtual objects moving within the media item.

60. The method according to claim 59, wherein the trajectories of the plurality of objects moving within the media item are modified based on the presence of at least one of the objects within the media item or a visual effect applied to the media item.

61. The method according to any one of claims 57 to 60, wherein each of the aforementioned effect options corresponds to an effect for displaying one or more selectable graphic icons within the media item.

62. The method according to any one of claims 57 to 61, wherein the plurality of effect options include an option for adding a label to the media item.

63. The method according to any one of claims 57 to 62, wherein the plurality of effect options include an option for applying one or more image filters to the media item.

64. The method according to any one of claims 57 to 63, wherein the depth data corresponding to the media item is acquired by the camera of the electronic device after the pre-selection of the effect affordance has been detected.

65. The method according to any one of claims 57 to 64, wherein the media item is a recorded image or video, and the effect is applied based on the depth data after the media item has been recorded.

66. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device having a display device, wherein the one or more programs include instructions for performing the method according to any one of claims 57 to 65.

67. It is an electronic device, Display device and One or more processors, A memory that stores one or more programs configured to be executed by one or more processors, An electronic device comprising, wherein one or more programs include instructions for performing the method according to any one of claims 57 to 65.

68. It is an electronic device, Display device and One or more input devices, Means for carrying out the method described in any one of claims 57 to 65, An electronic device equipped with the following features.

69. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device equipped with a display device, wherein the one or more programs are A media user interface is provided via the display device, and the media user interface is A media display area including the representation of media items, Display a media user interface, including effect affordances. The gesture directed towards the aforementioned effect affordance is detected, The command includes, in response to detecting the gesture directed towards the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, In accordance with the determination that the media item is associated with the corresponding depth data, the plurality of effect options include each effect option for applying the effect based on the depth data, A non-temporary computer-readable storage medium, including the fact that, according to the determination that the image data does not contain the depth data, each of the effect options is unavailable for activation in the plurality of effect options.

70. It is an electronic device, Display device and One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are A media user interface is provided via the display device, and the media user interface is A media display area including the representation of media items, Display a media user interface, including effect affordances. The gesture directed towards the aforementioned effect affordance is detected, The command includes, in response to detecting the gesture directed towards the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, In accordance with the determination that the media item is associated with the corresponding depth data, the plurality of effect options include each effect option for applying the effect based on the depth data, An electronic device in which, according to the determination that the image data does not contain the depth data, each of the effect options is unavailable for activation in the plurality of effect options.

71. It is an electronic device, Display device and One or more input devices, A means for displaying a media user interface via the display device, wherein the media user interface is A media display area including the representation of media items, Means including effect affordances, A means for detecting a gesture directed towards the aforementioned effect affordance, The system includes means for displaying a plurality of effect options for applying an effect to a media item, simultaneously with the representation of the media item, in response to the detection of the gesture directed towards the effect affordance. In accordance with the determination that the media item is associated with the corresponding depth data, the plurality of effect options include each effect option for applying the effect based on the depth data, An electronic device in which, according to the determination that the image data does not contain the depth data, each of the effect options is unavailable for activation in the plurality of effect options.

72. It is a method, In an electronic device having a display device, The live video communication user interface of a live video communication application is displayed via the aforementioned display device, wherein the live video communication user interface is The expressions of the subjects participating in the live video communication session, The first affordance, and including, To detect a gesture directed towards the first affordance, In response to detecting the gesture directed towards the first affordance, Activate camera effects mode, To increase the size of the representation of the subject participating in the live video communication session, Methods that include...

73. Before displaying the first affordance, a first input is detected on the live video communication user interface corresponding to a request to display one or more options associated with the live video communication session, In response to detecting the first input, Displaying one or more options associated with the live video communication session, Displaying the first affordance, The method according to claim 72, further comprising:

74. While the aforementioned camera effect mode is activated, Detecting the selection of effect option affordances, In response to detecting the selection of the aforementioned effect option affordance, the appearance of the representation of the subject participating in the live video communication session is modified based on the visual effect associated with the selected effect option affordance. The method according to claim 72 or 73, further comprising:

75. The means of detecting a second input on the live video communication user interface, wherein the second input corresponds to a request to reduce the size of the representation of the subject participating in the live video communication session. In response to detecting the second input, The representation of the subject participating in the live video communication session having the modified appearance based on the visual effect associated with the selected effect option affordance, To simultaneously display one or more representations of each participant in the aforementioned live video communication session, The method according to claim 74, further comprising:

76. The method according to any one of claims 72 to 75, wherein the representation of the subject participating in the live video communication session includes image data captured by a camera associated with the electronic device.

77. The method according to any one of claims 72 to 75, wherein the representation of the subject participating in the live video communication session includes image data transmitted from a second electronic device to the electronic device.

78. The method according to any one of claims 72 to 77, further comprising: detecting the gesture directed toward the first affordance; displaying a first visual effect affordance associated with a first type of visual effect; and a second visual effect affordance associated with a second type of visual effect that is different from the first type of visual effect.

79. To detect a selection of one of the aforementioned affordances associated with a certain type of visual effect, In response to detecting the selection of the affordance associated with the visual effect, display a plurality of visual effect options corresponding to the visual effect, The method according to claim 78, further comprising:

80. While the aforementioned camera effect mode is activated, The method according to any one of claims 72 to 79, further comprising changing the appearance of the representation of the subject participating in the live video communication session to display one or more visual effects.

81. While the aforementioned camera effect mode is activated, To change the appearance of the representation of the subject participating in the live video communication session and to display a virtual avatar, The electronic device detects changes in faces within the field of view of one or more cameras, Based on the detected changes in the face, the appearance of the virtual avatar is modified. The method according to any one of claims 72 to 80, further comprising:

82. The aforementioned live video communication user interface is: The representation of the second participant in the aforementioned live video communication session, The representation of the third participant in the aforementioned live video communication session, The method according to any one of claims 72 to 81, further comprising:

83. Multiple participants are taking part in the live video communication session, and the multiple participants include the subject and a first remote participant, and the live video communication user interface further includes a representation of the first remote participant, and the method is The method according to any one of claims 72 to 82, further comprising reducing the size of the representation of the first remote participant in response to the detection of the gesture directed toward the first affordance.

84. While the aforementioned camera effect mode is activated, Displaying a first visual effect on the representation of the subject participating in the live video communication session, Detecting the input corresponding to the first visual effect, In response to detecting the input corresponding to the first visual effect, In accordance with the determination that the input is of the first type, the position of the first visual effect in the representation of the subject participating in the live video communication session is changed based on the magnitude and direction of the input. In accordance with the determination that the input is of the second type, the size of the first visual effect is changed based on the size of the input. The method according to any one of claims 72 to 83, further comprising:

85. Changing the position of the first visual effect is Before detecting the end of the input, the movement of the first visual effect is displayed based on the magnitude and direction of the input. In accordance with the determination that the first visual effect has moved across a boundary region at a predetermined position, an indication is generated showing that the first visual effect has intersected the boundary region. The method according to claim 84, including the method described in claim 84.

86. The method according to claim 85, wherein the indication is at least one of tactile feedback or visual feedback.

87. Changing the position of the first visual effect includes displaying the movement of the first visual effect from the first position to the second position. The method according to claim 84, wherein changing the size of the first visual effect includes displaying a transition of the first visual effect from a first display size to a second display size.

88. Changing the position of the first visual effect includes transitioning the first visual effect from its appearance at the first position c to its appearance at the second position without displaying the first visual effect appearing at the intermediate position. The method according to claim 84, wherein changing the size of the first visual effect includes transitioning the first visual effect from a first display size to a second display size without displaying the first visual effect having an intermediate size.

89. Displaying the live video communication user interface without the first affordance, To detect touch input on the live video communication user interface, In response to detecting the aforementioned touch input, In accordance with the determination that the camera effect mode is activated, a live video communication option user interface including the first affordance and a plurality of visual effect affordances is displayed. In accordance with the determination that the camera effect mode is not activated, the live video communication option user interface is displayed, which includes the first affordance and excludes the plurality of visual effect affordances. The method according to any one of claims 72 to 88, further comprising:

90. Multiple participants are taking part in the live video communication session, and these multiple participants include the subject and a first remote participant, and the representation of the subject is a live preview of the field of view of the camera of the electronic device, and the method is After changing the appearance of the representation of the subject participating in the live video communication session based on the visual effect associated with the selected effect option affordance, transmit data corresponding to the changed appearance of the representation of the subject participating in the live video communication session to at least the remote participant among the plurality of participants. The method according to claim 74, further comprising:

91. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device having a display device, wherein the one or more programs include instructions for performing the method according to any one of claims 72 to 90.

92. It is an electronic device, Display device and One or more processors, An electronic device comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 72 to 90.

93. It is an electronic device, Display device and One or more input devices, Means for carrying out the method described in any one of claims 72 to 90, An electronic device equipped with the following features.

94. A non-temporary computer-readable storage medium that stores one or more programs configured to be executed by one or more processors of an electronic device equipped with a display device, wherein the one or more programs are The live video communication user interface of a live video communication application is transmitted via the aforementioned display device, and the live video communication user interface is The expressions of the subjects participating in the live video communication session, Display a live video communication user interface, including the first affordance. The gesture directed towards the first affordance is detected, In response to detecting the gesture directed towards the first affordance, Activate camera effects mode, A non-temporary computer-readable storage medium containing instructions for increasing the size of the representation of the subject participating in the live video communication session.

95. It is an electronic device, Display device and One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The live video communication user interface of a live video communication application is transmitted via the aforementioned display device, and the live video communication user interface is The expressions of the subjects participating in the live video communication session, Display a live video communication user interface, including the first affordance. The gesture directed towards the first affordance is detected, In response to detecting the gesture directed towards the first affordance, Activate camera effects mode, To increase the size of the representation of the subject participating in the live video communication session, An electronic device that includes instructions.

96. It is an electronic device, Display device and One or more input devices, A means for displaying the live video communication user interface of a live video communication application via the display device, wherein the live video communication user interface is The expressions of the subjects participating in the live video communication session, The means, including the first affordance, A means for detecting a gesture directed towards the first affordance, In response to detecting the gesture directed towards the first affordance, Activate camera effects mode, Means for increasing the size of the representation of the subject participating in the live video communication session, An electronic device equipped with the following features.

97. It is a method, In an electronic device having one or more cameras and a display device, Displaying a representation of image data captured via one or more cameras via the display device, wherein the representation includes a representation of a subject, and the image data corresponds to depth data including depth data of the subject. Displaying a representation of a virtual avatar via the display device, which is displayed in place of at least a portion of the representation of the subject, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the subject, determined based on the depth data of the subject, and the displaying of the representation of the virtual avatar is Based on the depth data, in accordance with the determination that the first portion of the virtual avatar satisfies a set of depth-based display criteria, which includes the requirement that the depth data of the subject indicates that the first portion of the virtual avatar has a simulated depth in front of the corresponding first portion of the subject, the representation of the first portion of the virtual avatar is included as part of the representation of the virtual avatar, displayed in place of the first portion of the subject, so that the depth-based display criteria are satisfied. Based on the depth data, if it is determined that the first portion of the virtual avatar does not satisfy the set of depth-based display criteria for the first portion of the subject, the representation of the first portion of the virtual avatar is removed from the representation of the virtual avatar, and the first portion of the subject is displayed in the area that the first portion of the virtual avatar would have occupied. Methods that include...

98. The method according to claim 97, wherein the first portion of the virtual avatar moves based on the movement of the subject.

99. The method according to claim 97, wherein the representation of the virtual avatar includes a representation of the second portion of the virtual avatar displayed above the corresponding second portion of the representation of the subject, regardless of whether the depth data indicates that the second portion of the virtual avatar has a simulated depth in front of or behind the second portion of the representation of the subject.

100. The method according to claim 98, wherein the second portion of the virtual avatar is continuously displayed in place of the corresponding second portion of the representation of the subject.

101. The method according to claim 98, wherein the second portion of the virtual avatar moves based on the movement of the subject.

102. The method according to any one of claims 99 to 101, wherein the first portion of the virtual avatar is a first sub-part of a first avatar feature, and the second portion of the virtual avatar is a second sub-part of the first avatar feature.

103. The virtual avatar includes avatar hair features, which include the first portion of the virtual avatar. The method according to any one of claims 97 to 102, wherein displaying the representation of the virtual avatar includes displaying the first portion of the avatar hair feature and conditionally displaying the second portion of the avatar hair feature based on whether the simulated depth of the second portion of the avatar hair feature is in front of or behind the third portion of the representation of the subject, based on the depth data of the subject.

104. The virtual avatar includes avatar neck features, which include the first portion of the virtual avatar. The method according to any one of claims 97 to 103, wherein displaying the representation of the virtual avatar includes conditionally displaying the portion of the avatar's neck feature based on whether the simulated depth of the portion of the avatar's neck feature is in front of or behind the fourth portion of the representation of the subject, based on the depth data of the subject.

105. The method according to any one of claims 97 to 104, wherein the first portion of the virtual avatar includes an obscured portion of the virtual avatar that is not displayed when the portion of the representation of the subject is in a position facing one or more cameras.

106. To detect changes in the posture of the subject's head, The method according to any one of claims 97 to 105, further comprising: detecting the change in the posture of the head portion of the subject, and changing the amount of the virtual avatar that is excluded from the representation of the virtual avatar based on the depth data and the change in posture.

107. The method according to any one of claims 97 to 106, further comprising displaying the representation of the virtual avatar to change the visual appearance of a third portion of the virtual avatar adjacent to the first portion of the virtual avatar to an appearance based on both the appearance of the avatar and the appearance of the representation of the subject.

108. The electronic device includes one or more depth sensors, and the method is Before displaying the representation of the virtual avatar, initial depth data of the subject is captured via one or more depth sensors. The process involves generating depth data for the subject by modifying the initial depth data for the subject, The method according to any one of claims 97 to 107, further comprising:

109. The method according to claim 108, wherein modifying the initial depth data for the subject involves performing one or more transformations on the initial depth data, selected from the group consisting of blurring the initial depth data, fading out the initial depth data, and smoothing the initial depth data.

110. To detect changes in the posture of the subject, In response to detecting the change in the posture of the subject, the displayed representation of the virtual avatar is modified based on the change in posture, Based on the depth data that takes into account the changes in the posture of the subject, the appearance of the representation of the virtual avatar is updated from a first appearance excluding the first part of the virtual avatar to a second appearance including the first part of the virtual avatar, in accordance with the determination that the first part of the virtual avatar satisfies the set of depth-based display criteria, and This includes updating the appearance of the representation of the virtual avatar to a fourth appearance, which excludes the first part of the virtual avatar, from a third appearance including the first part of the virtual avatar, based on the depth data that takes into account the changes in the posture of the subject, in accordance with the determination that the first part of the virtual avatar does not satisfy the set of depth-based display criteria, The method according to any one of claims 97 to 109, further comprising:

111. To detect a change in the portion of the representation of the subject, The appearance of the virtual avatar is modified based on the detected changes in the portion of the representation of the subject, The method according to any one of claims 97 to 110, further comprising:

112. To display, via the display device, a representation of the shadow projected by the virtual avatar, which is displayed on at least a fifth portion of the representation of the subject, The method according to any one of claims 97 to 111, further comprising:

113. The method according to claim 112, wherein one or more characteristics of the displayed representation of the shadow are based on the shape of the virtual avatar.

114. The method according to claim 112, wherein one or more characteristics of the displayed representation of the shadow are based on the illumination conditions within the field of view of the one or more cameras.

115. The method according to claim 112, wherein one or more characteristics of the displayed representation of the shadow are based on the depth data.

116. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device comprising a display device and one or more cameras, wherein the one or more programs include instructions for performing the method according to any one of claims 97 to 115.

117. It is an electronic device, One or more cameras, Display device and One or more processors, An electronic device comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the method according to any one of claims 97 to 115.

118. It is an electronic device, One or more cameras, Display device and Means for carrying out the method described in any one of claims 97 to 115, An electronic device equipped with the following features.

119. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of an electronic device comprising one or more cameras and a display device, wherein the one or more programs are The display device displays a representation of image data captured via one or more cameras, wherein the representation includes a representation of a subject, and the image data corresponds to depth data including depth data of the subject. The command includes, via the display device, displaying a representation of a virtual avatar that is displayed in place of at least a portion of the representation of the subject, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the subject, determined based on the depth data of the subject, and displaying the representation of the virtual avatar is, Based on the depth data, in accordance with the determination that the first portion of the virtual avatar satisfies a set of depth-based display criteria, which includes the requirement that the depth data of the subject indicates that the first portion of the virtual avatar has a simulated depth in front of the corresponding first portion of the subject, the representation of the first portion of the virtual avatar is included as part of the representation of the virtual avatar, displayed in place of the first portion of the subject, so that the depth-based display criteria are satisfied. Based on the depth data, if it is determined that the first portion of the virtual avatar does not satisfy the set of depth-based display criteria for the first portion of the subject, the representation of the first portion of the virtual avatar is removed from the representation of the virtual avatar, and the first portion of the subject is displayed in the area that the first portion of the virtual avatar would have occupied. Non-temporary computer-readable storage media, including [specific type of storage medium].

120. It is an electronic device, One or more cameras, Display device and One or more processors, The system comprises a memory that stores one or more programs configured to be executed by one or more processors, and the one or more programs are The display device displays a representation of image data captured via one or more cameras, wherein the representation includes a representation of a subject, and the image data corresponds to depth data including depth data of the subject. The command includes, via the display device, displaying a representation of a virtual avatar that is displayed in place of at least a portion of the representation of the subject, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the subject, determined based on the depth data of the subject, and displaying the representation of the virtual avatar is, Based on the depth data, in accordance with the determination that the first portion of the virtual avatar satisfies a set of depth-based display criteria, which includes the requirement that the depth data of the subject indicates that the first portion of the virtual avatar has a simulated depth in front of the corresponding first portion of the subject, the representation of the first portion of the virtual avatar is included as part of the representation of the virtual avatar, displayed in place of the first portion of the subject, so that the depth-based display criteria are satisfied. Based on the depth data, if it is determined that the first portion of the virtual avatar does not satisfy the set of depth-based display criteria for the first portion of the subject, the representation of the first portion of the virtual avatar is removed from the representation of the virtual avatar, and the first portion of the subject is displayed in the area that the first portion of the virtual avatar would have occupied. Electronic devices, including those mentioned above.

121. It is an electronic device, One or more cameras, Display device and A means for displaying a representation of image data captured via one or more cameras via the display device, wherein the representation includes a representation of a subject, and the image data corresponds to depth data including depth data of the subject. Means for displaying a representation of a virtual avatar, which is displayed via the display device in place of at least a portion of the representation of the subject, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the subject, determined based on the depth data of the subject, and the display of the representation of the virtual avatar is Based on the depth data, in accordance with the determination that the first portion of the virtual avatar satisfies a set of depth-based display criteria, which includes the requirement that the depth data of the subject indicates that the first portion of the virtual avatar has a simulated depth in front of the corresponding first portion of the subject, the representation of the first portion of the virtual avatar is included as part of the representation of the virtual avatar, displayed in place of the first portion of the subject, so that the depth-based display criteria are satisfied. Based on the depth data, if it is determined that the first portion of the virtual avatar does not satisfy the set of depth-based display criteria for the first portion of the subject, the representation of the first portion of the virtual avatar is removed from the representation of the virtual avatar, and the first portion of the subject is displayed in the area that the first portion of the virtual avatar would have occupied. Electronic devices, including those mentioned above.

Images