Creative Camera

Abstract

The present disclosure relates to "creative cameras". The present disclosure as a whole relates to displaying visual effects in image data. In some examples, the visual effect includes an avatar displayed on a user's face. In some examples, the visual effect includes a sticker applied to the image data. In some examples, the visual effect includes a screen effect. In some examples, the visual effect is modified based on depth data in the image data.

Description

[0001] This application is a divisional application of the invention patent application with application number 201811157549.6, application date September 28, 2018, and invention name “Creative Camera”. Technical Field

[0002] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for displaying visual effects. Background Art

[0003] Visual effects are used to enhance the user experience when capturing and viewing media using electronic devices. Visual effects can change the appearance of image data or represent idealized or completely fictitious representations of the environment captured in the image. Summary of the Invention

[0004] However, some techniques for displaying visual effects using electronic devices are often cumbersome and inefficient. For example, some existing techniques use complex and time-consuming user interfaces that may include multiple keystrokes or button presses. These techniques require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0005] Thus, the present technology provides electronic devices with faster, more efficient methods and interfaces for displaying visual effects. Such methods and interfaces optionally supplement or replace other methods for displaying visual effects. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time between battery charges.

[0006] A method is described. The method is performed at an electronic device having a camera, a display device, and one or more input devices. The method includes: displaying an instant messaging user interface of a messaging session via the display device, the messaging session including at least a first participant, the instant messaging user interface including a camera enable indication; detecting a first input directed to the camera enable indication via the one or more input devices; in response to detecting the first input, displaying a camera user interface, the camera user interface including a capture enable indication; detecting a second input directed to the capture enable indication via the one or more input devices; in response to detecting the second input: capturing image data using the camera; ceasing to display the capture enable indication; and displaying a send enable indication in the camera user interface at a position previously occupied by the capture enable indication; detecting a third input directed to the send enable indication via the one or more input devices; and in response to detecting the third input, initiating a process for sending the captured image data to the first participant.

[0007] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, a display device, and one or more input devices, the one or more programs including instructions for: displaying, via the display device, an instant messaging user interface of a messaging conversation including at least a first participant, the instant messaging user interface including a camera-enabled representation; detecting, via the one or more input devices, a first input directed toward the camera-enabled representation; in response to detecting the first input, displaying a camera user interface, the camera user interface including a capture-enabled representation; detecting, via the one or more input devices, a second input directed toward the capture-enabled representation; in response to detecting the second input, capturing image data using the camera; ceasing to display the capture-enabled representation; and displaying a send-enabled representation in the camera user interface at a location previously occupied by the capture-enabled representation; detecting, via the one or more input devices, a third input directed toward the send-enabled representation; and in response to detecting the third input, initiating a process for sending the captured image data to the first participant.

[0008] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, a display device, and one or more input devices, the one or more programs including instructions for: displaying, via the display device, an instant messaging user interface of a messaging conversation including at least a first participant, the instant messaging user interface including a camera-enabled representation; detecting, via the one or more input devices, a first input directed toward the camera-enabled representation; in response to detecting the first input, displaying a camera user interface, the camera user interface including a capture-enabled representation; detecting, via the one or more input devices, a second input directed toward the capture-enabled representation; in response to detecting the second input, capturing image data using the camera; ceasing to display the capture-enabled representation; and displaying a send-enabled representation in the camera user interface at a location previously occupied by the capture-enabled representation; detecting, via the one or more input devices, a third input directed toward the send-enabled representation; and in response to detecting the third input, initiating a process for sending the captured image data to the first participant.

[0009] The present invention describes an electronic device. The electronic device includes: a camera; a display device; one or more input devices; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying an instant messaging user interface of a messaging conversation including at least a first participant via the display device, the instant messaging user interface including a camera enable representation; detecting a first input directed to the camera enable representation via the one or more input devices; in response to detecting the first input, displaying a camera user interface, the camera user interface including a capture enable representation; detecting a second input directed to the capture enable representation via the one or more input devices; in response to detecting the second input: capturing image data using the camera; ceasing to display the capture enable representation; and displaying a send enable representation in the camera user interface at a position previously occupied by the capture enable representation; detecting a third input directed to the send enable representation via the one or more input devices; and in response to detecting the third input, initiating a process for sending the captured image data to the first participant.

[0010] The present invention describes an electronic device. The electronic device includes: a camera; a display device; one or more input devices; a device for displaying an instant messaging user interface of a messaging session via the display device, the messaging session including at least a first participant, the instant messaging user interface including a camera enable representation; a device for detecting a first input directed to the camera enable representation via the one or more input devices; a device for displaying a camera user interface in response to detecting the first input, the camera user interface including a capture enable representation; a device for detecting a second input directed to the capture enable representation via the one or more input devices; a device for performing the following operations in response to detecting the second input: capturing image data using the camera; ceasing to display the capture enable representation; and displaying a send enable representation in the camera user interface at a position previously occupied by the capture enable representation; a device for detecting a third input directed to the send enable representation via the one or more input devices; and a device for initiating a process for sending the captured image data to the first participant in response to detecting the third input.

[0011] A method is described. The method is performed at an electronic device having a camera and a display device. The method includes: displaying a camera user interface via the display device, the camera user interface including: a camera display area including a representation of image data captured by the camera; and a first affordance associated with a first camera display mode; detecting a gesture directed at the first affordance when an object is positioned within the field of view of the camera and the representation of the object and a background are displayed in the camera display area; activating the first camera display mode in response to detecting the gesture directed at the first affordance, wherein activating the first camera display mode includes: displaying an avatar selection area including a selected one of a plurality of avatar options; and displaying a representation of the selected avatar option on the representation of the object in the camera display area; detecting a posture change of the object while the first camera display mode is active; and in response to detecting the posture change of the object, changing the appearance of the displayed representation of the selected avatar option based on the detected posture change of the object while maintaining display of the background.

[0012] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera and a display device, the one or more programs including instructions for: displaying a camera user interface via the display device, the camera user interface comprising: a camera display area including a representation of image data captured by the camera; and a first affordance associated with a first camera display mode; detecting a gesture directed toward the first affordance when an object is positioned within a field of view of the camera and the representation of the object and a background are displayed in the camera display area; activating the first camera display mode in response to detecting the gesture directed toward the first affordance, wherein activating the first camera display mode comprises: displaying an avatar selection area including a selected one of a plurality of avatar options; and displaying a representation of the selected avatar option over the representation of the object in the camera display area; while the first camera display mode is active, detecting a posture change of the object; and in response to detecting the posture change of the object, changing the appearance of the displayed representation of the selected avatar option based on the detected posture change of the object while maintaining display of the background.

[0013] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera and a display device, the one or more programs including instructions for: displaying a camera user interface via the display device, the camera user interface comprising: a camera display area including a representation of image data captured by the camera; and a first affordance associated with a first camera display mode; detecting a gesture directed toward the first affordance when an object is positioned within a field of view of the camera and the representation of the object and a background are displayed in the camera display area; activating the first camera display mode in response to detecting the gesture directed toward the first affordance, wherein activating the first camera display mode comprises: displaying an avatar selection area including a selected one of a plurality of avatar options; and displaying a representation of the selected avatar option on the representation of the object in the camera display area; while the first camera display mode is active, detecting a change in posture of the object; and in response to detecting the change in posture of the object, changing an appearance of the displayed representation of the selected avatar option based on the detected change in posture of the object while maintaining display of the background.

[0014] The present invention describes an electronic device. The electronic device includes a camera, a display device, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying a camera user interface via the display device, the camera user interface including: a camera display area including a representation of image data captured by the camera; and a first affordance associated with a first camera display mode; detecting a gesture directed at the first affordance when an object is positioned within a field of view of the camera and the representation of the object and a background are displayed in the camera display area; activating the first camera display mode in response to detecting the gesture directed at the first affordance, wherein activating the first camera display mode includes: displaying an avatar selection area including a selected one of a plurality of avatar options; and displaying a representation of the selected avatar option on the representation of the object in the camera display area; while the first camera display mode is active, detecting a posture change of the object; and in response to detecting the posture change of the object, changing the appearance of the displayed representation of the selected avatar option based on the detected posture change of the object while maintaining display of the background.

[0015] The present invention describes an electronic device. The electronic device includes: a camera; a display device; one or more input devices; means for displaying a camera user interface via the display device, the camera user interface including: a camera display area including a representation of image data captured by the camera; and a first affordance associated with a first camera display mode; means for detecting a gesture directed at the first affordance when an object is positioned within the field of view of the camera and the representation of the object and a background are displayed in the camera display area; means for activating the first camera display mode in response to detecting the gesture directed at the first affordance, wherein activating the first camera display mode includes: displaying an avatar selection area including a selected one of a plurality of avatar options; and displaying a representation of the selected avatar option on the representation of the object in the camera display area; means for detecting a change in posture of the object while the first camera display mode is active; and means for changing the appearance of the displayed representation of the selected avatar option based on the detected change in posture of the object while maintaining display of the background in response to detecting the change in posture of the object.

[0016] A method is described. The method is performed at an electronic device having a display device. The method includes: displaying a media user interface via the display device, the media user interface including: a media display area, the media display area including a representation of a media item; and an effect enable representation; detecting a gesture directed to the effect enable representation; in response to detecting the gesture directed to the effect enable representation, displaying a plurality of effect options, the plurality of effect options being used to apply an effect and a representation of the media item to the media item simultaneously, including: based on determining that the media item is associated with corresponding depth data, the plurality of effect options including a corresponding effect option for applying an effect based on the depth data; and based on determining that the image data does not include the depth data, the corresponding effect option is not available for activation of the plurality of effect options.

[0017] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs including instructions for the following operations: displaying a media user interface via the display device, the media user interface including: a media display area, the media display area including a representation of a media item; and an effect enable representation; detecting a gesture directed toward the effect enable representation; in response to detecting the gesture directed toward the effect enable representation, displaying a plurality of effect options, the plurality of effect options being used to simultaneously apply an effect and a representation of the media item to the media item, including: based on determining that the media item is associated with corresponding depth data, the plurality of effect options including a corresponding effect option for applying an effect based on the depth data; and based on determining that the image data does not include the depth data, the corresponding effect option is not available for activation of the plurality of effect options.

[0018] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs including instructions for: displaying a media user interface via the display device, the media user interface including: a media display area including a representation of a media item; and an effect affordance; detecting a gesture directed toward the effect affordance; in response to detecting the gesture directed toward the effect affordance, displaying a plurality of effect options for applying an effect and a representation of the media item to the media item simultaneously, including: based on determining that the media item is associated with corresponding depth data, the plurality of effect options including a corresponding effect option for applying an effect based on the depth data; and based on determining that the image data does not include the depth data, the corresponding effect option is unavailable for activation of the plurality of effect options.

[0019] The present invention describes an electronic device. The electronic device includes: a display device; one or more processors; and a memory, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying a media user interface via the display device, the media user interface including: a media display area, the media display area including a representation of a media item; and an effect enable representation; detecting a gesture directed to the effect enable representation; in response to detecting the gesture directed to the effect enable representation, displaying multiple effect options, the multiple effect options being used to apply an effect and a representation of the media item to the media item simultaneously, including: based on determining that the media item is associated with corresponding depth data, the multiple effect options including corresponding effect options for applying an effect based on the depth data; and based on determining that the image data does not include the depth data, the corresponding effect options are not available for activation of the multiple effect options.

[0020] The present invention describes an electronic device. The electronic device includes: a display device; one or more input devices; a device for displaying a media user interface via the display device, the media user interface including: a media display area, the media display area including a representation of a media item; and an effect enable representation; a device for detecting a gesture directed to the effect enable representation; in response to detecting the gesture directed to the effect enable representation, a device for displaying a plurality of effect options, the plurality of effect options being used to apply an effect and a representation of the media item to the media item simultaneously, including: based on determining that the media item is associated with corresponding depth data, the plurality of effect options including a corresponding effect option for applying an effect based on the depth data; and based on determining that the image data does not include the depth data, the corresponding effect option is not available for activation of the plurality of effect options.

[0021] A method is described. The method is performed at an electronic device having a display device. The method includes: displaying a real-time video communication user interface of a real-time video communication application via the display device, the real-time video communication user interface including: representations of objects participating in a real-time video communication session, and a first affordance; detecting a gesture directed toward the first affordance; and in response to detecting the gesture directed toward the first affordance: activating a camera effects mode; and increasing the size of the representation of the objects participating in the real-time video communication session.

[0022] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs including instructions for: displaying a real-time video communication user interface of a real-time video communication application via the display device, the real-time video communication user interface including: representations of objects participating in a real-time video communication session, and a first affordance; detecting a gesture directed toward the first affordance; and in response to detecting the gesture directed toward the first affordance: activating a camera effects mode; and increasing the size of the representation of the objects participating in the real-time video communication session.

[0023] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs including instructions for: displaying a real-time video communication user interface of a real-time video communication application via the display device, the real-time video communication user interface including: representations of objects participating in a real-time video communication session, and a first affordance; detecting a gesture directed toward the first affordance; and in response to detecting the gesture directed toward the first affordance: activating a camera effects mode; and increasing the size of the representation of the objects participating in the real-time video communication session.

[0024] The present invention describes an electronic device. The electronic device includes: a display device; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying a real-time video communication user interface of a real-time video communication application via the display device, the real-time video communication user interface including: representations of objects participating in a real-time video communication session, and a first enable representation; detecting a gesture directed toward the first enable representation; and in response to detecting the gesture directed toward the first enable representation: activating a camera effects mode; and increasing the size of the representation of the objects participating in the real-time video communication session.

[0025] The present invention describes an electronic device. The electronic device includes: a display device; one or more input devices; means for displaying a real-time video communication user interface of a real-time video communication application via the display device, the real-time video communication user interface including: representations of objects participating in a real-time video communication session, and a first affordance; means for detecting a gesture directed toward the first affordance; and means for, in response to detecting the gesture directed toward the first affordance, activating a camera effects mode and increasing the size of the representation of the objects participating in the real-time video communication session.

[0026] A method is described. The method is performed at an electronic device having a camera and a display device. The method includes: displaying, via the display device, a representation of image data captured via the one or more cameras, wherein the representation includes a representation of an object, and the image data corresponds to depth data, the depth data including depth data of the object; displaying, via the display device, a representation of a virtual avatar, the virtual avatar being displayed in place of at least a portion of the representation of the object, wherein the virtual avatar is placed at a simulated depth relative to the representation of the object, the simulated depth being determined based on the depth data of the object, displaying the representation of the virtual avatar includes: determining, based on the depth data, that a first portion of the virtual avatar satisfies a set of depth-based display criteria, wherein the depth-based display criteria The criteria include requiring that the depth data of the object indicate that the first part of the virtual avatar has a simulated depth in front of the corresponding first part of the object, and in order for the depth-based display criteria to be met, including a representation of the first part of the virtual avatar to be displayed in place of the first part of the object as part of the representation of the virtual avatar; and based on a determination based on the depth data that the first part of the virtual avatar does not meet the set of depth-based display criteria for the first part of the object, excluding the representation of the first part of the virtual avatar from the representation of the virtual avatar and displaying the first part of the object in the area that should have been occupied by the first part of the virtual avatar.

[0027] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera and a display device, the one or more programs including instructions for the following operations: displaying, via the display device, a representation of image data captured by the one or more cameras, wherein the representation includes a representation of an object and the image data corresponds to depth data, the depth data including depth data of the object; displaying, via the display device, a representation of a virtual avatar, the virtual avatar being displayed in place of at least a portion of the representation of the object, wherein the virtual avatar is placed at a simulated depth relative to the representation of the object, the simulated depth being determined based on the depth data of the object, displaying the representation of the virtual avatar including: displaying, via the display device, a representation of the virtual avatar based on the depth data determined based on the depth data; The first part of the virtual avatar satisfies a set of depth-based display criteria, wherein the depth-based display criteria include requiring that the depth data of the object indicate that the first part of the virtual avatar has a simulated depth in front of the corresponding first part of the object, and in order for the depth-based display criteria to be satisfied, a representation of the first part of the virtual avatar displayed in place of the first part of the object is included as part of the representation of the virtual avatar; and based on a determination based on the depth data that the first part of the virtual avatar does not satisfy the set of depth-based display criteria for the first part of the object, the representation of the first part of the virtual avatar is excluded from the representation of the virtual avatar and the first part of the object is displayed in the area that should have been occupied by the first part of the virtual avatar.

[0028] A transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera and a display device, the one or more programs including instructions for: displaying, via the display device, a representation of image data captured by the one or more cameras, wherein the representation includes a representation of an object and the image data corresponds to depth data, the depth data including depth data of the object; displaying, via the display device, a representation of a virtual avatar, the virtual avatar being displayed in place of at least a portion of the representation of the object, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the object, the simulated depth being determined based on the depth data of the object, displaying the representation of the virtual avatar including: The first part of the virtual avatar satisfies a set of depth-based display criteria, wherein the depth-based display criteria include requiring that the depth data of the object indicate that the first part of the virtual avatar has a simulated depth in front of the corresponding first part of the object, and in order for the depth-based display criteria to be satisfied, a representation of the first part of the virtual avatar displayed in place of the first part of the object is included as part of the representation of the virtual avatar; and based on a determination based on the depth data that the first part of the virtual avatar does not satisfy the set of depth-based display criteria for the first part of the object, the representation of the first part of the virtual avatar is excluded from the representation of the virtual avatar and the first part of the object is displayed in the area that should have been occupied by the first part of the virtual avatar.

[0029] The present invention describes an electronic device. The electronic device includes: a camera; a display device; one or more processors; and a memory, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for the following operations: displaying a representation of image data captured by the one or more cameras via the display device, wherein the representation includes a representation of an object and the image data corresponds to depth data, the depth data including depth data of the object; displaying a representation of a virtual avatar via the display device, the virtual avatar being displayed in place of at least a portion of the representation of the object, wherein the virtual avatar is placed at a simulated depth relative to the representation of the object, the simulated depth being determined based on the depth data of the object, and displaying the representation of the virtual avatar includes: determining that the first portion of the virtual avatar satisfies a set of depth-based display criteria, wherein the depth-based display criteria include requiring that the depth data of the object indicate that the first portion of the virtual avatar has a simulated depth in front of the corresponding first portion of the object, and including a representation of the first portion of the virtual avatar that is displayed in place of the first portion of the object in order for the depth-based display criteria to be satisfied; and excluding the representation of the first portion of the virtual avatar from the representation of the virtual avatar and displaying the first portion of the object in the area that should have been occupied by the first portion of the virtual avatar based on a determination based on the depth data that the first portion of the virtual avatar does not satisfy the set of depth-based display criteria for the first portion of the object.

[0030] The present invention describes an electronic device. The electronic device includes: a camera; and a display device; a device for displaying, via the display device, a representation of image data captured by the one or more cameras, wherein the representation includes a representation of an object and the image data corresponds to depth data, the depth data including depth data of the object; a device for displaying, via the display device, a representation of a virtual avatar, the virtual avatar being displayed in place of at least a portion of the representation of the object, wherein the virtual avatar is positioned at a simulated depth relative to the representation of the object, the simulated depth being determined based on the depth data of the object, displaying the representation of the virtual avatar includes: determining, based on the depth data, that a first portion of the virtual avatar satisfies a set of depth-based display criteria, wherein the depth data corresponds to depth data. Depth-based display criteria include requiring that the depth data of the object indicate that the first part of the virtual avatar has a simulated depth in front of the corresponding first part of the object, and in order for the depth-based display criteria to be met, a representation of the first part of the virtual avatar to be displayed instead of the first part of the object is included as part of the representation of the virtual avatar; and based on the determination that the first part of the virtual avatar does not meet the set of depth-based display criteria for the first part of the object, a device is used to exclude the representation of the first part of the virtual avatar from the representation of the virtual avatar and display the first part of the object in the area that should have been occupied by the first part of the virtual avatar.

[0031] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0032] Thus, a faster and more efficient method and interface is provided for devices to display visual effects, thereby improving the effectiveness, efficiency, and user satisfaction of such devices.Such methods and interfaces can supplement or replace other methods for displaying visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0034] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0035] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0036] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0037] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0038] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0039] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0040] Figure 5A A personal electronic device according to some embodiments is shown.

[0041] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0042] Figures 6A to 6BQ An exemplary user interface for displaying visual effects in an instant messaging application is shown.

[0043] Figure 7A and Figure 7B is a flow chart illustrating a method for displaying visual effects in an instant messaging application.

[0044] Figures 8A to 8BQ An exemplary user interface for displaying visual effects in a camera application is shown.

[0045] Figure 9A and Figure 9B is a flow chart illustrating a method for displaying visual effects in a camera application.

[0046] Figures 10A to 10AL An exemplary user interface for displaying visual effects in a media item viewing mode is shown.

[0047] Figure 11A and Figure 11B is a flow chart illustrating a method for displaying visual effects in a media item viewing mode.

[0048] Figures 12A to 12AP An exemplary user interface for displaying visual effects in a real-time video communication session is shown.

[0049] Figure 13A and Figure 13B is a flow chart illustrating a method for displaying visual effects in a real-time video communication session.

[0050] Figures 14A to 14M An exemplary user interface for displaying visual effects in a camera application is shown.

[0051] Figure 15A and Figure 15B is a flow chart illustrating a method for displaying visual effects in a camera application. DETAILED DESCRIPTION

[0052] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0053] Electronic devices need to provide more efficient methods and interfaces for displaying visual effects. For example, while programs for displaying visual effects already exist, these programs are inefficient and difficult to use compared to the following technology that allows users to display visual effects within various applications. Such technology can reduce the cognitive burden on users who display visual effects within applications, thereby improving productivity. Furthermore, such technology can reduce processor and battery power that would otherwise be wasted on redundant user input.

[0054] under Figure 1A to Figure 1B 、 Figure 2 、 Figure 3 、 Figures 4A to 4B and Figures 5A to 5B A description of an exemplary device for performing techniques for management event notification is provided.

[0055] Figures 6A to 6BQ An exemplary user interface for displaying visual effects in an instant messaging application is shown. Figure 7A and Figure 7B is a flowchart illustrating a method for displaying visual effects in an instant messaging application according to some embodiments. Figures 6A to 6BQ The user interface in the Figure 7A and Figure 7B The process is described below in the process.

[0056] Figures 8A to 8BQ An exemplary user interface for displaying visual effects in a camera application is shown. Figure 9A and Figure 9B is a flowchart illustrating a method for displaying visual effects in a camera application according to some embodiments. Figures 8A to 8BQ The user interface in the Figure 9A and Figure 9BThe process is described below in the process.

[0057] Figures 10A to 10AL An exemplary user interface for displaying visual effects in a camera application is shown. Figure 11A and Figure 11B is a flowchart illustrating a method for displaying visual effects in a media item viewing mode according to some embodiments. Figures 10A to 10AL The user interface in the Figure 11A and Figure 11B The process is described below in the process.

[0058] Figures 12A to 12AP An exemplary user interface for displaying visual effects in a real-time video communication session is shown. Figure 13A and Figure 13B is a flow chart illustrating a method for displaying visual effects in a real-time video communication session according to some embodiments. Figures 12A to 12AP The user interface in the Figure 13A and Figure 13B The process is described below in the process.

[0059] Figures 14A to 14M An exemplary user interface for displaying visual effects in a camera application is shown. Figure 15A and Figure 15B is a flowchart illustrating a method for displaying visual effects in a camera application according to some embodiments. Figures 14A to 14M The user interface in the Figure 15A and Figure 15B The process is described below in the process.

[0060] Although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first touch may be named a second touch, and similarly, a second touch may be named a first touch, without departing from the scope of the various embodiments described. Both the first touch and the second touch are touches, but they are not the same touch.

[0061] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0062] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0063] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod equipment, and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0064] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0065] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0066] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0067] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and external ports 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0068] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0069] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0070] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 100 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0071] Memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0072] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0073] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with communication networks and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as by a short-range communication radio component. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0074] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0075] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripherals interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in ).

[0076] A quick press of the push button optionally releases the lock on the touch screen 112 or optionally initiates the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0077] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0078] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0079] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® Touch Controller from Apple Inc. (Cupertino, California). and iPod The technology used in

[0080] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, whereas a touch-sensitive touchpad does not provide visual output.

[0081] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following patent applications: (1) U.S. patent application 11 / 381,313, filed on May 2, 2006, entitled “Multipoint Touch Surface Controller”; (2) U.S. patent application No. 10 / 840,862, filed on May 6, 2004, entitled “Multipoint Touchscreen”; (3) U.S. patent application No. 10 / 903,964, filed on July 30, 2004, entitled “Gestures For Touch Sensitive Input Devices”; (4) U.S. patent application No. 11 / 048,264, filed on January 31, 2005, entitled “Gestures For Touch Sensitive Input Devices”; (5) U.S. patent application No. 11 / 050,264, filed on January 18, 2005, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, entitled “Virtual Input Device Placement On A Touch Screen User Interface”; (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, entitled “Operation Of A Computer With A Touch Screen Interface”; (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, entitled “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”; and (9) U.S. Patent Application No. 11 / 228,768, filed March 3, 2006, entitled “Multi-Functional Hand-Held All of these applications are incorporated herein by reference in their entirety.

[0082] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, a finger, or the like, to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the user's desired action.

[0083] In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0084] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0085] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user while viewing other video conference participants on the touch screen display for video conferencing. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0086] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1A A depth camera sensor is shown coupled to a depth camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data from the environment to create a three-dimensional model of objects (e.g., faces) within the scene from a viewpoint (e.g., the depth camera sensor). In some embodiments, in conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map for different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located on the front of the device 100, enabling the user to optionally obtain an image with depth information for video conferencing while viewing other video conference participants on the touchscreen display, and for capturing selfies using the depth map data. In some embodiments, the depth camera sensor 175 is located on the rear of the device, or on both the rear and front of the device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), enabling the depth camera sensor 175 to be used in conjunction with the touchscreen display for both video conferencing and still and / or video image acquisition.

[0087] In some embodiments, a depth map (e.g., a depth map image) includes information (e.g., values) related to the distance of objects in a scene from a viewpoint (e.g., a camera, an optical sensor, a depth camera sensor). In one embodiment of a depth map, each depth pixel defines the position of the two-dimensional pixel corresponding to the depth pixel in the Z axis of the viewpoint. In some embodiments, the depth map is composed of pixels, where each pixel is defined by a value (e.g., 0-255). For example, a value of "0" represents the pixel located farthest away in a "three-dimensional" scene, and a value of "255" represents the pixel located closest to the viewpoint (e.g., a camera, an optical sensor, a depth camera sensor) in a "three-dimensional" scene. In other embodiments, the depth map represents the distance between objects in the scene and the viewpoint plane. In some embodiments, the depth map includes information about the relative depths of various features of an object of interest in the depth camera's field of view (e.g., the relative depths of the eyes, nose, mouth, and ears of a user's face). In some embodiments, the depth map includes information that enables the device to determine the outline of the object of interest in the z direction.

[0088] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100 opposite touch screen display 112 located on the front of device 100.

[0089] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: No. 11 / 241,839, entitled “Proximity Detector In Handheld Device”; No. 11 / 240,788, entitled “Proximity Detector In Handheld Device”; No. 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; No. 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and No. 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals”, which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0090] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile output on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0091] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0092] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications, if any, are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture.

[0093] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0094] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0095] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single-finger contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multiple-finger contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0096] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, a mouse "click" threshold for a touchpad or touchscreen can be set to any one of a large range of predefined thresholds without changing the touchpad or touchscreen display hardware. In addition, in some embodiments, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0097] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0098] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0099] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0100] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0101] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0102] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as the weather widget, the local yellow pages widget, and the map / navigation widget).

[0103] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0104] Contacts module 137 (sometimes called an address book or contact list);

[0105] Telephone module 138;

[0106] Video conferencing module 139;

[0107] Email client module 140;

[0108] Instant messaging (IM) module 141;

[0109] Fitness support module 142;

[0110] A camera module 143 for still and / or video images;

[0111] Image management module 144;

[0112] Video player module;

[0113] Music player module;

[0114] Browser module 147;

[0115] Calendar module 148;

[0116] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6;

[0117] A widget creator module 150 for forming user-created widgets 149-6;

[0118] Search module 151;

[0119] Video and music player module 152, which combines the video player module and the music player module;

[0120] Notepad module 153;

[0121] Map module 154; and / or

[0122] Online video module 155.

[0123] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0124] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0125] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0126] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the touch / motion module 130, the graphics module 132, the text input module 134, the contacts module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0127] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0128] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0129] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing, and transmitting fitness data.

[0130] In combination with the touch screen 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0131] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0132] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0133] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0134] In conjunction with the RF circuit 108, the touch screen 112, the display system controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., the Yahoo! desktop widget).

[0135] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., to transfer a user-specified portion of a web page into a desktop widget).

[0136] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0137] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0138] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0139] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0140] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos. Additional descriptions of online video applications can be found in U.S. provisional patent application 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. patent application 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0141] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as separate software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0142] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0143] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, when the user touches the touchpad, the device 100 is navigated to a main menu, home menu, or root menu from any user interface displayed on the device 100. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0144] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0145] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0146] In some embodiments, the application internal state 192 includes additional information, such as one or more of: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information ready for display by the application, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0147] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (via audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0148] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0149] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0150] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0151] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0152] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0153] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0154] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0155] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0156] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 is part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0157] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0158] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0159] The event comparator 184 compares the event information with a predefined event or sub-event definition and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0160] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view (in which three user interface objects are displayed on touch display 112), when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0161] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0162] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0163] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0164] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag obtains the flag and performs a predefined process.

[0165] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0166] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0167] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0168] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0169] Figure 2A portable multifunction device 100 with a touch screen 112 is shown according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0170] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally being executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0171] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0172] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes a circuit system (sometimes referred to as a chipset) that interconnects the system components and controls communication between the system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The tactile output generator 167 described above), sensor 359 (e.g., optical sensor, acceleration sensor, proximity sensor, touch sensor and / or contact intensity sensor (similar to the above reference Figure 1A The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310. In some embodiments, the memory 370 stores data related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0173] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the memory device mentioned previously.Each module in the above-mentioned modules corresponds to the instruction set for performing the above-mentioned functions.Above-mentioned modules or programs (for example, instruction set) need not be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0174] Attention is now turned to an embodiment of a user interface, optionally implemented on, for example, portable multifunction device 100 .

[0175] Figure 4A An exemplary user interface for an applications menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0176] One or more signal strength indicators 402 of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0177] Time 404;

[0178] Bluetooth indicator 405;

[0179] Battery status indicator 406;

[0180] A tray 408 with icons for commonly used applications, such as:

[0181] o an icon 416 of the phone module 138 labeled “Phone,” which optionally includes an indicator 414 of the number of missed calls or voice messages;

[0182] o an icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0183] o An icon 420 labeled "Browser" of the browser module 147; and

[0184] o an icon 422 labeled “iPod” for the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0185] Icons for other apps, such as:

[0186] o Icon 424 of the IM module 141 labeled “Messages”;

[0187] o Icon 426 labeled “Calendar” of calendar module 148;

[0188] o Icon 428 labeled “Photos” of the image management module 144;

[0189] o an icon 430 labeled “Camera” of the camera module 143;

[0190] o Icon 432 labeled “Online Video” of the online video module 155;

[0191] o Icon 434 labeled "Stock Market" of the Stock Market widget 149-2;

[0192] o an icon 436 labeled “Map” of the map module 154;

[0193] o Icon 438 labeled “Weather” of the Weather widget 149-1;

[0194] o Icon 440 labeled “Clock” of the Alarm Clock widget 149-4;

[0195] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0196] o An icon 444 of the notepad module 153 labeled "Notepad"; and

[0197] o An icon 446 labeled “Settings” for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0198] It should be indicated that Figure 4A The icon labels shown are exemplary only. For example, icon 422 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0199] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet or touchpad 355) of the device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451, and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0200] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B , 460 corresponds to 468 and 462 corresponds to 470 ) at contact with touch-sensitive surface 451 (e.g., Figure 4B 460 and 462 in FIG. 4 ). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of the multi-function device ( Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0201] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0202] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include a body 502 relative to the devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in . In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. As an alternative to or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of the contact (e.g., touch) applied. The one or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on their intensity, which means that touches of different intensities can invoke different user interface operations on device 500.

[0203] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0204] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0205] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a reference Figure 1A 、 Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. The I / O portion 514 can be connected to a display 504, which can have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, the I / O portion 514 can be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, the input mechanism 508 is optionally a button.

[0206] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to I / O portion 514.

[0207] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the following techniques, including processes 700, 900, 1100, 1300, and 1500 ( 7A to 7B 、 Figures 9A to 9B 、 Figures 11A to 11B 、 13A to 13B ,as well as FIG. 15A to FIG. 15B ). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5Bcomponents and configurations, but may include other components or additional components in a variety of configurations.

[0208] As used herein, the term "indicator" refers to an indication that is optionally provided on device 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figures 5A to 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0209] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in FIG, the particular user interface element is adjusted according to the detected input. In the case of a touch screen display (e.g., a touch screen display) that enables direct interaction with user interface elements on the touch screen display Figure 1A touch-sensitive display system 112 or Figure 4A In some implementations of the touch screen 112 in FIG. 1 , a detected contact on the touch screen acts as a “focus selector” such that when input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0210] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0211] In some embodiments, a portion of a gesture is identified for determining the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic strength.

[0212] The intensity of the contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface, without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0213] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0214] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the intensity of the contact (or multiple contacts) to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact to above the press input intensity threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the press input intensity threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0215] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0216] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0217] Attention is now turned to an embodiment of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0218] Figures 6A to 6BQ An exemplary user interface for displaying visual effects in an instant messaging application according to some embodiments is shown. The user interface in these figures is used to illustrate an example of a user interface including 7A to 7B The process is described below in the process.

[0219] Figure 6ADevice 600 is shown having a display 601, which in some cases is a touch-sensitive display. In some embodiments, device 600 also includes a camera 602, which includes at least an image sensor capable of capturing data representing a portion of a light spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, camera 602 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing the sensed light, in some embodiments, camera 602 is also capable of capturing other types of data, such as depth data. For example, in some embodiments, camera 602 also uses techniques based on speckle, time of flight, parallax, or focus to capture depth data. The image data captured by device 600 using camera 602 includes data corresponding to a portion of the light spectrum of the scene within the camera's field of view. In addition, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes data sufficient to determine or generate depth data for the data for the portion of the light spectrum. In some embodiments, device 600 includes one or more features of device 100, 300, or 500.

[0220] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a projector), such as an IR floodlight, a structured light projector, or a combination thereof. The light emitting device is optionally used to illuminate the object during the capture of the image by the visible light camera and the depth camera (e.g., an IR camera), and uses the information from the depth camera and the visible light camera to determine a depth map of different parts of the object captured by the visible light camera. In some embodiments, the depth map (e.g., a depth map image) contains information (e.g., a value) related to the distance of objects in the scene from the viewpoint (e.g., a camera). In one embodiment of the depth map, each depth pixel defines the position of the two-dimensional pixel corresponding to the depth pixel in the Z axis of the viewpoint. In some examples, the depth map is composed of pixels, where each pixel is defined by a value (e.g., 0-255). For example, a value of "0" represents the pixel farthest away in the "three-dimensional" scene, and a value of "255" represents the pixel located closest to the viewpoint (e.g., a camera) in the "three-dimensional" scene. In other examples, the depth map represents the distance between objects in the scene and the viewpoint plane. In some embodiments, the depth map includes information about the relative depths of various features of the object of interest in the field of view of the depth camera (e.g., the relative depths of the eyes, nose, mouth, ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the outline of the object of interest in the z direction. In some embodiments, parallax information from two cameras (e.g., two visible light cameras) is used for the rear image to display the lighting effects described herein, and a combination of depth information from the depth camera and image data from the visible light camera is used for the front image (e.g., a selfie image) to display the lighting effects described herein. In some embodiments, the same user interface is used when two visible light cameras are used to determine depth information and when a depth camera is used to determine depth information, thereby providing a consistent experience to the user even when significantly different technologies are used to determine the information used in generating the lighting effects. In some embodiments, when the camera user interface is displayed with one of the lighting effects applied, the device detects selection of a camera switch indication and switches from the front-pointing camera (e.g., a depth camera and a visible light camera) to the rear-pointing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa) while maintaining display of user interface controls for applying the lighting effect and replacing the field of view of the front-pointing camera with the field of view of the rear-pointing camera.

[0221] exist Figure 6A, device 600 displays an instant messaging user interface 603 of an instant messaging application. Instant messaging user interface 603 includes a message display area 604 that includes messages 605 sent to participants (represented by recipient identifiers 606) in the messaging session. Instant messaging user interface 603 also includes a message composition field 608 for displaying input (e.g., text input, multimedia input, etc.) sent to the participants in the messaging session. Instant messaging user interface 603 also includes a camera application affordance 609, an application dock affordance 610, a keyboard display area 612, and a text suggestion area 614.

[0222] exist Figure 6B , device 600 detects input 616 (e.g., touch input on display 601) at a location corresponding to camera application enable indication 609.

[0223] exist Figure 6C , in response to detecting input 616, device 600 launches a camera application associated with camera application enable indication 609 and displays camera application user interface 615. Camera application user interface 615 includes an image display area 620 that displays a representation of image data, such as streaming image data (e.g., a live camera preview, a live camera recording, or a live video communication session) representing objects within the field of view of a camera (e.g., a rear-facing camera or camera 602); or displays media items, such as photos or video recordings. Figure 6C In the illustrated embodiment, the image display area 620 displays a real-time camera preview 620 - 1 ′ from the rear-facing camera of the device 600 .

[0224] The camera application user interface 615 also includes an area above the image display area 620 that includes camera-specific enable indicators 617 and a completion indicator 618 for exiting the camera application user interface 615. The camera-specific enable indicators include an enable indicator 617-1 associated with a camera flash function, an enable indicator 617-2 associated with a camera mode function, an enable indicator 617-3 associated with a timer function, and an enable indicator 617-4 associated with a filter function.

[0225] The camera application user interface 615 also includes a camera options area 625 positioned below the image display area 620. The camera options area 625 includes a camera selector enable representation 627 for switching between cameras (e.g., the rear-facing camera and camera 602), and camera option enable representations 619 associated with different capture modes in which the camera can record image data. For example, the video enable representation 619-1 is associated with a function for activating a video recording capture mode for the camera, and the photo enable representation 619-2 is associated with a function for activating a still image capture mode for the camera. Figures 6C to 6AX In the discussed embodiments, device 600 is in a still image capture mode associated with photo-enabled representation 619-2. However, unless otherwise indicated, these embodiments also apply to a video recording mode associated with video-enabled representation 619-1.

[0226] Camera options area 625 also includes effects affordances 622 for enabling and disabling a mode of device 600 in which device 600 is enabled or disabled for displaying visual effects in image display area 620. Such modes of device 600 are generally referred to herein as effects modes.

[0227] The camera options area 625 also includes a capture enable indication 621, which can be selected to capture the image data represented in the image display area 620. In some embodiments, the device 600 captures image data in a manner based on the currently enabled capture mode (e.g., video recording capture mode or image capture mode). In some embodiments, the device 600 captures image data based on the type of gesture detected on the capture enable indication 621. For example, if the device 600 detects a tap gesture on the capture enable indication 621, the device 600 captures a static image of the image data represented in the image display area 620 at the beginning of the tap gesture. If the device 600 detects a tap and hold gesture on the capture enable indication 621, the device 600 captures a video record of the image data represented in the image display area 620 for the period of time that the tap and hold gesture lasts. In some embodiments, the video recording stops when the finger is lifted off the enable indication. In some embodiments, the video recording continues until a subsequent input (e.g., a tap input) is detected at a location corresponding to the enable indication. In some embodiments, the captured image (e.g., a still image or a video recording) is then inserted into the message composition field 608 and then sent to the participants in the messaging conversation. In some embodiments, the captured image is sent directly to the participants in the messaging conversation without inserting the captured image into the message composition field 608.

[0228] exist Figure 6D, device 600 detects input 623 on effect indicator 622 (e.g., a tap gesture on display 601).

[0229] exist Figure 6E , in response to detecting input 623, the device 600 activates the camera 602 (e.g., switches from the rear-facing camera) and updates the image display area 620 to display a real-time camera preview 620-1 from the camera 602, thereby showing a representation of an object 632 positioned in the field of view of the camera 602 and a background 636 displayed behind the object 632. As described herein, in some embodiments, the image data captured using the camera 602 includes depth data that can be used to determine the depth of objects in the field of view of the camera 602. In some embodiments, the device 600 interprets the depth of the detected objects (e.g., in the image data) based on their depth and uses this determination to apply the visual effects discussed herein. For example, the device 600 may classify the object 632 as being in the foreground of the real-time camera preview 620-1 and classify objects positioned behind the user as being in the background of the real-time camera preview 620-1. These background objects are generally referred to herein as the background 636.

[0230] Device 600 also highlights effect enable indication 622 to indicate that visual effects are enabled for display, and updates camera options area 625 by replacing camera options enable indication 619 with visual effect option enable indication 624. Visual effect option enable indications include avatar effect enable indication 624-1 and sticker effect enable indication 624-2. Visual effect option enable indications 624 correspond to different visual effects that can be applied to the image displayed in image display area 620. By selecting one of the visual effect option enable indications (e.g., 624-1 or 624-2), a menu is displayed with visual effect options corresponding to the selected visual effect option enable indication.

[0231] The user can activate or deactivate the effects mode of device 600 by selecting an effects enable indication 622. When effects enable indication 622 is highlighted, the effects mode of device 600 is enabled to display the visual effects in image display area 620. If the user selects the highlighted enable indication 622, effects enable indication 622 is no longer highlighted, and effects mode is disabled so that the visual effects are not enabled for display in image display area 620. In some embodiments, when effects mode is enabled, device 600 updates the image shown in image display area 620 to display one or more visual effects that have been applied to the image (including visual effects applied to a real-time image stream), and when effects mode is disabled, device 600 removes or hides the visual effects from the image shown in image display area 620.

[0232] exist Figure 6F, device 600 detects input 626 on avatar effect affordance 624 - 1 (eg, a tap gesture on display 601 ).

[0233] exist Figure 6G In response to detecting input 626, device 600 displays an avatar options menu 628 having a scrollable list of avatar options 630. Avatar options menu 628 also includes a selection area 629 for indicating the selected avatar option among avatar options 630. Figure 6G As shown, the robot avatar option 630-3 is positioned in the selection area 629, which indicates that the robot avatar option 630-1 is selected. Figure 6G Other avatar options 630 shown include a customized female avatar option 630-1, a blank avatar option 630-2, a rabbit avatar option 630-4, and an alien avatar option 630-5. As described in detail below, the blank avatar option 630-2 corresponds to an avatar option that does not display an avatar on the object in the image display area 620. In some embodiments, when the avatar options menu 628 is displayed, the blank avatar option 630-2 is the default selected avatar option.

[0234] The avatar options 630 correspond to virtual avatar visual effects applied to the representation of the object in the image display area 620. Specifically, each avatar option 630 corresponds to a virtual avatar that, when selected, is transformed onto the face of the object in the image display area, while other portions of the image in the image display area (such as the background of the user or other portions, such as the user's body) remain displayed. A user (e.g., subject 632) positioned in the field of view of the camera 602 can control visual aspects of the virtual avatar by changing the pose (e.g., rotation or orientation) of their face, including moving various facial features (e.g., winking, sticking out the tongue, smiling, etc.). Details for controlling the display and movement of virtual avatars are provided in U.S. patent application serial number 15 / 870,195, which is hereby incorporated by reference herein for all purposes.

[0235] In some embodiments, the virtual avatar is a representation of the user that can be graphically depicted (e.g., a graphical representation of the user). In some embodiments, the virtual avatar is non-photorealistic (e.g., cartoonish). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond to (e.g., are mapped to) one or more physical features of the user's face, such that detected movement of the user's physical features (e.g., as determined based on a camera such as a depth sensing camera) affects the avatar features (e.g., affects the graphical representation of the features).

[0236] In some examples, the user is able to manipulate the characteristics or features of the virtual avatar using camera sensors (e.g., camera module 143, optical sensor 164) and optionally a depth sensor (e.g., depth camera sensor 175). When the user's physical characteristics (such as facial features) and position (such as head position, head rotation, or head tilt) change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical characteristics and position. In some embodiments, the changes to the user's physical characteristics and position indicate a variety of expressions, emotions, scenes, tones, or other non-verbal communications. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, scenes, tones, or other non-verbal communications.

[0237] In some embodiments, the virtual avatar is a customizable avatar (e.g., customizable avatar 835). A customizable avatar is a virtual avatar that can be selected and customized by a user, for example, to achieve a desired appearance (e.g., to look like the user). A customizable avatar typically has the appearance of human characteristics, rather than non-human characteristics, such as anthropomorphic constructions of animals or other non-human objects. Additionally, if desired, an avatar editing user interface (e.g., such as described below with respect to Figures 8AA to 8AB In some embodiments, customizable avatars can be created and configured to achieve customized physical appearance, physical structure, or patterned behavior.

[0238] In some embodiments, a virtual avatar is a non-customizable avatar. A non-customizable avatar is a virtual avatar that can be selected by a user but is generally not fundamentally configurable, although their appearance can be changed through facial tracking, as described in more detail below. In contrast, a non-customizable avatar is pre-configured and generally does not have features that can be modified by the user. In some embodiments, a non-customizable avatar has an appearance of non-human characteristics, such as an anthropomorphic structure of an animal or other non-human object (e.g., see robot avatar 633, rabbit avatar 634). A non-customizable avatar cannot be created or modified by a user to achieve significant changes in the physical appearance, physical structure, or patterned behavior of the non-customizable avatar.

[0239] Because the robot avatar 630-3 is Figure 6G, so that the device 600 displays a robot avatar 633 positioned on the face of an object 632 (e.g., a user) displayed in the image display area 620. The device 600 displays the avatar so that the avatar remains on the face of the object as the object moves within the field of view of the camera 602. The device 600 also continuously modifies the avatar based on any detected changes in the user's face, including changes in facial posture or facial expression, while maintaining the displayed object's body and background 636 in the image shown in the image display area 620.

[0240] like Figure 6H As shown, if the device 600 detects that the user's head has moved beyond the field of view of the camera 602, the device 600 displays, for example, a prompt 638 including a representation 640 of the selected avatar (e.g., a robot avatar) and a message 642 instructing the user to reposition their head in the field of view of the camera 602. In some embodiments, the prompt 638 is displayed with a blurred background 644 (including blurred user 632 and background 636). In some embodiments, displaying the prompt 638 includes an animation showing the selected avatar returning to the center position of the image display area 620, and showing the slow movement of the avatar and its features as if they were settled at a certain stop based on the physical model of the avatar's features.

[0241] Once the device 600 detects that the user's head has returned to the field of view of the camera 602, the device continues to update the selected avatar (e.g., robot avatar 633) based on the changes detected in the user's face, as shown in FIG. Figure 6I In some embodiments, when the device 600 detects that the user's face returns to the field of view of the camera 602, the device 600 displays the avatar 633 moving from the center position of the image display area 620 to the position of the user's face, and restores the modification to the avatar based on the detected change to the user's face.

[0242] exist Figures 6I to 6K , the device 600 detects input 644 (e.g., a scroll gesture on the display 601) on the displayed list of avatar options 630. As the input 644 moves in a leftward direction across the display 601, the device 600 displays the avatar options 630 scrolled to the left to display additional avatar options (e.g., a poop avatar option and a fox avatar option). In some embodiments, when one of the displayed avatar options enters the selection area 629, the device generates tactile feedback (e.g., tactile output with or without audio output) to indicate to the user that terminating the input 644 at that time will result in selection of the avatar option currently positioned within the selection area 629. For example, in Figure 6K , when the alien head option 630 - 5 is positioned within the selection area 629 , the device 600 generates tactile feedback 646 .

[0243] exist Figure 6J , the device 600 detects the termination of input 644 and determines that the alien avatar option 630-5 is located within the selection area 629. Therefore, the device 600 replaces the robot avatar 633 with the alien avatar 635 corresponding to the selected alien avatar option 630-5, while maintaining the display of the subject's body and background 636. In some embodiments, replacing the previously selected avatar (e.g., robot avatar 633) with the currently selected avatar (e.g., alien avatar 635) includes an animation of the previously selected avatar moving off the screen (e.g., out of the image display area 620) and an animation of the currently selected avatar moving from the avatar options menu 628 to the user's face in the image display area 620. In some embodiments, facial tracking of the currently selected avatar begins when the avatar moves from the avatar options menu 628 to the user's face. In some embodiments, the background is blurred when the avatar switching operation occurs.

[0244] exist Figures 6M to 6N , the device 600 detects a vertical gesture 646 on the avatar options menu 628 (e.g., a vertical swipe or touch and drag on the display 601), and in response, enlarges the avatar options menu 628 into a magnified view 628-1, as shown in FIG. Figure 6N As shown. Figure 6N In FIG, the avatar options 630 are arranged in a matrix and can be as follows Figure 6O As shown, the robot avatar option 630 - 3 is selected (eg, gesture 648 selects the robot avatar option 630 - 3 ) to display a different avatar in the image display area 620 .

[0245] exist Figure 6P , in response to gesture 648, the device 600 displays (e.g., by shrinking the menu) the avatar options menu 628 with the robot avatar option 630-3 selected (positioned in the selection area 629) and the robot avatar 633 is displayed on the user's head in the image display area 620.

[0246] exist Figure 6Q , the device 600 detects input 649 on the close icon 650 (e.g., a tap gesture on the display 601) to close the application options menu 628 and return to Figure 6R6, which shows a camera user interface 615 showing a camera options area 625 while maintaining a robot avatar 633 displayed over the user's face in the image stream (e.g., live camera preview) on the image display area 620. The camera options area 625 shows a capture affordance 621, a visual effects options affordance 624, and a highlight effects affordance 622. The user can select the avatar menu affordance 624-1 to display the avatar options menu 628 again, as shown below. Figure 6S and Figure 6T As shown in .

[0247] For example, in Figure 6S , device 600 detects input 652 on avatar effect affordance 624-1 (e.g., a tap gesture on display 601). Figure 6T In response to detecting input 652, device 600 displays avatar options menu 628 having avatar options 630, which replaces camera options area 625 including capture affordance 621, effects affordance 622, and visual effects options affordance 624. Avatar options menu 628 can be closed again by selecting close affordance 650 as previously discussed, returning to the previous display. Figure 6U The camera application user interface 615 is shown.

[0248] exist Figure 6U , device 600 detects input 654 on sticker effect affordance 624-2 (e.g., a tap gesture on display 601). Figure 6V , in response to detecting input 654, the device 600 displays a sticker options menu 656 having a scrollable list of stickers 658. Stickers are static graphical objects that can be selected by a user and applied to an image in the image display area 620. In some embodiments, a sticker can be selected by a touch and drag gesture that starts on a selected sticker, drags the sticker to a desired location, and then places the sticker at the desired location by terminating the drag (e.g., lifting a finger). In some embodiments, a sticker can be selected by a tap gesture that then displays the corresponding sticker at a location on the image display area 620. An example of such an embodiment is shown in FIG. Figures 6W to 6X , which shows a tap gesture 660 on a helmet sticker 658 - 1 , which is then displayed in the center of the image display area 620 .

[0249] Figures 6Y to 6Z A placed sticker (e.g., sticker 658-1) can be moved to a new location by dragging the sticker (see Moving Sticker 658-1 to a new location). Figure 6Z The image is repositioned on the image display area 620 by a drag gesture 662 to a new position in the image.

[0250] Figure 6AA It is shown that a placed sticker (eg, sticker 658 - 1 ) can be rotated using a two-finger gesture 664 .

[0251] Figure 6AB It is shown that the placed sticker (e.g., sticker 658-1) can be resized using a pinch gesture or an expand gesture. Figure 6AB , helmet sticker 658-1 is enlarged in response to expand gesture 666. Figure 6AC The resized, rotated, and repositioned helmet sticker 658-1 is shown placed on the head of the robot avatar.

[0252] exist Figure 6AD , the device 600 detects input 668 on the close icon 670 (e.g., a tap gesture on the display 601) to close the sticker options menu 656 and return to the Figure 6AE The camera application user interface 615 shown includes a camera options area 625 that displays a capture enable indication 621, a visual effect option enable indication 624, and a highlight effect enable indication 622 that indicates that visual effects are enabled for display. Figure 6AE , device 600 shows a representation of the user in the image on image display area 620, but the representation is modified with visual effects including a robot avatar 633 (e.g., selected from avatar options menu 628) and a helmet sticker 658-1 (e.g., selected from sticker options menu 656). The display of the visual effect is dynamic. For example, device 600 continuously modifies robot avatar 633 in response to detecting changes in the user's face. In some embodiments, the positioning of sticker 658-1 is also dynamic and changes based on detected user movement, as discussed in more detail below.

[0253] When visual effects are enabled on device 600 (e.g., effects enable representation 622 is shown as being highlighted), the applied visual effects (e.g., avatars and stickers) may be removed or hidden from image display area 620 by deselecting the highlighted effects enable representation 622 (e.g., selecting the effects enable representation 622 when it is highlighted to disable the visual effects). Figure 6AF , the device detects input 672 on the highlighted effect affordance 622 (e.g., a tap gesture on display 601). In response, Figure 6AG As shown, the device 600 removes the displayed visual effects (eg, Figure 6AF632 and background 636 in the field of view of the camera 602 (e.g., the real-time camera preview 620-1).

[0254] In some embodiments, after visual effects mode is disabled (e.g., by deselecting highlighted effects enable representation 622), the removed visual effects can be restored, for example, by reselecting effects enable representation 622 within a predetermined amount of time. Figure 6AH , the device detects input 674 (e.g., a tap gesture on display 601) on the unhighlighted effect-enabled representation 622. In response to detecting input 674, device 600 highlights effect-enabled representation 622 and displays helmet sticker 658-1 and robot avatar 633 again, while continuing to display the image stream (e.g., live camera preview 620-1) including the body of object 632 and background 636, as shown in FIG. Figure 6AI shown.

[0255] exist Figure 6AJ In FIG. 6 , device 600 detects input 676 on capture-enabled representation 621 (eg, a tap gesture on display 601 ). In response to detecting input 676, device 600 captures Figure 6AJ The real-time camera preview 620-1 images in Figure 6AK 620 - 2 is displayed in the media item 620 - 2 (e.g., a still image of the state of the real-time camera preview 620 - 1 when the device 600 detects input 676 on the capture enable representation 621 ).

[0256] In some embodiments, when an image is captured (e.g., stored as a media item), the device 600 encodes the depth data into the media item. Storing the depth data in the media allows depth-based effects (e.g., effects based on the position of an object (e.g., a user's face) in the z-direction) to be applied later. In some embodiments, when an image is captured with an effect applied, the effect is encoded directly into the visual (e.g., RGB) information to improve compatibility with other devices.

[0257] exist Figure 6AK , device 600 displays camera application user interface 615 showing an image (e.g., media item 620-2) captured in image display area 620 (e.g., media item 620-2 replaces live camera preview 620-1 shown in image display area 620). Media item 620-2 is a representation of live camera preview 620-1 when capture affordance 621 is selected. Thus, media item 620-2 includes helmet sticker 658-1 and robot avatar 633 displayed on the face of subject 632, as well as background 636.

[0258] If the user subsequently sends the media item as described below, device 600 also replaces the camera-specific affordance (e.g., Figure 6AJ ), the recipient identifier indicates the currently selected recipient of the media item 620-2.

[0259] Device 600 displays a camera options area 625 that includes visual effects options affordances 624. Visual effects options affordances 624 can be selected to display their respective options menus that can be used to modify captured media item 620-2 (and recorded video media item 620-4, described below).

[0260] Device 600 also updates camera options area 625 to replace capture affordance 621 and camera selector affordance 627 with tag affordance 677, edit affordance 678, and send affordance 680. Tag affordance 677 allows the user to tag media item 620-2. Edit affordance 678 allows the user to edit media item 620-2, such as by cropping the image or adjusting other properties of media item 620-2. Figure 6AK As shown, the send enable indication 680 is displayed in the same position / area as the capture enable indication 621, so that the send enable indication 680 replaces the capture enable indication 621 on the display.

[0261] Send affordance 680 allows the user to immediately send media item 620-2 to the recipient indicated by recipient identifier 606. For example, Figure 6AL , device 600 detects input 682 on send affordance 680 (e.g., a tap gesture on display 601) and immediately sends media item 620-2 to contact John corresponding to recipient identifier 606, as shown in FIG. Figure 6AM In some embodiments, device 600 sends media item 620-2 without displaying instant messaging user interface 603. In some embodiments, such as in Figure 6AM In the embodiment shown in FIG, the device sends media item 620-2 and displays an instant messaging user interface 603 to indicate that media item 620-2 is sent to a participant (e.g., John) in the messaging conversation, the instant messaging user interface having media item 620-2 in message display area 604 and displaying an application dock 690 positioned above keyboard display area 612. Application dock 690 includes application affordances 692 that can be selected to access different applications.

[0262] In some embodiments, media item 620-2 is not immediately sent to the participants in the instant messaging session. Figure 6AN In the example shown in FIG. 6 , instead of selecting send affordance 680, the user selects completion affordance 618. In response to detecting input 684 on completion affordance 618 (e.g., a tap gesture on display 601), device 600 displays a representation of media item 685 in message composition field 608 of instant messaging user interface 603, as shown in FIG. Figure 6AO In this embodiment ( Figures 6AO to 6AR In the example shown in FIG. 6 , instead of displaying the application dock 690, the device 600 displays a text suggestion area 614 and a keyboard display area 612 to allow the user to write text (e.g., 687) or optionally add other message content (e.g., multimedia content) before sending the message. In response to detecting input 686 on the send icon 688 (e.g., a tap gesture on the display 601), the device 600 sends a representation of the media item 685 to the participants in the message conversation, such as Figures 6AQ to 6AR shown.

[0263] In some embodiments, sending the media item 685 includes sending the media item along with the encoded depth data in the media item. Sending the media item along with the depth data in the media allows depth-based effects (e.g., effects based on the position of an object (e.g., a user's face) in the z-direction) to be applied later (e.g., by a recipient). In some embodiments, when the media item is sent, the effects are encoded directly in the visual (e.g., RGB) information to improve compatibility with other devices.

[0264] In some embodiments, after sending the representation of the media item 685, the device 600 displays the following Figure 6AR Instant messaging user interface 603 is shown, having a text suggestion area 614 displayed above keyboard display area 612. In response to receiving input 694 on application dock affordance 610 (e.g., a tap gesture on display 601), device 600 replaces text suggestion area 614 with application dock 690, as shown in FIG. Figure 6AT shown.

[0265] In some embodiments, the application dock 690 remains displayed in the instant messaging user interface 603 until the user selects the message composition field 608 or the keyboard area 612 (an action associated with composing message text). Figure 6AU As shown, the device 600 detects input 696 on the keyboard display area 612 (e.g., a tap gesture on the device 601). In response, as shown Figure 6AVAs shown, the device 600 removes the application dock 690 and displays the text suggestion area 614 above the keyboard display area 612. In some embodiments, if the input 696 is an input on the message composition field (rather than the keyboard display area 612), the device 600 also removes the application dock 690 and displays the text suggestion area 614, as shown in FIG. Figure 6AV shown.

[0266] Figure 6AW The application dock indicator 610 is shown to be selected (e.g., in response to Figure 6AS Instant messaging user interface 603 after input 694 in . Application dock 690 is displayed above keyboard selection area 612 (e.g., replacing text suggestion area 614). In response to detecting input 698 on application affordance 692-1 (e.g., a tap gesture on display 601), device 600 displays application display area 699 at the location previously occupied by text suggestion area 614 and keyboard display area 612. Application display area 699 optionally includes application dock 690, which includes an indication 6100 of the selected application affordance (e.g., application affordance 692-1), as shown in FIG. Figure 6AX shown.

[0267] In some embodiments, the application display area 699 includes graphical objects 6102 that can be selected for use in the instant messaging user interface 603. In some embodiments, the type of graphical object displayed in the application display area 699 depends on the application affordance selected to invoke the display of the application display area. Figure 6AX In the illustrated embodiment, the selected application affordance 692-1 corresponds to an avatar application. Therefore, graphical objects 6102-1 and 6102-2 displayed in application display area 699 correspond to avatars that can be selected for use in the instant messaging user interface. If the application affordance selected from application dock 610 corresponds to a sticker application, graphical object 6102 corresponds to a sticker that can be selected for use in the instant messaging user interface.

[0268] As mentioned above, about Figures 6C to 6AX The foregoing embodiments described are discussed with respect to device 600 operating in a still image capture mode of operation associated with photo-enabled representation 619-2. However, unless otherwise indicated, these embodiments also apply to the operation of device 600 when the device is operating in a video recording mode associated with video-enabled representation 619-1. Figures 6AY to 6BQThe following description, simplified where appropriate to avoid redundancy, illustrates many of the aforementioned embodiments as applied to device 600 when operating in a video recording mode associated with video enable representation 619-1.

[0269] Figures 6AY to 6AZ Device 600 is shown switching to video recording mode in response to detecting input 6104 on video affordance 619-1 in camera application user interface 615. When switching to video recording mode, device 600 removes camera-specific affordances 617-2, 617-3, and 617-4 and displays timer 6122.

[0270] Figures 6AZ to 6BB The device 600 is shown enabling effects mode in response to input 6106 on the effects enable indication 622 and subsequently displaying the visual effects options enable indication 624 in the camera options area 625 when the video recording mode is enabled.

[0271] Figures 6BC to 6BE Device 600 is shown displaying avatar options menu 628 in response to detecting input 6108 on avatar effect enable representation 624-1. Avatar options menu 628 includes avatar options 630 that can be selected to display a corresponding avatar on the subject's face in video mode. Robot avatar option 630-3 is selected, and device 600 displays robot avatar 633 on the subject's face while maintaining display of the subject's body and background 636. Selecting close icon 650 using input 6110 closes avatar options menu 628 and returns to camera options area 625 with effects enable representation 622 highlighted and visual effects options enable representation 624 displayed.

[0272] Figures 6BF to 6BH Device 600 is shown generating a recorded video of image data represented in image display area 620. Device 600 detects input 6112 on capture-enabled representation 621 that initiates capture of image data represented in image display area 620. In some embodiments, device 600 generates a video recording of the image data based on determining that a video recording mode of device 600 is enabled. In some embodiments, device 600 generates a video recording or an optional still image capture based on input 6112 detected on capture-enabled representation 621. For example, if input 6112 is detected as a tap gesture, device 600 captures a still image of the image data represented in image display area 620. If input 6112 is detected as a tap and hold gesture, device 600 captures a video recording of the image data, as discussed in more detail above.

[0273] exist Figure 6BG, in response to detecting input 6112 on capture enable indication 621, device 600 initiates a process for recording image data represented in image display area 620, the image data including the object, background, and visual effects. Device 600 also modifies camera application user interface 615 by removing camera effects area 625, expanding image display area 620, and displaying stop enable indication 6114 and, optionally, image capture enable indication 6116 for capturing a still image of the live camera recording. Figure 6BG and Figure 6BH 6, the image display area 620 shows a real-time camera recording 620-3 of objects within the field of view of the camera 602, wherein the addition of visual effects (e.g., a robot avatar 633) is enabled according to various selections made as described above. These visual effects may also include stickers and full-screen effects, which are displayed in the image display area 620 as part of the real-time image data and recorded to form part of the resulting media item. As described herein, these visual effects may be optionally displayed or hidden in the media item after it is recorded.

[0274] exist Figure 6BH In the embodiment of the present invention, the device 600 detects changes in the user's face and modifies the robot avatar 633 based on the detected changes in the user's face while recording the real-time image stream. In response to detecting input 6118 on the stop enable indication 6114, the device 600 stops recording the image data represented in the image display area 620 and displays Figure 6BI The playback interface shown in .

[0275] exist Figure 6BI , if the user subsequently sends a media item as described herein, the device 600 displays the recorded video media item 620-4 in the image display area 620 and displays a retake indication 679 and a recipient identifier 606 that indicates the currently selected recipient of the video media item 620-4.

[0276] Device 600 also removes stop enable indicator 6114 and image capture enable indicator 6116 and displays camera options area 625 having video locator 6120 for recorded video media item 620-4, effects enable indicator 622, edit enable indicator 678, tag enable indicator 677, and send enable indicator 680. Camera options area 625 also includes visual effects option enable indicator 624. Visual effects option enable indicator 624 can be selected to display their corresponding options menus, which can be used to modify captured media item 620-4. Figures 6BJ to 6BN An example of changing the visual effects in captured video media item 620-4 is shown. Figure 6BIAs shown, the send enable indication 680 is displayed in the same position / area as the capture enable indication 6116, so that the send enable indication 680 replaces the capture enable indication 6116 on the display.

[0277] Figures 6BJ to 6BN Device 600 is shown displaying avatar options menu 628 in response to detecting input 6124 on avatar effects affordance 624-1. Figure 6BK As shown, the avatar options menu 628 shows avatar options 630, including the selected robot avatar option 630-3. Figure 6BL As shown, device 600 detects input 6126 selecting rabbit avatar option 630-4 and updates video media item 620-4 to show a rabbit avatar 634 positioned on the subject's head while still displaying the subject's body and background 636, as shown. Figure 6BM As shown. Figure 6BN As shown, in response to detecting input 6128 on the close icon 650, the device 600 closes the avatar options menu 628 and displays Figure 6BO The camera options area 625. The camera options area 625 includes a video locator 6120, which is displayed from Figure 6BJ , updates to show a rabbit avatar 634. In some embodiments, in response to input directed to the video locator 6120 (e.g., a swipe or drag input, including movement of contact starting at the location of the video locator 6120), the device quickly previews (scrubs through) the recorded video (e.g., changes the appearance of the video preview and moves the locator, or moves the playhead in the locator based on the input). In some embodiments, the quick preview of the recorded video shows how the effects applied to the video change over time (e.g., how the stickers and / or avatars move and / or change expression as the user's head / face moves and / or changes expression in the field of view of one or more cameras). In some embodiments, the video locator 6120 includes an optional enable indication for cropping the video (e.g., by changing the start or end time of the video locator by dragging an edit enable indication at the end of the video locator). For example, in response to Figure 8AW The user inputs the middle finger to the video locator 8120, and the device will quickly preview the Figures 8AW to 8AZ The video with the avatar and stickers is shown, but the quick preview is at a rate determined based on the input change (e.g., the magnitude and direction of contact movement on the touch-sensitive display). Interaction with the video locator 6120 described above (and the video locator 8120 described below) is optionally applicable in a similar manner to interaction with the other locators described herein.

[0278] As mentioned above about Figures 6AK to 6AOAs discussed, video media item 620-4 can be immediately sent to the participants in the messaging session in response to detecting an input on send affordance 680 (e.g., input 6130), such as Figure 6BP and Figure 6BQ In response to completing input on affordance 618 , a representation of video media item 620 - 4 may be displayed in message composition field 608 .

[0279] Figure 7A and Figure 7B 700 is a flow chart illustrating a method for displaying visual effects in an instant messaging application using an electronic device according to some embodiments. Method 700 is performed at a device (e.g., 100, 300, 500, 600) having a camera, a display device, and one or more input devices. Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0280] As described below, method 700 provides an intuitive way to display visual effects in an instant messaging application. The method reduces the cognitive burden on users of applying visual effects to images for sending in an instant messaging application, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to display visual effects in images more quickly and efficiently saves power and increases the time between battery charges.

[0281] An electronic device (e.g., 600) displays (702) via a display device (e.g., 601) an instant messaging user interface (e.g., 603) of a messaging conversation including at least a first participant, wherein the instant messaging user interface includes a camera enable indication (e.g., 609, an optional icon associated with a function of activating a camera application).

[0282] An electronic device (e.g., 600) detects (704) via one or more input devices a first input directed toward a camera-enabled representation (e.g., 616, a touch gesture at a location on the touch screen display corresponding to the camera-enabled representation).

[0283] In response to detecting the first input (e.g., 616), the electronic device (e.g., 600) displays (706) a camera user interface (e.g., 615). The camera user interface includes (708) a capture affordance (e.g., 621, a selectable icon associated with a function for capturing image data using a camera of the electronic device).

[0284] In some embodiments, the camera user interface (e.g., 615) includes (710) an effects mode enable representation (e.g., 622, a selectable icon associated with functionality for activating a mode in which various visual effects can be used to modify image data) associated with a mode in which visual effects are enabled for display in captured image data. Including the effects mode enable representation in the camera user interface enables the user to recognize that certain effects (e.g., visual effects) can be applied to an image via the camera user interface. Providing additional control over the device enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the visual effect can be added to a representation of image data within the camera's field of view. In some embodiments, the visual effect can be added to the captured image data. In some embodiments, the visual effect is based on depth data. In some embodiments, the electronic device detects selection of the effects mode enable representation (e.g., 623) via one or more input devices. In some embodiments, in response to detecting the selection of the effects mode enable indication, the electronic device converts the electronic device from a first camera mode (e.g., standard camera mode) to a second camera mode different from the first camera mode (e.g., effects camera mode; a mode in which multiple visual effects can be used to modify image data). In some embodiments, when the device is in the second camera mode, a visual indication that the second camera mode is running is displayed (e.g., the effects mode enable indication is highlighted).

[0285] In some embodiments, further in response to detecting a selection (e.g., 623) of an effect mode enable representation (e.g., 622), the electronic device (e.g., 600) stops displaying one or more camera mode enable representations. In some embodiments, further in response to detecting a selection of an effect mode enable representation, the electronic device displays multiple effect option enable representations (e.g., selectable icons 624 associated with functions for creating visual effects). In some embodiments, the effect option enable representation includes a sticker enable representation (e.g., 624-2) and / or an avatar enable representation (e.g., 624-1) at a position previously occupied by one or more camera mode enable representations in the camera user interface (e.g., 615). In some embodiments, the location where the effect option enable representation is displayed is any location in a specific area (e.g., camera effect area 625) where the camera mode enable representation was previously displayed. In some embodiments, the location where the corresponding effect option enable representation is displayed is the same location in the area where the corresponding camera mode enable representation is displayed. In some embodiments, displaying the effect option enable representation includes replacing the camera mode enable representation with the effect option enable representation).

[0286] In some embodiments, the electronic device (e.g., 600) detects a selection (e.g., 654) of a first effect option enable representation (e.g., sticker enable representation 624-2) in the effect option enable representation via the one or more input devices. In some embodiments, in response to detecting the selection of the first effect option enable representation in the effect option enable representation, the electronic device stops displaying multiple effect option enable representations (e.g., 624) and displays multiple selectable graphic icons (e.g., 658, stickers). Stopping displaying multiple effect option enable representations and displaying multiple selectable graphic icons in response to detecting the selection of the first effect option enable representation in the effect option enable representation enables the user to quickly and easily recognize that the first effect option enable representation in the effect option enable representation is associated with the graphic icon (e.g., sticker) option, thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, displaying stickers includes displaying an area (e.g., sticker menu 656) above the effect options enable representation, wherein the area includes a plurality of sticker options that can be selected to be displayed on the image represented in the camera user interface. In some embodiments, the user selects (e.g., 660) the sticker by tapping on the sticker (e.g., 658-1), and the sticker is automatically displayed on the image (e.g., in a default position, such as the center of the image). In some embodiments, the user selects the sticker by touching the sticker and dragging the sticker from the sticker menu onto the image. In some embodiments, while displaying the capture enable representation and further in response to detecting a selection of the first effect option enable representation in the effect options enable representation, the electronic device stops displaying the capture enable representation.

[0287] In some embodiments, the electronic device (e.g., 600) detects a selection (e.g., 626) of a second effect option enable representation (e.g., avatar enable representation 624-1) in the effect option enable representation via the one or more input devices. In some embodiments, in response to detecting the selection of the second effect option enable representation in the effect option enable representation, the electronic device stops displaying the plurality of effect option enable representations (e.g., 624) and displays an avatar selection area (e.g., avatar menu 628) having a plurality of avatar enable representations (e.g., 630, displayed in a linear arrangement) (e.g., enable representations representing avatars). Ceasing to display multiple effect option enable representations and displaying an avatar selection area in response to detecting a selection of a second effect option enable representation in the effect option enable representation enables the user to quickly and easily recognize that the second effect option enable representation in the effect option enable representation is related to the avatar selection, thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the avatar enable representation corresponds to a customizable avatar, a non-customizable avatar, or a combination thereof. In some embodiments, displaying the avatar selection area includes displaying the area (e.g., avatar menu 628) above the effect option enable representation, the area including multiple avatar enable representations that can be selected to display corresponding avatars on the image represented in the camera user interface.

[0288] In some embodiments, the electronic device (e.g., 600) detects a swipe input (e.g., 646, a vertical swipe gesture) on the avatar selection area (e.g., 628) via the one or more input devices. In some embodiments, in response to detecting the swipe input on the avatar selection area, the electronic device increases the size of the avatar selection area (e.g., 628-1) and displays multiple avatar affordances (e.g., 630, arranged in a matrix). Increasing the size of the avatar selection area and displaying multiple avatar affordances arranged in a matrix in response to detecting the swipe input on the avatar selection area enables the user to (simultaneously) view one or more additional selectable avatars that are not (simultaneously) visible in the avatar selection area. Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, increasing the size of the avatar display area includes extending the avatar display area in a vertical direction to present a full screen display of the avatar display area, wherein the avatar indicator is represented by being displayed in a matrix in the avatar display area.

[0289] In some embodiments, the camera user interface (e.g., 615) further includes a first representation of the image data (e.g., a live camera preview 620-1). Providing the first representation of the image data (e.g., the live camera preview) provides visual feedback about one or more modifications made by the user (to the image) before saving / confirming the modifications. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, further in response to detecting a selection of the effect mode enable indication (e.g., 624-1, 624-2), the electronic device (e.g., 600) stops displaying the first representation of the image data based on determining that the first representation of the image data corresponds to image data obtained from the second camera (e.g., the rear-facing camera), and displays a second representation of the image data (e.g., the live camera preview), the second representation of the image data corresponding to image data obtained from the camera (e.g., the front-facing camera). In some embodiments, the representation of image data corresponding to the forward-pointing camera includes a representation of a user positioned in the field of view of the forward-pointing camera.

[0290] In some embodiments, when the electronic device (e.g., 600) is in a second camera mode (e.g., an effects camera mode; a mode in which various visual effects can be used to modify image data), the electronic device receives a request to transition to a first camera mode (e.g., normal mode, non-effects mode) (e.g., selection of an active visual effect enable representation). In some embodiments, in response to receiving the request to transition to the first camera mode, the electronic device transitions the electronic device from the second camera mode to the first camera mode. In some embodiments, based on the first visual effect being active (e.g., actively applied to captured image data or a preview of captured image data), the electronic device deactivates the first visual effect (e.g., disables, stops displaying the displayed first visual effect).

[0291] In some embodiments, after deactivating the first visual effect, the electronic device (e.g., 600) detects a subsequent selection of an effect mode enable representation (e.g., 624-1, 624-2) via the one or more input devices. In some embodiments, in response to detecting a subsequent selection of the effect mode enable representation, the electronic device reactivates the first visual effect based on determining that the subsequent selection of the effect mode enable representation occurs within a predetermined amount of time after deactivating the first visual effect. Reactivating the first visual effect based on determining that the subsequent selection of the effect mode enable representation occurs within a predetermined amount of time after deactivating the first visual effect enables the user to quickly and easily revert to the previous visual effect (e.g., without having to reselect / recreate the effect). Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, after an effects mode enabling indication is selected to remove a visual effect from an image, if the effects mode enabling indication is selected again within a predetermined time period, the removed visual effect is restored to the image.

[0292] The electronic device (e.g., 600) detects (712) a second input (e.g., 676, a touch gesture at a location on the touch screen display corresponding to the capture enable representation) directed toward the capture enable representation (e.g., 621) via one or more input devices.

[0293] In response to detecting the second input (714), the electronic device (e.g., 600) uses the camera (e.g., 602) to capture (716) image data. In some embodiments, capturing image data includes capturing image data in a first image capture mode (e.g., photo capture mode, still image capture mode) based on the value of the characteristic of the second input (e.g., contact duration) meeting the first capture mode standard (e.g., less than a threshold duration). In some embodiments, capturing image data includes capturing image data in a second image capture mode (e.g., video capture mode, continuous capture mode) based on the value of the characteristic of the second input meeting the second capture mode standard (e.g., greater than a second threshold duration). In some embodiments, the capture enable representation (e.g., 621) is a multi-function capture enable representation. In some embodiments, when a tap on the capture enable representation is detected, the electronic device captures a photo (e.g., a still image). In some embodiments, when a tap and hold gesture on the capture enable representation is detected, the electronic device captures a video (e.g., a continuous image).

[0294] In response to detecting the second input (e.g., 676) (714), the electronic device (e.g., 600) stops (718) displaying the capture enable representation (e.g., 621).

[0295] In some embodiments, after image data is captured using a camera (e.g., 602), the electronic device (e.g., 600) displays a mark enable indication (e.g., 677), an edit enable indication (e.g., 678), and a retake enable indication (e.g., 679). In some embodiments, when displaying the mark enable indication, the edit enable indication, and the retake enable indication, the electronic device receives a fourth user input via one or more input devices. In some embodiments, in response to detecting the fourth user input, the electronic device initiates a process of editing the captured image data according to the fourth user input corresponding to the edit enable indication. In some embodiments, the editing process includes displaying one or more enable indications for editing the captured image data. In some embodiments, in response to detecting the fourth user input, the electronic device initiates a process of marking the captured image data according to the fourth user input corresponding to the mark enable indication. In some embodiments, the marking process includes displaying one or more enable indications for marking the captured image data. In some embodiments, in response to detecting the fourth user input, the electronic device initiates a process of retaking the captured image data according to the fourth user input corresponding to the retake enable indication. In some embodiments, initiating a process of retaking the captured image data includes capturing new image data and replacing the captured image data with the new image data.

[0296] In response to detecting the second input (e.g., 676) (714), the electronic device (e.g., 600) displays (720) a send enable indication (e.g., 680, a selectable icon associated with a function for sending the captured image data to a participant in the conversation or for presenting the captured image data in a composition area before subsequent sending) in the camera user interface (e.g., 615) at the location previously occupied by the capture enable indication (e.g., 621). Displaying the send enable indication in the camera user interface at the location previously occupied by the capture enable indication provides visual feedback that the captured image is ready to be sent to the intended recipient. Providing visual feedback to the user without cluttering the UI enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the location where the send enable indication (e.g., 680) is displayed is any location in the specific area (e.g., camera effects area 625) where the capture enable indication was previously displayed. In some embodiments, the location where the send affordance is displayed is the same location in the area where the capture affordance is displayed. In some embodiments, displaying the send affordance includes replacing the capture affordance with the send affordance.

[0297] In some embodiments, when displaying a send affordance (e.g., 680), the electronic device (e.g., 600) displays (722) a representation of the first participant (e.g., an icon, picture, avatar, or other identifier associated with the first participant). Displaying the representation of the first participant when displaying the send affordance enables the user to quickly and easily identify the intended recipient, thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the representation of the first participant serves as an indication to the user that the captured photo will be sent to the first participant. In some embodiments, the representation of the first participant is not displayed (724) until image data is captured using the camera (e.g., 602). In some embodiments, the camera user interface (e.g., 615) includes camera-specific affordances (e.g., 619, corresponding to filters, lighting options, timer options, etc.) that are displayed before image data is captured. In some embodiments, displaying a representation of the first participant replaces the displayed camera-specific enabling representation with the representation of the first participant.

[0298] The electronic device (e.g., 600) detects (726) via one or more input devices a third input directed toward the send enable indication (e.g., a touch gesture at a location on the touch screen display corresponding to the send enable indication 680).

[0299] In response to detecting the third input, the electronic device (e.g., 600) initiates (728) a process of sending the captured image data to the first participant (e.g., immediately sending the captured image data or presenting the captured image data in the writing area before subsequent sending).

[0300] In some embodiments, before detecting a third input directed to a send enable indication (e.g., 680), the electronic device (e.g., 600) displays a completion enable indication (e.g., an optional icon associated with a function for closing a camera user interface to display an instant messaging user interface). In some embodiments, the electronic device detects selection of the completion enable indication via one or more input devices. In some embodiments, in response to detecting selection of the completion enable indication, the electronic device displays an instant messaging user interface (e.g., 603), which has a message writing area (e.g., 608). In some embodiments, in response to detecting selection of the completion enable indication, the electronic device displays a representation of the captured image data in the message writing area. In some embodiments, selecting the completion enable indication causes the camera user interface to be closed and the captured image data to be displayed in the message writing field of the instant messaging user interface without sending the captured image data.

[0301] In some embodiments, before detecting the third user input, the electronic device (e.g., 600) is in a first image capture mode (e.g., photo capture mode, video capture mode). In some embodiments, further in response to detecting the third user input, the electronic device maintains the first image capture mode. In some embodiments, the electronic device can be configured (e.g., by the user) to capture image data according to multiple modes (e.g., photo capture mode, video capture mode). In some embodiments, the selection of the image capture mode is persistent even when the electronic device transitions from a first camera mode (e.g., standard camera mode, non-effect camera mode) to a second camera mode (e.g., effect camera mode).

[0302] In some embodiments, initiating the process of sending the captured image data to the first participant includes sending the captured image data to the first participant (e.g., without displaying the instant messaging user interface 615). In some embodiments, selecting the send affordance (e.g., 680) from the camera user interface immediately sends the captured image data to the other participant in the messaging conversation without displaying any intervening user interface or requiring further input from the user.

[0303] In some embodiments, initiating the process of sending the captured image data to the first participant includes redisplaying an instant messaging user interface (e.g., 615), wherein the instant messaging user interface also includes a keyboard area (e.g., 612) and an application menu enable representation (e.g., 610, an optional icon associated with a function for displaying the application menu user interface). Redisplaying the instant messaging user interface as part of initiating the process of sending the captured image data to the first participant provides visual feedback that the captured image data is being sent via the messaging session. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the application menu enable representation is displayed near the message composition field in the instant messaging user interface. In some embodiments, the electronic device (e.g., 600) detects selection of the application menu enable representation via one or more input devices. In some embodiments, in response to detecting selection of an application menu enable representation, the electronic device displays an application menu area near (e.g., above) the keyboard area, wherein the application menu area has multiple application enable representations (e.g., each associated with a selectable icon for initiating a function of an application associated with the corresponding application enable representation). In some embodiments, the application enable representations include sticker enable representations and avatar enable representations.

[0304] In some embodiments, the electronic device (e.g., 600) detects a fourth input on the instant messaging user interface (e.g., 615). In some embodiments, based on determining that the fourth input corresponds to the position of the keyboard area (e.g., 612) or the position of the message writing area (e.g., 608) in the instant messaging user interface, the electronic device stops displaying the application menu area and displays a text suggestion area (e.g., an area with a list of suggested words for the user to easily select) at the position previously occupied by the application menu area in the instant messaging user interface.

[0305] In some embodiments, the electronic device (e.g., 600) detects a selection of one of a plurality of application enable representations (e.g., sticker enable representations or avatar enable representations) in an application menu area. In some embodiments, in response to detecting the selection of an application enable representation (e.g., 610), the electronic device stops displaying a keyboard area (e.g., 612) and displays an application display area in an instant messaging user interface (e.g., 615) at a position previously occupied by the keyboard area, wherein the application display area includes a plurality of graphical objects (e.g., avatars or stickers) corresponding to the selected application enable representation. In some embodiments, the selected application enable representation is a sticker enable representation, and the graphical object displayed in the application display area is a sticker. In some embodiments, the selected application enable representation is a sticker enable representation, and the graphical object displayed in the application display area is an avatar (e.g., a customizable avatar and / or a non-customizable avatar).

[0306] Note that the above descriptions with respect to method 700 (e.g., 7A to 7B ) also applies in a similar manner to the methods described below. For example, methods 900, 1100, 1300, and 1500 optionally include one or more features of the various methods described above with reference to method 700. For example, visual effects such as stickers and virtual avatars are displayed in image data in a camera application user interface, a media user interface, and a user interface for a real-time video communication session. For the sake of brevity, these details are not repeated below.

[0307] Figures 8A to 8BQ An exemplary user interface for displaying visual effects in a camera application is shown according to some embodiments. The user interface in these figures is used to illustrate the Figure 9A and Figure 9B The process is described below.

[0308] exist Figure 8A , the device 600 displays a home screen 800 and detects input 801 on a camera application enable indication 802.

[0309] exist Figure 8B In response to detecting input 801, device 600 launches a camera application associated with camera application affordance 802 and displays camera application user interface 815. Camera application user interface 815 is similar to that described above with respect to Figures 6A to 6BQ The illustrated embodiment discusses a camera application user interface 615 .

[0310] The camera application user interface 815 includes an image display area 820 that displays a representation of image data, such as streaming image data (e.g., a live camera preview, a live camera recording, or a live video communication session) representing objects within the field of view of a camera (e.g., a rear-facing camera or camera 602), or displays media items, such as photos or video recordings. Figure 8B In the illustrated embodiment, the image display area 820 displays a real-time camera preview 820 - 1 ′ from the rear-facing camera of the device 600 .

[0311] The camera application user interface 815 also includes an area above the image display area 820 that includes camera-specific enable indications 817. The camera-specific enable indications include an enable indication 817-1 associated with a camera flash function, an enable indication 817-2 associated with a camera mode function, an enable indication 817-3 associated with a timer function, and an enable indication 817-4 associated with a filter function.

[0312] Camera application user interface 815 also includes a camera options area 825 (similar to camera options area 625) positioned below image display area 820 (similar to image display area 620). Camera options area 825 includes a camera selector enable representation 827 for switching between cameras (e.g., a rear-facing camera and camera 602), and camera options enable representations 819 associated with different capture modes in which the camera can record image data. For example, video enable representation 819-1 is associated with functionality for activating a video recording capture mode for the camera, and photo enable representation 819-2 is associated with functionality for activating a still image capture mode for the camera. Figures 8B to 8AQ In the discussed embodiments, device 600 is in a still image capture mode associated with photo-enabled representation 819-2. However, unless otherwise indicated, these embodiments also apply to a video recording mode associated with video-enabled representation 819-1.

[0313] Camera options area 825 also includes an effects enable representation 822 for enabling and disabling a mode (visual effects mode, effects mode) of device 600 in which device 600 is enabled or disabled for displaying visual effects in image display area 820. Effects enable representation 822 is similar to effects enable representation 622 and, therefore, has the same functionality as effects enable representation 622, unless otherwise indicated. Thus, effects enable representation 822 can be selected to enable display of visual effects and can be deselected to disable display of visual effects.

[0314] The camera options area 825 also includes a capture enable indication 821, which functions in a manner similar to the capture enable indication 621 described above. The capture enable indication 821 can be selected to capture image data represented in the image display area 820. In some embodiments, the device 600 captures image data in a manner based on the currently enabled capture option (e.g., video recording capture mode or image capture mode). In some embodiments, the device 600 captures image data based on the type of gesture detected on the capture enable indication 821. For example, if the device 600 detects a tap gesture on the capture enable indication 821, the device 600 captures a still image of the image data represented in the image display area 820 at the beginning of the tap gesture. If the device 600 detects a tap and hold gesture on the capture enable indication 821, the device 600 captures a video recording of the image data represented in the image display area 820 for the period of time that the tap and hold gesture lasts. In some embodiments, the video recording stops when the finger is lifted off the enable indication. In some embodiments, video recording continues until subsequent input (e.g., a tap input) is detected at a location corresponding to the affordance. In some embodiments, the captured image (e.g., a still image or video recording) can be shared with other devices, such as using an instant messaging application.

[0315] exist Figure 8C In FIG. 8 , device 600 detects input 823 on effect enable representation 822 .

[0316] exist Figure 8D , in response to detecting input 823, the device 600 activates the camera 602 (e.g., switches from the rear-facing camera) and updates the image display area 820 to display a real-time camera preview 820-1 from the camera 602, thereby showing a representation of an object 832 positioned in the field of view of the camera 602 and a background 836 displayed behind the object 832. As described herein, in some embodiments, the image data captured using the camera 602 includes depth data that can be used to determine the depth of objects in the field of view of the camera 602. In some embodiments, the device 600 interprets the depth of the detected objects (e.g., in the image data) based on their depth and uses this determination to apply the visual effects discussed herein. For example, the device 600 may classify the object 832 as being in the foreground of the real-time camera preview 820-1 and classify objects positioned behind the user as being in the background of the real-time camera preview 820-1. These background objects are generally referred to herein as the background 836.

[0317] Device 600 also highlights effect enable indication 822 to indicate that visual effects are enabled for display, and updates camera options area 825 by replacing camera options enable indication 819 with visual effect option enable indication 824. Visual effect option enable indication includes avatar effect enable indication 824-1 and sticker effect enable indication 824-2. Visual effect option enable indication 824 is similar to visual effect option enable indication 624 described above. Visual effect option enable indication 824 corresponds to different visual effects that can be applied to the image displayed in image display area 820. By selecting one of the visual effect option enable indications (e.g., 824-1 or 824-2), a menu is displayed with visual effect options corresponding to the selected visual effect option enable indication.

[0318] exist Figures 8E to 8F , device 600 detects input 826 on avatar effect affordance 824-1 and displays avatar options menu 828. Avatar options menu 828 is similar to avatar options menu 628. Avatar options menu 828 includes a scrollable list of avatar options 830 and a selection area 829 for indicating one of the avatar options 830 that is selected. The displayed avatar options include a customizable female avatar 830-1, a blank avatar option 830-2, a robot avatar option 830-3, a rabbit avatar option 830-4, and a second customizable female avatar 830-5. Figure 8F As shown, empty avatar option 830-2 is selected as the default avatar option 830. Thus, the representation of the subject 832 displayed in the image display area 820 is shown without an avatar displayed on the subject's face. The avatar options 830 may be preconfigured, such as avatar options 830-3 or 830-4, or they may be customizable avatars, such as avatar options 830-1 and 830-5.

[0319] Avatar options 830 have similar functionality to avatar options 630. Thus, avatar options 830 correspond to virtual avatar visual effects applied to the representation of an object in image display area 820. Specifically, each avatar option 830 corresponds to a virtual avatar that, when selected, is transformed onto the face of the object in the image display area, while other portions of the image in the image display area (such as the user's background or other portions, such as the user's body) remain displayed. A user (e.g., subject 832) positioned in the field of view of camera 602 can control the visual aspects of the virtual avatar by changing the pose (e.g., rotation or orientation) of their face as described above.

[0320] The avatar options 830 can be scrolled by gestures on the avatar options menu 828. For example, horizontal gesture 844a is Figure 8G830-3, which is then displayed as a robot avatar 833 on the subject's face in the image display area 820. Figure 8H Continuing to scroll by horizontal gesture 844b, additional avatar options 830 are scrolled onto the displayed avatar options menu 828. In some embodiments, as the user scrolls the avatar options 830, the image display area 820 shows the avatar (e.g., Figure 8J The robot avatar 833 in FIG. 8 has a blur effect 803 applied to the image data represented behind the avatar.

[0321] exist Figure 8K , device 600 displays customizable female avatar option 830-5 positioned within selection area 829 just before being selected (e.g., gesture 844b has not yet terminated), and edit-enabled representation 804 displayed below customizable female avatar option 830-5. Device 600 continues to display robot avatar 833 with blur effect 803. In some embodiments, when the customizable avatar option is positioned in selection area 829, device 600 displays edit-enabled representation 804 in selection area 829. Edit-enabled representation 804 can be selected to display an avatar editing user interface, such as described below with respect to Figure 8AA discussed.

[0322] exist Figure 8L , after avatar option 830-5 is selected (e.g., gesture 844b is terminated). The device 600 removes the robot avatar 833 from the image display area 820 (while continuing the blur effect 803) and displays the customizable female avatar 835 that appears and moves to the image display area 820 (e.g., at a central position in the image display area 820, or at a position corresponding to the subject's face) in accordance with the avatar option 830-5 within the selection area 829, wherein the avatar 835 is modified based on the detected changes in the subject's face. In some embodiments, the animation includes starting as a static appearance (e.g., similar to a sticker matching the displayed avatar option) and then transitioning to a dynamic state (in Figure 8L , wherein the avatar (e.g., 835) is modified based on changes in the user's face detected by the device 600. In some embodiments, the transition from static to dynamic occurs before the avatar moves from the selection area. In some embodiments, the transition from static to dynamic occurs when the avatar moves from the selection area 829 to the image display area 820.

[0323] exist Figure 8M and Figure 8NAs shown, when the avatar 835 is applied to the subject's face, the device 600 modifies the avatar 835 based on detected changes to the subject's face, including changes in the facial position within the field of view of the camera 602.

[0324] In some embodiments, when the device 600 no longer detects the subject's face within the field of view of the camera 602, the device 600 again applies the blur effect 803 (similar to the blur effect 644) to the background and displays a prompt 838 instructing the user to return their face to the field of view of the camera 602. Figure 8P In the illustrated embodiment, device 600 displays a prompt 838 that includes a representation 840 of the selected avatar (e.g., a customizable female avatar) and a message 842 instructing the user to reposition their head in the field of view of camera 602. In some embodiments, displaying prompt 838 includes an animation showing the selected avatar (e.g., 835) returning to a center position in image display area 820, and illustrating the slow movement of the avatar and its features as they appear to settle at a stop based on a physical model of the avatar's features.

[0325] In some embodiments, when the device 600 detects that the user's face returns to the field of view of the camera 602, the device 600 displays the avatar 835 moving from the center position of the image display area 820 to the said position of the user's face, and restores the modification of the avatar based on the detected changes to the user's face.

[0326] In some embodiments, the avatar options menu 828 may be expanded with a vertical gesture (e.g., 805) to display the avatar options menu. Figures 8Q to 8T 830-3 ). The device 600 then displays the corresponding avatar (e.g., robot avatar 833) on the user's face when the enlarged version of the avatar options menu returns to its original (e.g., compressed) state. Figure 8U shown.

[0327] In some embodiments, in response to detecting a swipe gesture on the image display area 820, the device 600 displays a different avatar on the user's head. Figure 8V, the device 600 detects a swipe gesture 806 that moves in a right direction across the image display area 820. In response to detecting the swipe gesture 806, the device 600 scrolls the avatar options 830 in the avatar options menu 828. In some embodiments, the scrolling amount is determined by the amount of the swipe gesture. Thus, the user can scroll through multiple avatars based on the amount of the swipe gesture. In addition, the scrolling direction and the corresponding movement of the avatars on the image display area 820 are determined based on the direction of the swipe gesture 806. Thus, if the swipe is in the left direction, the scrolling of the avatar options and the corresponding movement of the transition avatars can be in the left direction, or if the swipe is in the right direction, the scrolling of the avatar options and the corresponding movement of the transition avatars can be in the right direction (e.g., Figure 8V and Figure 8W shown).

[0328] When the avatar options 830 begin to scroll, the currently selected avatar option (e.g., Figure 8V 830-3) leaves the selection area 829, and as scrolling continues, new avatar options (e.g., Figure 8W 830-1) enters the selection area 829. As the avatar options scroll, the device 600 also displays an animation in the image display area 820 of the currently selected avatar (e.g., robot avatar 833) moving away from the user's face and leaving the right side of the image display area 820, while the next avatar (e.g., customizable avatar 831) enters the image display area 820 from the left and moves toward the user's face. In some embodiments, the device 600 applies a blur effect 803 to the image display area 820 as the avatars scroll across the image display area 820.

[0329] exist Figure 8V and Figure 8W , the empty avatar option 830-2 is not selected to better illustrate the transition from the currently selected avatar to the new avatar that appears on the image display area 820. However, it should be understood that the empty avatar option 830-2 can be selected using a swipe gesture on the image display area 820. In such a case, the currently selected avatar moves off the screen while the background blurs, and then the blur effect is removed to display the user's face without the avatar.

[0330] Figure 8X and Figure 8Y A selection of a different customizable avatar option (e.g., customizable avatar option 830-5) and a corresponding displayed customizable avatar 835 are shown. When customizable avatar option 830-5 is displayed in selection area 829, edit affordance 804 is displayed. Edit affordance 804 may be selected (e.g., by input 807) to display the Figure 8AA808. The avatar editing user interface 808 provides an interface for editing an avatar (e.g., customizable avatar 835) corresponding to a customizable avatar option (e.g., 830-5) associated with the selected edit affordance 804.

[0331] In some embodiments, the applied visual effects may include lighting effects, such as a shadow 851 shown on the subject's neck beneath the applied customized avatar 835 or light reflections on glasses. As the device 600 modifies the avatar 835 to mirror the user's real-time movements, the device 600 also modifies the lighting effects on the avatar 835 and those projected onto the subject, including moving the displayed position of the reflected shadow based on the relative positions of the modeled light source and the avatar 835.

[0332] like Figure 8AA As shown, the avatar editing user interface 808 includes a customizable avatar 835 and representations of various avatar feature options 810 (e.g., a facial color option 810-1 and a facial shape option 810-2), which represent the currently selected avatar feature option and the available feature options that can be selected to modify the avatar 835. The avatar feature options correspond to values ​​for various aspects of particular avatar features, particularly the currently selected avatar features, such as the avatar head features indicated by the highlighted avatar head enable indication 809-1. The avatar editing user interface 808 indicates the selected facial color option 810-1a and the selected facial shape option 810-2a represented in the avatar 835. The displayed avatar feature options can be changed by selecting different avatar features. For example, in Figure 8AB , device 600 updates avatar editing user interface 808 to display different avatar hair feature options (eg, hair quality option 811-1 and hair color option 811-2) when avatar hair affordance 809-2 is selected.

[0333] The device 600 modifies the avatar 835 when different avatar feature options are selected. For example, Figure 8AB , the device 600 detects the selection of the straight hair quality option 811-2a and updates the avatar 835 to have the selected straight hair quality option. In some embodiments, the device 600 modifies the representation of the avatar 835 shown in the avatar editing user interface 808 in response to detecting a change in the user's face.

[0334] In response to detecting selection of the completion enable indication 812, the device 600 exits the avatar editing user interface 808 and returns to the camera application user interface 815, which shows the selected avatar option 830-5 and the corresponding avatar 835 updated based on the hair quality option selected in the avatar editing user interface.

[0335] Figures 8AD to 8AF An embodiment is shown in which a new custom avatar can be created and added to the avatar options menu 828. In response to detecting selection of the new avatar affordance 813, the device 600 displays an avatar editing user interface 808 having a representation of a default avatar with no avatar feature options selected, such as Figure 8AE The device 600 may modify the default avatar based on the user's selection of the avatar feature option and save the avatar as a new avatar that can be selected from the avatar option menu 828. Figure 8AF , the device 600 detects the selection to cancel the affordance and therefore abandons saving the modified avatar in the avatar options menu 828, as shown in FIG. Figure 8AG shown.

[0336] In response to detecting Figure 8AG For selection of the cancel icon 850, the device 600 closes the avatar options menu 828 and displays the following Figure 8AH The camera options area 825 is shown.

[0337] In response to detecting selection of label effect affordance 824-2, Figure 8AI 850. The device 600 displays a sticker options menu 856 with a scrollable list of stickers 858. The sticker options menu 856 and stickers 858 are similar to the sticker options menu 656 and stickers 658 discussed above. Stickers are static graphical objects that can be selected by a user and applied to an image in the image display area 820. In some embodiments, stickers can be selected by a point-and-click gesture, which then displays the corresponding sticker at a location on the image display area 820. In some embodiments, stickers can be selected by a touch and drag gesture that starts on a selected sticker, drags the sticker to a desired location, and then places the sticker at the desired location by terminating the drag (e.g., lifting a finger). An example of such an embodiment is shown in FIG. Figures 8AI to 8AK , where the helmet sticker 858-1 is selected (e.g., input 853) and dragged to a position above the custom avatar 835 (e.g., away from the custom avatar). Figure 8AK In the example, the device 600 detects that the object 832 is Figure 8AJ In response, device 600 displays helmet sticker 858-1 and avatar 835, both of which move laterally based on the lateral movement (e.g., direction and magnitude) of the representation of the object.

[0338] exist Figure 8AK In the embodiment, device 600 detects input 854 on close enable indication 855. In response, Figure 8ALThe device 600 closes the sticker options menu 856 to display the camera options area 825. Figure 8AL In FIG, device 600 shows object 832 being displaced back to the center of the screen with the object's head tilted. In response to detecting this movement of object 832, device 600 displays helmet sticker 858-1 in FIG. Figure 8AL and shifted laterally (from Figure 8AK ). The device 600 displays the helmet sticker 858-1, which maintains a relative spacing relative to the avatar 835. However, the helmet sticker 858-1 does not rotate (e.g., tilt) in response to the rotation (e.g., tilt) of the subject's head, while the avatar 835 is rotated (e.g., tilted), as shown in FIG. Figure 8AL shown.

[0339] exist Figure 8AL , the device 600 detects a selection of the capture enable indication 821, which causes the device 600 to capture an image of the live camera preview 820-1 displayed in the image display area 820, the image including visual effects (e.g., the customized avatar 835 and the helmet sticker 858-1) and other image data including the object 832 and the background 836.

[0340] exist Figure 8AM , device 600 displays camera application user interface 815 showing an image (e.g., media item 820-2) captured in image display area 820 (e.g., media item 820-2 replaces live camera preview 820-1 shown in image display area 820). Media item 820-2 is a representation of live camera preview 820-1 when capture affordance 821 is selected. Thus, media item 820-2 includes helmet sticker 858-1 and avatar 835 displayed on the face of subject 832, as well as background 836.

[0341] Device 600 also replaces the camera-specific affordance (e.g., Figure 8AL ), and saves an affordance 818 for saving the captured media item 820-2.

[0342] Device 600 also updates camera options area 825 to replace capture affordance 821 and camera selector affordance 827 with tag affordance 877, edit affordance 878, and share affordance 880. Tag affordance 877 allows the user to tag media item 820-2. Edit affordance 878 allows the user to edit media item 820-2, such as by cropping the image or adjusting other properties of media item 820-2. Share affordance 880 allows the user to send media item 820-2 to another device, such as in an instant messaging application or an email application.

[0343] Device 600 displays a camera options area 825 that includes visual effects options affordances 824. Visual effects options affordances 824 can be selected to display their corresponding options menus that can be used to modify captured media item 820-2 (and recorded video media item 820-4, described below). For example, Figures 8AN to 8AP Device 600 is shown adding a sticker to media item 820-2. Sticker effect affordance 824-2 is shown in Figure 8AN is selected, which causes the device 600 to Figure 8AO The sticker options menu 856 is displayed. Figure 8AO Also shown is a selection of a bunny sticker 858-2 positioned on object 832 in media item 820-2. Figure 8AP As shown. Sticker options menu 856 in Figure 8AP is closed in Figure 8AQ The camera options area 825 is displayed.

[0344] exist Figure 8AQ , device 600 also displays media item 820 - 2 having a bunny sticker 858 - 2 positioned on object 832 and other visual effects (eg, avatar 835 and helmet sticker 858 - 1 ).

[0345] Stickers can be added to media items recorded in video format in a manner similar to that described above for media item 820-2 (still image). For example, Figure 8AR An embodiment in which video media item 820-4 is a recorded video is shown. Media item 820-4 can be created in a manner similar to that described above for media item 620-4.

[0346] exist Figure 8AR, device 600 displays recorded video media item 820-4 in image display area 820. Device 600 also displays camera options area 825 having a video locator 8120, effects affordances 822, edit affordances 878, tag affordances 877, and share affordances 880 for recorded video media item 820-4. Camera options area 825 also includes visual effects options affordances 824. Visual effects options affordances 824 can be selected to display their corresponding options menus, which can be used to modify captured media item 820-4. For example, Figures 8AR to 8AV Device 600 is shown adding stickers 858 - 2 and 858 - 3 to media item 820 - 4 .

[0347] In some embodiments, displayed stickers may have different patterned behaviors within a video media item (or live video stream). For example, some stickers have the appearance of being applied to a display (e.g., 601) and remaining static while objects in the image data move. An example of such a sticker is represented by Figures 8AX to 8BC In some embodiments, the sticker has this static behavior when it is placed in a media item (e.g., 820-4) or live camera preview (e.g., 820-1) when the device 600 does not detect the presence of all or a portion of the subject representation (e.g., a head or face) in the media item, live camera preview, or field of view of camera 602. For example, if the subject's face is completely off the screen (e.g., not detected in the field of view of camera 602) and a sticker is placed on image display area 820, then once the face is detected in the field of view of camera 602 and displayed on image display area 820 along with previously placed stickers, the sticker is static and does not track any movement of the subject's face.

[0348] Other stickers have the appearance of being applied to the display and moving with an object in the image (e.g., an item in the camera's field of view, including, for example, an avatar or representation of an object). In some embodiments, the sticker is placed at a location away from the object that the sticker follows. An example of such a sticker is Figure 8AK and Figures 8AX to 8BC 858-1 in the helmet sticker. The helmet sticker 858-1 follows the lateral movement of the object 832, but does not follow the rotational movement of the object. Another example of such a sticker is Figures 8BE to 8BQ When positioned away from the subject's face, the starfish sticker 858-5 follows the movement of the subject's face while maintaining its relative position relative to the face (e.g., the starfish sticker 858-5 follows the forward, backward, and side-to-side movement of the subject's face, but does not follow the rotational movement of the subject's face).

[0349] Still other stickers have the appearance of being applied to an object in the field of view of the camera 602 and moving to follow the object in the field of view (e.g., having the appearance of depth as the sticker adjusts to the object in the image). An example of such a sticker is provided by Figures 8AX to 8BC The rabbit sticker 858-2 moves with the lateral and rotational movement of the object (specifically, the object's shoulder). Another example of such a sticker is shown by Figures 8BG to 8BM The glasses sticker 858-4 is shown here. Figures 8BG to 8BM When positioned on the subject's face, the glasses sticker 858-4 follows the lateral movement (e.g., side to side, up / down), forward and backward movement, and rotational movement of the subject's face. The glasses sticker 858-4 has the appearance of being physically applied (e.g., glued) to the subject's face.

[0350] In some embodiments, the behavior of a sticker changes based on its position relative to an object in the media item (or live camera preview or camera field of view). In some embodiments, the behavior of a sticker changes in response to detecting a change in the sticker's position relative to an object. Examples of such stickers are described in more detail below. Figures 8BE to 8BQ Shown in.

[0351] Figures 8AW to 8BC 835, a helmet sticker 858-1, a bunny sticker 858-2, and a heart sticker 858-3 are shown. As the media item 820-4 advances during playback, the heart sticker 858-3 remains static, having the appearance of being applied to the display 601 and not moving. The helmet sticker 858-1 has the appearance of being applied to the display but moving (e.g., translating) in response to the change in position of the avatar 835. The bunny sticker 858-2 has the appearance of being applied to the shoulder of the object 832 in the media item 820-4. As the shoulder moves, the bunny sticker 858-2 changes position, including orientation and display size, to give the appearance of following the shoulder of the object during playback of the media item 820-4.

[0352] In some embodiments, a sticker or other virtual object applied to the field of view of one or more cameras including a representation of a corresponding object (e.g., the face, hand, or other body part of a device user) that is tracked in three dimensions (e.g., via depth information from a depth sensor) is attached to the corresponding object so that the size and / or orientation of the virtual object changes as the distance of the corresponding object from the one or more cameras and / or the orientation of the corresponding object relative to the one or more cameras changes, in addition to moving laterally (e.g., side to side and / or up and down) as the corresponding object moves laterally (e.g., side to side and / or up and down) in the field of view of the one or more cameras. For example, Figures 8AS to 8AT As shown, the bunny sticker 858-2 is placed on a representation of the subject's shoulder. Figures 8AX to 8AY As the rabbit sticker 858-2 moves upward and toward the camera, the rabbit sticker 858-2 moves laterally (upward) with the shoulder and enlarges as the shoulder moves toward the camera. In some embodiments, a sticker or other virtual object applied to a position representing a corresponding object (e.g., a face, hand, or other body part of a device user) that is tracked in three dimensions (e.g., via depth information from a depth sensor) in the field of view of one or more cameras moves laterally (e.g., side to side and / or up and down) as the corresponding object moves laterally (e.g., side to side and / or up and down) in the field of view of the one or more cameras, but is not attached to the corresponding object, so that the size and / or orientation of the virtual object does not change as the distance of the corresponding object from the one or more cameras and / or the orientation of the corresponding object relative to the one or more cameras changes. For example, as Figure 8AX and Figure 8AY As shown, the helmet sticker 858-1 is placed at a position away from the object representation. Figure 8AX and Figure 8AY Moving sideways and closer to the camera, helmet sticker 858-1 moves laterally based on the movement of the user's representation, but does not change in orientation or size.

[0353] exist Figure 8AY , the device 600 detects input 882 (e.g., a swipe or drag gesture) on the locator 8120 and quickly previews the media item 820-4 based on the input. Figure 8AZ As shown, in response to input 882, device 600 reverses through media item 820-4 to display the media item 820-4 previously viewed. Figure 8AX Device 600 detects input 884 on edit enable representation 878.

[0354] exist Figure 8BA, the device 600 enters an editing mode (for editing media item 820-4) and displays an optional enable indication 885 for trimming the video by dragging the enable indication 885 at the end of the video locator 8120 (via input 883) to change the end time of the video locator.

[0355] like Figure 8BB As shown, video locator 8120 is trimmed in response to input 883. Device 600 detects input 886 on completion affordance 887 and exits edit mode.

[0356] exist Figure 8BC , the video locator 8120 is shown as being trimmed (e.g., having a smaller size than that of the entry Figure 8AZ Device 600 detects input 888 on effects enable indication 822, which exits effects mode.

[0357] like Figure 8BD As shown, the visual effect displayed in media item 820-4 is hidden in response to detecting input 888 deselecting effect enable representation 822.

[0358] In some embodiments, a sticker may have a behavior that is determined based on the conditions of the sticker's placement (e.g., the sticker's position relative to other objects, the presence (or absence) of an object when the sticker is placed, etc.). For example, in some embodiments, a sticker may have a first type of behavior when positioned away from an object or area, and a second type of behavior when positioned on the object or area. In some embodiments, if no object is present when the sticker is placed, the sticker may have a third type of behavior. In some embodiments, the behavior of a sticker may change based on a change in the sticker's position (e.g., a change in its position relative to an object).

[0359] For example, Figures 8BE to 8BQ An exemplary embodiment is shown that illustrates the behavior of various stickers displayed in a live camera preview 820-1 and the change in behavior of the stickers based on the change in the position of the stickers relative to the subject's face. Figures 8BE to 8BQ In the illustrated embodiment, the behavior of the sticker is determined based on the sticker's position relative to the subject's face.

[0360] exist Figure 8BE , the device 600 displays a camera application user interface 815 having a camera application user interface similar to Figure 8D, but with stickers 858 displayed in image display area 820. Image display area 820 shows a live camera preview 820-1 from camera 602, showing a representation of object 832 positioned in the field of view of camera 602 and a background 836 displayed behind object 832. Glasses sticker 858-4, starfish sticker 858-5, and baseball sticker 858-6 are each displayed away from (e.g., not on) the face of object 832. Figures 8BE to 8BQ The stickers shown in FIG can be placed and moved within the image display area according to various embodiments discussed herein.

[0361] When stickers 858 are positioned away from the subject's face (e.g., not on the subject's face), they exhibit a first type of behavior. For example, the stickers follow the lateral (e.g., side-to-side and up / down), forward (e.g., toward camera 602), and backward (e.g., away from camera 602) movements of the subject's face, but do not follow rotational movements of the subject's face (e.g., do not follow pitch and yaw of the subject's face).

[0362] exist Figure 8BF , device 600 detects input 889 on glasses sticker 858-4 (e.g., a touch and drag gesture on display 601). In response to detecting input 889, device 600 displays glasses sticker 858-4 moving with the input (e.g., in the direction and magnitude of movement of input 889), as shown in FIG. Figure 8BG and Figure 8BH shown.

[0363] In some embodiments, device 600 provides an indication when the sticker moves to a location corresponding to the object that the sticker is following. Figure 8BG , the device 600 shows that the glasses sticker 858-4 moves from a position away from the subject's face to a position on the subject's face. The device 600 displays brackets 890 around the subject's face and optionally generates tactile feedback 892 (e.g., tactile output with or without audio output) when the glasses sticker 858-4 moves onto the subject's face. In some embodiments, the brackets 890 provide a visual indication of the area in which the glasses sticker 858-4 can be placed so that they will be displayed on the subject's face (causing the behavior of the glasses sticker 858-4 to be modified). Therefore, the brackets 890 (and tactile feedback 892) can indicate to the user when the user drags the sticker to the placement area (e.g., within the brackets) that positions the sticker on the subject's face. In some embodiments, the tactile feedback 892 indicates that the glasses sticker 858-4 has moved from an area outside the subject's face (e.g., an area where the sticker has a first type of behavior) to an area on the subject's face (e.g., an area where the sticker has a second type of behavior).

[0364] In some embodiments, when device 600 detects that the position of a sticker (or other visual effect) moves to a position corresponding to an object that the sticker is following, device 600 modifies the appearance of the sticker based on the position of the object and modifies the behavior of the sticker (in some embodiments, the behavior of the sticker is modified after detecting termination of input 889). Device 600 also modifies the appearance and behavior of the sticker in the opposite manner when the sticker moves from a position corresponding to the object that the sticker is following to a position away from the object.

[0365] exist Figure 8BH , when the device 600 detects that the glasses sticker 858-4 is located on the subject's face (e.g., positioned within bracket 890), the device 600 modifies the sticker based on the position (e.g., size and orientation) of the subject's face. For example, the glasses sticker 858-4 is increased in size (e.g., to proportionally match the size of the subject's face) and rotated from its static position in FIG. BG to match the subject's face. Figure 8BH In some embodiments, the sticker is positioned on a plane corresponding to the subject's nose and its coordinates change as the face moves. In some embodiments, a facial mesh (e.g., a depth mask) is used to determine the location of the nose. In some embodiments, the sticker is positioned on the plane but does not conform to the shape of the subject's face.

[0366] Device 600 also changes the behavior of the sticker to a second type of behavior (e.g., a behavior that is different from the first type of behavior). Figure 8BH As shown, the second type of behavior may include a behavior in which the sticker maintains its 3D position on the face, follows rotational movement of the subject's face (e.g., follows tilt and yaw of the face), as well as lateral movement and forward / backward movement. The behavior of the other stickers remains unchanged. Thus, the baseball sticker 858-6 and the starfish sticker 858-5 continue to have the first type of behavior because they are not placed on the subject's face. Instead, the starfish sticker 858-5 and the baseball sticker 858-6 continue to maintain their relative spacing from the face.

[0367] In some embodiments, when input 889 is positioned on the subject's face (e.g., within a bracket), bracket 890 persists following input 889. Thus, when device 600 detects termination of input 889, device 600 stops displaying bracket 890, as shown in FIG. Figure 8BI shown.

[0368] Figure 8BI and Figure 8BJ The device 600 is shown causing the glasses sticker 858-4 to rotate with the subject's face, while the starfish sticker 858-5 and the baseball sticker 858-6 remain static (unlike the glasses sticker 858-4, they do not follow the rotational movement of the subject's face).

[0369] exist Figure 8BK , the subject moves closer to camera 602. Glasses sticker 858-4 moves as the subject's face moves forward, maintaining its rotational position with the subject's face. Similarly, baseball sticker 858-6 and starfish sticker 858-5 move as the subject's face moves forward, maintaining their relative spacing relative to the subject's face. Baseball sticker 858-6 and starfish sticker 858-5 do not rotate with the subject's face.

[0370] exist Figure 8BK , device 600 detects input 893 on capture enable indication 821 and, in response, captures an image of live camera preview 820-1.

[0371] exist Figure 8BL , device 600 displays a camera application user interface 815 showing an image (e.g., media item 820-5) captured in image display area 820 (e.g., media item 820-5 replaces live camera preview 820-1 shown in image display area 820). Media item 820-5 is a representation of live camera preview 820-1 when capture affordance 821 is selected. Figure 8BL The embodiment shown in Figure 8AM The embodiment shown in is similar, but has a media item 820-5 (instead of media item 820-2) that includes a glasses sticker 858-4, a starfish sticker 858-5, and a baseball sticker 858-6 displayed relative to the face of the object 832 and the background 836.

[0372] As described herein, the display of visual effects is similar across different embodiments. For example, unless otherwise indicated, visual effects can be displayed and manipulated in a similar manner in a camera application, an instant messaging application, an avatar editing application, a real-time video instant messaging application, or any other application discussed herein. Additionally, visual effects can be displayed and manipulated in a similar manner across different types of image data. For example, unless otherwise indicated, visual effects can be displayed and manipulated in a similar manner in a live camera preview, a media item, streaming image data, or any other image data discussed herein. For example, Figures 8BL to 8BQ Stickers are shown displayed in media item 820-5 in a manner similar to that discussed above for live camera preview 820-1. Specifically, Figures 8BL to 8BQ Stickers in the presentation media item (eg, 820 - 5 ) may be repositioned and their respective behaviors changed accordingly in a manner similar to that discussed above with respect to the live image preview 820 - 1 .

[0373] exist Figure 8BMIn , the device 600 detects input 894 on the eyeglass sticker 858-4 and displays brackets 890. Figure 8BN , input 894 moves the glasses sticker 858-4 away from the subject's face. When the glasses sticker moves away from the face, the device 600 generates tactile feedback 892, stops displaying the bracket 890, and modifies the glasses sticker 858-4 back to its original shape (e.g., relative to the original shape in FIG. Figure 8BE The device 600 also modifies the behavior of the glasses sticker 858-4 back to the first type of behavior (the behavior associated with a sticker placed away from the tracked object) and modifies the appearance of the sticker based on the changed behavior. Figures 8BN to 8BQ is shown as having a larger size (with Figures 8BE to 8BG to maintain the relative spacing of the position of the glasses sticker 858-4 in the media item 820-5 relative to the subject's face, similar to Figures 8BJ to 8BK The starfish sticker 858-5 and the baseball sticker 858-6 change in size based on the forward movement of the subject's face.

[0374] Figures 8BO to 8BQ Different stickers are shown placed on the subject's face in media item 820-5, and the different stickers are modified in response to being moved onto the subject's face based on different behaviors of the stickers. Figure 8BO , device 600 detects input 895 on baseball sticker 858-6. Figure 8BP When the baseball sticker 858-6 is dragged onto the subject's face, the device 600 displays brackets 890, generates tactile feedback 892 (e.g., tactile output), and repositions the baseball sticker 858-6 based on the angle of the subject's face. In other words, the behavior of the baseball sticker 858-6 changes so that it follows the rotational movement of the subject's face, similar to how the glasses sticker 858-4 follows the rotational movement of the subject's face when placed on the subject's face.

[0375] exist Figure 8BQ , input 895 terminates and device 600 stops displaying bracket 890.

[0376] Figure 9A and Figure 9B 900 is a flow chart illustrating a method for displaying visual effects in a camera application using an electronic device according to some embodiments. Method 900 is performed at a device having a camera and a display device (e.g., 100, 300, 500, 600). Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0377] As described below, method 900 provides an intuitive way to display visual effects in a camera application. This method reduces the cognitive burden on a user to apply visual effects to an image viewed in the camera application, thereby creating a more efficient human-computer interface. For battery-powered computing devices, enabling users to display visual effects in images more quickly and efficiently saves power and increases the time between battery charges.

[0378] An electronic device (e.g., 600) displays (902) a camera user interface (e.g., 815) via a display device (e.g., 601). The camera user interface includes (904) a camera display area (e.g., 820) including a representation (e.g., 835) of image data captured via the camera (e.g., 602).

[0379] In some embodiments, the image data includes (906) depth data for the captured image or video (e.g., image data including depth aspects (e.g., depth data independent of RGB data)). In some embodiments, the image data includes at least two components: an RGB component that encodes visual characteristics of the captured image, and depth data that encodes information about the relative spacing relationships of elements within the captured image (e.g., the depth data encodes that the user is in the foreground and background elements such as a tree behind the user are in the background). In some embodiments, the depth data is a depth map. In some embodiments, the depth map (e.g., a depth map image) contains information (e.g., values) related to the distance of objects in the scene from a viewpoint (e.g., a camera). In one embodiment of the depth map, each depth pixel defines the position in the z-axis of the viewpoint at which the two-dimensional pixel to which the depth pixel corresponds is located. In some examples, the depth map is composed of pixels, where each pixel is defined by a value (e.g., 0-255). For example, a value of "0" represents a pixel located farthest away in a "three-dimensional" scene, and a value of "255" represents a pixel located closest to the viewpoint (e.g., a camera) in the "three-dimensional" scene. In other examples, the depth map represents the distances between objects in the scene and the viewpoint plane. In some embodiments, the depth map includes information about the relative depths of various features of an object of interest in the depth camera's field of view (e.g., the relative depths of the eyes, nose, mouth, and ears of a user's face). In some embodiments, the depth map includes information that enables the device to determine the outline of the object of interest in the z-direction.

[0380] In some embodiments, the depth data has a first depth component (e.g., a first portion of the depth data encoding the spatial location of an object in the camera display area; a plurality of depth pixels forming a discrete portion of a depth map, such as foreground or a particular object), the first depth component including a representation of the object in the camera display area (e.g., 820). In some embodiments, the depth data has a second depth component (e.g., a second portion of the depth data encoding the spatial location of the background in the camera display area; a plurality of depth pixels forming a discrete portion of the depth map, such as the background), separate from the first depth component, the second depth aspect including a representation of the background in the camera display area. In some embodiments, the first depth aspect and the second depth aspect are used to determine a spatial relationship between the object in the camera display area and the background in the camera display area. This spatial relationship can be used to distinguish the object from the background. This distinction can be exploited, for example, to apply different visual effects (e.g., visual effects having a depth component) to the object and the background. In some embodiments, all areas of the image data that do not correspond to the first depth component (e.g., areas of the image data that are outside the range of the depth camera) are segmented out of (e.g., excluded from) the depth map.

[0381] In some embodiments, the representation (e.g., 835) of image data captured via a camera (e.g., 602) is a live camera preview (e.g., a stream of image data representing what is in the field of view of the camera).

[0382] In some embodiments, when the first camera display mode is active, the electronic device (e.g., 600) detects a swipe gesture on the camera display area (e.g., 820). In some embodiments, in response to detecting a swipe gesture on the camera display area (e.g., the electronic device (e.g., 600) changes the appearance of the displayed representation of the selected avatar option in the camera display area from a first appearance (e.g., an appearance based on the currently selected avatar option) to a second appearance (e.g., an appearance based on a different avatar option (e.g., an empty avatar option or an avatar option corresponding to a different avatar, including different types of avatars (e.g., customizable avatars, non-customizable avatars))), where the second appearance corresponds to a different one of the multiple avatar options (e.g., a different avatar option included in the avatar selection area). Changing the appearance of the displayed representation of the selected avatar option in response to detecting a swipe gesture on the camera display area provides a user with a quick and easy way to change the representation of the selected avatar. Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0383] In some embodiments, when a different one of the multiple avatar options is an empty avatar option, the device (e.g., 600) stops displaying the representation of the avatar on the representation of the object (e.g., the device abandons replacing the image data of the user's head with the virtual avatar). In some embodiments, when a different one of the multiple avatar options is an avatar option for a different avatar character (including a customizable or non-customizable avatar character), the device replaces the selected avatar character with the different avatar character (e.g., the device replaces the representation of the avatar with a representation of the different avatar). In some embodiments, replacing the selected avatar character with the different avatar character includes displaying an animation of the different avatar character moving to the center of the screen. In some embodiments, replacing the selected avatar character with the different avatar character includes displaying an animation of the different avatar character moving to the user's head. In some embodiments, replacing the selected avatar character with the different avatar character includes blurring the background when the selected avatar is replaced. Displaying an animation once / when the selected avatar character is replaced with the different avatar character (e.g., the different avatar character moves to the center of the screen, the different avatar character moves to the user's head, and the background is blurred) provides visual feedback that the avatar character is changing. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the currently selected avatar option corresponds to a first type of avatar (e.g., a customizable avatar), and the different avatar option corresponds to a second type of avatar (e.g., a non-customizable avatar).

[0384] In some embodiments, changing the appearance of the displayed representation of the selected avatar option in the camera display area (e.g., 820) from a first appearance to a second appearance includes moving a first version of the representation of the selected avatar option off the display, the first version having the first appearance. In some embodiments, changing the appearance of the displayed representation of the selected avatar option in the camera display area from the first appearance to the second appearance includes moving a second version of the representation of the selected avatar option to approximately the center of the display, the second version having the second appearance. Moving the first version of the representation of the selected avatar option off the display and moving the second version of the representation of the selected avatar option to approximately the center of the display provides visual feedback that the first version is being replaced by the second version. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0385] In some embodiments, changing the appearance of the displayed representation of the selected avatar option from a first appearance to a second appearance includes moving a first version of the representation of the selected avatar option off the display, the first version having the first appearance. In some embodiments, changing the appearance of the displayed representation of the selected avatar option from the first appearance to the second appearance includes moving the second version of the representation of the selected avatar option to approximately the location of the object representation displayed in the camera display area (e.g., 820), the second version having the second appearance. Moving the first version of the representation of the selected avatar option off the display and moving the second version of the representation of the selected avatar option to approximately the location of the object representation displayed in the camera display area provides visual feedback that the first version is being replaced by the second version. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0386] In some embodiments, changing the appearance of the displayed representation of the selected avatar option from a first appearance to a second appearance includes modifying the visual appearance of the background displayed in the camera display area (e.g., 820) (e.g., blurring the background, desaturating the background).

[0387] The camera user interface also includes (908) a first enable representation (e.g., an enable representation corresponding to the virtual avatar) associated with a first camera display mode (e.g., a mode in which image data of the user's head is replaced with a virtual avatar).

[0388] In some embodiments, the camera user interface (e.g., 815) also includes a sticker enable representation (e.g., 824-2, an enable representation corresponding to a function that enables sticker display) associated with a sticker display mode (e.g., a mode in which stickers are enabled to be applied to image data). In some embodiments, when the image data (and optionally a representation of the selected avatar option) is displayed in the camera display area (e.g., 820), the electronic device (e.g., 600) detects a gesture (e.g., 824-2) directed toward the sticker enable representation. Figure 8AH). In some embodiments, in response to detecting a gesture pointing to a sticker indication, the electronic device activates a sticker display mode, wherein activating the sticker display mode includes displaying a sticker selection area (e.g., 856) including a plurality of sticker options (e.g., 858, stickers), detecting a selection of one of the plurality of sticker options in the sticker selection area (e.g., 858-2, stickers), and in response to detecting the selection, displaying a representation of the selected sticker option (e.g., 858-2) on the image data in the camera display area. In some embodiments, the selected sticker option displayed on the image data has the appearance of being placed on a display screen (e.g., 601) (e.g., similar to the appearance of a user placing a physical sticker on the device screen) and does not move with or interact with an object represented in the image data. In some embodiments, the selected sticker option displayed on the image data has the appearance of being placed on a display screen (e.g., similar to the appearance of a user placing a physical sticker on the device screen), but moves based on the movement of the object displayed in the image data. For example, a sticker appears to be placed on the display and interact with an object (e.g., a person) represented in the image data, however, the sticker's movement is limited to movement along the x- and y-axes. In other words, the sticker appears to be applied to the device's screen, but can be moved around the screen by the object represented in the image data (e.g., a person in the image data appears to touch or drag the sticker around the screen). In some embodiments, the selected sticker option displayed on the image data has the appearance of being inserted into the image data as an interactive object forming part of the image data. Providing the appearance of being inserted into the image data as an interactive object forming part of the image data provides visual feedback that the selected sticker can function as an interactive object within the image. Providing improved visual feedback to the user enhances the device's operability and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the device's battery life by enabling the user to use the device more quickly and efficiently. In such embodiments, the sticker appears to move based on interaction with the object represented in the image data. For example, a sticker can appear as an object sitting on a person's shoulders. As the person moves, the sticker moves with the person's shoulders to maintain the appearance of sitting on the person's shoulders. This includes movement along the x-axis, y-axis, and z-axis.

[0389] In some embodiments, while displaying representations of selected sticker options (e.g., 858-1, 858-2, 858-3, 858-4, 858-5, 858-6) on image data in a camera display area (e.g., 820), a device (e.g., 600) detects lateral movement of an object (e.g., 832) within the field of view of one or more cameras (e.g., 602). In response to detecting lateral movement of the object within the field of view of the one or more cameras, the device moves the representation of the selected sticker option laterally in accordance with the movement of the object within the field of view of the one or more cameras (e.g., regardless of the relationship of the sticker to the object) (e.g., see Figure 8AV to Figure 8AY 858-1 in the helmet sticker. Lateral movement of the selected sticker option in accordance with lateral movement of the object in the field of view of one or more cameras provides visual feedback that the selected sticker can serve as an interactive object within the image. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0390] In some embodiments, when a representation of a selected sticker option (e.g., 858-1, 858-2, 858-3, 858-4, 858-5, 858-6) is displayed on image data in a camera display area (e.g., 820), a device (e.g., 600) detects a rotation of an object (e.g., 832) in the field of view of one or more cameras (e.g., 602) (e.g., a rotation relative to an axis perpendicular to the display; for example, the object turns its head). In response to detecting the rotation of the object in the field of view of the one or more cameras, the device performs one or more of the following steps. Based on determining that the representation of the selected sticker option has (e.g., is positioned to have) a first relationship with the object (e.g., the sticker was initially (or currently) positioned on the display at a location corresponding to the object; for example, the sticker is positioned on a representation of the object's face or other designated area (e.g., shown in brackets (e.g., 890))), the device rotates the representation of the selected sticker option according to the magnitude and direction of the rotation of the object (e.g., the sticker rotates and turns to follow the tilt and yaw of the object's face) (e.g., see Figures 8BH to 8BM Based on determining that the representation of the selected sticker option does not have (e.g., is not positioned to have) a first relationship with the object (e.g., the sticker was initially (or currently) positioned on the display away from the object; e.g., the sticker is positioned outside of a representation of the object's face or other designated area), the device forgoes rotating the representation of the selected sticker option according to the rotation magnitude and direction of the object (e.g., see Figures 8BE to 8BN 858-5 and baseball sticker 858-6 in the image). Selectively rotating the representation of the selected sticker option based on whether the representation of the selected sticker option has a first relationship to the object provides visual feedback regarding the behavior of the selected sticker option and indicates that the selected sticker option can serve as an interactive object within the image. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0391] In some embodiments, when a representation of a selected sticker option (e.g., 858-1, 858-2, 858-3, 858-4, 858-5, 858-6) is displayed on image data in a camera display area (e.g., 820), a device (e.g., 600) detects movement of an object (e.g., 832) toward (or away from) one or more cameras (e.g., 602). In response to detecting that the object moves toward (or away from) the one or more cameras, the device performs one or more of the following steps. Based on determining that the representation of the selected sticker option has (e.g., is placed to have) a first relationship with the object (e.g., the sticker was initially (or currently) placed at a position on the display corresponding to the object; for example, the sticker was placed when a representation of the object's face was present (e.g., detected) within the camera's field of view), the device enlarges (or shrinks) the representation of the selected sticker option based on the amount by which the object moves toward (or away from) the one or more cameras. For example, as Figure 8AX and Figure 8AY As shown, the bunny sticker 858-2 is enlarged in response to the representation of the subject's shoulders moving toward the camera. In another example, the starfish sticker 858-5 and the baseball sticker 858-6 are enlarged as the subject moves toward the camera. Figures 8BJ to 8BK Based on determining that the representation of the selected sticker option does not have (e.g., is not positioned to have) a first relationship with the object (e.g., the sticker was initially (or currently) positioned on the display at a location away from the object; for example, the sticker was positioned when a representation of the subject's face was not present (e.g., not detected) within the field of view of the camera), the device forgoes enlarging the representation of the selected sticker option based on the amount of movement of the subject toward (or away from) the one or more cameras (e.g., see Figure 8AV to Figure 8AY858-1 and heart sticker 858-3 in the image). Selectively enlarging (or shrinking) the representation of the selected sticker option based on whether the representation of the selected sticker option has a first relationship to the object provides visual feedback regarding the behavior of the selected sticker option and indicates that the selected sticker option can act as an interactive object within the image. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0392] When an object is positioned within the field of view of a camera (e.g., 602) and a representation of the object and background (e.g., objects other than the object in the field of view of the camera) are displayed in a camera display area (e.g., 820), the electronic device (e.g., 600) detects (910) a gesture directed toward a first affordance. In some embodiments, the electronic device detects (e.g., recognizes) that the object is positioned within the field of view.

[0393] In some embodiments, the camera user interface (e.g., 815) also includes a camera display area (e.g., 820) while displaying the capture enable representation, and the camera display area includes a representation of a real-time preview of the camera's field of view (e.g., an image data stream representing the content in the camera's field of view). In some embodiments, when an object is positioned within the camera's field of view (e.g., the electronic device detects / recognizes that the object is within the field of view) and the representation of the object and the background (e.g., objects in the camera's field of view other than the object) are displayed in the camera display area, the electronic device (e.g., 600) displays a representation of the selected avatar on the representation of the object in the camera display area (e.g., the displayed user's head or face is replaced (or covered (e.g., opaquely, transparently, semi-transparently)) by the head of a virtual avatar corresponding to the selected avatar). In some embodiments, when the representation of the selected avatar is displayed on the representation of the object in the camera display area, the electronic device receives a request to display an avatar selection area. In some embodiments, in response to receiving a request to display the avatar selection area, the electronic device stops displaying the capture-enabling representation and displays (e.g., at the location previously occupied by the capture-enabling representation in the camera user interface) an avatar selection area (e.g., avatar menu 828) having multiple avatar-enabling representations. In some embodiments, in response to (or in conjunction with) the avatar selection area no longer being displayed, the capture-enabling representation is displayed (e.g., redisplayed).

[0394] In some embodiments, the camera user interface (e.g., 815) also includes a camera display area (e.g., 820) while displaying a capture enable representation (e.g., 821), the camera display area including a representation of a real-time preview of the camera's field of view (e.g., an image data stream representing the content in the camera's field of view). In some embodiments, when an object is positioned within the camera's field of view and a representation of the object and a background (e.g., objects other than the object in the camera's field of view) are displayed in the camera display area, the electronic device (e.g., 600) displays a representation of the selected avatar on the representation of the object in the camera display area (e.g., the displayed user's head or face is replaced (or covered (e.g., opaquely, transparently, semi-transparently)) by the head of a virtual avatar corresponding to the selected avatar). In some embodiments, the electronic device detects (e.g., recognizes) that the object is positioned in the field of view. In some embodiments, when the representation of the selected avatar is displayed on the representation of the object in the camera display area, the electronic device detects a change in the object's posture (e.g., position and / or orientation). In some embodiments, the posture change is detected when the user moves their head or any facial feature. In some embodiments, in response to detecting the change in posture of the object, the electronic device changes the appearance of the displayed representation of the selected avatar option based on the detected change in posture of the object, while maintaining the display of the background (e.g., as described with respect to methods 900 and Figures 9A to 9B The present invention also provides a method for controlling the appearance of an avatar option displayed based on the detected change in posture of the object, while maintaining the display of the background so that the user can quickly and easily recognize that the movement of the avatar corresponds to and / or is based on the detected movement of the user. Providing additional control options enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0395] In response to detecting a gesture directed toward a first affordance, the electronic device (e.g., 600) activates a first camera display mode. Activating the first camera display mode includes displaying (914) an avatar selection area (e.g., 829) (e.g., including a selected one of a plurality of avatar options (e.g., affordances representing different virtual avatars that can be selected to appear above a user's head in a camera display area (e.g., 820) of a camera user interface (e.g., 815))).

[0396] In some embodiments, the avatar selection area (e.g., 829) also includes an option to stop displaying the representation of the selected avatar option on the representation of the object in the camera display. In some embodiments, the electronic device (e.g., 600) receives user input corresponding to a selection of an option to stop displaying the representation of the selected avatar option on the representation of the object in the camera display area (e.g., 820). In some embodiments, in response to receiving user input corresponding to a selection of an option to stop displaying the representation of the selected avatar option on the representation of the object in the camera display area, the electronic device stops displaying the representation of the selected avatar option on the representation of the object in the camera display area.

[0397] In some embodiments, the avatar selection area (e.g., 829) includes an empty avatar option (e.g., 830-2). When the empty avatar option is selected, no avatar is displayed on the representation of the object in the camera display area (e.g., 820) (e.g., the device abandons replacing the image data of the user's head with a virtual avatar). In some embodiments, the avatar selection area includes a "cancel" enable indication (e.g., an "x" icon located in a corner of the avatar selection area). When the cancel enable indication is selected, the device stops displaying the avatar selection area and, optionally, stops displaying any selected avatar on the representation of the object (e.g., the device abandons replacing the image data of the user's head with a virtual avatar).

[0398] In some embodiments, activating a first camera display mode (e.g., an avatar display mode that replaces image data of the user's head with a virtual avatar) further includes displaying (916) a representation of the object in the camera display area (e.g., 820) without displaying a representation of the selected avatar option on the representation of the object. In some embodiments, after entering the avatar display mode, the device initially displays a representation of the object without an avatar (e.g., the device abandons replacing the image data of the user's head with the virtual avatar). In some embodiments, the avatar option initially selected upon entering the avatar display mode corresponds to an empty avatar option. When the empty avatar option is selected, the device abandons replacing the image data of the object's head with the virtual avatar.

[0399] Activating the first camera display mode includes displaying (918) a representation of the selected avatar option on the representation of the object in the camera display area (e.g., 820) (e.g., the displayed head or face of the user is replaced (or overlaid (e.g., opaquely, transparently, semi-transparently)) with the head of a virtual avatar corresponding to the selected avatar). Displaying the representation of the selected avatar option on the representation of the object in the camera display area enables the user to quickly and easily recognize that the selected avatar option is associated with the representation of the object. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, displaying the representation of the selected avatar option on the representation of the object includes using depth information obtained using one or more depth cameras of the electronic device (e.g., 600).

[0400] In some embodiments, activating the first camera display mode also includes displaying the selected avatar option in the avatar selection area (e.g., 829) as having a static appearance (e.g., the avatar appearance does not change based on detected changes in the user's face). In some embodiments, activating the first camera display mode also includes updating the selected avatar option to have a dynamic appearance that changes based on detected changes in the posture of the object (e.g., the avatar changes to mirror detected changes in the user's face). In some embodiments, activating the first camera display mode also includes displaying an animation of the selected avatar with a dynamic appearance moving from the avatar selection area to a representation of an object (e.g., a representation of the user's face) in the camera display area (e.g., 820). In some embodiments, the avatar continues to track changes in the user's face during the animated movement from the avatar selection area to the user's face in the camera display area.

[0401] In some embodiments, updating a selected avatar option to have a dynamic appearance that changes based on a detected change in an object posture includes first displaying an avatar option with a dynamic appearance before changing the appearance of the avatar option based on a detected change in an object posture, wherein the avatar option has an initial posture that corresponds to (e.g., matches) a posture of the avatar option with a static appearance.

[0402] When the first camera display mode is active, the electronic device (e.g., 600) detects (920) a posture change (e.g., the position and / or orientation of the object). In some embodiments, the posture change is detected when the user moves his or her head or any facial features.

[0403] In response to detecting a change in posture of the object, the electronic device (e.g., 600) changes (922) the appearance of the displayed representation of the selected avatar option based on the detected change in posture of the object, while maintaining display of the background (e.g., 836) (e.g., a virtual avatar displayed on the user is responsive to detected changes in the user's head and face, such that changes in the user's head or face affect changes in the displayed virtual avatar, while still displaying the background). Changing the appearance of the displayed representation of the selected avatar option based on the detected change in posture of the object while maintaining display of the background enables the user to quickly and easily recognize that the movement of the avatar corresponds to and / or is based on the detected movement of the user. Providing additional control options enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the camera user interface includes information about Figures 6A to 6BQ and Figures 9A to 9B The illustrated embodiments describe one or more features / functionality of the camera user interface. For example, the camera user interface (e.g., 815) may include an effects mode affordance (e.g., 622).

[0404] In some embodiments, an electronic device (e.g., 600) detects a horizontal swipe gesture on an avatar selection area (e.g., 829). In some embodiments, in response to detecting the horizontal swipe gesture, the electronic device displays an avatar creation enablement associated with a function for adding a new avatar option to a plurality of avatar options. Displaying the avatar creation enablement associated with the function for adding a new avatar option to a plurality of avatar options in response to detecting the horizontal swipe gesture enables a user to quickly and easily access the avatar creation enablement from the avatar selection area. Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, a horizontal swipe gesture on the avatar selection area scrolls the displayed avatar options to display the avatar creation enablement. In some embodiments, the avatar creation enablement can be selected to create a new avatar. When a new avatar is created, a new avatar option representing the created avatar is added to the plurality of avatar options (eg, 830 ) in the avatar selection area.

[0405] In some embodiments, when the first camera display mode is active, the electronic device (e.g., 600) detects a swipe gesture on the avatar selection area (e.g., 829). In some embodiments, in response to detecting the swipe gesture on the avatar selection area, the electronic device changes the appearance of the displayed representation of the selected avatar option in the camera display area (e.g., 820) from a first appearance (e.g., based on the appearance of the currently selected avatar option) to a second appearance (e.g., based on the appearance of a different avatar option (e.g., an empty avatar option or an avatar option corresponding to a different avatar, including different types of avatars (e.g., customizable avatars, non-customizable avatars))), where the second appearance corresponds to a different one of the multiple avatar options (e.g., a different avatar option included in the avatar selection area). Changing the appearance of the displayed representation of the selected avatar option in response to detecting the swipe gesture on the avatar selection area enables the user to quickly and easily change the appearance of the selected avatar option. Providing additional control options without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, when a different one of the multiple avatar options is an empty avatar option (e.g., 830-2), the device stops displaying a representation of the avatar on the representation of the object (e.g., the device abandons replacing the image data of the user's head with a virtual avatar). In some embodiments, when a different one of the multiple avatar options is an avatar option for a different avatar character (including a customizable or non-customizable avatar character), the device replaces the selected avatar character with the different avatar character (e.g., the device replaces the representation of the avatar with a representation of the different avatar). In some embodiments, replacing the selected avatar character with a different avatar character includes displaying an animation of the different avatar character moving to the center of the screen. In some embodiments, replacing the selected avatar character with a different avatar character includes displaying an animation of the different avatar character moving to the user's head. In some embodiments, replacing the selected avatar character with a different avatar character includes blurring the background as the selected avatar is replaced (e.g., 836). In some embodiments, the currently selected avatar option corresponds to a first type of avatar (e.g., a customizable avatar), and the different avatar option corresponds to a second type of avatar (e.g., a non-customizable avatar).

[0406] In some embodiments, when the first camera display mode is act...

Claims

1. A method for displaying a media user interface, comprising: In an electronic device having a display device: The media user interface is displayed via the display device, wherein the media user interface includes: a media display area including a representation of a media item; and Effect affordances indicate; detecting a gesture directed toward the effect affordance; In response to detecting the gesture directed toward the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, comprising: Based on determining that the media item includes depth data for enabling an effect, the plurality of effect options includes a corresponding effect option for applying an effect based on the depth data; and Based on determining that the media item does not include the depth data, the corresponding effect option of the plurality of effect options is unavailable for activation. 2 . The method of claim 1 , wherein the corresponding effect option corresponds to an effect for displaying an avatar in the media item. 3 . The method according to claim 1 , wherein the corresponding effect option corresponds to an effect for displaying a plurality of virtual objects moving in the media item. 4 . The method of claim 3 , wherein the trajectories along which the plurality of objects move in the media item are modified based on the presence of at least one of the objects in the media item or a visual effect applied to the media item. 5 . The method of claim 1 , wherein the corresponding effect option corresponds to an effect for displaying one or more selectable graphical icons in the media item.

6. The method of any one of claims 1 to 2, wherein the plurality of effect options includes an option for adding a tag to the media item.

7. The method of any one of claims 1 to 2, wherein the plurality of effect options includes options for applying one or more image filters to the media item.

8. A method according to any one of claims 1 to 2, wherein the depth data corresponding to the media item is obtained by a camera of the electronic device after detecting a prior selection of the effect enable representation.

9. A method according to any one of claims 1 to 2, wherein the media item is a recorded image or video, and the effect is applied based on the depth data after the media item is recorded.

10. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a display device, the one or more programs comprising instructions for performing the following operations: Displaying a media user interface via the display device, the media user interface comprising: a media display area including a representation of a media item; and Effect affordances indicate; detecting a gesture directed toward the effect affordance; In response to detecting the gesture directed toward the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, comprising: Based on determining that the media item includes depth data for enabling an effect, the plurality of effect options includes a corresponding effect option for applying an effect based on the depth data; and Based on determining that the media item does not include the depth data, the corresponding effect option of the plurality of effect options is unavailable for activation.

11. The computer-readable storage medium of claim 10, wherein the corresponding effect option corresponds to an effect for displaying an avatar in the media item.

12. The computer-readable storage medium of any one of claims 10 to 11, wherein the corresponding effect option corresponds to an effect for displaying a plurality of virtual objects moving in the media item.

13. The computer-readable storage medium of claim 12, wherein the trajectory along which the plurality of objects move in the media item is modified based on the presence of at least one of the objects in the media item or a visual effect applied to the media item.

14. The computer-readable storage medium of any one of claims 10 to 11, wherein the corresponding effect option corresponds to an effect for displaying one or more selectable graphical icons in the media item.

15. The computer-readable storage medium of any one of claims 10-11, wherein the plurality of effect options includes an option for adding a tag to the media item.

16. The computer-readable storage medium of any one of claims 10-11, wherein the plurality of effect options includes an option for applying one or more image filters to the media item.

17. A computer-readable storage medium according to any one of claims 10 to 11, wherein the depth data corresponding to the media item is obtained by a camera of the electronic device after detecting a prior selection of the effect enable representation.

18. The computer-readable storage medium of any one of claims 10 to 11, wherein the media item is a recorded image or video, and the effect is applied based on the depth data after the media item is recorded.

19. An electronic device comprising: display device, one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a media user interface via the display device, the media user interface comprising: a media display area including a representation of a media item; and Effect affordances indicate; detecting a gesture directed toward the effect affordance; In response to detecting the gesture directed toward the effect affordance, displaying a plurality of effect options for applying an effect to the media item simultaneously with the representation of the media item, comprising: Based on determining that the media item includes depth data for enabling an effect, the plurality of effect options includes a corresponding effect option for applying an effect based on the depth data; and Based on determining that the media item does not include the depth data, the corresponding effect option of the plurality of effect options is unavailable for activation.

20. The electronic device of claim 19, wherein the corresponding effect option corresponds to an effect for displaying an avatar in the media item.

21. The electronic device according to any one of claims 19 to 20, wherein the corresponding effect option corresponds to an effect for displaying a plurality of virtual objects moving in the media item.

22. The electronic device of claim 21, wherein the trajectory along which the plurality of objects move in the media item is modified based on the presence of at least one of the objects in the media item or a visual effect applied to the media item.

23. The electronic device of any one of claims 19 to 20, wherein the corresponding effect option corresponds to an effect for displaying one or more selectable graphical icons in the media item.

24. The electronic device of any one of claims 19-20, wherein the plurality of effect options includes an option for adding a tag to the media item.

25. The electronic device of any one of claims 19-20, wherein the plurality of effect options includes an option for applying one or more image filters to the media item.

26. An electronic device according to any one of claims 19 to 20, wherein the depth data corresponding to the media item is obtained by a camera of the electronic device after detecting a prior selection of the effect enable representation.

27. An electronic device as claimed in any one of claims 19 to 20, wherein the media item is a recorded image or video and the effect is applied based on the depth data after the media item is recorded.

28. An electronic device comprising: Display device; one or more input devices; and Module for executing the method according to any one of claims 1 to 2.

29. A computer program product comprising a computer program, which, when executed by a processor, causes the processor to perform the method as claimed in any one of claims 1 to 2.

Images