Emoji Recording and Sending

Abstract

The present disclosure generally relates to generating and modifying virtual avatars. An electronic device having a camera and a display device displays a virtual avatar whose appearance changes in response to changes in a face within a field of view of the camera. In response to detecting a change in one or more physical features of the face within the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims priority to U.S. Provisional Application No. 62 / 507,177, entitled "Emoji Recording and Sending", filed on May 16, 2017; U.S. Provisional Application No. 62 / 556,412, entitled "Emoji Recording and Sending", filed on September 9, 2017; and U.S. Provisional Application No. 62 / 557,121, filed on September 11, 2017. This patent application also claims priority to Danish Patent Application No. PA201770393, entitled "Emoji Recording and Sending", filed on May 29, 2017; Danish Patent Application No. PA201770720, entitled "Emoji Recording and Sending", filed on September 22, 2017; and Danish Patent Application No. PA201770721, entitled "Emoji Recording and Sending", filed on September 22, 2017. The contents of these patent applications are hereby incorporated by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to computer user interfaces, and more particularly to techniques for generating, recording, and sending emojis and avatars. Background Art

[0004] Sometimes multimedia content, such as emojis and avatars, is sent as part of messaging communication. Emojis and avatars represent a variety of predefined people, objects, actions, and / or other things. Some messaging applications allow users to select from a predefined library of emojis and avatars and send them as part of a message that may contain other content (e.g., other multimedia and / or text content). Stickers are another type of multimedia content that is sometimes sent with messaging applications. In some respects, stickers are similar to emojis and avatars in that they can represent people, objects, actions, and / or other things. Some stickers and / or messaging applications allow stickers to be associated with previously sent or received messages. Summary of the Invention

[0005] However, some techniques for using an electronic device to generate, send, and receive emojis and avatars are generally cumbersome and inefficient. For example, some existing techniques use complex and time-consuming user interfaces that may include multiple button presses or keystrokes. Some other existing techniques use complex and time-consuming methods to manipulate and generate emojis and avatars, which include requiring the user to provide a large amount of input to achieve a desired emoji (e.g., a desired animated emoji or a dynamic emoji). The existing techniques take more time than necessary, which results in a waste of the user's time and device energy. This latter consideration is particularly important in battery-powered devices.

[0006] Accordingly, the present technology provides faster and more efficient methods and interfaces for an electronic device to send and receive emojis and avatars. Such methods and interfaces optionally supplement or replace other methods for sending and receiving emojis. Such methods and interfaces reduce the cognitive burden imposed on the user and result in a more effective human-machine interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0007] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying an avatar generation interface; displaying a preview of an avatar in the avatar generation interface, wherein the preview of the avatar responds to changes in the facial appearance in the camera field of view; while displaying the preview of the avatar, detecting an input in the avatar generation interface; in response to detecting the input in the avatar generation interface: generating a static avatar representing a facial expression in the camera field of view at a corresponding time based on determining that an input has started on the preview of the avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated avatar representing a sequence of changes in the facial expression in the camera field of view over a period of time based on determining that the input includes an activation of a recording affordance in the avatar generation interface, wherein the period of time is determined based on the timing of the input.

[0008] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting an input in the virtual avatar generation interface; in response to detecting an input in the virtual avatar generation interface: generating a static virtual avatar representing the facial expression in the camera field of view at a corresponding time according to determining that an input starts on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar according to determining that the input includes activation of a recording enabling representation in the virtual avatar generation interface, the animated virtual avatar representing a sequence of changes in the facial expression in the camera field of view over a period of time, wherein the period of time is determined based on the timing of the input.

[0009] According to some embodiments, a transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting an input in the virtual avatar generation interface; in response to detecting an input in the virtual avatar generation interface: generating a static virtual avatar representing the facial expression in the camera field of view at a corresponding time according to determining that an input starts on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar according to determining that the input includes activation of a recording enabling representation in the virtual avatar generation interface, the animated virtual avatar representing a sequence of changes in the facial expression in the camera field of view over a period of time, wherein the period of time is determined based on the timing of the input.

[0010] According to some embodiments, an electronic device is described. The electronic device includes: a display; a camera; one or more processors; and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting an input in the virtual avatar generation interface; in response to detecting an input in the virtual avatar generation interface: generating a static virtual avatar representing the facial expression in the camera field of view at a corresponding time according to determining that an input starts on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar according to determining that the input includes an activation of a recording enablement indication in the virtual avatar generation interface, the animated virtual avatar representing a sequence of changes in the facial expression in the camera field of view over a period of time, wherein the period of time is determined based on the timing of the input.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and for displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; means for performing the following operation: while displaying the preview of the virtual avatar, detecting an input in the virtual avatar generation interface; and means for performing the following operation: in response to detecting an input in the virtual avatar generation interface: generating a static virtual avatar representing the facial expression in the camera field of view at a corresponding time according to determining that an input starts on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar according to determining that the input includes an activation of a recording enablement indication in the virtual avatar generation interface, the animated virtual avatar representing a sequence of changes in the facial expression in the camera field of view over a period of time, wherein the period of time is determined based on the timing of the input.

[0012] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar within the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in the facial appearance within the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face within the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face within the camera's field of view; after recording the facial expressions of the face within the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate an animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0013] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar within the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in the facial appearance within the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face within the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face within the camera's field of view; after recording the facial expressions of the face within the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate an animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0014] According to some embodiments, a transient computer-readable storage medium is described. A non-transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera. The one or more programs include instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0015] According to some embodiments, an electronic device is described. The electronic device includes: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and for displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; means for receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; means for, in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; means for, after recording the facial expressions of the face in the camera field of view, causing the display of a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0017] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar within the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance within the camera's field of view; while displaying the preview of the virtual avatar, detecting an input within the virtual avatar generation interface; in response to detecting an input within the virtual avatar generation interface: generating a static virtual avatar representing the facial expression within the camera's field of view at a corresponding time based on determining that an input has started on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in the facial expression within the camera's field of view over a period of time based on determining that the input includes activation of a recording enablement indication within the virtual avatar generation interface, wherein the period of time is determined based on the timing of the input.

[0018] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar within the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance within the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face within the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face within the camera's field of view; after recording the facial expressions of the face within the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate an animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0019] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying, via the display device, a virtual avatar that changes appearance in response to changes in the face within the field of view of the one or more cameras, wherein the virtual avatar includes: a first portion and a second portion different from the first portion; while displaying the virtual avatar via the display device, detecting a change in the facial pose within the field of view of the one or more cameras; in response to detecting the change in facial pose, changing the appearance of the virtual avatar, which includes: based on determining that the change in facial pose includes a first type of change in facial pose, changing the appearance of the virtual avatar includes moving the first portion of the virtual avatar relative to the second portion of the virtual avatar based on the magnitude of the first type of change in facial pose; and based on determining that the change in facial pose includes a second type of change in facial pose, changing the appearance of the virtual avatar includes moving the first portion of the virtual avatar and the second portion of the virtual avatar based on the magnitude of the second type of change in facial pose.

[0020] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of one or more cameras and a second physical feature of the face within the field of view of the one or more cameras; and a second avatar feature; while displaying the virtual avatar via the display device, detecting changes in one or more physical features of a face within the field of view of the one or more cameras; based on determining that the changes include a change in the first physical feature: modifying the first avatar feature of the virtual avatar based on the change in the first physical feature and forgoing modifying the second avatar feature based on the change in the first physical feature; and based on determining that the changes include a change in the second physical feature: modifying the first avatar feature based on the change in the second physical feature and forgoing modifying the second avatar feature based on the change in the second physical feature;

[0021] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of one or more cameras; a second avatar feature that responds to changes in the first physical feature; and a third avatar feature that does not primarily respond to changes in the first physical feature; while displaying the virtual avatar, detecting a change in the first physical feature; and in response to detecting the change in the first physical feature: modifying the first avatar feature based on the detected change in the first physical feature; modifying the second avatar feature based on the detected change in the first physical feature; and forgoing modifying the third avatar feature based on the detected change in the first physical feature.

[0022] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of the one or more cameras; a second avatar feature that responds in a different manner to changes in a second physical feature of the face, the manner depending on whether the change in the second physical feature occurs within a first range of changes of the second physical feature or within a second range of changes of the second physical feature, the second range of changes of the second physical feature being different from the first range of changes of the second physical feature; while displaying the virtual avatar, detecting a first change in the corresponding physical feature of the face within the field of view of the one or more cameras; and in response to detecting the first change in the corresponding physical feature, modifying the virtual avatar, which includes: modifying the first avatar feature to reflect the change in the first physical feature based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature; and changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and the change in the second physical feature is within the first range of changes; refraining from changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and the change in the second physical feature is within the second range of changes.

[0023] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes a plurality of avatar features that respond to changes in one or more physical features of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting changes in a plurality of physical features of the face, the plurality of physical features of the face including a first physical feature corresponding to one or more of the plurality of avatar features and a second physical feature not corresponding to any of the plurality of avatar features; and in response to detecting the changes in the plurality of physical features of the face: changing the appearance of the corresponding avatar feature of the plurality of avatar features, wherein the magnitude and / or direction of the change in the corresponding avatar feature is based on the magnitude or direction of the change in the first physical feature; and deforming a portion of the virtual avatar that does not include an avatar feature before detecting the changes in one or more physical features of the face, wherein the magnitude and / or direction of deforming a portion of the avatar feature is based on the magnitude and / or direction of the change in the second physical feature.

[0024] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting a first physical feature change having a first physical feature change magnitude; in response to detecting the change in the first physical feature: based on determining that the change in the first physical feature is within a first physical feature value range, causing the first avatar feature to change by a first avatar feature change magnitude based on the first physical feature change magnitude; and based on determining that the change in the first physical feature is within a second physical feature value range different from the first physical feature value range, causing the first avatar feature to change by a second avatar feature change magnitude, the second avatar feature change magnitude being different from the first avatar feature change magnitude and based on the first physical feature change magnitude.

[0025] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar has a corresponding spatial position within a reference frame, wherein the corresponding spatial position is based on the position of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting a corresponding amount of change in the position of the face within the field of view of the one or more cameras; in response to detecting the change in the position of the face within the field of view of the one or more cameras: based on determining that the change in the position of the face includes a first change component in a first direction, modifying the spatial position of the virtual avatar within the reference frame based on the first change component and the magnitude of a first modification factor; and based on determining that the change in the position includes a second change component in a second direction different from the first direction, modifying the spatial position of the virtual avatar within the reference frame based on the second change component and the magnitude of a second modification factor, the second modification factor being different from the first modification factor.

[0026] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar responds to changes in one or more physical features of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting a first configuration of the one or more physical features of the face; while detecting the first configuration of the one or more physical features of the face: based on determining that the first configuration of the one or more physical features meets an animation criterion, modifying the virtual avatar to include a first animation effect, the animation criterion including a requirement to maintain the first configuration for at least a first threshold amount of time for the animation criterion to be met; and based on the first configuration of the one or more physical features not meeting the animation criterion, forgoing modifying the virtual avatar to include the first animation effect.

[0027] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. The executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0028] Accordingly, faster and more efficient methods and interfaces are provided for a device to generate, send, and receive emojis, thereby enhancing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace other methods for sending and receiving emojis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To better understand the various described embodiments, reference should be made to the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals indicate corresponding parts in all the figures.

[0030] Figure 1A is a block diagram showing a portable multifunctional device having a touch-sensitive display in accordance with some embodiments.

[0031] Figure 1B is a block diagram showing exemplary components for event handling in accordance with some embodiments.

[0032] Figure 2 shows a portable multifunctional device having a touch screen in accordance with some embodiments.

[0033] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface in accordance with some embodiments.

[0034] Figure 4A shows an exemplary user interface for a menu of an application on a portable multifunctional device in accordance with some embodiments.

[0035] Figure 4B shows an exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display in accordance with some embodiments.

[0036] Figure 5A shows a personal electronic device in accordance with some embodiments.

[0037] Figure 5B is a block diagram showing a personal electronic device in accordance with some embodiments.

[0038] Figures 6A to 6MM shows an exemplary user interface for generating and sending emojis, stickers, virtual avatars, and / or other multimedia content.

[0039] 7A to 7J Shows an exemplary user interface for receiving emojis, stickers, virtual avatars, and / or other multimedia content.

[0040] FIG. 8A to FIG. 8B Is a flowchart showing a method for generating and sending emojis, stickers, virtual avatars, and / or other multimedia content.

[0041] FIG. 9A to FIG. 9B Is a flowchart showing a method for generating and sending emojis, stickers, virtual avatars, and / or other multimedia content.

[0042] FIG. 10A to FIG. 10I Shows an exemplary user interface for generating and modifying poo virtual avatars.

[0043] FIG. 11A to FIG. 11C Shows an exemplary user interface for generating and modifying bear virtual avatars.

[0044] FIG. 12A to FIG. 12C Shows an exemplary user interface for generating and modifying alien virtual avatars.

[0045] Fig.13 Shows an exemplary user interface for generating and modifying rabbit virtual avatars.

[0046] FIG. 14A to FIG. 14D Shows an exemplary user interface for generating and modifying robot virtual avatars.

[0047] FIG. 15A to FIG. 15B Shows an exemplary user interface for generating and modifying unicorn virtual avatars.

[0048] FIG. 16A to FIG. 16B Shows an exemplary user interface for generating and modifying chicken virtual avatars.

[0049] FIG. 17A to FIG. 17B Shows an exemplary user interface for generating and modifying pig virtual avatars.

[0050] FIG. 18A to FIG. 18B Is a flowchart showing a method for generating and modifying virtual avatars based on a face detected by one or more cameras.

[0051] Fig.19 Is a flowchart showing a method for generating and modifying virtual avatars based on a face detected by one or more cameras.

[0052] Fig. 20 Is a flowchart showing a method for generating and modifying virtual avatars based on a face detected by one or more cameras.

[0053] Fig.21It is a flowchart showing a method for generating and modifying a virtual avatar based on a face detected by one or more cameras.

[0054] Fig. 22 It is a flowchart showing a method for generating and modifying a virtual avatar based on a face detected by one or more cameras.

[0055] Fig.23 It is a flowchart showing a method for generating and modifying a virtual avatar based on a face detected by one or more cameras.

[0056] Fig.24 It is a flowchart showing a method for generating and modifying a virtual avatar based on a face detected by one or more cameras.

[0057] Fig.25 It is a flowchart showing a method for generating and modifying a virtual avatar based on a face detected by one or more cameras. Detailed Description

[0058] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but rather is provided as a description of exemplary embodiments.

[0059] Sending a message with multimedia content and text content or only multimedia content may better convey the sender's message. For example, multimedia content such as a virtual avatar (e.g., an animated or static emoji or sticker) can provide context and / or tone that is difficult or impossible to convey with text alone (e.g., it can be referred to as "non-verbal communication"). In some cases, pre-defined virtual avatars can be used to provide some of this context and / or tone. However, pre-defined content cannot cover all cases or provide finely tuned context or tone. Therefore, there is a need for an electronic device that provides effective methods and interfaces for generating, sending, and receiving virtual avatars as part of a message. Such techniques can reduce the cognitive burden on users sending and receiving messages, thereby increasing productivity. In addition, such techniques can reduce the processor power and battery power otherwise wasted on redundant user input.

[0060] Below, Figure 1A to Figure 1B 、 Figure 2 、 Figure 3 、 FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5B provides a description of an exemplary device for performing techniques for generating, sending, and receiving virtual avatars. Figures 6A to 6MM and 7A to 7J show an exemplary user interface for receiving, generating, modifying, and sending virtual avatars. FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9Bis a flowchart showing an exemplary method for receiving, generating, modifying, and sending virtual avatars. Figures 6A to 6MM and 7A to 7J The user interfaces in are used to show the processes described below, which include FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9B the processes in. FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B and FIG. 17A to FIG. 17B show exemplary user interfaces for generating and modifying virtual avatars according to some embodiments. The user interfaces in these figures are used to show the processes described below, which include FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25 the processes in. FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B and FIG. 17A to FIG. 17B the user interfaces of and FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25 The processes of can be used to generate virtual avatars used in the Figures 6A to 6MM and 7A to 7J interfaces and FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9B the processes of.

[0061] Although the following description uses terms such as "first", "second", etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch could be named a second touch and similarly a second touch could be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0062] The terms used in the description of the various embodiments herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the term "comprises" ("includes", "including", "comprises" and / or "comprising") when used in this specification specifies the presence of the stated features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0063] Depending on the context, the term "if" is optionally interpreted to mean "when", "upon", or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is detected" is optionally interpreted to mean "when it is determined..." or "in response to determining..." or "when [the stated condition or event] is detected" or "in response to detecting [the stated condition or event]".

[0064] Embodiments of electronic devices, user interfaces for such devices, and related processes of using such devices are described herein. In some embodiments, the device is a portable communication device that also includes other functions such as PDA and / or music player functions, such as a mobile phone. Exemplary embodiments of the portable multifunctional device include, but are not limited to, devices from Apple Inc. (Cupertino, California) devices, iPod devices, and devices. Other portable electronic devices are optionally used, such as a laptop or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0065] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0066] The device generally supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, gaming applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0067] The various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or within the respective applications. Thus, a common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0068] Attention is now turned to an embodiment of a portable device having a touch-sensitive display. Figure 1A FIG. is a block diagram of a portable multifunctional device 100 having a touch-sensitive display system 112 in accordance with some embodiments. The touch-sensitive display 112 is sometimes called a "touch screen" for convenience, and is sometimes referred to as or called a "touch-sensitive display system". The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of contacts on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). The device 100 optionally includes one or more tactile output generators 167 for generating tactile outputs on the device 100 (e.g., generating tactile outputs on a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100 or the touchpad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0069] As used in this specification and the claims, the "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of methods and a variety of sensors or combinations of sensors. For example, one or more force sensors beneath or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in size of the contact area detected on the touch-sensitive surface, the capacitance and / or change in capacitance of the touch-sensitive surface near the contact, and / or the resistance and / or change in resistance of the touch-sensitive surface near the contact are optionally used as a surrogate for the force or pressure of a contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of user input allows a user to access additional device functions that would otherwise be inaccessible on a smaller device with a limited footprint, the smaller device being used to (e.g., on a touch-sensitive display) display affordances and / or receive user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or physical / mechanical controls such as a knob or button).

[0070] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device detected by the user using the user's sense of touch, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of the component relative to the center of mass of the device. For example, in the case of contact between the device or a component of the device and a surface of the user sensitive to touch (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a sense of touch corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or a touchpad) may optionally be interpreted by the user as a "press click" or "release click" of a physical actuation button. In some cases, the user will feel a sense of touch, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such interpretations of touch by the user will be limited by the user's individual sensory perception, many sensory perceptions of touch are common to most users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or its component that would generate the stated sensory perception of a typical (or ordinary) user.

[0071] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0072] Memory 102 may optionally include high-speed random access memory and may also optionally include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Memory controller 122 may optionally control access to memory 102 by other components of device 100.

[0073] The peripheral device interface 118 can be used to couple the input and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0074] The RF (Radio Frequency) circuit 108 receives and transmits RF signals, which are also referred to as electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with a communication network and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 108 optionally communicates with the network and other devices via wireless communication, and these networks are such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular phone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The RF circuit 108 optionally includes well-known circuits for detecting a near field communication (NFC) field, such as detecting via a short-range communication radio component. The wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), WiMAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the date of submission of this document.

[0075] The audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuitry 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves audible to humans. The audio circuitry 110 also receives the electrical signal converted from sound waves by the microphone 113. The audio circuitry 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuitry 108 by the peripheral interface 118. In some embodiments, the audio circuitry 110 also includes an earphone jack (e.g., Figure 2 212 in

[0076] ). The earphone jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or an earphone having both an output (e.g., a mono or stereo earphone) and an input (e.g., a microphone). Figure 2 The I / O subsystem 106 couples input / output peripheral devices on the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, a depth controller 169, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / transmit electrical signals to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to (or not coupled to) any of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208 in

[0077] Rapidly pressing the depress button optionally disengages the lock of the touch screen 112 or optionally starts the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application No. 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. Pressing and holding the depress button (e.g., 206) optionally powers on or powers off the device 100. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0078] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0079] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and the display controller 156 (together with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0080] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, but uses other display technologies in other embodiments. The touch screen 112 and the display controller 156 optionally use any of a variety of touch sensing technologies currently known or later developed, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. The variety of touch sensing technologies includes, but is not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in from Apple Inc. (Cupertino, California) and iPod The technology used in

[0081] In some embodiments of the touch screen 112, the touch-sensitive display is optionally similar to the multi-touch sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.) and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays the visual output from the device 100, while the touch-sensitive touchpad does not provide a visual output.

[0082] In some embodiments, the touch-sensitive display of the touchscreen 112 is as described in the following patent applications: (1) U.S. Patent Application No. 11 / 381,313, entitled "Multipoint Touch Surface Controller," filed on May 2, 2006; (2) U.S. Patent Application No. 10 / 840,862, entitled "Multipoint Touchscreen," filed on May 6, 2004; (3) U.S. Patent Application No. 10 / 903,964, entitled "Gestures For Touch Sensitive Input Devices," filed on July 30, 2004; (4) U.S. Patent Application No. 11 / 048,264, entitled "Gestures For Touch Sensitive Input Devices," filed on January 31, 2005; (5) U.S. Patent Application No. 11 / 038,590, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," filed on January 18, 2005; (6) U.S. Patent Application No. 11 / 228,758, entitled "Virtual Input Device Placement On A Touch Screen User Interface," filed on September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700, entitled "Operation Of A Computer With A Touch Screen Interface," filed on September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737, entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed on September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749, entitled "Multi-Functional Hand-Held Device," filed on March 3, 2006. All of these applications are hereby incorporated by reference in their entirety.

[0083] The touch screen 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contact and gestures, which may not be as precise as stylus-based input due to the larger contact area of the finger on the touch screen. In some embodiments, the device converts the finger-based rough input into an accurate pointer / cursor position or command for performing the action desired by the user.

[0084] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device, which, unlike the touch screen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0085] The device 100 also includes a power system 162 for powering various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0086] The device 100 optionally further includes one or more optical sensors 164. Figure 1AAn optical sensor coupled to the optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representative of an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite the touch screen display 112 on the front of the device, such that the touch screen display can be used as a viewfinder for still image and / or video image capture. In some embodiments, the optical sensor is located on the front of the device such that an image of the user can optionally be acquired for a video conference while the user views other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) such that a single optical sensor 164 can be used with the touch screen display for both video conferencing and still image and / or video image capture.

[0087] The device 100 optionally further includes one or more contact intensity sensors 165. Figure 1A A contact intensity sensor coupled to the intensity sensor controller 159 in the I / O subsystem 106 is shown. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., the touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the rear of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0088] The device 100 optionally further includes one or more proximity sensors 166. Figure 1AA proximity sensor 166 is shown coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 optionally operates as described in the following U.S. patent applications: No. 11 / 241,839, titled "Proximity Detector In Handheld Device"; No. 11 / 240,788, titled "Proximity Detector In Handheld Device"; No. 11 / 620,702, titled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, titled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, titled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor turns off and disables the touch screen 112.

[0089] Device 100 optionally further includes one or more tactile output generators 167. Figure 1AShows a haptic output generator coupled to the haptic feedback controller 161 in the I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components; and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output on the device). The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., the touch-sensitive display system 112), and optionally generates a haptic output by moving the touch-sensitive surface vertically (e.g., into / out of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0090] The device 100 optionally further includes one or more accelerometers 168. Figure 1A Shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally operates as described in the following U.S. Patent Publications: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are hereby incorporated by reference in their entireties. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on an analysis of data received from one or more accelerometers. The device 100 optionally further includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the location and orientation (e.g., portrait or landscape) of the device 100.

[0091] The device 100 optionally further includes one or more depth camera sensors 175. Figure 1AA depth camera sensor coupled to the depth camera controller 169 in the I / O subsystem 106 is shown. The depth camera sensor 175 receives data projected through the sensor from the environment. In conjunction with the imaging module 143 (also referred to as the camera module), the depth camera sensor 175 optionally determines depth maps for different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located at the front of the device 100 such that a user image with depth information is optionally acquired for video conferencing while the user views other video conferencing participants on the touch screen display, and a selfie with depth map data is captured. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) such that the depth camera sensor 175 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.

[0092] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and application programs (or instruction sets) 136. Additionally, in some embodiments, the memory 102 ( Figure 1A ) or 370 ( Figure 3 ) stores the device / global internal state 157, as Figure 1A and Figure 3 shown. The device / global internal state 157 includes one or more of the following: an active application state that indicates which applications (if any) are currently active; a display state that indicates what applications, views, or other information occupy the various regions of the touch screen display 112; a sensor state that includes information obtained from the various sensors and input control devices 116 of the device; and location information regarding the location and / or orientation of the device.

[0093] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between the various hardware components and software components.

[0094] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used on devices of

[0095] (Apple Inc.'s trademark).

[0096] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or a physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple finger contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0096] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., determining whether the user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of the touchpad or touchscreen can be set to any threshold within a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some implementations, software settings are provided to the user of the device for adjusting one or more of the intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click on an "intensity" parameter).

[0097] The contact / motion module 130 optionally detects a user's gesture input. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contact). Thus, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same position (or substantially the same position) as the finger press event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift (lift-off) event.

[0098] The graphics module 132 includes various known software components for presenting and displaying graphics on the touch screen 112 or other displays, including components for changing the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphics" includes any object that can be displayed to a user, which non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0099] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data as necessary, and then generates screen image data for output to the display controller 156.

[0100] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0101] The text input module 134, which is optionally a component of the graphics module 132, provides a soft keyboard for entering text in various application programs (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0102] The GPS module 135 determines the location of the device and provides this information for use in various application programs (e.g., provided to the phone 138 for use in location-based dialing; provided to the camera 143 as picture / video metadata; and provided to application programs that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0103] The application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0104] · Contact module 137 (sometimes referred to as an address book or contact list);

[0105] · Phone module 138;

[0106] · Video conferencing module 139;

[0107] · Email client module 140;

[0108] · Instant messaging (IM) module 141;

[0109] · Fitness support module 142;

[0110] · Camera module 143 for still images and / or video images;

[0111] · Image management module 144;

[0112] · Video player module;

[0113] · Music player module;

[0114] · Browser module 147;

[0115] · Calendar module 148;

[0116] · Desktop widget module 149, which optionally includes one or more of the following: weather desktop widget 149-1, stock market desktop widget 149-2, calculator desktop widget 149-3, alarm clock desktop widget 149-4, dictionary desktop widget 149-5, and other desktop widgets obtained by the user, as well as user-created desktop widget 149-6;

[0117] · Desktop widget creator module 150 for forming user-created desktop widget 149-6;

[0118] · Search module 151;

[0119] · Video and music player module 152, which combines the video player module and the music player module;

[0120] · Notepad module 153;

[0121] · Map module 154; and / or

[0122] · Online video module 155.

[0123] Examples of other applications 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice reproduction.

[0124] In conjunction with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the contacts module 137 is optionally used to manage an address book or a contacts list (e.g., in the application internal state 192 of the contacts module 137 stored in the memory 102 or the memory 370), including: adding one or more names to the address book; deleting names from the address book; associating a phone number, an email address, a physical address, or other information with a name; associating an image with a name; categorizing and classifying names; providing a phone number or an email address to initiate and / or facilitate communication via the phone 138, the video conferencing module 139, the email 140, or the IM 141; and so on.

[0125] In conjunction with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the phone module 138 is optionally used to input a character sequence corresponding to a phone number, access one or more phone numbers in the contacts module 137, modify the entered phone number, dial the corresponding phone number, conduct a session, and disconnect or hang up when the session is completed. As described above, the wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies.

[0126] In conjunction with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contacts module 137, and the phone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between the user and one or more other participants according to user instructions.

[0127] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with static images or video images captured by the camera module 143.

[0128] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying a previously input character, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving an instant message, and viewing the received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0129] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., having time, distance, and / or calorie burn goals); communicating with a fitness sensor (exercise device); receiving fitness sensor data; calibrating the sensors for monitoring fitness; selecting and playing music for the fitness; and displaying, storing, and transmitting fitness data.

[0130] In combination with the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for the following operations: capturing a still image or video (including a video stream) and storing them in the memory 102, modifying the characteristics of a still image or video, or deleting a still image or video from the memory 102.

[0131] In combination with the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0132] In combination with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching for, linking to, receiving, and displaying web pages or portions thereof, and linking to attachments and other files of web pages.

[0133] In combination with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and data associated with the calendars (e.g., calendar entries, to-do items, etc.) according to user instructions.

[0134] In combination with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop widget module 149 is a mini-application (e.g., weather desktop widget 149-1, stock market desktop widget 149-2, calculator desktop widget 149-3, alarm clock desktop widget 149-4, and dictionary desktop widget 149-5) optionally downloaded and used by the user or a mini-application created by the user (e.g., user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (HyperText Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop widget).

[0135] In combination with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop widget creator module 150 is optionally used by the user to create desktop widgets (e.g., transforming a user-specified portion of a web page into a desktop widget).

[0136] In combination with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions for searching for text, music, sound, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0137] In combination with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0138] In combination with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, notepad module 153 includes executable instructions to create and manage notepads, to-do lists, etc. according to user instructions.

[0139] In combination with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data related to stores and other points of interest at or near a particular location, and other location-based data) according to user instructions.

[0140] In combination with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, e-mail client module 140, and browser module 147, online video module 155 includes instructions for performing the following operations: allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, the instant message module 141 is used instead of the e-mail client module 140 to send a link to a particular online video. Other descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed Jun. 20, 2007, and titled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," and U.S. Patent Application No. 11 / 968,067, filed Dec. 31, 2007, and titled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," the contents of which are hereby incorporated by reference in their entirety.

[0141] Each of the above modules and applications corresponds to a set of executable instructions for performing one or more of the above functions and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, the video player module is optionally combined with the music player module into a single module (e.g., Figure 1A the video and music player module 152 in). In some embodiments, memory 102 optionally stores a subgroup of the above modules and data structures. In addition, memory 102 optionally stores additional modules and data structures not described above.

[0142] In some embodiments, device 100 is a device in which the operation of a predefined set of functions on the device is performed uniquely via a touch screen and / or a touchpad. By using the touch screen and / or the touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0143] The predefined set of functions performed uniquely via the touch screen and / or the touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 from any user interface displayed on device 100 to a main menu, a home menu, or a root menu. In such embodiments, the touchpad is used to implement a "menu button". In some other embodiments, the menu button is a physical push button or other physical input control device rather than the touchpad.

[0144] Figure 1B is a block diagram showing exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in operating system 126) and corresponding application 136-1 (e.g., any one of the foregoing applications 137 to 151, 155, 380 to 390).

[0145] Event classifier 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of application 136-1. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views that are displayed on the touch-sensitive display 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by event classifier 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0146] In some embodiments, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by application 136-1, a state queue for enabling the user to return to a previous state or view of application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0147] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (e.g., a user touch on the touch-sensitive display 112 as part of a multi-touch gesture). The peripheral interface 118 transmits information that it receives from the I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via the audio circuitry 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or a touch-sensitive surface.

[0148] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (e.g., a received input that is above a predetermined noise threshold and / or a received input that exceeds a predetermined duration).

[0149] In some embodiments, the event classifier 170 further includes a hit view determination module 172 and / or an active event recognizer determination module 173.

[0150] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where within one or more of the views a sub-event has occurred. Views are composed of controls and other elements that a user can see on the display.

[0151] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct inputs is optionally determined at least in part based on the hit view of the initial touch that begins the touch-based gesture.

[0152] The hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchical structure that should handle the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or a potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view generally receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0153] The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views and, thus, determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with a particular view, higher views in the hierarchy will still remain as actively participating views.

[0154] The event dispatcher module 174 distributes event information to event recognizers (e.g., event recognizer 180). In embodiments that include the active event recognizer determination module 173, the event dispatcher module 174 delivers the event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, the event dispatcher module 174 stores the event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0155] In some embodiments, the operating system 126 includes the event classifier 170. Alternatively, the application 136-1 includes the event classifier 170. In yet another embodiment, the event classifier 170 is an independent module or is part of another module (such as the contact / motion module 130) stored in the memory 102.

[0156] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of an independent module that is a higher-level object such as a user interface toolkit or from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. The event handler 190 optionally utilizes or invokes the data updater 176, the object updater 177, or the GUI updater 178 to update the internal state 192 of the application. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included within a corresponding application view 191.

[0157] A corresponding event recognizer 180 receives event information (e.g., event data 179) from an event classifier 170 and identifies an event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0158] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as a touch or a touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device pose).

[0159] The event comparator 184 compares the event information with predefined event or sub - event definitions and, based on that comparison, determines an event or sub - event or determines or updates the state of an event or sub - event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 contains the definition of an event (e.g., a predefined sequence of sub - events), such as event 1 (187 - 1), event 2 (187 - 2), and others. In some embodiments, the sub - events in an event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi - touch. In one example, the definition of event 1 (187 - 1) is a double - tap on a displayed object. For example, a double - tap includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift - off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift - off (touch end) of a predetermined duration. In another example, the definition of event 2 (187 - 2) is a drag on a displayed object. For example, a drag includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch - sensitive display 112, and lift - off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0160] In some embodiments, the event definition 187 includes the definition of an event for a corresponding user interface object. In some embodiments, the event comparator 184 performs a hit test to determine which user interface object is associated with a sub - event. For example, in an application view that displays three user interface objects on the touch - sensitive display 112, when a touch is detected on the touch - sensitive display 112, the event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub - event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, the event comparator 184 selects the event handler associated with the sub - event and the object that triggered the hit test.

[0161] In some embodiments, the definition of a corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub - event sequence does or does not correspond to the event type of the event recognizer.

[0162] When the corresponding event recognizer 180 determines that the sub - event sequence does not match any event in the event definition 186, the corresponding event recognizer 180 enters an event - impossible, event - failed, or event - ended state, after which subsequent sub - events of the touch - based gesture are ignored. In such a case, other event recognizers (if any) for which the hit view remains active continue to track and process sub - events of the ongoing touch - based gesture.

[0163] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists indicating how the event delivery system should perform sub - event delivery to the event recognizers actively participating. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists indicating how event recognizers interact with each other or can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists indicating whether sub - events are delivered to different levels in the view or the programmatic hierarchy.

[0164] In some embodiments, when one or more specific sub - events of an event are recognized, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferring to send) sub - events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a token associated with the recognized event, and the event handler 190 associated with that token retrieves the token and executes a predefined process.

[0165] In some embodiments, the event delivery instruction 188 includes a sub - event delivery instruction to deliver event information about the sub - event without activating the event handler. Instead, the sub - event delivery instruction delivers the event information to the event handler associated with the sub - event sequence or to the actively participating view. The event handler associated with the sub - event sequence or with the actively participating view receives the event information and executes a predetermined process.

[0166] In some embodiments, the data updater 176 creates and updates data used in the application 136-1. For example, the data updater 176 updates the phone numbers used in the contacts module 137 or stores the video files used in the video player module. In some embodiments, the object updater 177 creates and updates objects used in the application 136-1. For example, the object updater 177 creates new user interface objects or updates the positions of user interface objects. The GUI updater 178 updates the GUI. For example, the GUI updater 178 prepares the display information and sends the display information to the graphics module 132 for display on the touch-sensitive display.

[0167] In some embodiments, the event handler 190 includes or has access to the data updater 176, the object updater 177, and the GUI updater 178. In some embodiments, the data updater 176, the object updater 177, and the GUI updater 178 are included in a single module of the corresponding application 136-1 or the application view 191. In other embodiments, they are included in two or more software modules.

[0168] It should be understood that the above discussion regarding event handling of user touches on the touch-sensitive display also applies to other forms of user input for operating the multifunctional device 100 using an input device, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses optionally in cooperation with single or multiple keyboard presses or holds; contact movement on a touchpad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; voice commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0169] Figure 2FIG. 0 shows a portable multifunctional device 100 having a touch screen 112, according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this and other embodiments described below, a user can select one or more of these graphics by making gestures on the graphics, such as by using one or more fingers 202 (not drawn to scale in the figures) or one or more styli 203 (not drawn to scale in the figures). In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, right to left, up, and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, left to right, up, and / or down). In some implementations or in some cases, inadvertently contacting a graphic does not select the graphic. For example, when the gesture corresponding to selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0170] The device 100 optionally further includes one or more physical buttons, such as a “home” or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any of a set of applications 136 that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0171] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a depressible button 206 for powering on / off the device and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The depressible button 206 is optionally used to power on / off the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions via a microphone 113. The device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0172] Figure 3is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, in accordance with some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home or industrial controller). Device 300 generally includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes termed a chipset) that interconnects the system components and controls the communication between the system components. Device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to one or more of the haptic output generators 167 described above with reference to Figure 1A ), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors (similar to one or more of the contact intensity sensors 165 described above with reference to Figure 1A ). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, those stored in memory 102 of portable multifunctional device 100 ( Figure 1A ). Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunctional device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while memory 102 of portable multifunctional device 100 ( Figure 1A ) optionally does not store these modules.

[0173] Figure 3Each of the above elements in [element name] is optionally stored in one or more of the previously mentioned memory devices of the memory device. Each of the above modules corresponds to an instruction set for performing the above functions. The above modules or programs (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, the memory 370 optionally stores a subgroup of the above modules and data structures. Additionally, the memory 370 optionally stores additional modules and data structures not described above.

[0174] Attention is now turned to embodiments of a user interface optionally implemented on, for example, the portable multifunctional device 100.

[0175] Figure 4A An exemplary user interface of an application menu on the portable multifunctional device 100 according to some embodiments is shown. A similar user interface is optionally implemented on the device 300. In some embodiments, the user interface 400 includes the following elements or subsets or supersets thereof:

[0176] · One or more signal strength indicators 402 of one or more wireless communications such as cellular signals and Wi-Fi signals;

[0177] · Time 404;

[0178] · Bluetooth indicator 405;

[0179] · Battery status indicator 406;

[0180] · A tray 408 with icons for common applications, such as:

[0181] ο An icon 416 labeled "Phone" for the phone module 138, which icon 416 optionally includes an indicator 414 of the number of missed calls or voicemails;

[0182] ο An icon 418 labeled "Mail" for the email client module 140, which icon 418 optionally includes an indicator 410 of the number of unread emails;

[0183] ο An icon 420 labeled "Browser" for the browser module 147; and

[0184] ο An icon 422 labeled "iPod" for the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0185] · Icons for other applications, such as:

[0186] The icon 424 marked "Message" of the οIM module 141;

[0187] The icon 426 marked "Calendar" of the οCalendar module 148;

[0188] The icon 428 marked "Photo" of the οImage Management module 144;

[0189] The icon 430 marked "Camera" of the οCamera module 143;

[0190] The icon 432 marked "Online Video" of the οOnline Video module 155;

[0191] The icon 434 marked "Stock Market" of the οStock Market Desktop Applet 149-2;

[0192] The icon 436 marked "Map" of the οMap module 154;

[0193] The icon 438 marked "Weather" of the οWeather Desktop Applet 149-1;

[0194] The icon 440 marked "Clock" of the οAlarm Desktop Applet 149-4;

[0195] The icon 442 marked "Fitness Support" of the οFitness Support module 142;

[0196] The icon 444 marked "Notepad" of the οNotepad module 153; and

[0197] The icon 446 marked "Settings" of the οSettings Application or Module, which provides access to the settings of the device 100 and its various applications 136.

[0198] It should be noted that Figure 4A The icon labels shown in are merely exemplary. For example, the icon 422 of the Video and Music Player module 152 is marked "Music" or "Music Player". Other labels may be optionally used for the various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0199] Figure 4B A device is shown having a touch-sensitive surface 451 (e.g., Figure 3 a tablet or touchpad 355) separate from the display 450 (e.g., Figure 3Exemplary user interfaces on device 300). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contacts on the touch-sensitive surface 451 and / or one or more haptic output generators 357 for generating haptic output for a user of device 300.

[0200] Although some of the following examples will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as Figure 4B shown. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451 in ) has a major axis (e.g., Figure 4B 452 in ) corresponding to the major axis (e.g., Figure 4B 453 in ) on the display (e.g., 450). According to these embodiments, the device detects contacts (e.g., Figure 4B 460 and 462 in ) with the touch-sensitive surface 451 at positions corresponding to corresponding positions on the display (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, when the touch-sensitive surface (e.g., Figure 4B 451 in ) is separate from the display of the multifunctional device ( Figure 4B 450 in ), user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0201] In addition, although the following examples are mainly given with reference to finger input (e.g., finger contact, single-finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is above the position of the tap gesture (e.g., instead of detecting a contact, followed by stopping detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and a finger contact are optionally used simultaneously.

[0202] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include some or all of the features described with respect to devices 100 and 300 (e.g., Figures 1A to 4B ). In some embodiments, the device 500 has a touch-sensitive display screen 504 hereinafter referred to as a touch screen 504. As an alternative or addition to the touch screen 504, the device 500 has a display and a touch-sensitive surface. As in the case of devices 100 and 300, in some embodiments, the touch screen 504 (or the touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). One or more intensity sensors of the touch screen 504 (or the touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of the device 500 may respond to a touch based on the intensity of the touch, meaning that touches of different intensities may invoke different user interface operations on the device 500.

[0203] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial Number PCT / US2013 / 040061, filed May 8, 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application", published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial Number PCT / US2013 / 069483, filed November 11, 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships", published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0204] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508, if included, may be in physical form. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, may allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0205] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, device 500 may include some or all of the components described with reference Figure 1A , Figure 1B and Figure 3 Device 500 has a bus 512 that operatively couples the I / O section 514 to one or more computer processors 516 and a memory 518. The I / O section 514 may be connected to a display 504, which may have a touch-sensitive component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). Additionally, the I / O section 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 is optionally a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 is optionally a button.

[0206] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0207] The memory 518 of personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the following techniques, including processes 800, 900, 1800, 1900, 2000, 2100, 2200, 2300, 2400, and 2500 (corresponding to FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25)。A computer-readable storage medium can be any medium that tangibly contains or stores computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transient computer-readable storage medium. Non-transient computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B the components and configurations thereof, but may include other components or additional components in a variety of configurations.

[0208] As used herein, the term "indicative representation" refers to a user-interactive graphical user interface object optionally displayed on the display screen of devices 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and FIG. 5A to FIG. 5B ). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an indicative representation.

[0209] As used herein, the term "focus selector" refers to an input element for indicating the current part of the user interface with which the user is interacting. In some specific embodiments including a cursor or other position marker, the cursor acts as the "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., Figure 3 the touchpad 355 in Figure 4B ) or Figure 1A the touch-sensitive surface 451 in Figure 4AIn some specific implementations of the touch screen 112), the detected contact on the touch screen serves as a "focus selector", such that when an input (e.g., a press input made by a contact) is detected at the position of a specific user interface element (e.g., a button, a window, a slider, or other user interface element) on the touch screen display, the specific user interface element is adjusted according to the detected input. In some specific implementations, the focus moves from one area of the user interface to another area of the user interface without a corresponding movement of the cursor or a movement of the contact on the touch screen display (e.g., moving the focus from one button to another button by using the tab key or arrow keys); in these specific implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to deliver the interaction with the user interface expected by the user (e.g., by indicating to the device the element of the user interface that the user desires to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or a touch screen), the position of the focus selector (e.g., a cursor, a contact, or a selection box) above the corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0210] As used in the specification and claims, the term "feature strength" of a contact refers to a feature of the contact based on one or more strengths of the contact. In some embodiments, the feature strength is based on a plurality of strength samples. The feature strength is optionally based on a predefined number or set of strength samples collected during a predefined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after detecting the contact, before detecting the contact lift-off, before or after detecting the contact start to move, before detecting the contact end, before or after detecting the contact strength increase, and / or before or after detecting the contact strength decrease). The feature strength of the contact is optionally based on one or more of the following: the maximum value of the contact strength, the mean value of the contact strength, the average value of the contact strength, the value at the top 10% of the contact strength, the half maximum value of the contact strength, the 90% maximum value of the contact strength, etc. In some embodiments, the duration of the contact is used in determining the feature strength (e.g., when the feature strength is the average value of the contact strength over time). In some embodiments, the feature strength is compared with a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact with a feature strength not exceeding the first threshold results in a first operation, a contact with a feature strength exceeding the first strength threshold but not exceeding the second strength threshold results in a second operation, and a contact with a feature strength exceeding the second threshold results in a third operation. In some embodiments, the comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to forgo performing the corresponding operation) rather than for determining whether to perform a first operation or a second operation.

[0211] In some embodiments, a portion of the gesture is identified for determining the feature strength. For example, the touch-sensitive surface optionally receives a continuous swiping contact that transitions from a starting position to an ending position at which the contact strength increases. In this example, the feature strength of the contact at the ending position is optionally based on only a portion of the continuous swiping contact rather than the entire swiping contact (e.g., only the portion of the swiping contact at the ending position). In some embodiments, a smoothing algorithm is optionally applied to the strength of the swiping contact before determining the feature strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the strength of the swiping contact for the purpose of determining the feature strength.

[0212] Optionally, characterize the contact intensity on the touch-sensitive surface relative to one or more intensity thresholds such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move a focus selector based on the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.

[0213] An increase in contact characteristic intensity from an intensity below the light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact characteristic intensity from an intensity below the deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact characteristic intensity from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact characteristic intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0214] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or contacts), where the corresponding press input is detected at least in part based on the intensity of the detected contact (or contacts) increasing above a press input intensity threshold. In some embodiments, a corresponding operation is performed in response to detecting the intensity of a corresponding contact increasing above a press input intensity threshold (e.g., the "down stroke" of the corresponding press input). In some embodiments, a press input includes an increase in the intensity of a corresponding contact above a press input intensity threshold and a subsequent decrease in the intensity of the contact below the press input intensity threshold, and a corresponding operation is performed in response to detecting the subsequent decrease in the intensity of the corresponding contact below the press input threshold (e.g., the "up stroke" of the corresponding press input).

[0215] In some embodiments, the device employs strength hysteresis to avoid unexpected inputs sometimes referred to as "jitter", where the device defines or selects a hysteresis strength threshold that has a predefined relationship to a press input strength threshold (e.g., the hysteresis strength threshold is X strength units lower than the press input strength threshold, or the hysteresis strength threshold is 75%, 90%, or some reasonable proportion of the press input strength threshold). Thus, in some embodiments, a press input includes the strength of a corresponding contact increasing above the press input strength threshold and the strength of that contact subsequently decreasing below the hysteresis strength threshold corresponding to the press input strength threshold, and a corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases below the hysteresis strength threshold (e.g., the "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from a strength equal to or lower than the hysteresis strength threshold to a strength equal to or higher than the press input strength threshold and optionally the contact strength subsequently decreases to a strength equal to or lower than the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., depending on the context, the contact strength increases or the contact strength decreases).

[0216] For ease of explanation, optionally, a description of an operation performed in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is triggered in response to detecting any one of the following various situations: the contact strength increases above the press input strength threshold, the contact strength increases from a strength lower than the hysteresis strength threshold to a strength higher than the press input strength threshold, the contact strength decreases below the press input strength threshold, and / or the contact strength decreases below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in an example where an operation is described as being performed in response to detecting that the strength of a contact decreases below the press input strength threshold, the operation is optionally performed in response to detecting that the strength of the contact decreases below the hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0217] Attention is now turned to embodiments of a user interface ("UI") implemented on an electronic device (such as portable multifunctional device 100, device 300, or device 500) and associated processes.

[0218] Figures 6A to 6MM An exemplary user interface for generating and sending a virtual avatar is shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, which include FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9B the processes in. 7A to 7J An exemplary user interface for receiving and playing a virtual avatar is shown in accordance with some embodiments. Figures 6A to 6MM and 7A to 7J Using the virtual avatar as a specific example of a virtual avatar.

[0219] Fig. 6A A device 600 is depicted having a display 601 (in some cases, the display is a touch-sensitive display) and a camera 602 (the camera includes at least an image sensor capable of capturing data representing a portion of a light spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, the camera 602 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, the camera 602 is capable of capturing other types of data such as depth data. For example, in some embodiments, the camera 602 also captures depth data using techniques based on speckle, time of flight, parallax, or focus. The image data captured by the device 600 using the camera 602 includes data corresponding to a portion of the light spectrum of a scene within the camera's field of view. In addition, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes data sufficient to determine or generate depth data for the data for that portion of the light spectrum. In some embodiments, the device 600 includes one or more features of the device 100, 300, or 500.

[0220] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate the object during the capture of the image by the visible light camera and the depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine the depth map of different parts of the object captured by the visible light camera. In some embodiments, the lighting effects described herein are displayed using parallax information from two cameras (e.g., two visible light cameras) for backward images, and the depth information from the depth camera is combined with the image data from the visible light camera for forward images (e.g., selfie images). In some embodiments, the same user interface is used when the depth information is determined using two visible light cameras and when the depth information is determined using a depth camera, thereby providing a consistent experience for the user even when the information used to produce the lighting effect is determined using completely different technologies. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects selection of a camera switching indicator and switches from a front-facing camera (e.g., a depth camera and a visible light camera) to a rearward-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa) while maintaining display of user interface controls for applying the lighting effect and replacing the field of view display of the front-facing camera with the field of view of the rearward-facing camera (or vice versa).

[0221] exist Fig. 6AIn [description], device 600 is displaying a home screen interface with multiple icons for various applications, including icon 603 for a messaging application. In response to a gesture on icon 603 (e.g., tap gesture 604), device 600 displays Figure 6B the user interface corresponding to the messaging application associated with icon 603.

[0222] In Figure 6B [description], device 600 is displaying a messaging interface 608. Elements 605-1 to 605-6 correspond to previous messaging communications. Each of elements 605-1 to 605-6 represents a communication with one or more remote users, each of the one or more remote users being associated with their own electronic device. In response to a gesture on a particular element (e.g., tap gesture 606), device 600 updates messaging interface 608 to display a portion of the previous messaging communications with one or more remote users that were part of the communication, as Figure 6C depicted in [description].

[0223] In Figure 6C [description], device 600 is displaying a messaging interface 608 for messaging communication with a remote user named "John" (and having the initials or monogram "JA"). Messaging interface 608 includes a message area 609 that includes four previously exchanged messages 610-1 to 610-3 (message 610-3 was sent from the user of device 600 to "John", while the other two messages were received by device 600 from "John"). Messaging interface 608 also includes a message composition area 612 and message option icons to the left of message composition area 612 (including icon 614, e.g., an icon to access an interface for selecting stickers and / or other multimedia elements for a message). In some embodiments, the message option icons allow sending different types of messages, including photos, emojis, stickers, and other forms of non-text messages, such as those described below.

[0224] In response to device 600 detecting (e.g., via Figure 6C tap gesture 616) a selection of message composition area 612, as Fig.6D depicted in [description], messaging interface 608 is updated. For example, in Fig.6D [description], the message option icons are hidden (but can be redisplayed by selecting button 618), a suggested message response 620 is displayed, and a virtual keyboard 622 is displayed. In some cases, virtual keyboard 622 is used to input a new message to be sent to the remote user.

[0225] In Fig. 6EIn this case, the message composition area 612 includes the text "will come late. For example, this text is input via the virtual keyboard 622 or other methods such as voice input. In response to the selection of the send button 621, the device 600 sends the text as part of the message to one or more participants associated with the communication in the message 609. In Fig. 6E the case of, the device 600 sends the message to the user named "John". In Fig. 6F this case, the device 600 has an updated message area 609 to reflect the sending of the message by updating the message area 612 to include the message 610-4.

[0226] In some cases, the message option icon is accessed to add or compose a new message (e.g., by adding non-text content to the message). For example, in response to the device 600 detecting the selection of the enabling representation 618 (e.g., via Fig. 6F the tap gesture 624 in), the message option icon including the icon 614 is displayed again, as depicted in Figure 6G this case. In response to the selection of the icon 614 (e.g., via a gesture such as Figure 6H the tap gesture 626 in), the device 600 updates the messaging interface 608 by replacing the virtual keyboard 622 with the multimedia item interface 628 that currently displays the recent items menu 629 (sometimes referred to as the "tray" of recent items), as depicted in Fig.6I this case. The multimedia item interface includes previously sent multimedia items (e.g., Fig.6I the stickers 630-1 to 630-4 in, but may also include other types of multimedia items such as sounds, animations, or videos). Using this interface, the user can select a previously sent multimedia item to send again. For example, the user can via a tap gesture on the selected sticker in Fig.6ISelect a sticker in the recent items menu 629. In response to such a selection, the device 600 places the sticker in the message composition area 612 or sends the selected sticker to one or more remote users involved in the communication represented in the message area 609. In some embodiments, a tap and drag gesture is used to place the selected sticker (or other multimedia item) in the message composition area 612 or the message area 609 (and in some cases, on a specific message). For example, a specific sticker is selected via a tap gesture. Without breaking contact with the touch-sensitive display 601, the sticker is dragged to the message composition area 612 or the message area 609 via a drag gesture. Once the sticker reaches the desired location, contact with the touch-sensitive display 601 is stopped, and the sticker is placed at the final contact location. If the final contact location is in the message area 609, the sticker is sent to one or more remote users associated with the communication represented in the message area 609. Optionally, the sticker and the data associating the sticker with a specific message are sent to the remote user (e.g., the sticker is sent together with data indicating that the sticker is "pasted" at a specific location on a specific message). These techniques are not specific to selecting and sending stickers. It can also be applied to other types of multimedia items that can be selected from the recent items menu 629 or other locations.

[0227] In Fig.6I the multimedia item interface 628 also includes a menu selection button 632 (which allows selection of a menu or interface other than the recent items menu 629 by displaying buttons or other selectable items corresponding to available menus) and a full screen button 634 (which allows the multimedia item interface 628 to expand to occupy more of the display (or the entire display)). The full screen button 634 is further described below.

[0228] In addition to using the menu selection button 632 to switch between menus or interfaces, gestures are also optionally used to switch between menus. For example, in response to a swipe gesture (e.g., a swipe represented by movement of contact 636 across the multimedia item interface 628 as depicted in Figure 6J and Figure 6K ), the device updates the multimedia item interface 628 to replace the display of the recent items menu 629 with a virtual avatar menu 638. Although the recent items menu 629 is being replaced with the virtual avatar menu 638, a scroll indicator 639 provides feedback on how many other menus are available in the multimedia item interface 628.

[0229] In Figure 6L the virtual avatar menu 638 has completely replaced the display of the recent items menu 629. In response to the device 600 detecting (e.g., via a gesture such as Figure 6M a tap gesture 642) a selection of the continue enabling representation 640, as in Figure 6N depicts the display of the virtual avatar interface 643. This interface allows the user to generate a new virtual avatar that reflects the user's facial movements and expressions, as further described below. In some embodiments, the virtual avatar menu 638 is not displayed at all. Instead, the virtual avatar interface 643 is displayed without first displaying the virtual avatar menu 638.

[0230] Figure 6N The virtual avatar interface 643 includes avatar template representations 644-1 through 644-7 corresponding to different avatar frames (e.g., avatar characters with different appearances and behaviors). Each avatar template representation can map detected facial movements and expressions to the avatar frame thereon. An indicator 645 corresponds to the currently selected avatar template. The virtual avatar preview 646 is a "live" preview of the virtual avatar, as it is updated to reflect the user's current facial movements and expressions. For example, in some embodiments, using the camera 602, the device 600 continuously captures image data from the camera 602. The captured image data includes visible light data and depth data. The device 600 analyzes the captured image data to identify facial movements (e.g., muscle movements, head orientation, gaze direction, etc.) and / or facial expressions (e.g., smile, frown, angry expression, sad expression, confused expression, etc.). Then, the device 600 updates the avatar preview 646 based on the parameters of the avatar frame currently associated with the virtual avatar preview 646 to reflect the detected user characteristics. In some embodiments, the device 600 automatically begins continuously updating the virtual avatar preview 646 in response to the virtual avatar interface 643 being the first one being executed or displayed. Detecting a selection of a different avatar template representation will cause the device 600 to update the virtual avatar preview 646 based on the newly selected avatar template.

[0231] Fig.6O depicts several examples of the user's face in the captured image data 650-1 through 650-5 and the corresponding updates 651-1 through 651-5 to the virtual avatar preview. These are examples of the device 600 updating the emoji preview 646 to reflect the user's facial movements, expressions, and poses. In the captured image data 650-1, the device 600 detects (e.g., based on facial features, facial muscles, facial movements, and / or facial expressions) that the user is looking straight ahead, smiling, and / or happy. In response, the device 600 updates the virtual avatar preview to reflect the user's smile and / or happy expression in addition to updating the eyes of the virtual avatar preview to look straight ahead, as depicted in update 651-1. Although the physical characteristics of the user detected in the captured image data are sometimes updated such that the virtual avatar reflects the same physical characteristics of the user in the virtual avatar, in other cases, changes in the detected physical characteristics of the user cause different types of physical characteristics of the virtual avatar to be updated. For example, in Fig.6OIn [the example], since the monkey has no eyebrows, changes in the user's eyebrows are mapped to the monkey's ears (or other features), as shown in 650-2 and 651-2. In this example, the user's mouth and eyes are mapped to the monkey's mouth and eyes. In the examples of image data 650-3 and update 651-3, the user's unhappy expression and / or frown are reflected in the corresponding features of the virtual avatar preview. In some embodiments, if the user maintains a facial expression or facial pose, as depicted in images 650-3 and 650-4, the virtual avatar preview is updated with additional features (such as tears in the case of update 651-4). This type of predefined update can also occur in response to detected lack of movement. In some embodiments, the update is also based on detected user movement in the image data. For example, a device 600 that detects rotation of the user's head generates an update that causes the virtual avatar preview to rotate similarly. In some embodiments, the update is also based on a physical model for the features of the virtual avatar. For example, in image data 650-5, the device 600 detects that the user is shaking their head. In response, the device 600 generates an update 651-5 to reflect the head shake. Additionally, in update 651-5, the puppy's ears also stick out due to the physical model applied to the puppy's ears.

[0232] In Figure 6P [the example], the device 600 detects selection of the record button 652 via a gesture (e.g., a tap gesture represented by contact 653). In response, the virtual avatar interface 643 is updated to show that an animated virtual avatar is being generated, as Figure 6Q depicted. For example, the record button 652 is replaced with a stop button 654, the avatar template representations 644-1 to 644-7 are no longer displayed, and a recording progress indicator 656 is shown, which indicates how long the animated emoji has been recorded and the relative amount of time remaining to record the virtual avatar. Recording can be stopped by any of a variety of methods, such as expiration of a predetermined amount of time (e.g., 15 seconds) or by selecting the stop button 654. In some embodiments, during recording, the device 600 is detecting and / or storing a series of data points for creating the animated virtual avatar. For example, in some embodiments, the device 600 records a time series of facial movements and / or facial expressions (e.g., as values within a range of possible values, where each value within the range of possible values corresponds to a predetermined movement or expression), and then maps the facial movements and / or facial expressions to an avatar template to create the animated virtual avatar. Alternatively, when the device 600 updates the virtual avatar preview to reflect the user's facial movements and / or expressions, the device 600 records the animated virtual avatar by creating a video recording of the virtual avatar preview. In some embodiments, the device 600 also records sound captured by the device 600's microphone such that the recorded animated virtual avatar includes sound that can be played back along with the recorded animation of the virtual avatar.

[0233] Figure 6R Depicts a later time point during the recording of an animated virtual avatar. The virtual avatar preview 646 has been updated to reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0234] Figure 6R-1 Depicts a device 600 that has detected that the user has changed their position relative to the device during avatar recording. Specifically, at the time point corresponding to Figure 6R-1 the user's face is no longer within the field of view of the camera. In response, the device 600 displays the virtual avatar at the edge of the avatar interface 643 (e.g., the edge corresponding to the position of the last detected user face), displays frame corners 653 around the virtual avatar, and displays a message 655A ("Please place your face in view") to prompt the user to adjust their alignment relative to the device. In some embodiments, even after the user's face is no longer detected within the field of view of the camera, the recording of the virtual avatar continues, but the virtual avatar remains stationary (or assumes a predetermined pose (e.g., a neutral pose)) when the user's face is not detected.

[0235] Figure 6R-2 Depicts the device 600 after the user has been outside the camera's field of view for longer than a predetermined threshold time. In response to detecting that the user has been outside the camera's field of view for longer than a predetermined time, the device 600 pauses the recording of the virtual avatar. As Figure 6R-2 shown, in accordance with the recording pause, the device 600 has replaced the stop button 654 with a record button 648. The device 600 also displays a message 655B ("Tap to resume"), which indicates to the user that the recording has been paused. In some embodiments, the user can resume the recording by tapping anywhere within the avatar interface 643 (including tapping the record button 648). Pausing the recording of the virtual avatar when the user has been outside the camera's field of view for longer than a predetermined threshold amount of time and requesting another input to resume the recording reduces energy usage as well as the use of the depth camera, thus extending the battery life of a battery-powered device and extending the life of the depth camera.

[0236] Figure 6S Depicts yet another later time point during the recording of an animated virtual avatar. The virtual avatar preview 646 has been updated to further reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0237] At Figure 6SUpon receiving a request to stop recording an animated virtual avatar gesture (e.g., a tap gesture represented by contact 658), in response, device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, as Figure 6T depicted. In other cases, in response to the expiration of a predetermined time period (e.g., 15 seconds), device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, as Figure 6T depicted.

[0238] Figure 6S-1 to Figure 6S-3 Another embodiment of the virtual avatar interface 643 for recording (e.g., generating) a virtual avatar is depicted. As Figure 6S-1 shown, device 600 displays a timer 659 (e.g., showing 10 seconds remaining), which indicates the time remaining in the current avatar recording session (e.g., a session initiated by activating the record button 648). In Figure 6S-2 , for the same recording session of 4 seconds, the timer 659 currently shows 6 seconds remaining in the avatar recording session. In Figure 6S-3 , the recording session has ended (i.e., Figure 6S-3 is a time point 10 seconds later than Figure 6S-1 ). In response to the end of the recording session, device 600 replaces the timer 659 with a trash can affordance 660-1, which can be activated (e.g., by a tap gesture) to discard the completed recording session. In some embodiments, the function of the trash can affordance 660 is similar to Figure 6T and Figure 6U to Figure 6U-1 's discard affordance 660.

[0239] In Figure 6T , the virtual avatar interface 643 is currently playing the recorded animated virtual avatar 659 instead of showing a virtual avatar preview, as described by three snapshots of the playback of the animated virtual avatar 659. In some embodiments, the recorded animated virtual avatar plays in a loop (e.g., plays the recorded animated virtual avatar at least twice without user input, as Figure 6T(as indicated by the arrow in). The virtual avatar interface 643 also includes a discard button 660, a mute button 662, and a confirmation button 664 (the confirmation button is shown in place of the record button 652). The discard button 660 discards the displayed recorded animated avatar without saving the avatar or sending it to a remote user. The mute button 662 allows the user to mute the playback of the recorded animated avatar. The confirmation button 664 allows the recorded animated avatar to be sent to a remote user (e.g., in response to activation of the confirmation button 664 or by moving to the message composition area 612 before the user sends a message, directly sent to one or more users associated with the communication displayed in the message area 609). After the device 600 detects a selection of the confirmation button 664, the virtual avatar interface 643 is updated to return to the state relative to Figure 6N described. In some embodiments, the confirmation button 664 includes a glyph or icon similar to or the same as the send glyph or icon (e.g., Figure 6V 670 in) displayed in the send button for sending a message in the message composition area, which is used to indicate that the recorded animated avatar can be sent to a remote user by selecting the confirmation button 664.

[0240] While the animated avatar 659 is being played, in response to a tap gesture 665 on the representation of an avatar template different from the currently selected template, the animated avatar is updated to reflect the new avatar template without having to re-record the animated avatar. This is depicted in Figure 6U where Figure 6U an animated avatar 659 that has been replaced by an animated avatar 666 is depicted. The facial muscles, facial movements, facial features, and facial expressions recorded for generating Figure 6T the animated avatar 659 in are reapplied to Figure 6U the newly selected avatar template in.

[0241] In Figure 6U-1 , while the animated avatar 666 is being played, the device 600 detects a tap gesture 661 corresponding to a selection of the discard enable representation 660. In response, the device 600 discards the captured animated avatar data (e.g., abandons adding the animated avatar to the message composition area 612) and switches to Figure 6U-2 the interface. In Figure 6U-2 , the device 600 displays a pre-recorded avatar interface 643, which is similar to the interface seen in Figure 6P (e.g., includes a record button). Compared with Figure 6P , the avatar template remains as the avatar template of the robot (e.g., virtual avatar 666) instead of returning to the monkey virtual avatar 659. That is, the Figure 6T and Figure 6UThe detection result of the change from avatar 659 to avatar 666 during playback as depicted.

[0242] Refer back Figure 6T , in response to a gesture (e.g., a tap gesture represented by contact 667), device 600 adds the recorded animated virtual avatar 668 to the message composition area 612 (see Figure 6V ) and returns the virtual avatar interface 643 to Figure 6N the state described in Figure 6V ). Then, the user can add more message content (e.g., text or other multimedia items) to the message (see Figure 6W ) before device 600 sends the message (e.g., in response to a tap gesture represented by contact 672 on the send enable indication 670, as depicted in Figure 6V ). Alternatively, when device 600 detects a selection of the confirmation button 664, device 600 sends the recorded animated virtual avatar to one or more remote users associated with the communication displayed in the message area 609, and then updates the message area to reflect that the animated virtual avatar 668 has been sent to one or more users associated with the communication included in the message area 609, as depicted in Figure 6X .

[0243] Figures 6Y to 6BB depicts the response of the virtual avatar interface 643 to user input for scrolling through the avatar template list. For example, in response to a swipe gesture (e.g., a representation of contact 676 moving vertically across the avatar templates, as depicted in Figure 6Y to Figure 6AA ), device 600 scrolls the avatar templates and changes the currently selected avatar template. The avatar template indicator 645 is updated with the avatar templates based on the swipe gesture. In response to detecting that a new avatar template is being selected, device 600 updates the virtual avatar preview. For example, in Figure 6Z , when device 600 detects a selection of the avatar template representation 644-5, the virtual avatar preview 678 (which is based on the avatar template corresponding to representation 644-5) is displayed, and in Figure 6AA , when the avatar template representation 644-8 is selected, the virtual avatar preview 680 (which is based on the avatar template corresponding to representation 644-8) is displayed.

[0244] In addition to generating recordings of animated puppet emojis, the emoji interface 643 also allows for the generation of static virtual avatars (e.g., stickers with an expression / appearance based on the state of the virtual avatar). For example, in Figure 6CCIn the embodiment, in response to user input on the virtual avatar preview 680 (e.g., a tap and hold gesture represented by contact 682), the device 600 generates a sticker corresponding to the state of the virtual avatar preview 680 at a time associated with the user input (e.g., when the input is received, when the user input ends, or at some other time associated with the user input). In an embodiment, the device 600 displays a sticker 683 ( Figure 6DD and Figure 6EE ) to indicate that a sticker has been generated and / or that the user can place a sticker.

[0245] After the device 600 generates the sticker, the user optionally selects from several actions for the sticker. For example, the user can cause the device 600 to place the sticker in a recent menu or other similar interface that allows for later use. The user can also cause the device 600 to place the sticker in the message composition area 612 before the device 600 sends the message containing the sticker, and the user can place the sticker in the message area 609 (and optionally) on a specific message to cause the device 600 to send the sticker to one or more users participating in the communication in the message area 609.

[0246] For example, in Figure 6FF , device 600 has detected the lift-off of contact 682 while the contact is still above virtual avatar preview 680. In response, device 600 has saved the generated sticker to device 600, such as in a database or library in the memory of device 600, which can be accessed through recent items menu 629 ( Fig.6I ) accesses the database or library so that stickers can be optionally selected through the recent items menu 629 or via other interfaces on the device 600. The device 600 optionally displays that the stickers are saved locally via animations with different graphic versions 684 and 686, such as Figures 6FF to 6GG As depicted in , move toward menu selection button 632.

[0247] Another example: Figures 6HH to 6KK An example is shown where the device 600 sends the generated sticker to one or more users participating in the communication represented in the message area 609. Fig. 6II In the embodiment, the device 600 detects a user input on the virtual avatar preview 680 (e.g., by Figures 6HH to 6JJ 688 in the message area 609). When the device 600 detects that the user drags the sticker representation into the message area 609, the sticker representation 690 follows the contact 688. Once the device 600 detects that the contact 688 in the message area 609 is lifted off, the device 600 sends the sticker 691 to one or more remote users who are participants in the communication represented in the message area 609, such asFigure 6KK as depicted in

[0248] In Figure 6LL device 600 updates the virtual avatar interface 643 to display more of the screen (or display in full screen mode) in response to a selection of the full screen button 634 (e.g., a tap gesture represented by contact 692 in Figure 6LL ). Figure 6MM depicts the virtual avatar interface 643 after being enlarged to occupy more of the display 601. When button 692 is selected, the button causes device 600 to return the virtual avatar interface 643 to its previous configuration.

[0249] Figures 7A to 7J depicts the messaging interface 608 after receiving an animated emoji from a remote user. Although Figures 7A to 7J device 600 using Figures 6A to 6MM as an example, the user interfaces and features depicted in Figures 7A to 7J also apply to other devices (e.g., devices 100, 300, or 500), including those that have not previously sent stickers or animated avatars.

[0250] Figure 7A depicts the messaging interface 608 after receiving an animated virtual avatar 700 from a remote user named "John" (and having the initials or monogram "JA") just before playing the animated virtual avatar. In some embodiments, after receiving the animated virtual avatar 700, device 600 automatically plays the animated virtual avatar. When the mute button 702 is selected, the mute button causes any sound associated with the animated virtual avatar 700 to disappear. In some embodiments, if the animated virtual avatar scrolls off the display, any sound also disappears. In some embodiments, the virtual avatar interface 643 is displayed as described with respect to Figures 6A to 6MM (e.g., the virtual avatar interface 643 includes a preview virtual avatar based on detected facial movements / expressions and a selected avatar template).

[0251] In Figure 7B in response to playing the animated virtual avatar 700 once (e.g., playing from start to end once), a still frame 703 of the animated virtual avatar 700 is displayed in place of the animated virtual avatar 700. A playback button 704 is also displayed in the message area 609 and allows the animated virtual avatar 700 to be played again, for example, by a tap gesture represented by contact 706 in Figure 7C . Figure 7D depicts device 600 playing the animated virtual avatar 700 again (with respect to Figure 7A describing playing an animated emoji).

[0252] In some embodiments, when the animated virtual avatar 700 is being played, if the device 600 receives a user input on the mute button 702 (e.g., a tap gesture represented by Figure 7E contact 708), the device 600 stops playing any sound associated with the animated virtual avatar 700 while continuing to play the animated virtual avatar 700 (e.g., the animated virtual avatar 700 still moves without sound). In some embodiments, in response to a selection of the mute button 702 (or if the sound on the device 600 is turned off or the device 600 has an enabled accessibility feature), the transcript button 714 is displayed, as Figure 7H depicted. In response to a selection of the transcript button 714 (e.g., a tap gesture represented by Figure 7G contact 716), the transcript 718 for the sound of the animated virtual avatar 700 is displayed, as Figure 7H depicted. The content of the transcript 718 is generated locally or remotely on the device 600 (e.g., using remote server computing resources to generate).

[0253] In response to a user input on the animated virtual avatar 700 (e.g., a tap and hold gesture represented by Figure 7I contact 720), the device 600 displays an option menu associated with the animated virtual avatar 700, as Figure 7J depicted. For example, the menu 722 includes several response buttons 723-1 to 723-6, and the device 600 can send them to one or more remote users participating in the communication represented in the messaging area 609. Additionally, the menu 724 is also displayed, which has a copy button 726, a save button 728, and a more button 730. The copy button 726 copies the animated virtual avatar 700 to the clipboard of the device 600. The save button 728 saves the animated virtual avatar 700 to the device 600 (e.g., saves it to a database or library that can be accessed later by an application installed on the device 600). The more button 730 displays additional operations that can be performed with respect to the animated virtual avatar 700.

[0254] Figures 8A to 8B is a flowchart showing a method 800 of using an electronic device according to some embodiments. The method 800 is executed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in the method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0255] As described below, method 800 provides an intuitive way to generate and send emojis such as virtual avatars. The method reduces the cognitive burden on the user to generate and send emojis, thus creating a more effective human-machine interface. For battery-powered computing devices, enabling the user to generate and send emojis more quickly and effectively saves power and increases the time interval between two battery charges.

[0256] An electronic device (e.g., 600) having a camera (e.g., 602) (e.g., configured with one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (802) a virtual avatar generation interface (e.g., Figure 6N 643) (e.g., an interface for selecting emojis (animated or static), generating static stickers, and / or recording animated virtual avatars). The electronic device displays (804) a preview of the virtual avatar (e.g., Figure 6N 646) (e.g., a 2D or 3D computer-generated graphic object, which in some cases is intended to convey a non-verbal message, such as a mood or reaction) (e.g., an animated virtual avatar selected from a plurality of different available virtual avatar templates) in the virtual avatar generation interface. The preview of the virtual avatar responds to changes in the appearance of the face in the camera field of view (e.g., Figure 6O )(e.g., the animated virtual avatar will reflect the user's head movement, facial expression, and orientation detected in the image data from one or more image sensors in the camera). While displaying the preview of the virtual avatar, the electronic device detects (806) an input in the virtual avatar generation interface (e.g., contact 652, 682, or 690). In response to (808) detecting an input in the virtual avatar generation interface and based on determining that the input starts on the preview of the virtual avatar (e.g., 682 or 690) (e.g., a touch-and-hold input on the animated virtual avatar or a touchpad input controlling a cursor), the electronic device generates (810) a static virtual avatar sticker (e.g., 683 or 691) (e.g., a still image of an animated emoji that can be "stuck" at a specific location in the message area), which represents the expression of the face in the camera field of view at the corresponding time. In some embodiments, the corresponding time is determined based on the timing of the input (e.g., when the input is first received, when the input ends, when the gesture corresponding to the input starts to move across the touch-sensitive surface, or any other time associated with the input). Based on determining that the input includes an activation of a recording affordance (e.g., 648) in the virtual avatar generation interface (e.g., a tap on the recording affordance), the electronic device generates (812) an animated virtual avatar (e.g., 668), which represents a sequence of changes in the expression of the face in the camera field of view over a period of time (e.g., as Figures 6Q to 6Sas shown). In some embodiments, a time period is determined based on the timing of an input (e.g., a time period starting when the start of the input is detected, when the end of the input is detected, when a certain type of movement of the input is detected (such as when the input is a gesture on a touch-sensitive surface), or some other time period based on the input). In some embodiments, the virtual avatar is three-dimensional. In some embodiments, a preview of the virtual avatar (e.g., 646) or an animated virtual avatar (e.g., 659) is displayed in 3D form. Eliminating the ambiguity between user inputs for two possible styles (e.g., animated virtual avatars and static virtual avatars) avoids the need for a separate interface to generate each type of message content. Reducing the number of inputs required to convey the desired message enhances the operability of the device and makes the user device interface more efficient (e.g., by allowing multiple types of multimedia communication from a single interface to help the user achieve the intended communication), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0257] In some embodiments, an electronic device displays (814) a messaging interface (e.g., interface 608) (e.g., a messaging application such as Apple's Messages) that includes a message area (e.g., 609). The message area includes messages (e.g., 610-1 to 610-4) from two or more participants (e.g., in Figure 6N the communication in the message area 609) (e.g., a messaging thread) (e.g., messages sent from a user of the electronic device and messages received from a remote user of a different electronic device). The virtual avatar generation interface is displayed simultaneously with the messaging interface (e.g., Figure 6C )(e.g., the virtual avatar generation interface is displayed in the lower half of the messaging interface). In some embodiments, a virtual avatar preview (e.g., 646) is automatically displayed as part of the initial display of the virtual avatar generation interface. Figure 6N In some embodiments, the messaging interface includes a message composition area (e.g., 612) (e.g., a message input area for entering text, emojis, and other content before sending a message to a recipient) and the input is a tap on the virtual avatar preview (e.g., 646). In response to detecting an input in the virtual avatar generation interface, the electronic device displays a static virtual avatar (e.g., 683 or 691) in the message composition area. In some embodiments, displaying the virtual avatar generation interface includes replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with the display of the virtual avatar generation interface (e.g., from

[0258] In some embodiments, the messaging interface includes a message composition area (e.g., 612) (e.g., a message input area for entering text, emojis, and other content before sending a message to a recipient) and the input is a tap on the virtual avatar preview (e.g., 646). In response to detecting an input in the virtual avatar generation interface, the electronic device displays a static virtual avatar (e.g., 683 or 691) in the message composition area. In some embodiments, displaying the virtual avatar generation interface includes replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with the display of the virtual avatar generation interface (e.g., from Figures 6H to 6NConversion without intervening in the data). Displaying the multimedia content of a message before sending the message reduces the likelihood of error messages and allows the user to add more content (e.g., via text or other content) before sending the message. Reducing the number of messages required to deliver the desired message enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user achieve the intended communication while reducing the number of messages required for communication), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0259] In some embodiments, the avatar generation interface includes a static virtual avatar area (e.g., 629) (e.g., a tray of previously generated stickers displayed at the bottom of the avatar generation interface), and the static virtual avatar area includes a collection of one or more previously generated virtual avatars (e.g., 630-1 to 630-4). In response to user input (e.g., Figure 6CC of 682), the electronic device adds the generated virtual avatar to (612) the collection of one or more previously generated virtual avatars (e.g., send the avatar, favorite the virtual avatar, or otherwise mark the virtual avatar to include it in the collection of virtual avatars). In some embodiments, in response to user input (e.g., 626) (e.g., a selection that enables a representation of the collection of virtual avatars in the avatar generation user interface or the messaging user interface), a collection of virtual avatars (e.g., stickers) is displayed (e.g., including a miniature version of the newly generated stickers in the tray). In some embodiments, the tray of previously generated stickers is hidden until an input is received from the user requesting display of the tray (e.g., input 626) or until some other event detected on the electronic device indicates that the tray may be relevant to the current state of the message interface or the avatar generation interface. In some embodiments, after adding a virtual avatar to the collection of virtual avatars, the electronic device receives a request from the user to share the collection of virtual avatars with a second user, and in response, the electronic device sends the collection of virtual avatars to the second user. Maintaining the previously sent message multimedia content allows the user to add and reuse the previous content when applicable to a new message. Not requiring recreation of the content enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding duplicate generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0260] In some embodiments, the input starts (816) on a preview of the virtual avatar (e.g., 680) and ends at a location within the message area (e.g., see Figures 6HH to 6KK)(For example, a gesture that starts with a finger touching a preview of a virtual avatar (e.g., 680), continues with the finger being dragged to the message area, and ends with the finger being lifted off in the message area (in some cases, the gesture may end on a specific message in the message area, and the sticker is associated with that specific message and optionally moves with the message as it moves in the conversation)). The electronic device sends (818) a static virtual avatar (e.g., 691) to the participants associated with the communication (e.g., one or more remote users) (e.g., Figure 6KK ). In some embodiments, an animation is displayed ([[]] Figures 6CC to 6FF [[]]) in response to a gesture showing the static virtual avatar being peeled away from the virtual avatar preview. Figures 6CC to 6FF )

[0261] In some embodiments, the static virtual avatar (e.g., 691) has an appearance determined based on the expression of the face in the camera field of view when an input (e.g., 688) is detected on the virtual avatar preview (e.g., 680). In some embodiments, in response to the start of an input being detected on the preview of the virtual avatar, the electronic device causes the preview of the virtual avatar to stop (820) reacting to changes in the appearance of the face in the camera field of view. This indicates to the user that a sticker has been generated and previews for the user the appearance of the sticker that will be sent when the sticker is dragged into the communication displayed in message 609. By showing a preview of the sticker to be generated without the user having to perform additional interactions or complete a full sticker generation gesture before seeing the resulting sticker, the operability of the device is enhanced, which improves and makes more efficient the human-machine interface. This reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0262] In some embodiments, after the input moves away from the virtual avatar, the virtual avatar preview (e.g., 680) resumes reacting to changes (e.g., when dragging the static avatar towards the message session, the animation resumes). In some embodiments, the virtual avatar preview (e.g., 680) stops reacting to changes in the face appearance until the input for dragging the static avatar ends. Resuming the update of the virtual avatar preview enables the user to compare the appearance of the generated sticker with other possible appearances of the virtual avatar, which may potentially be the basis for different / additional stickers. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback on other content that can be generated before the user sends the generated content to help the user achieve the desired result), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0263] In some embodiments, in response to the expiration of a time period for generating an animated virtual avatar (e.g., expiration of a timer of 5 seconds, 10 seconds, or 15 seconds, or user input that stops the time period), the electronic device displays a send or confirm affordance (e.g., 664) in place of a record affordance (e.g., no longer displays a virtual record button, but instead displays a virtual send button in place of the virtual record button). In response to receiving an input that selects the send or confirm affordance (e.g., a tap gesture on the send affordance on a touch-sensitive display), the electronic device sends (824) the generated animated virtual avatar to a remote user (e.g., see Figure 6U and Figure 6X , rather than first sending the animated virtual avatar to the message composition area 612, as Figure 6V and Figure 6W illustrate) (e.g., sends the animated virtual avatar to a remote user associated with a messaging thread or session without first placing the animated virtual avatar in another area of the messaging interface such as the message composition area). Displaying a send or confirm button in place of a record button after completion of recording the animated virtual avatar enables more information to be displayed in the interface by reusing the area occupied by buttons not applicable to the current state of the interface and by providing the user with more contextually relevant functionality. This enhances the operability of the device and makes the user device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0264] In some embodiments, in response to the expiration of a time period for generating an animated virtual avatar, the electronic device displays (822) a confirm affordance (e.g., 664) in place of a record affordance. In response to receiving an input that selects the send affordance (e.g., via contact 667) (e.g., a tap gesture on the send affordance on a touch-sensitive display), the electronic device displays a representation of the animated virtual avatar (e.g., a static graphic element or the animated virtual avatar) in the message composition area (e.g., 612) of the messaging interface (e.g., Figure 6V ) (e.g., the area of the messaging interface that displays text typed on a keyboard). In some embodiments, the time period is based on a predetermined amount of time (e.g., the amount of time represented by the progress indicator 656). After generating the animated virtual avatar, the electronic device stops displaying a preview of the virtual avatar and displays a looping version of the animated virtual avatar (e.g., Figure 6T)。The display of the looped version of the animated virtual avatar includes displaying the animation sequence two or more times (e.g., as described below with respect to method 900). Displaying a send or confirmation button in place of the record button after the recording of the animated virtual avatar is completed enables more information to be displayed in the interface by reusing the area occupied by buttons that are not applicable (or less applicable) to the current state of the interface. This enhances the operability of the device and makes the user device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0265] In some embodiments, a first virtual avatar template (e.g., the template represented by element 644-4, or another element indicated by Figure 6N indicator 645) is used to display the looped version of the animated virtual avatar. The electronic device displays representations of a plurality of other virtual avatar templates (e.g., elements 644-1 through 644-7) (e.g., miniature generic versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), which representations include a representation of a second virtual avatar template (e.g., Figure 6Z element 644-8), where the second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the plurality of virtual avatar templates are based on emojis that can be sent via a messaging application. After starting to display the looped version of the animated virtual avatar (e.g., Figure 6T ) and in response to receiving a user input (e.g., 665) selecting the representation of the first virtual avatar template, the electronic device updates the display of the looped version of the animated virtual avatar to reflect the second virtual avatar template (e.g., Figure 6U ) (e.g., when the animated virtual avatar is still based on a sequence of facial expression changes, changing the appearance of the animated virtual avatar to reflect the new virtual avatar template). For example, the animated virtual avatar can change from a monkey to a robot, but it will still reflect the sequence of facial expression changes over time in the same camera field of view (e.g., see the transition from Figures 6T to 6U ). Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the selection of the style of the animated virtual avatar after recording the movements and actions of the animated virtual avatar. Not requiring the re-recording of the animated virtual avatar to try new avatar templates enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding the repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0266] In some embodiments, a virtual avatar preview (e.g., 646) is automatically displayed in response to launching a virtual avatar generation interface. In some embodiments, once the virtual avatar generation interface is displayed, the virtual avatar preview is displayed without user input.

[0267] In some embodiments, in response to detecting a characteristic (e.g., position, orientation, movement) of a first physical characteristic of a face in a camera field of view (e.g., a mouth smile, a tongue sticking out, an ear wiggling, an eyebrow raising, or any other movement of any other physical characteristic), the electronic device updates a first physical characteristic of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the first physical characteristic of the face (e.g., eyes, eyebrows, mouth, tongue, ears) is the same as the type of the first physical characteristic of the displayed preview. In some embodiments, if the user's mouth is open, the mouth of the virtual avatar will open correspondingly (e.g., Figure 6O at 650-1 and 651-1 of). Similar results can be based on facial expressions. For example, if one or more movements of physical characteristics or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be updated to reflect the predefined movement by updating the corresponding physical characteristics or characteristics to reflect the detected facial expression. Mapping the user's physical characteristics to similar physical characteristics of the virtual avatar enables the user to provide movements, expressions, and poses that provide input to the system, which are intuitively mapped onto the virtual avatar without the need for cumbersome or time-consuming touch or key input. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user achieve the desired result by mapping the user's characteristics to the virtual avatar in a predictable manner and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0268] In some embodiments, in response to detecting a characteristic of a second physical characteristic of a face in a camera field of view, the electronic device updates a second physical characteristic of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the second physical characteristic of the face is different from the type of the second physical characteristic of the displayed preview (e.g., at Figure 6OAmong them are the movement of the eyebrows 650-2 and the movement of the ears 651-2). In some embodiments, if the user is smiling, and this indicates that the user is happy, different features of the virtual avatar (such as the horn of a unicorn or the lights on a robot) can be changed to reflect the smile. Similar results can be based on facial expressions. For example, if one or more movements of the physical features or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the preview of the displayed virtual avatar can optionally be updated to reflect the predefined movement by updating different sets of physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to different physical features of the virtual avatar enables the user to provide movements, expressions, and poses that provide input to the system, which are mapped to avatar features that the user would not otherwise be able to easily control. This enhances the operability of the device and makes the user device interface more effective (e.g., by mapping the user's features to the virtual avatar to help the user achieve the desired result by controlling additional features of the virtual avatar and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0269] In some embodiments, in response to detecting movement of a face in the camera field of view, the electronic device updates a third physical feature of the preview of the displayed virtual avatar based on a physical model for the virtual avatar and the detected movement (e.g., Figure 6O at 650-5 and 651-5 in). In some embodiments, for example, if the virtual avatar is based on a virtual avatar template for a puppy, when it is detected that the user's face is shaking, the face of the virtual avatar will shake, and the ears of the virtual avatar can stick out even if the user's ears do not stick out in response to the shaking to reflect the physics of the shaking movement. In some embodiments, the same physical feature of the preview of the displayed virtual avatar is updated based on the movement of the corresponding feature of the face in the camera field of view and the physical model (e.g., the ears move based on the movement of the user's ears but also based on the physical model for the large ears of a puppy). Updating the virtual avatar based on the physical model for the virtual avatar enables the user to create a realistic and interactive virtual avatar that can convey a wider range of non-verbal information. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user convey a predefined message using a more realistic virtual avatar movement), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0270] In some embodiments, the preview of the virtual avatar is based on a predefined virtual avatar template (e.g., associated with Figure 6Nrepresentation of the avatar template associated with 644-4). The electronic device updates the preview of the virtual avatar based on one or more predefined behaviors associated with a predefined virtual avatar template. In some embodiments, if a movement or change in facial expression is detected in the face not in the camera field of view (e.g., 650-3 and 650-4), the preview of the virtual avatar displays a predefined response (e.g., 651-4), such as blinking, rotating the head, making a facial expression, or other actions.

[0271] In some embodiments, in response to determining that a face is no longer detected in the camera field of view (e.g., because the face has moved out of the camera field of view, face tracking has failed; the face is occluded and not present in the camera field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device fades out the display of the virtual avatar preview (e.g., the virtual avatar preview 646 will fade out). In some embodiments, the device makes other modifications to the virtual avatar preview to indicate that the user's face can no longer be tracked, such as degrading the virtual avatar preview based on the last information detected by the device, including changing the size, rotation, movement, etc. of the virtual avatar preview. In some embodiments, in response to determining that a face is no longer detected in the camera field of view (e.g., because the face has moved out of the camera field of view, face tracking has failed; the face is occluded and not present in the camera field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device displays a message indicating that the camera can no longer accurately detect the face (above or instead of Figure 6N the virtual avatar 643 of Figure 6NPreview of the virtual avatar 646 is displayed (display prompt). In some embodiments, in response to determining that no face is detected in the camera field of view (e.g., because the face has moved out of the camera field of view, face tracking fails; the face is occluded and does not appear in the camera field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device updates the display of the preview of the virtual avatar based on the changes in the facial appearance that occurred during a period of time before the face could no longer be detected in the field of view (e.g., immediately before or shortly before the face could no longer be detected in the field of view). For example, device 600 repeatedly displays the transition from smiling to frowning or eye movement. In some embodiments, updating the display of the preview of the virtual avatar based on the changes in the facial appearance that occurred during a period of time before the face could no longer be detected in the camera field of view includes: gradually slowing down the update of the preview of the virtual avatar over time such that the update of the virtual avatar gradually stops (e.g., a rotating avatar slowly stops rotating, eyes that are opening or closing slowly stop opening or closing, a mouth that is opening or closing slowly stops opening or closing). Displaying feedback on whether the preview of the virtual avatar is tracking the user's face enables the user to determine whether the device is being held correctly and whether the conditions for detecting the user's face are correct. Providing improved feedback to the user about the device state enhances the operability of the device and makes the user device interface more effective by providing better continuity of the user interface with an indication that the device is still attempting to track the user's face. This provides a better and more intuitive human-machine interface and enables the user to continue interacting with the device even when the device cannot track the user's face.

[0272] Note that the processes described above with respect to method 800 (e.g., Figures 8A to 8B ) also apply to the methods described below in a similar manner. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, the generation of the sticker described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. Also, for example, muting the sound before sending the animated virtual avatar (e.g., animated virtual avatar) as described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For the sake of brevity, these details are not repeated below.

[0273] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described with respect to Figure 1A 、 Figure 3 and Figure 5A ) or an application-specific chip. Additionally, as referred to above Figure 8A and Figure 8BThe operations described above are optionally implemented by Figures 1A to 1B the components depicted in Figures 1A to 1B . For example, detecting an input (806) in the virtual avatar generation interface is optionally implemented by the event classifier 170, the event recognizer 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects a contact on the touch-sensitive surface 604, and the event dispatcher module 174 delivers event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or invokes the data updater 176 or the object updater 177 to update the internal application state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those of ordinary skill in the art will readily appreciate how other processes can be implemented based on Figures 1A to 1B the components depicted in Figures 1A to 1B .

[0274] Figures 9A to 9B FIG. Figures 9A to 9B is a flowchart showing a method 900 of using an electronic device according to some embodiments. The method 900 is performed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in the method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0275] As described below, the method 900 provides an intuitive way to generate and send emojis such as virtual avatars. The method reduces the cognitive burden on the user to generate and send emojis, thereby creating a more effective human-machine interface. For battery-powered computing devices, enabling the user to generate and send emojis more quickly and effectively saves power and increases the time interval between two battery charges.

[0276] An electronic device (e.g., 600) having a camera (e.g., configured with one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (902) a virtual avatar generation interface (e.g., Figure 6N 643 of Figure 6N ) (e.g., an interface for selecting emojis, generating static emojis, and recording animated emojis). The electronic device displays (904) a preview of the virtual avatar in the virtual avatar generation interface (e.g., Figure 6N646) (e.g., 2D or 3D computer-generated graphical objects, which in some cases are intended to convey non-verbal messages, such as emotions or reactions) (e.g., a moving emoji selected from a plurality of different available emoji styles or templates). A preview of the virtual avatar responds to changes in the appearance of the face in the camera's field of view (e.g., Figure 6O ) (e.g., the animated emoji will reflect the user's head movement, facial expression, and orientation detected in the image data from one or more image sensors). The electronic device receives (906) a request to generate an animated virtual avatar based on a change in the facial expression of the face in the camera's field of view (e.g., contact 652). In response to receiving the request to generate an animated virtual avatar, the electronic device records (908) (e.g., Figures 6Q to 6S ) a sequence of facial expressions of the face in the camera's field of view (e.g., a sequence including a series of data points that provides a mapping of points that can be applied to a virtual avatar template to generate an animated virtual avatar). After recording the facial expressions of the face in the camera's field of view, the electronic device displays (910) an animated virtual avatar (e.g., 659) including a looped version of the animated sequence (e.g., Figure 6T and 6U ), the animated sequence being based on the sequence of facial expressions recorded in response to the request to generate an animated virtual avatar (e.g., the animated virtual avatar is displayed by sequentially mapping a series of data points representing the recorded facial expressions to a predefined animated virtual avatar template). The electronic device displays the looped version of the animated virtual avatar two or more times. In some embodiments, recording the sequence of facial expressions includes recording a time series of values of discrete mapping points for a predefined virtual avatar template. Displaying the looped version of the animated virtual avatar enables the user to subsequently view the animated virtual avatar content to understand whether an appropriate message is being conveyed. Providing improved visual feedback to the user enhances the operability of the device, reduces error instances, and makes the user device interface more effective (e.g., by providing feedback indicating the output that will cause the device to generate the expected result to help the user achieve the expected result and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0277] In some embodiments, the electronic device stops recording the sequence of facial expressions in response to a timer expiration (e.g., indicated by progress indicator 656) (e.g., a 5-second, 10-second, or 15-second timer). In some embodiments, the electronic device stops recording the sequence of facial expressions in response to receiving user input (e.g., contact 658) (e.g., a tap by the user on a virtual button displayed on the display). Limiting the time for recording the animated virtual avatar enables the user to create the animated virtual avatar recording while limiting the impact on the computing resources (e.g., storage) of the device. This enhances the operability of the device by conserving the computing resources of the device.

[0278] In some embodiments, the electronic device replaces (912) the display of the preview with a display of a looped version of the animated virtual avatar (e.g., see the transition from Figures 6S to 6T (e.g., in response to completion of recording of the sequence of facial expressions, the generated animated virtual avatar is played back to the user in a loop). Playing the looped version of the recorded animated emoji automatically enables the user to view the animated emoji before deciding whether to send, delete, or save the animated emoji. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback indicating the result to help the user achieve the desired result before the user submits the result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0279] In some embodiments, the request to generate the animated virtual avatar includes selecting a record affordance representation (e.g., 648) displayed in the virtual avatar generation interface. After recording the facial expression of the face in the camera view, the electronic device replaces the display of the record affordance representation with a send or confirm affordance representation (e.g., 664). In some embodiments, the send affordance representation operates as explained above with respect to method 800. Recording the animated virtual avatar in response to the selection of the record affordance representation enables the user to use the virtual avatar preview to verify that the device is tracking the user and that the currently selected virtual avatar template is consistent with the message the user wishes to convey. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing a preview of the desired result to help the user achieve the desired result before the user generates the result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0280] In some embodiments, a looping version of an animated virtual avatar is displayed (918) using a first virtual avatar template (e.g., the avatar template corresponding to element 644-4). The electronic device displays representations of a plurality of other virtual avatar templates (e.g., elements 644-1 through 644-7) (e.g., miniature generic versions of different virtual avatar templates such as smiley faces, animals, robots, or other objects), and these representations include a representation of a second virtual avatar template (e.g., Figure 6Z element 644-8). The second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the plurality of virtual avatar templates are based on emojis that can be sent via a messaging application. After starting to display the looping version of the animated virtual avatar and in response to receiving user input selecting the representation of the first virtual avatar template, the electronic device updates (922) the display of the looping version of the animated virtual avatar to reflect the second virtual avatar template (e.g., changing the appearance of the animated virtual avatar to reflect the new virtual avatar template while the animated virtual avatar is still based on a sequence of changing facial expressions). For example, the animated virtual avatar can change from a monkey to a robot, but it will still reflect the sequence of changing facial expressions over time within the same camera field of view (e.g., see the transition from Figures 6T to 6U ). In some embodiments, the electronic device displays (914) representations of virtual avatar templates (e.g., miniature generic versions of different virtual avatar templates such as smiley faces, animals, robots, or other objects), and these representations include a representation of the first virtual avatar template. In response to receiving user input corresponding to the selection of the representation of the first virtual avatar template, the electronic device updates (916) the display of the looping version of the animated virtual avatar to correspond to the first virtual avatar template (e.g., changing the animation to a robot animation based on the robot virtual avatar template instead of a puppy animation based on the puppy virtual avatar template without the user having to re-record any facial expressions). Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for an intended message by allowing selection of the style of the animated virtual avatar after recording the movement and actions of the animated virtual avatar. Not having to re-record the animated virtual avatar to preview the new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding the repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0281] In some embodiments, the preview virtual avatar is based on the second avatar template. In response to detecting a face in the camera field of view (e.g., Figure 6OThe first characteristic of the first physical characteristic of (e.g., movement of the user's eyebrows in 650-2), the electronic device updates (920) the first physical characteristic of the displayed virtual avatar preview based on the detected first characteristic (e.g., causing the virtual avatar preview to move its eyebrows). The first physical characteristic of the displayed preview has a first characteristic type (e.g., eyebrows). After receiving a user input corresponding to the selection of a first graphical element (e.g., switching the avatar template from a puppy to a monkey) and in response to detecting a second characteristic of the first physical characteristic of a face in the camera field of view (e.g., movement of the user's eyebrows), the electronic device updates (922) the second physical characteristic of the displayed virtual avatar preview based on the detected second characteristic (e.g., Figure 6O of 651-2) (e.g., moving the ears of the monkey), wherein the second physical characteristic of the displayed preview has a second characteristic type (e.g., ears) different from the first characteristic type (e.g., eyebrows). Mapping the same physical characteristic of the user to different physical characteristics of different avatar templates enables the user to have a wider range of options for transmitting messages by generating multiple virtual avatars using the same input. This enhances the operability of the device and makes the user device interface more effective (e.g., by providing more choices for conveying messages to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0282] In some embodiments, in response to receiving a user input corresponding to a request to scroll (e.g., via Figures 6Y to 6AA contact 676) multiple representations of an avatar template, the electronic device scrolls the display of the multiple representations of the avatar template to display a second graphical element that is not part of the multiple representations of the avatar template. In some embodiments, the scrolling is based on the speed of the user input corresponding to the request. Scrolling through the avatar template enables the user to quickly view different options for the virtual avatar. Additionally, scrolling the display of the multiple representations of the avatar template enables the user to view the previous avatar template and the next avatar template. This enhances the operability of the device and makes the user device interface more effective (e.g., by providing feedback indicating the output that will cause the device to generate the intended result to help the user achieve the intended result, and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0283] In some embodiments, after detecting the end of user input, the scrolling speed gradually decreases over time (e.g., the scrolling gradually stops as if the multiple representations of the virtual avatar have inertia that gradually slows down due to friction). Gradually decreasing the scrolling speed over time enables the user to continue viewing different virtual avatar template options without providing additional input. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback on possible results without additional interaction to help the user achieve the desired outcome and by reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0284] In some embodiments, in response to receiving user input corresponding to a request, the electronic device generates an auditory output and / or a tactile output corresponding to the currently selected virtual avatar template that changes from one virtual avatar template to a different virtual avatar template. For example, when each of the multiple representations of the virtual avatar template scrolls past the position indicating the currently selected virtual avatar template, an auditory output and / or a tactile output is generated. Generating the auditory or tactile feedback enables the user to determine when a new selection occurs. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback indicating when a new selection is made to help the user achieve the desired outcome and by reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0285] In some embodiments, in response to receiving user input on an animated virtual avatar, the input corresponding to a request to save the animated virtual avatar, the electronic device stores the data for the animated virtual avatar in a database on the electronic device (e.g., Figures 6CC to 6GG ). For example, the electronic device stores the data representing the animated virtual avatar in a directory or library in the non-volatile storage device in the electronic device.

[0286] In some embodiments, the electronic device receives a request from a remote user to send the animated virtual avatar to a remote device (e.g., contact 688) (e.g., Figures 6HH to 6KK)。Based on determining that the remote device meets a first set of criteria (e.g., the remote device has the required application with the required version for playing the first version of the animated virtual avatar), the electronic device sends the first version of the animated virtual avatar to the user of the remote device (e.g., sends non-graphic data representing the recorded sequence of facial expressions and an indication of the virtual avatar template such that the remote device can reproduce the animated virtual avatar). Based on determining that the remote device does not meet the first set of criteria (e.g., the remote device does not have a suitable messaging application or a suitable version of the messaging application for playing the first version of the animated virtual avatar), the electronic device sends a second version of the animated virtual avatar that is different from the first version (e.g., sends a video file representing the animated virtual avatar) to the user of the remote device. Determining which version of the animated virtual avatar to send to the remote user saves the resources of the device by sending only the minimal amount of compatible information to the remote user. Additionally, doing so reduces the need for the user to resend the information in a more compatible format (e.g., in response to the remote user indicating that the initial format is not viewable). Efficient and effective data transfer enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0287] In some embodiments, when displaying a looped version of the animated virtual avatar, the electronic device plays (924) audio data based on the sound recorded during the recording of the sequence of facial expressions and based on an audio filter associated with a predefined avatar template. In some embodiments, when displaying a looped version of the animated virtual avatar, the electronic device plays audio data based on the sound recorded during the recording of the sequence of facial expressions. In response to receiving user input corresponding to a selection of a mute enabling indication (e.g., Figure 6T and Figure 6U of 662), the electronic device stops playing the audio data. Playing filtered audio for the animated virtual avatar based on a filter specific to the avatar template used for the virtual avatar enables the user to more effectively convey messages by providing more options on how to convey the message and a more engaging animated virtual avatar. This enhances the operability of the device and makes the user device interface more effective (e.g., by providing the user with more options on how to convey the message to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0288] In some embodiments, in response to receiving a request to send an animated virtual avatar to a remote user (926) and based on determining that the request to send the animated virtual avatar to the remote user is received while muting audio data associated with the display of the looped version of the animated virtual avatar (e.g., contact 667), the electronic device sends (928) data representing the animated virtual avatar to the remote user without sending the sound data for the animated virtual avatar. Based on determining that the request to send the animated virtual avatar to the remote user is received while not muting the audio data associated with the display of the looped version of the animated virtual avatar, the electronic device sends (930) the data representing the animated virtual avatar to the remote user together with the sound data for the animated virtual avatar. Sending the animated virtual avatar without sound when the user mutes the playback of the animated virtual avatar enables the user to effectively select whether to include sound in the message sent to the remote user. This enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user achieve the intended message with minimal interaction), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0289] In some embodiments, in response to receiving a request to generate an animated virtual avatar (e.g., contact 652), the electronic device records a first facial movement of a face in the camera's field of view, wherein displaying a looped version of the animated virtual avatar includes animating the virtual avatar based on a physical model for the animated virtual avatar and the first facial movement (e.g., see image data 650-5, and Figure 6O update 651-5). Updating the animated virtual avatar based on the physical model for the virtual avatar enables the user to create a realistic and interactive virtual avatar that can convey a wider range of non-verbal communication. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user convey a predetermined message using more realistic virtual avatar movements), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0290] In some embodiments, in response to detecting that a particular feature of a face in the camera's field of view remains in a particular pose for more than a threshold amount of time while recording a sequence of facial expressions (e.g., see Figure 6O 650-4 and 650-5), the electronic device adds a predefined animated expression corresponding to the particular pose of the face to the animated virtual avatar (e.g., see Figure 6Oof 651-5). For example, if the face has a neutral expression for a predetermined period of time, predefined movements (such as head turns or blinks) are added to the animated virtual avatar. As another example, if the face has an angry expression for a predetermined period of time, one or more additional features indicating anger (such as the ears changing color or steam emerging from the ears) are added to the animated virtual avatar. Updating the animated virtual avatar based on the device detecting that the features of the user's face remain in a particular pose for a threshold amount of time enables the user to add more actions to the animated virtual avatar than may be achievable with just facial expressions, features, and movements. This enhances the operability of the device and makes the user device interface more effective (e.g., by providing an additional mechanism for conveying actions that do not otherwise correspond to readily available facial expressions, movements, or features to help the user achieve the desired result), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0291] In some embodiments, when recording a sequence of facial expressions, in response to receiving user input via an input mechanism separate from the camera (e.g., a touch on a touch-sensitive surface, movement of the electronic device detected by a motion sensor, activation of a button, or other input), the electronic device adds a first facial expression to the sequence of facial expressions (e.g., records a happy facial expression, a facial expression with the tongue sticking out, or any other facial expression not actually recorded as a face in the camera's field of view when recording facial expressions for inclusion in the animated virtual avatar). The first facial expression is based on the user input received via the input mechanism. In some embodiments, when the animated virtual avatar is looping, the user can use touchscreen controls to add additional expressions to the animated virtual avatar such that as the animated virtual avatar loops, the user can gradually add expressions to the animated virtual avatar so that even if the changes in those expressions are different from the changes in expressions recorded based on the face (e.g., the user's face) in the camera's field of view when initially creating the animated virtual avatar, the animated virtual avatar includes the changes in the expressions selected by the user. Updating the animated virtual avatar based on user input rather than with camera capture enables the user to add more actions to the animated virtual avatar than may be achievable with just facial expressions, features, and movements. This enhances the operability of the device and makes the user device interface more effective (e.g., by providing an additional mechanism for conveying actions that do not otherwise correspond to readily available facial expressions, movements, or features to help the user achieve the desired result), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0292] Note that the processes described above with respect to method 900 (e.g., Figures 9A to 9B) details can also be applied to the methods described above in a similar manner. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, displaying an animated virtual avatar preview based on a recorded sequence of facial features, movements, and / or expressions and based on a frame associated with the avatar template described with respect to method 900 can be applied to the sticker and animated virtual avatar interfaces described above with respect to method 800.

[0293] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as described with respect to Figure 1A , Figure 3 and Figure 5A described) or an application-specific chip. Additionally, the operations described above with reference to Figure 9A and Figure 9B are optionally implemented by Figures 1A to 1B depicted in the components. For example, receiving a request to generate an animated virtual avatar (906) is optionally implemented by event classifier 170, event recognizer 180, and event handler 190. Event monitor 171 in event classifier 170 detects a contact on touch-sensitive surface 604, and event distributor module 174 delivers the event information to application 136-1. The corresponding event recognizer 180 of application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, event recognizer 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally utilizes or invokes data updater 176 or object updater 177 to update the application internal state 192. In some embodiments, event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those of ordinary skill in the art will clearly know how to implement other processes based on Figures 1A to 1B depicted in the components.

[0294] Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B and Figures 17A to 17B show exemplary user interfaces for generating and modifying virtual avatars according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, which include Figures 18A to 18B ,Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 , Figure 24 and Figure 25 the processes in.

[0295] In some embodiments, a virtual avatar is a user representation that can be graphically depicted. In some embodiments, the virtual avatar is non-photorealistic (e.g., cartoonish). In some embodiments, the avatar is an anthropomorphic construct such as a stylized animal (e.g., avatars 1100, 1300, 1500, 1600, and 1700), a stylized robot (e.g., avatar 1400), or a stylization of an inanimate object in general (e.g., avatar 1000). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., map) to one or more physical features of the user's face such that movement of the detected physical features of the user affects the avatar features (e.g., affects the graphical representation of the features).

[0296] In some examples, a user is able to manipulate the properties or features of a virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164). When the physical features (such as facial features) and location (such as head position or head tilt) of the user change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical features and location. In some embodiments, the change in the user's body features and location indicates various expressions, moods, contexts, tones, or other non-verbal communications. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, moods, contexts, tones, or other non-verbal communications.

[0297] Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B and Figures 17A to 17B illustrate corresponding changes in exemplary user inputs and exemplary virtual avatars (e.g., poop avatar, bear avatar, alien avatar, rabbit avatar, robot avatar, unicorn avatar, chicken avatar, and pig avatar) displayed on an electronic device. In some embodiments, the electronic device includes one or more elements and / or features of devices 100, 300, and 500. Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B andFigures 17A to 17B The image on the left represents an image of the user detected by the electronic device when the user is within the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In other words, the image of the user is from the perspective of the camera (e.g., camera module 143, optical sensor 164), which in some embodiments may be positioned on the electronic device (e.g., devices 100, 300, and 500), and in other embodiments may be positioned separately from the electronic device (e.g., an external camera or sensor that transmits data to the electronic device). In some embodiments, Figures 10A to 10I 、 Figures 11A to 11C 、 Figures 12A to 12C 、 Figure 13 、 Figures 14A to 14D 、 Figures 15A to 15B 、 Figures 16A to 16B and Figures 17A to 17B the boundary of the image on the left represents the boundary of the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In some embodiments, the image of the user is displayed on a display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504). In some embodiments, the image of the user is transmitted to an external electronic device for display. In some embodiments, the external electronic device includes one or more elements and / or features of devices 100, 300, and 500. In some embodiments, the image data of the user is collected and processed by the device but is not immediately displayed on the electronic device or transmitted to an external device.

[0298] Figures 10A to 10I 、 Figures 11A to 11C 、 Figures 12A to 12C 、 Figure 13 、 Figures 14A to 14D 、 Figures 15A to 15B 、 Figures 16A to 16B and Figures 17A to 17B each of the images on the left shows a virtual avatar (e.g., a poo avatar) in a state presented (e.g., displayed after being modified) based on the detected corresponding user image located on the left side of the figure. In some embodiments, the virtual avatar is displayed from the perspective of the user observing the virtual avatar. In some embodiments, the virtual avatar is displayed on a display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504). In some embodiments, the virtual avatar is transmitted to an external electronic device for display. In some embodiments, Figures 10A to 10I 、 Figures 11A to 11C 、 Figures 12A to 12C 、 Figure 13 、 Figures 14A to 14D 、 Figures 15A to 15B 、Figures 16A to 16B and Figures 17A to 17B the image on the right represents the position of the virtual avatar within the display area of a display of an electronic device (e.g., touch screen 112, display 340, display 450, display 504), and Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B and Figures 17A to 17B the boundary of the image on the right represents the boundary of the display area that includes the virtual avatar. In some embodiments, the display area represented in the right corresponds to the avatar display area of an application user interface, such as the virtual avatar interface 643, the message composition area 612, the message area 609 (or a portion thereof) discussed above.

[0299] In some embodiments, the magnitude of the response of an avatar feature (e.g., a discrete element of the avatar that can be discretely moved or modified relative to other avatar features) corresponds to the magnitude of the change in a user's physical feature (e.g., a detected or tracked feature such as a user's muscle, muscle group, or anatomical feature such as an eye). For example, in some embodiments, the magnitude of the change in the physical feature is determined based on the potential range of motion of the physical feature, where the magnitude represents the relative position of the physical feature within the range of motion of the physical feature (e.g., a predicted or modeled range of motion). In such embodiments, the magnitude of the response of the avatar feature is similarly the relative position of the avatar feature within the range of motion of the avatar feature. In some embodiments, the magnitude of the change is determined based on a comparison or measurement (e.g., distance) of the starting and ending positions of the change in the physical feature. In such embodiments, the change in the physical feature can be translated into a modification of the first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly, or as a scaled or adjusted value).

[0300] In some embodiments, the modification of the avatar feature has a magnitude component and a direction component, wherein the direction component of the modification of the avatar feature is based on the direction component of the change of one or more physical features to which the avatar feature responds. In some embodiments, the direction of the response of the avatar feature corresponds to (e.g., directly corresponds to or conversely corresponds to) the relative direction of the change of the user's physical features, wherein the relative direction of the change of the physical features is determined based on the direction of movement of the physical features starting from an initial position (e.g., the neutral position, the rest position of the physical features, or in some embodiments, the position of the physical features initially detected by the device). In some embodiments, the direction of the response of the avatar feature directly corresponds to the relative direction of the change of the physical features (e.g., if the physical features move upward, the avatar feature also moves upward). In other embodiments, the direction of the response of the avatar feature conversely corresponds to the relative direction of the change of the physical features (e.g., if the physical features move upward, the avatar feature moves downward).

[0301] In some embodiments, the direction component of the change of the avatar feature is mirrored with respect to the direction component of the change of the physical features. For example, when the physical feature (e.g., the user's mouth) moves to the left, the avatar feature (e.g., the avatar's mouth) moves to the right. In some embodiments, for movement along the vertical axis, the direction component of the change of the avatar feature is the same as the direction component of the change of the physical features, and for movement along the horizontal axis, the direction component of the change of the avatar feature is in a mirror image relationship with the direction component of the change of the physical features, similar to the effect seen when looking in a mirror. In some embodiments, the neutral rest position of the user's iris is determined relative to a specific position (e.g., a centered position) around the user's eyeball.

[0302] Figure 10A An exemplary embodiment is shown that illustrates an electronic device modifying the poo avatar 1000 in response to detecting a change in the user's facial features. The poo avatar is shown as having four display states (1011A, 1011B, 1011C, and 1011D), wherein each of the four display states of the poo avatar corresponds to a respective one of the four detection states (1001A, 1001B, 1001C, and 1001D) of the user. Specifically, in Figure 10A each of the display states, the electronic device locates or modifies the features of the poo avatar in response to detecting the position or change in position of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. In Figure 10A the illustrated embodiment, the detected facial features of the user include the user's mouth 1020 (having corners 1020A and 1020B) and the user's eyebrows 1022. In some embodiments, the physical features being tracked may include other facial features such as eyelids, irises, muscles, muscle groups, etc. InFigure 10A In an embodiment, the features of the poop avatar include the avatar face 1000C, the mouth 1030, the avatar eyes 1032, the upper part 1034 of the avatar, the lower part 1036 of the avatar, and (in certain states) the avatar eyebrows 1038.

[0303] As shown in 1001A, the electronic device detects a neutral facial expression of the user. For example, the electronic device detects that the user's mouth 1020 and eyebrows 1022 are positioned in a relaxed neutral state rather than in positions associated with a particular facial expression (e.g., smiling or frowning). In response to detecting the user's neutral facial expression, the electronic device displays a poop avatar 1000 with a neutral expression (e.g., neutral state) in 1011A. Specifically, the electronic device displays a poop avatar having an avatar mouth 1030 in a relaxed neutral state rather than in a position typically associated with a particular facial expression (e.g., smiling or frowning). In some embodiments, the neutral position of the avatar corresponds to other representations of the associated image, such as a static poop emoji that may be found in a messaging application. Additionally, the electronic device displays the face 1000C of the poop avatar without the eyebrows 1038, and displays the avatar eyes 1032 looking in a direction perpendicular to the focal plane of the display (or the camera (e.g., camera 143, optical sensor 164)). The electronic device also displays the upper part 1034 of the poop avatar in a neutral vertical position above the lower part 1036.

[0304] As shown in 1001B, the electronic device detects the position of the user's mouth 1020 for forming a smiling facial expression (e.g., one or both of the corners 1020A and 1020B of the user's mouth are positioned in an upward posture (e.g., upward position) to form a smiling facial expression). In response to detecting the position of the user's mouth 1020, the electronic device modifies the display of the avatar mouth 1030 to have a smiling expression such that the avatar mouth 1030 has an open smiling position, as shown in 1011B. As shown in 1011B, the eyes 1032, the upper part 1034, and the lower part 1036 remain unchanged relative to their original positions in 1011A.

[0305] As shown in 1001C, the electronic device detects the user's frowning facial expression. In some embodiments, the electronic device detects the frowning facial expression by detecting that one or both of the corners 1020A and 1020B of the user's mouth are positioned in a downward posture (e.g., a downward position) and the user's eyebrows 1022 are in a downward position (e.g., compared to the position of the eyebrows 1022 in the relaxed neutral state in 1001A and 1001B, the user's eyebrows are wrinkled or positioned lower on the user's face). In response to detecting the user's frowning facial expression, the electronic device modifies the poop avatar to have a frowning and drooping face, as shown in 1011C. For example, the electronic device modifies the poop avatar such that the corners of the avatar's mouth 1030 turn downward to give the mouth 1030 a slightly open posture, and the lower part 1036 of the poop avatar bends downward, similar to the downward turning posture of the poop avatar's mouth 1030. In Figure 10A embodiments, when the electronic device detects the downward position of the user's eyebrows 1022, the anatomical corresponding part of the poop avatar above the avatar's eyes 1032 is not modified.

[0306] In some embodiments, the electronic device modifies the poop avatar by displaying an animation of the mouth 1030 turning downward and the lower part 1036 bending downward as the mouth 1030 moves to the downward turning position, to give it a drooping face as shown in 1011C. In some embodiments, the electronic device further modifies the poop avatar such that when the poop avatar presents a drooping face, the tip 1040 of the poop avatar collapses or tilts downward. In some embodiments, the position of the tip 1040 is specifically based on the position of the user's eyebrows 1022 (a physical feature that is not anatomically corresponding to the tip 1040). In some embodiments, when the user no longer makes a frowning expression, the electronic device modifies the poop avatar to return it to the neutral position. In such embodiments, the electronic device modifies the poop avatar by displaying an animation of the mouth 1030 moving to the neutral position and the lower part 1036 moving back to its neutral position, to return it to the neutral state in 1011A. In some embodiments, returning the poop avatar from the drooping face to the neutral state includes the electronic device displaying the tip 1040 of the poop avatar being straightened to its neutral position.

[0307] As shown in 1001D, the electronic device detects that the corners 1020A and 1020B of the user's mouth are slightly raised and the user's eyebrows 1022 are in a raised position (e.g., at a higher position on the user's face compared to the position of the eyebrows 1022 when in the relaxed neutral state shown in 1001A and 1001B). In response to detecting the positions of the corners 1020A and 1020B of the user's mouth 1020, the electronic device modifies the display of the poop avatar such that the corners of the avatar's mouth 1030 are slightly raised to match the positions of the corners 1020A and 1020B of the user's mouth 1020. In response to detecting the raised position of the user's eyebrows 1022, the electronic device modifies the poop avatar by introducing the eyebrows 1038 located above the eyes 1032 of the poop avatar to a raised position (e.g., to convey an expression that the poop avatar 1000 is raising its eyebrows 1038) and extending the upper portion 1034 of the poop avatar in an upward direction (e.g., by extending the tip portion 1040 of the poop avatar while maintaining the original position of the lower portion 1036). In the embodiments shown in 1001D and 1011D, when the user's eyebrows 1022 are raised, the electronic device introduces the avatar eyebrows 1038 and extends the tip portion 1040 of the poop avatar. In some embodiments, when the user's eyebrows 1022 return to their neutral position, the electronic device removes the avatar eyebrows 1038 and relaxes the tip portion 1040. In some embodiments, the electronic device removes the eyebrows 1038 of the poop avatar by animating the process of moving the eyebrows 1038 downward toward the eyes 1032 of the poop avatar and disappearing into the poop avatar face 1000C above the eyes 1032.

[0308] Figure 10B An exemplary embodiment is shown, which shows an electronic device modifying a poop avatar in response to detecting a change in a user's facial feature, wherein the modification of the poop avatar includes moving the avatar features in an exaggerated manner. In some embodiments, exaggerating the features of a virtual avatar allows a user to affect the maximum change made to the avatar features without uncomfortably changing the corresponding features of their face. For example, as Figure 10B shown, the user can open the avatar's mouth as wide as possible (e.g., in a surprised expression) without uncomfortably opening their mouth (e.g., without having to open their mouth to the maximum extent of a predicted or determined range of the user's mouth movement).

[0309] Figure 10B shows the poop avatar as having three display states (1012A, 1012B, and 1012C), wherein each of the three display states of the poop avatar corresponds to three detected states (1002A, 1002B, and 1002C) of the user. Specifically, in Figure 10BIn each display state, the electronic device locates or modifies the features of the poop avatar in response to detecting the location or change in location of one or more physical features of the user (such as facial features) detected in the corresponding state of the user. In Figure 10B the illustrated embodiment, the detected user facial feature includes the user's mouth 1020. In some embodiments, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, and the like. In Figure 10B the embodiment, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, an upper portion 1034 of the avatar, and a lower portion 1036 of the avatar.

[0310] As shown in 1002A, the electronic device detects a first state of the user in which the user's mouth 1020 is slightly open (e.g., 10% of the maximum range of the predicted or determined range of motion of the user's mouth opening). In response to detecting the slightly open user mouth 1020, the electronic device modifies the poop avatar as shown in 1012A such that the mouth 1030 of the poop avatar has an open position (20% of the maximum range of the simulated range of motion of the avatar's mouth, which is greater than (e.g., within its corresponding range of motion) the opening degree of the user's mouth 1020), while keeping other features of the avatar (such as the eyes 1032, upper portion 1034, and lower portion 1036 of the poop avatar) unchanged.

[0311] As shown in 1002B, the electronic device detects a change in the user's facial feature in which the user's mouth 1020 is opened wider than in state 1002A (e.g., 25% of the maximum range of the predicted or determined range of motion of the user's mouth opening). In response to detecting the user's mouth 1020 transitioning from the slightly open position in 1002A to the wider open position in 1002B, the electronic device modifies the mouth 1030 of the poop avatar to increase its size as shown in 1012B such that the mouth 1030 has an open position that is larger (e.g., 50% of the maximum range of the simulated range of motion of the avatar's mouth) than the position shown in 1012A, while still maintaining the positioning of the other avatar features (including the eyes 1032, upper portion 1034, and lower portion 1036 of the poop avatar).

[0312] As shown in 1002C, the electronic device detects yet another change in the user's facial features, in which the user's mouth 1020 is even wider open than in 1002B (e.g., 50% of the maximum range of the predicted or determined range of motion of the user's open mouth). In response to detecting the conversion of the user's mouth 1020 from the open position in 1002B to the wider open position in 1002C, the electronic device modifies the mouth 1030 of the poop avatar to further increase its size, as shown in 1012C, such that the mouth 1030 has an open position that is larger (e.g., 100% of the maximum range of the simulated range of motion of the avatar's mouth) than the position shown in 1012B. However, in 1012C, the opening of the mouth 1030 is greater than the height of the lower portion 1036. Thus, the electronic device expands the lower portion 1036 of the poop avatar 1000 at 1042 in response to the user's open mouth 1020 to maintain the structural integrity of the poop avatar 1000. In other words, to maintain the consistent positioning of the avatar and its features in response to the user's open mouth 1020, the device modifies one or more interconnected portions of the virtual avatar (e.g., the lower portion 1036). For example, as shown in 1012C, the electronic device expands the lower portion 1036 of the poop avatar adjacent to the avatar's mouth 1030 at region 1042 to accommodate the increased size of the enlarged avatar's mouth 1030. If the electronic device does not modify the lower portion 1036 in this manner, the enlarged mouth 1030 will extend beyond the structure of the virtual avatar, potentially hindering the context and / or tone that the user intends to convey using the virtual avatar.

[0313] In some embodiments, the increase in the size of the avatar's mouth 1030 (e.g., from the position shown in 1012A to the position shown in 1012B, or from the position shown in 1012B to the position shown in 1012C) is not proportional to the increase in the size of the user's mouth 1020 (e.g., from the position shown in 1002A to the position shown in 1002B, or from the position shown in 1002B to the position shown in 1002C), but is scaled to provide an exaggerated rate of size change. For example, in some embodiments, the scaling factor is a doubling factor of 2, such that the relative open position of the avatar's mouth is twice that of the user's mouth. For example, if the user's mouth is open 10% of the maximum range of motion of the user's mouth, the electronic device displays the avatar's mouth open 20% of the maximum range of the simulated range of motion of the avatar's mouth.

[0314] Figure 10CAn exemplary implementation is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics. The modification of the poop avatar includes rotating the upper part of the poop avatar forward and backward (e.g., tilting) while keeping the lower part of the poop avatar stationary. The electronic device displays the poop avatar as having three display states (1013A, 1013B, and 1013C), where each of the three display states of the poop avatar corresponds to three detection states (1003A, 1003B, and 1003C) of the user, respectively. Specifically, in Figure 10C each display state, the electronic device locates or modifies the features of the poop avatar in response to detecting the position or a change in the position of one or more physical characteristics (such as facial features) of the user detected in the corresponding state of the user. In Figure 10C the implementation shown, the user's physical characteristics include the user's face 1024, chin 1026, and head 1028. In some implementations, the physical characteristics to be tracked may include other facial features such as eyelids, irises, muscles, muscle groups, etc. In Figure 10C the implementation, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, the upper part 1034 of the avatar (including the pointed end 1040 of the poop avatar), and the lower part 1036 of the avatar.

[0315] As shown in 1003A, the electronic device detects the user's face 1024 in a position that is rotated in the upward direction. For example, the user raises their chin 1026 and tilts their head 1028 backward (e.g., away from the camera (e.g., camera 143, optical sensor 164) in the field of view plane that is tilted) to position their face 1024 in the upward direction. In response to detecting the upward position of the user's face 1024, the electronic device modifies the poop emoji to a looking-up state by tilting the upper part 1034 of the poop emoji in a direction away from the display (e.g., backward from the camera (e.g., camera 143, optical sensor 164) focal plane), as shown in 1013A. In this backward-tilted position, the electronic device displays the tip 1040 of the poop emoji that is positioned toward the rear side of the poop emoji 1000 to show the offset position of the tip 1040 when the poop emoji is modified to the looking-up state. Additionally, the electronic device modifies the eyes 1032 of the poop emoji to the looking-up state (e.g., by offsetting the pupil or iris 1032A of the poop emoji toward the top of the emoji eyes 1032), as shown in 1013A. When the electronic device tilts the upper part 1034 of the poop emoji and modifies the eyes 1032 to the looking-up state, the electronic device keeps other features of the poop emoji unchanged. For example, the electronic device keeps the position of the emoji mouth 1030 and fixes the position of the lower part 1036 of the poop emoji, such that the electronic device shows the upper part 1034 rotating backward about an axis (e.g., the x-axis) that extends along the width of the lower part 1036.

[0316] As shown in 1003B, the electronic device detects the user's face 1024 in a position rotated in the downward direction. For example, the user lowers (retracts) their chin 1026 and tilts their head 1028 forward (e.g., toward the focal plane of the camera (e.g., camera 143, optical sensor 164)) to position their face 1024 in the downward direction. In response to detecting the downward position of the user's face 1024, the electronic device modifies the poop emoji to a looking - down state by tilting the upper part 1034 of the poop emoji in a direction toward the display (e.g., forward toward the focal plane of the camera (e.g., camera 143, optical sensor 164)), as shown in 1013B. In this forward - tilted position, the electronic device displays the tip 1040 of the poop emoji positioned toward the front side of the poop emoji to show the offset position of the tip 1040 when the poop emoji is modified to the looking - down state. Additionally, the electronic device modifies the eyes 1032 of the poop emoji to the looking - down state (e.g., by offsetting the pupil or iris 1032A of the poop emoji toward the bottom of the emoji eyes 1032), as shown in 1013B. When the electronic device tilts the upper part 1034 of the poop emoji and modifies the eyes 1032 to the looking - down state, the electronic device keeps other features of the poop emoji unchanged. For example, the electronic device keeps the position of the emoji mouth 1030 and fixes the position of the lower part 1036 of the poop emoji, such that the electronic device shows the upper part 1034 rotating forward about an axis (e.g., the x - axis) that extends along the width of the lower part 1036.

[0317] As shown in 1003C, the electronic device detects that the user's position (specifically, the positions of the user's face 1024 and head 1028) is in the downward - rotated position shown in 1003B, but the user's position is also downward - offset from the position shown in 1003B within the field of view of the camera (e.g., camera 143, optical sensor 164). In response, the electronic device displays the poop emoji 1000 in the forward - tilted position shown in 1013B, but the poop emoji is also downward - offset within the display area in 1013C to mirror the downward direction of the user's offset within the camera's field of view.

[0318] Figure 10D An exemplary embodiment is shown of an electronic device modifying a poop emoji in response to detecting a change in a user's physical characteristic, where the modification of the poop emoji includes rotating the upper part of the poop emoji while keeping the lower part of the poop emoji stationary. The poop emoji is shown as having four display states (1014A, 1014B, 1014C, and 1014D), where each of the four display states of the poop emoji corresponds to a respective one of four detected states of the user (1004A, 1004B, 1004C, and 1004D). In Figure 10DIn each display state, the electronic device locates or modifies the features of the poop avatar in response to detecting the location or change in location of one or more physical features of the user (such as facial features) detected in the corresponding state of the user. In Figure 10D the embodiment shown in, the physical features of the user include the user's mouth 1020, face 1024, head 1028, and shoulders 1021. In some embodiments, the physical features tracked may include other facial features such as eyelids, irises, muscles, muscle groups, and the like. In Figure 10D the embodiment of, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, an upper portion 1034 of the avatar (including the pointed end portion 1040 of the poop avatar), and a lower portion 1036 of the avatar.

[0319] As shown in 1004A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the right side of the user while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the right (while keeping the lower portion 1036 stationary), as shown in 1014A, such that the twisting motion of the virtual avatar mirrors the rightward rotational movement of the user's head 1028 and face 1024. The electronic device also detects a smiling pose of the user's mouth 1020 and modifies the avatar mouth 1030 to a smiling state. As shown in 1014A, the electronic device twists the upper portion 1034 of the poop avatar about an axis 1051 (e.g., the y-axis) that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the right, the electronic device also offsets the pointed end portion 1040 of the poop avatar to the left, moves the avatar eyes 1032 to the right, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, thereby giving the poop avatar a distorted appearance that includes a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnect relationship between the upper portion 1034 and the lower portion 1036. Even though parts of the avatar (e.g., the pointed end portion 1040 of the poop avatar) may not necessarily correspond anatomically to the physical features of the user, these modifications to the poop avatar provide an animated effect that mimics the physical movement of the user.

[0320] 1004B and 1014B show a similar effect, where the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the left side of the user, while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the left (while keeping the lower portion 1036 stationary), as shown in 1014B, such that the twisting motion of the virtual avatar mirrors the leftward rotational movement of the user's head 1028 and face 1024. As shown in 1014B, the electronic device twists the upper portion 1034 of the poop avatar about an axis 1051 that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the left, the electronic device also offsets the tip portion 1040 of the poop avatar to the right, moves the eyes 1032 of the poop avatar to the left, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, thereby giving the poop avatar a distorted appearance that includes a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnect relationship between the upper portion 1034 and the lower portion 1036.

[0321] In some embodiments, the electronic device does not track the movement (e.g., rotational movement) or positioning of the user's shoulders 1021, such that the user can affect the change of the virtual avatar without having to maintain a fixed orientation or position in front of the camera (e.g., camera 143, optical sensor 164). For example, as shown in 1004C, the user's shoulders 1021 are tilted or rotated to the right side of the user, but the lower portion 1036 of the poop avatar remains fixed, as shown in 1014C. However, the electronic device detects that the user's head 1028 and optionally the user's face 1024 are rotated to the right side of the user. Accordingly, as shown in 1014C, the electronic device modifies the poop avatar accordingly as discussed above with respect to 1014A, without further modifying the poop avatar in response to the user rotating their shoulders 1021. Similar effects are shown in 1004D and 1014D, where the user's shoulders 1021 are tilted or rotated to the left side of the user together with the user's head 1028, and the electronic device modifies the poop avatar as discussed above with respect to 1014B, without further modifying the poop avatar (e.g., the lower portion 1036) in response to the user rotating their shoulders 1021.

[0322] Figure 10EAn exemplary implementation is shown where an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics. The modification of the poop avatar includes tilting the upper part of the poop avatar while keeping the lower part of the poop avatar stationary. The poop avatar is shown as having four display states (1015A, 1015B, 1015C, and 1015D), where each of the four display states of the poop avatar corresponds to four detection states (1005A, 1005B, 1005C, and 1005D) of the user respectively. In each display state, the electronic device locates or modifies the features of the poop avatar in response to detecting the position or change in position of one or more physical characteristics (such as facial features) of the user detected in the corresponding state of the user. In Figure 10E the implementation shown, the user's physical characteristics include the user's mouth 1020, face 1024, head 1028, shoulders 1021, eyes 1023, and neck 1025. In some implementations, the physical characteristics tracked may include other facial features such as eyelids, irises, muscles, muscle groups, etc. In Figure 10E the implementation, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, the upper part 1034 of the avatar (including the pointed end 1040 and the middle part 1031 of the poop avatar), and the lower part 1036 of the avatar.

[0323] As shown in 1005A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical characteristics including the face 1024) tilt to the user's right side, while the user's shoulders 1021 remain positioned forward. The electronic device also detects that the user's neck 1025 tilts slightly to the user's right. In response, the electronic device modifies the poop avatar by tilting the upper part 1034 of the poop avatar (the upper part 1034 of the poop avatar includes the pointed end 1040 and the middle part 1031) to the right while keeping the lower part 1036 stationary, as shown in 1015A, such that the tilting motion of the virtual avatar mirrors the rightward tilting of the user's head 1028 (and / or face 1024) and neck 1025.

[0324] Except for the mirrored tilt direction, the electronic device also modifies the virtual avatar to account for the different degrees of tilt present in the various physical characteristics of the user shown in 1005A. For example, the upper part of the user (e.g., the user's head 1028) is tilted to a greater degree than the lower part of the user (e.g., the user's neck 1025). Thus, as shown in 1015A, the electronic device modifies the virtual avatar such that the amount of movement or tilt at the top of the virtual avatar (e.g., at the tip 1040) is the greatest and the amount of movement or tilt at the bottom of the virtual avatar (e.g., at the lower part 1036) is the smallest. In other words, the amount of tilt gradually decreases from the top to the bottom of the virtual avatar, which is consistent with the change in the degree of tilt shown in the user in 1005A. This is demonstrated by the tip 1040 of the poop avatar in 1015A having a large degree of tilt, the middle part 1031 having a smaller degree of tilt than the tip 1040, and the non-tilted lower part 1036. Even though a part of the avatar (e.g., the tip 1040 of the poop avatar) may not necessarily correspond anatomically to the physical characteristics of the user, these modifications to the poop avatar provide an animated effect that mimics the physical movement of the user. Additionally, for example when the user tilts their head 1028 and neck 1025, the electronic device modifies the virtual avatar to have different degrees of tilt to mimic the reduced range of motion of the user's physical characteristics.

[0325] 1005B and 1015B show a similar effect, where the electronic device detects that the user's head 1028 and neck 1025 are tilted to the left of the user. In response, the electronic device modifies the poop avatar by tilting the upper part 1034 of the poop avatar to the left at different degrees of tilt (e.g., the tip 1040 is tilted to a greater degree than the middle part 1031), while keeping the lower part 1036 stationary, as shown in 1015B, such that the tilt movement of the virtual avatar mirrors the leftward tilt of the user's head 1028 and neck 1025, as discussed in more detail with respect to 1005A and 1015A.

[0326] As shown in 1005C, the electronic device detects a user's eye 1023 (e.g., the iris or pupil of the user's eye) that has shifted to the right of the user, and a smiling facial expression formed by the corners 1020A and 1020B of the user's mouth 1020 positioned in an upward pose. In response to detecting the rightward shift of the user's eye 1023, the electronic device modifies the eyes 1032 of the poop avatar to a looking-right state (e.g., by shifting the pupil or iris 1032A of the poop avatar toward the right side of the avatar's eyes 1032), as shown in 1015C. In response to detecting one or both of the corners 1020A and 1020B of the user's mouth positioned in an upward pose, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, where the avatar's mouth 1030 has an open smiling position, as shown in 1015C. As shown in 1015C, the upper part 1034 and the lower part 1036 remain unchanged relative to their respective neutral positions (shown in 1011A).

[0327] 1005D and 1015D show a similar effect, where the electronic device detects a smiling facial expression of the user and a user's eye 1023 (e.g., the iris or pupil of the user's eye) that has shifted to the left of the user. In response to detecting the leftward shift of the user's eye 1023, the electronic device modifies the eyes 1032 of the poop avatar to a looking-left state (e.g., by shifting the pupil or iris 1032A of the poop avatar toward the left side of the avatar's eyes 1032), as shown in 1015D. In response to detecting one or both of the corners 1020A and 1020B in an upward pose, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, as discussed above with respect to 1015C. Similarly, in 1015D, the upper part 1034 and the lower part 1036 remain unchanged relative to their respective neutral positions (shown in 1011A and 1015C).

[0328] Figure 10F An exemplary embodiment is shown in which the electronic device modifies the poop avatar in response to detecting an offset in the position of the user within the field of view of a camera (e.g., camera 143, optical sensor 164). The modification of the poop avatar includes shifting (e.g., translating) the poop avatar in a direction corresponding to the offset in the position of the user within the field of view of the camera. The poop avatar is shown as having four display states (1016A, 1016B, 1016C, and 1016D) in four display regions, where each of the four display states of the poop avatar corresponds to a respective four detection states of the user (1006A, 1006B, 1006C, and 1006D). In Figure 10FIn each of the display states, the electronic device locates or modifies the poop avatar in response to detecting the position or a change in the position of the user detected within the camera's field of view while the user is in the corresponding state. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1016A, 1016B, 1016C, and 1016D) represent the boundaries of the display area that includes the virtual avatar.

[0329] As shown in 1006A, the electronic device detects that the position of the user is horizontally centered within the field of view of the camera (e.g., camera 143, optical sensor 164). In response to detecting the horizontally centered position of the user within the camera's field of view, the electronic device displays a poop avatar with a horizontally centered position within the display area as shown in 1016A.

[0330] In 1006B, the electronic device detects that the position of the user is off - center (e.g., offset or translated) in the right - hand direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is offset in the user's left - hand direction (e.g., offset to the right relative to the camera's field of view), but remains fully visible within the camera's field of view. In response to detecting the offset position of the user in 1006B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) such that the poop avatar is displayed at an offset - to - the - left position as shown in 1016B, so as to mirror the user's directional offset to the user's left - hand direction. As shown in 1006B, the user is offset to their left, bringing their left shoulder 1021A close to the right edge of the field of view. Accordingly, the electronic device displays a poop avatar near the left edge of the display area of 1016B that mirrors the direction of the user's offset position within the camera's field of view. In some embodiments, the user's offset position is mirrored by an offset of the virtual avatar in both direction and magnitude. In some embodiments, the user's offset position is mirrored by an offset of the virtual avatar only in direction, and the magnitude of the offset of the virtual avatar is adjusted (e.g., attenuated) to keep the position of the virtual avatar within the boundaries of the display area. Examples of such embodiments are discussed below with respect to 1006C, 1016C, 1006D, and 1016D.

[0331] In 1006C, the electronic device detects that the user's position is off-center (e.g., offset or translated) in the rightmost direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is far off to the user's left direction (e.g., offset to the right relative to the field of view of the camera), but offset such that the user's left shoulder 1021A is no longer within the camera's field of view. In response to detecting the user's significantly offset position in 1006C, the electronic device offsets the horizontal position of the poop avatar such that the entire poop avatar (including both the upper part 1034 and the lower part 1036) is shown in a fully left-offset position, where the outermost edge of the poop avatar (e.g., edge 1036A of the lower part 1036) is positioned against the left boundary of the display area shown in 1016C. The offset display of the poop avatar in 1016C mirror-reflects the direction of the user's offset to the user's left direction, rather than offsetting the poop avatar such that a part of the avatar extends beyond the display area in 1016C (as the user did in 1006C), and the device positions the poop avatar at the edge of the display area 1016C. By keeping the position of the virtual avatar within the display area (e.g., 1016A, 1016B, 1016C, and 1016D) even when a part of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect the change of the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0332] Similar effects are shown in 1006D and 1016D. In 1006D, the electronic device detects that the user is offset to the right (e.g., offset to the left within the field of view of the camera), such that the user's right shoulder is no longer within the camera's field of view. In response to detecting the user's far-offset position, the electronic device offsets the horizontal position of the poop avatar such that the entire poop avatar (including both the upper part 1034 and the lower part 1036) is shown in a fully right-offset position, where the outermost edge of the poop avatar (e.g., edge 1036B of the lower part 1036) is positioned against the right boundary of the display area shown in 1016D. As described above, by keeping the position of the virtual avatar within the display area 1016D even when a part of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect the change of the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0333] Figure 10G An exemplary implementation is shown in which the electronic device modifies the poop avatar in response to detecting an offset in the position of the user's physical characteristics within the field of view of the camera (e.g., camera 143, optical sensor 164). The modification of the poop avatar includes offsetting (e.g., translating) the poop avatar in a direction corresponding to the offset in the position of the user's physical characteristics within the camera's field of view.Figure 10G The embodiments shown herein are similar to those discussed above with respect to Figures 10D to 10F in that: the electronic device tracks the movement and positioning of the user's head 1028 (e.g., rotational movement and / or translational movement), but does not track the movement or positioning of the user's shoulders 1021 and optionally the user's neck 1025. Additionally, Figure 10G the embodiments shown are similar to those in Figure 10F in that: the modification of the virtual avatar mirror reflects the user's movement in direction, but not necessarily the movement in magnitude. By implementing these techniques, the electronic device allows the user to affect the change of the virtual avatar without having to maintain a fixed orientation or position in front of the camera (e.g., camera 143, optical sensor 164).

[0334] The poop avatar is shown as having two display states (1017A and 1017B) in two display areas, wherein each of the two display states of the poop avatar corresponds to two detection states (1007A and 1007B) of the user respectively. In Figure 10G each display state, the device locates or modifies the poop avatar in response to detecting the position or change in position of the physical features of the user detected within the camera's field of view in the corresponding state of the user. In each of the two display states, the boundary of the displayed state (e.g., the boundary between 1017A and 1017B) represents the boundary of the display area including the virtual avatar.

[0335] In 1007A, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 deviate (e.g., translate) in the left direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset in the right direction of the user (e.g., offset to the left relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007A, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper part 1034 and the lower part 1036), such that the poop avatar is displayed at an offset position to the right, as shown in 1017A, so as to mirror the offset of the user's head in the right direction of the user.

[0336] Similar effects are shown in 1007B and 1017B. In 1007B, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset (e.g., translated) in the rightward direction within the field of view of a camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset in the leftward direction of the user (e.g., offset to the right relative to the field of view of the camera). In response to detecting the offset positions of the user's head 1028 and optionally the user's neck 1025 in 1007B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) such that the poop avatar is displayed at an offset position to the left, as shown in 1017B, in order to mirror the offset of the user's head in the leftward direction of the user.

[0337] Figure 10H Exemplary embodiments are shown in which the electronic device modifies the poop avatar in response to detecting a change in the position of the user's physical characteristics within the field of view of a camera (e.g., camera 143, optical sensor 164). Modifications to the poop avatar include increasing or decreasing the size of the poop avatar and offsetting (e.g., translating) the poop avatar in a direction corresponding to the offset of the position of the user's physical characteristics within the field of view of the camera. The poop avatar is shown as having four display states (1018A, 1018B, 1018C, and 1018D) in four display regions, where each of the four display states of the poop avatar corresponds to a respective one of four detection states (1008A, 1008B, 1008C, and 1008D) of the user. In Figure 10H each of the display states, the electronic device locates or modifies the poop avatar in response to detecting the position or a change in the position of the user's physical characteristics detected within the field of view of the camera in the respective state of the user. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1018A, 1018B, 1018C, and 1018D) represent the boundaries of the display region including the virtual avatar.

[0338] In 1008A, the electronic device detects that the user's head 1028 is offset (e.g., translated) in the upward direction relative to its shoulders 1021 within the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is stretching their neck 1025 upward). In response to detecting the upward offset position of the user's head 1028 in 1008A, the electronic device offsets the vertical position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) such that the entire poop avatar is displayed as being in the upward offset position, where the uppermost edge of the poop avatar (e.g., the edge 1040A of the tip portion 1040) is positioned close to the upper boundary of the display area shown in 1018A, so as to mirror the offset of the user's head 1028 in the upward direction shown in 1008A.

[0339] In 1008B, the electronic device detects that the user's head 1028 is offset (e.g., translated) in the downward direction relative to the user's shoulders 1021 within the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is lowering their head 1028). In response to detecting the downward offset position of the user's head 1028 in 1008B, the electronic device offsets the vertical position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) such that the entire poop avatar is displayed as being in the downward offset position, where the lowermost edge of the poop avatar (e.g., the edge 1036C of the lower part 1036) is positioned close to the lower boundary of the display area shown in 1018B, so as to mirror the offset of the user's head 1028 in the downward direction shown in 1008B.

[0340] In 1008C, the electronic device detects that the size of the user's head 1028 increases within the field of view of the camera (e.g., camera 143, optical sensor 164), such as when the user's head 1028 is positioned closer to the camera. In response to detecting the increase in the size of the user's head 1028 in 1008C, the electronic device increases the size of the entire poop avatar. In some embodiments, the electronic device increases the size of the poop avatar based on the detected change in the size of the user's head 1028 from one detected state (e.g., Figure 10A the neutral state in 1001A) to another state (e.g., the detected state 1008C). In 1018C, the electronic device increases the size of the poop avatar to fill the display area without causing a part of the poop avatar to extend beyond the boundaries of the display area. In some embodiments, the electronic device increases the size of the virtual avatar to give the impression that the avatar is positioned very close to the display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504).

[0341] For example, the electronic device increases the size of the poop avatar in 1018C such that the tip portion 1040 of the poop avatar is adjacent to the upper boundary of the display area at 1040A, the lower portion 1036 of the poop avatar is adjacent to the lower boundary of the display area at 1036C, the left edge of the lower portion 1036 is close to the left boundary of the display area at 1036A, and the right edge of the lower portion 1036 is close to the right boundary of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018C, the electronic device proportionally increases the size of the poop avatar such that the relative positions of various avatar features (e.g., the eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036 of the avatar) are not distorted relative to the shape of the poop avatar. For example, when the electronic device increases the size of the poop avatar in 1018C, the sizes of the avatar eyes 1032, mouth 1030, upper portion 1034 (including the tip portion 1040), and lower portion 1036 also increase, but other parts remain unchanged.

[0342] In 1008D, the electronic device detects that, for example, when the user's head 1028 is positioned away from the camera, the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) decreases. In response to detecting the decrease in the size of the user's head 1028 in 1008D, the electronic device decreases the size of the entire poop avatar. In some embodiments, the electronic device decreases the size of the poop avatar based on the detected change in the size of the user's head 1028 from one detected state (e.g., Figure 10A the neutral state in 1001A) to another state (e.g., the detected state 1008D). In some embodiments, such as in 1018D, the electronic device decreases the size of the poop avatar to give the impression that the virtual avatar is positioned away from the display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504).

[0343] For example, the electronic device reduces the size of the poop avatar in 1018D such that the tip portion 1040 of the poop avatar is positioned away from the upper boundary of the display area at 1040A, the lower portion 1036 of the poop avatar is positioned away from the lower boundary of the display area at 1036C, the left edge of the lower portion 1036 is positioned away from the left boundary of the display area at 1036A, and the right edge of the lower portion 1036 is positioned away from the right boundary of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018D, the electronic device proportionally reduces the size of the poop avatar such that the relative positions of various avatar features (e.g., the eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036 of the avatar) are not distorted relative to the shape of the poop avatar. For example, when the electronic device reduces the size of the poop avatar in 1018D, the sizes of the avatar eyes 1032, mouth 1030, upper portion 1034 (including the tip portion 1040), and lower portion 1036 are also reduced, but other parts remain unchanged.

[0344] Figure 10I An exemplary embodiment is shown that illustrates the electronic device modifying the poop avatar 1000 in response to detecting a change in a user's physical characteristic, such as a facial characteristic. The poop avatar is shown as having one display state 1019 corresponding to the detected state 1009 of the user. As shown in 1009 and 1019, the electronic device detects that the user makes a pouting expression, and in response, the electronic device modifies the poop avatar 1000 by replacing the mouth 1030 of the poop avatar with a set of puckered lips 1050. In some embodiments, such as the embodiment shown in 1009, the electronic device determines that the user is making a pouting facial expression by detecting that the user's jaw 1027 is in a closed position and detecting that the corners 1020A and 1020B of the user's mouth 1020 move towards each other such that the user's lips 1029 (e.g., the user's upper lip 1029A and lower lip 1029B) extend outward from the user's mouth 1020 in a pouting posture. Although not shown in Figure 10I In some embodiments, the electronic device modifies the poop avatar so as to emit a heart from the puckered lips 1050 in a manner similar to that shown in Figure 16B and discussed in more detail below.

[0345] Figure 11A An exemplary embodiment is shown that illustrates the electronic device modifying the bear avatar 1100 in response to detecting a change in the user's facial characteristic. The bear avatar is shown as having four display states (1111A, 1111B, 1111C, and 1111D), wherein each of the four display states of the bear avatar corresponds to a respective one of four detected states (1101A, 1101B, 1101C, and 1101D) of the user. In Figure 11AIn each display state, the electronic device locates or modifies the features of the bear avatar in response to detecting the position or change in position of one or more physical features of the user (such as facial features) detected in the corresponding state of the user. In Figure 11A the illustrated embodiment, the detected facial features of the user include the user's mouth 1120 (having corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, and the like. In Figure 11A the embodiment, the features of the bear avatar include an avatar mouth 1130, avatar eyes 1132, avatar ears 1133, avatar nose 1137, avatar head 1135, and (in certain states) avatar eyebrows 1138.

[0346] As shown in 1101A, the electronic device detects a neutral facial expression of the user. For example, the device detects that the user's mouth 1120 and eyebrows 1122 are positioned in a relaxed neutral state rather than in positions associated with a particular facial expression (such as smiling or frowning). In response to detecting the neutral facial expression of the user, the electronic device displays in 1111A a bear avatar 1100 having a neutral expression or state. Specifically, the electronic device displays a bear avatar having an avatar mouth 1130 in a relaxed neutral state rather than in a position typically associated with a particular facial expression (e.g., smiling or frowning). In some embodiments, the neutral position of the avatar corresponds to other representations of the associated image, such as a static bear emoji that may be found in a messaging application. In 1111A, the neutral state of the bear's mouth 1130 is indicated by lines 1130A and 1130B that extend horizontally from the outer sides of the bear's mouth 1130 or "muzzle" region and then curve slightly upward at the base of the bear's nose 1137. The electronic device also displays a bear avatar having a nose 1137 and ears 1133, where the nose is located above the relaxed mouth 1130 and the ears are in a relaxed neutral state, positioned along the sides of the bear's head 1135 and not curled or stretched. Additionally, the electronic device displays a bear avatar without eyebrows 1138 and displays avatar eyes 1132 looking in a direction perpendicular to the focal plane of the display (or the camera (e.g., camera 143, optical sensor 164)).

[0347] As shown in 1101B and 1111B, the electronic device detects the position of the user's mouth 1120 (e.g., detects that one or both of the corners 1120A and 1120B of the user's mouth are positioned in an upward posture (e.g., upward position) to form a smiling facial expression), and in response, the electronic device modifies two features of the bear avatar. For example, in response to detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture, the electronic device modifies both the bear ears 1133 and the bear mouth 1130. The electronic device modifies the bear's mouth 1130 to have a smiling expression by turning lines 1130A and 1130 upward, and the upward turning of lines 1130A and 1130 indicates the smiling expression of the bear mouth 1130, as shown in 1111B. The electronic device modifies the bear's ears 1133 to "stand up" or extend in the upward direction. In some embodiments, such as the embodiment shown in 1111B, the modification of the bear ears 1133 also includes slightly narrowing the width of each ear 1133 and changing the vertical position of the ears 1133 on the side of the bear head 1135 such that when the ears stand up, the ears 1133 are positioned at a higher position on the bear head 1135 (compared to the vertical position of the ears 1133 on the side of the bear head 1135 in the neutral state in 1111A). As shown in 1111B, the eyes 1132 and the nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user smiles, the electronic device changes the bear's mouth 1130 to a smiling state and makes the ears 1133 stand up. When the user stops smiling, the electronic device returns the mouth 1130 and the ears 1133 to their neutral positions.

[0348] As shown in 1101C, the electronic device detects that the user's eyebrows 1122 are in a raised position (e.g., at a higher position on the user's face compared to the position of the eyebrows 1122 in the relaxed neutral state shown in 1101A and 1101B). In response to detecting the raised eyebrows 1122, the electronic device modifies two features of the bear avatar. For example, the electronic device modifies the bear ears 1133 to stand up and modifies the bear avatar to introduce eyebrows 1138 located above the bear eyes 1132, thereby conveying the impression that the bear avatar 1100 is raising its eyebrows 1138, as shown in 1111C. As shown in 1111B, the bear mouth 1130 returns to its neutral position, and the eyes 1132 and the nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's eyebrows 1122 are raised, the electronic device introduces the eyebrows 1138 of the avatar and makes the ears 1133 stand up. Thus, when the user's eyebrows 1122 return to their neutral position, the electronic device removes the avatar eyebrows 1138 and relaxes the ears 1133.

[0349] In 1101D, the electronic device detects that the user's eyebrows 1122 are raised and one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture. In response, the electronic device modifies the bear avatar 1100 to achieve ears that are extremely upright, introduces raised eyebrows 1138, and makes it smile, as shown in 1111D. The electronic device modifies the ears 1133 to achieve extremely upright ears by significantly reducing the width of each ear 1133, extending the ears in the upward direction, and changing the vertical position of the ears 1133 on the sides of the bear's head 1135 such that the ears 1133 are positioned on the bear's head 1135 at a position even higher than the position when in the upright positions shown in 1111B and 1111C. This combination of modifications to the bear ears 1133 causes the appearance of the ears 1133 to be stretched significantly in the upward direction, to a greater extent than the ear uprightness shown in 1111B and 1111C. It should be understood that in this embodiment, when the user raises their eyebrows 1122 and smiles (e.g., one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture), the electronic device introduces the avatar's eyebrows 1138, modifies the bear's mouth 1130 to a smiling state, and makes the ears 1133 extremely upright. Thus, when the user's eyebrows 1122 and mouth 1120 return to their neutral positions, the electronic device removes the avatar eyebrows 1138 and relaxes the ears 1133 and the mouth 1130.

[0350] Figure 11B An exemplary embodiment is shown that illustrates the electronic device modifying the bear avatar 1100 in response to detecting changes in the user's facial features. The bear avatar is shown as having four display states (1112A, 1112B, 1112C, and 1112D), where each of the four display states of the bear avatar corresponds to a respective one of four detection states (1102A, 1102B, 1102C, and 1102D) of the user. In Figure 11B each display state, the electronic device locates or modifies the features of the bear avatar in response to detecting the position or change in position of one or more physical features (such as facial features) of the user detected in the user's corresponding state. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1112A, 1112B, 1112C, and 1112D) represent the boundaries of the display area that includes the virtual avatar.

[0351] In Figure 11BIn the illustrated embodiments, the detected facial features of the user include the user's mouth 1120 (having corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, etc. In Figure 11B the embodiments of, the features of the bear avatar include an avatar mouth 1130, avatar eyes 1132, avatar ears 1133, avatar nose 1137, avatar head 1135, and (in certain states) avatar eyebrows 1138.

[0352] As shown in 1102A, the electronic device detects that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward pose (e.g., downward position), and in response, the electronic device modifies two features of the bear avatar. In some embodiments, the electronic device determines that the user is making a sad facial expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth are in a downward position, and optionally detecting that one or both of the user's eyebrows 1122 are not in a lower position (e.g., the eyebrows 1122 are raised or in their neutral position). In response to detecting the downward pose or position of the corners 1120A and 1120B of the user's mouth 1120, the electronic device modifies both the bear's mouth 1130 and the bear's ears 1133, as shown in 1112A.

[0353] The electronic device modifies the bear's mouth 1130 to have a sad expression by rotating lines 1130A and 1130B downward, and the downward rotation of lines 1130A and 1130B forms the sad expression of the bear's mouth 1130, as shown in 1112A. The electronic device modifies the bear's ears 1133 to "curl" or "flop" in the downward direction by rotating the outer edges 1133A and 1133B of the bear's ears 1133 downward (e.g., folding, rotating, or tilting). In some embodiments, curling the bear's ears 1133 also includes changing the vertical position of the ears 1133 on the sides of the bear's head 1135 such that when the ears are curled, the ears 1133 are positioned at a lower position on the bear's head 1135 (compared to the Figure 11A vertical position of the ears 1133 on the sides of the bear's head 1135 in the neutral state of 1111A). As shown in 1112A, the bear's eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's mouth 1120 forms a sad facial expression, the electronic device changes the bear's mouth 1130 to a sad expression and curls the bear's ears 1133. When the user no longer makes a sad facial expression, the electronic device returns the bear's mouth 1130 and ears 1133 to their neutral positions.

[0354] In 1102B and 1112B, the electronic device detects that the user holds a sad facial expression fo...

Claims

1. A method, comprising: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of the one or more cameras and changes in a second physical feature of the face within the field of view of the one or more cameras, and a second avatar feature; when displaying the virtual avatar via the display device, detecting changes in one or more physical features of the face within the field of view of the one or more cameras; based on determining that the changes include a change in the first physical feature: modifying the first avatar feature of the virtual avatar based on the change in the first physical feature, wherein modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes modifying the first avatar feature based on the magnitude of the change in the first physical feature, and wherein the magnitude of the change in the first physical feature includes the degree of change in the position of the first physical feature, and refraining from modifying the second avatar feature based on the change in the first physical feature; and based on determining that the changes include a change in the second physical feature: modifying the first avatar feature based on the change in the second physical feature, wherein modifying the first avatar feature based on the change in the second physical feature includes modifying the first avatar feature based on the magnitude of the change in the second physical feature, and wherein the magnitude of the change in the second physical feature includes the degree of change in the position of the second physical feature, and refraining from modifying the second avatar feature based on the change in the second physical feature.

2. The method according to claim 1, wherein the second avatar feature responds to changes in a third physical feature different from the first physical feature and the second physical feature, and the method further comprises: based on determining that the changes in the one or more physical features of the face include a change in the third physical feature of the face: modifying the second avatar feature based on the change in the third physical feature.

3. The method according to claim 2, wherein the first avatar feature is anatomically different from at least one of the first physical feature and the second physical feature, and wherein the second avatar feature corresponds anatomically to the third physical feature.

4. The method according to claim 2, wherein: modifying the second avatar feature based on the change in the third physical feature includes modifying the second avatar feature based on the magnitude of the change in the third physical feature.

5. The method according to any one of claims 1 to 4, wherein: Modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes modifying at least a part of the pose of the first avatar feature based on the direction of the pose change of the first physical feature; and modifying the first avatar feature of the virtual avatar based on the change in the second physical feature includes modifying at least a part of the pose of the first avatar feature based on the direction of the pose change of the second physical feature.

6. The method according to any one of claims 2 to 4, wherein: Modifying the second avatar feature of the virtual avatar based on the change in the second physical feature includes modifying at least a part of the pose of the second avatar feature based on the direction of the pose change of the third physical feature.

7. The method according to any one of claims 1 to 4, wherein the change in the first physical feature includes at least a vertical displacement of the first physical feature of the face, and modifying the first avatar feature based on the change in the first physical feature includes moving the first avatar feature in a direction including at least one of a vertical displacement of at least a part of the first avatar feature and a horizontal displacement of at least a part of the first avatar feature.

8. The method according to any one of claims 1 to 4, wherein the change in the first physical feature includes at least one displacement of the first physical feature of the face, and modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes rotating at least a part of the first avatar feature.

9. The method according to any one of claims 1 to 4, wherein: the change in the one or more physical features of the face includes a change in a first magnitude of the first physical feature and a change in a second magnitude of the second physical feature, and modifying the first avatar feature of the virtual avatar includes modifying the first avatar feature by a certain modification magnitude based on both the first magnitude and the second magnitude.

10. The method according to claim 8, wherein: the first avatar feature includes avatar ears; the first physical feature includes at least one corner region of the user's mouth; the change in the first physical feature includes at least one displacement of the at least one corner region of the user's mouth; and modifying the first avatar feature based on the change in the first physical feature includes rotating at least a part of the avatar ears based on the magnitude of the displacement of the at least one corner region of the user's mouth.

11. The method according to any one of claims 1 to 4, wherein: the first avatar feature includes avatar ears; the first physical feature includes at least a part of the user's eyebrows; the change in the one or more physical features of the face includes a vertical offset in the position of the user's eyebrows; and modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes horizontally offsetting the position of the avatar ears.

12. The method according to claim 8, wherein: The second physical feature includes at least a part of the user's eyebrows; The change of the one or more physical features of the face includes a vertical offset of the position of the user's eyebrows; and Modifying the first avatar feature based on the change of the second physical feature includes vertically offsetting the position of the avatar ears.

13. The method according to any one of claims 1 to 4, wherein: The first avatar feature includes avatar eyes; The first physical feature includes at least a part of the user's eyes; The second physical feature includes at least a part of the user's eyebrows; The change of the first physical feature includes at least one of a displacement of the iris part of the user's eyes and a change in the size of at least a part of the user's eyes; The change of the second physical feature at least includes a vertical displacement of at least a part of the user's eyebrows; Modifying the first avatar feature based on the change of the first physical feature includes translating at least a part of the avatar eyes when the change of the first physical feature includes the displacement of the iris part of the user's eyes; Modifying the first avatar feature based on the change of the first physical feature includes adjusting the size of at least a part of the avatar eyes when the change of the first physical feature includes a change in the size of at least a part of the user's eyes; and Modifying the first avatar feature based on the change of the second physical feature includes adjusting the rotation degree of at least a part of the avatar eyes, wherein the rotation degree is based on the vertical displacement of at least a part of the user's eyebrows.

14. The method according to claim 2, wherein: The first avatar feature includes an avatar mouth; The second avatar feature includes at least a part of the avatar teeth; The first physical feature includes at least a part of the user's lips; The third physical feature includes at least a part of the user's mouth; The change of the first physical feature includes a displacement of at least a part of the user's lips from a first position to a second position; Modifying the first avatar feature of the virtual avatar based on the change of the first physical feature includes modifying the shape of the avatar mouth based on the second position of at least a part of the user's lips; The change of the third physical feature includes opening or closing at least a part of the user's mouth; and wherein modifying the second avatar feature based on the change of the three physical features includes modifying the vertical distance between a first part of the avatar teeth and a second part of the avatar teeth, wherein the magnitude of the vertical distance is based on the magnitude of the opening or closing of at least a part of the user's mouth.

15. The method according to any one of claims 1 to 4, wherein: The first avatar feature includes at least a part of an avatar nose; The first physical feature includes at least a part of the user's eyebrows; The second physical feature includes at least a part of the user's lips; The change in the first physical feature at least includes a vertical displacement of at least a portion of the user's eyebrows; Modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes modifying the position of the avatar nose based on the direction of the vertical displacement of at least a portion of the user's eyebrows, wherein the position of the avatar nose is modified in a direction determined based on the direction of the vertical displacement of at least a portion of the user's eyebrows; The change in the second physical feature includes at least one of a horizontal displacement of at least a portion of the user's lips and a vertical displacement of at least a portion of the user's lips; and Modifying the first avatar feature of the virtual avatar based on the change in the second physical feature includes further modifying the position of the avatar nose based on the direction of the horizontal displacement of at least a portion of the user's lips and based on the direction of the vertical displacement of at least a portion of the user's lips, wherein the position of the avatar nose is further modified in a direction corresponding to the direction of the horizontal displacement of at least a portion of the user's lips and in a direction corresponding to the direction of the vertical displacement of at least a portion of the user's lips.

16. The method according to any one of claims 1 to 4, wherein: The first avatar feature includes an avatar hair feature; The first physical feature includes at least a portion of the user's eyebrows; The second physical feature includes at least a portion of the user's head; The change in the first physical feature at least includes a vertical displacement of at least a portion of the user's eyebrows; The change in the second physical feature includes a change in the posture of the user's head; Modifying the first avatar feature of the virtual avatar based on the change in the first physical feature includes displacing at least a portion of the avatar hair feature based on the direction of the vertical displacement of at least a portion of the user's eyebrows; and Modifying the first avatar feature of the virtual avatar based on the change in the second physical feature includes rotating at least a portion of the avatar hair feature based on the direction or magnitude of the change in the posture of the user's head.

17. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs including instructions for performing the method according to any one of claims 1 to 16.

18. An electronic device, comprising: One or more cameras; A display device; One or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 16.

19. An electronic device, comprising: One or more cameras; A display device; and An apparatus for performing the method according to any one of claims 1 to 16.

20. A computer program product storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs including instructions for performing the method according to any one of claims 1 to 16.

Images