Emoji recording and sending

Abstract

The present disclosure relates to recording and sending emoticons. The present disclosure generally relates to generating and modifying virtual avatars. An electronic device having a camera and a display device displays a virtual avatar that changes appearance in response to changes in a face in the camera's field of view. In response to detecting changes in one or more physical features of the face in the camera's field of view, the electronic device modifies one or more features of the virtual avatar.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of January 23, 2018, the national application number 201880004632.1, and the invention name “Emoji Recording and Sending”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application claims priority to U.S. Provisional Application No. 62 / 507,177, entitled “Emoji Recording and Sending,” filed on May 16, 2017, U.S. Provisional Application No. 62 / 556,412, entitled “Emoji Recording and Sending,” filed on September 9, 2017, and U.S. Provisional Application No. 62 / 557,121, filed on September 11, 2017. This patent application also claims priority to Danish Patent Application No. PA201770393, entitled “Emoji Recording and Sending,” filed on May 29, 2017, Danish Patent Application No. PA201770720, entitled “Emoji Recording and Sending,” filed on September 22, 2017, and Danish Patent Application No. PA201770721, entitled “Emoji Recording and Sending,” filed on September 22, 2017. The contents of these patent applications are hereby incorporated by reference herein in their entirety. Technical Field

[0004] The present disclosure relates generally to computer user interfaces and, more particularly, to techniques for generating, recording, and sending emoticons and avatars. Background Art

[0005] Multimedia content (such as emojis and avatars) is sometimes sent as part of a messaging communication. Emojis and avatars represent a variety of predefined people, objects, actions, and / or other things. Some messaging applications allow users to select from a library of predefined emojis and avatars and send these emojis and avatars as part of a message that may contain other content (e.g., other multimedia and / or text content). Stickers are another type of multimedia content that is sometimes sent with messaging applications. In some respects, stickers are similar to emojis and avatars in that they can represent people, objects, actions, and / or other things. Some sticker and / or messaging applications allow stickers to be associated with previously sent or received messages. Summary of the Invention

[0006] However, some technologies for generating, sending and receiving emoticons and virtual avatars using electronic devices are often cumbersome and inefficient. For example, some prior art technologies use complex and time-consuming user interfaces that may include multiple keystrokes or keystrokes. Some other prior art technologies use complex and time-consuming methods to manipulate and generate emoticons and virtual avatars, including requiring the user to provide a large amount of input to achieve a desired emoticon (e.g., a desired animated emoticon or dynamic emoticon). Prior art technologies require more time than necessary, which results in a waste of user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Thus, the present technology provides electronic devices with faster, more efficient methods and interfaces for sending and receiving emoticons and avatars. Such methods and interfaces optionally supplement or replace other methods for sending and receiving emoticons. Such methods and interfaces reduce the cognitive burden on users and create a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces conserve power and increase the time between battery charges.

[0008] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting the input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time based on determining that the input began on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface. The animated virtual avatar represents a sequence of changes in facial expressions in the camera's field of view over a time period, wherein the time period is determined based on the timing of the input.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface.

[0010] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: based on determining that input begins on the preview of the virtual avatar, generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, wherein the corresponding time is determined based on the timing of the input; and based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, generating an animated virtual avatar, the animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period, wherein the time period is determined based on the timing of the input.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; a camera; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface. The animated virtual avatar represents a sequence of changes in facial expressions in the camera's field of view over a time period, wherein the time period is determined based on the timing of the input.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; a device for performing the following operations: while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; and a device for performing the following operations: in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera's field of view over a time period, wherein the time period is determined based on the timing of the input.

[0013] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of a face in the camera's field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0014] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of a face in the camera's field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0015] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of a face in the camera's field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in a camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of a face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; a device for the following operations: receiving a request to generate an animated virtual avatar based on changing the facial expression of a face in the camera's field of view; a device for the following operations: in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; and a device for the following operations: after recording the facial expression of the face in the camera's field of view, causing a looped version of the animated virtual avatar to be displayed, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0018] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera's field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period based on determining that the input includes activation of a recording enable indication in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0019] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changing facial expressions of a face in the camera's field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0020] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, the virtual avatar changing appearance in response to a change in a face in a field of view of the one or more cameras, wherein the virtual avatar includes: a first part and a second part different from the first part; while displaying the virtual avatar via the display device, detecting a change in facial posture within the field of view of the one or more cameras; in response to detecting the change in facial posture, changing the appearance of the virtual avatar includes: based on determining that the change in facial posture includes a first type of change in facial posture, changing the appearance of the virtual avatar includes moving the first part of the virtual avatar relative to the second part of the virtual avatar based on a magnitude of the first type of change in facial posture; and based on determining that the change in facial posture includes a second type of change in facial posture, changing the appearance of the virtual avatar includes moving the first part of the virtual avatar and the second part of the virtual avatar based on a magnitude of the second type of change in facial posture.

[0021] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face in a field of view of the one or more cameras and a second physical feature of a face in a field of view of the one or more cameras; and a second avatar feature; while displaying the virtual avatar via the display device, detecting changes in one or more physical features of a face in a field of view of the one or more cameras; based on determining that the changes include changes in the first physical feature: modifying the first avatar feature of the virtual avatar based on the changes in the first physical feature, and abandoning modifying the second avatar feature based on the changes in the first physical feature; and based on determining that the changes include changes in the second physical feature: modifying the first avatar feature based on the changes in the second physical feature, and abandoning modifying the second avatar feature based on the changes in the second physical feature;

[0022] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responds to a change in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature, the second avatar feature responds to a change in the first physical feature; and a third avatar feature, the third avatar feature does not primarily respond to a change in the first physical feature; while displaying the virtual avatar, detecting a change in the first physical feature; and in response to detecting the change in the first physical feature: modifying the first avatar feature based on the detected change in the first physical feature; modifying the second avatar feature based on the detected change in the first physical feature; and abandoning modifying the third avatar feature based on the detected change in the first physical feature.

[0023] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responding to a change in a first physical feature of the face within the field of view of the one or more cameras; a second avatar feature, the second avatar feature responding to a change in a second physical feature of the face in a different manner, the manner depending on whether the change in the second physical feature of the face occurs within a first change range of the second physical feature or within a second change range of the second physical feature, the second change range of the second physical feature being different from the first change range of the second physical feature; while displaying the virtual avatar, detecting the one or more cameras a first change in a corresponding physical feature of a face within the field of view; and in response to detecting the first change in the corresponding physical feature, modifying the virtual avatar, which includes: modifying the first avatar feature to reflect the change in the first physical feature based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature; and changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within a first change range; and abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within a second change range.

[0024] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes multiple avatar features that respond to changes in one or more physical features of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting changes in multiple physical features of the face, the multiple physical features of the face including a first physical feature corresponding to one or more of the multiple avatar features and a second physical feature that does not correspond to any of the multiple avatar features; and in response to detecting the changes in the multiple physical features of the face: changing the appearance of corresponding avatar features of the multiple avatar features, wherein the magnitude and / or direction of the change in the corresponding avatar features is based on the magnitude or direction of the change in the first physical feature; and before detecting the change in one or more physical features of the face, deforming a portion of the virtual avatar that does not include the avatar features, wherein the magnitude and / or direction of the deformation of a portion of the avatar features is based on the magnitude and / or direction of the change in the second physical feature.

[0025] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responding to a change in a first physical feature of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting a change in a first physical feature having a first physical feature change value; in response to detecting the change in the first physical feature: changing the first avatar feature by a first avatar feature change value based on the first physical feature change value based on determining that the change in the first physical feature is within a first physical feature value range; and changing the first avatar feature by a second avatar feature change value based on determining that the change in the first physical feature is within a second physical feature value range that is different from the first physical feature value range,

[0026] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar has a corresponding spatial position within a reference system, wherein the corresponding spatial position is based on the position of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting a corresponding quantitative change in the position of the face within the field of view of the one or more cameras; in response to detecting a change in the position of the face within the field of view of the one or more cameras: modifying the spatial position of the virtual avatar within the reference system based on the first change component and the magnitude of a first modification factor according to determining that the change in the position of the face includes a first change component in a first direction; and modifying the spatial position of the virtual avatar within the reference system based on the second change component and the magnitude of a second modification factor according to determining that the change in the position includes a second change component in a second direction different from the first direction, the second modification factor being different from the first modification factor.

[0027] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar responds to changes in one or more physical features of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting a first configuration of one or more physical features of the face; while detecting the first configuration of the one or more physical features of the face: based on determining that the first configuration of the one or more physical features satisfies an animation standard, modifying the virtual avatar to include a first animation effect, the animation standard including requiring the first configuration to be maintained for at least a first threshold amount of time in order for the animation standard to be satisfied; and based on the first configuration of the one or more physical features not satisfying the animation standard, abandoning modifying the virtual avatar to include the first animation effect.

[0028] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0029] Thus, a faster, more efficient method and interface is provided for devices for generating, sending, and receiving emoticons, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace other methods for sending and receiving emoticons. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0031] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0032] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0033] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0034] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0035] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0036] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0037] Figure 5A A personal electronic device according to some embodiments is shown.

[0038] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0039] Figures 6A to 6MM An exemplary user interface for generating and sending emojis, stickers, avatars, and / or other multimedia content is shown.

[0040] Figures 7A to 7J An exemplary user interface for receiving emojis, stickers, avatars, and / or other multimedia content is shown.

[0041] Figures 8A to 8B is a flow chart illustrating a method for generating and sending emoticons, stickers, avatars, and / or other multimedia content.

[0042] Figures 9A to 9B is a flow chart illustrating a method for generating and sending emoticons, stickers, avatars, and / or other multimedia content.

[0043] 10A to 10I An exemplary user interface for generating and modifying a poop avatar is shown.

[0044] Figures 11A to 11C An exemplary user interface for generating and modifying a bear avatar is shown.

[0045] 12A to 12C An exemplary user interface for generating and modifying an alien avatar is shown.

[0046] Figure 13 An exemplary user interface for generating and modifying a rabbit avatar is shown.

[0047] 14A to 14D An exemplary user interface for generating and modifying a robotic avatar is shown.

[0048] FIG. 15A to FIG. 15B An exemplary user interface for generating and modifying a unicorn avatar is shown.

[0049] 16A to 16B An exemplary user interface for generating and modifying a chicken avatar is shown.

[0050] 17A to 17B An exemplary user interface for generating and modifying a pig avatar is shown.

[0051] 18A to 18B is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0052] Figure 19 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0053] Figure 20 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0054] Figure 21is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0055] Figure 22 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0056] Figure 23 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0057] Figure 24 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0058] Figure 25 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras. DETAILED DESCRIPTION

[0059] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0060] Sending messages with multimedia content and text content, or with only multimedia content, has the potential to better convey the sender's message. For example, multimedia content such as virtual avatars (e.g., animated or static emoticons or stickers) can provide context and / or tone that is difficult or impossible to convey with text alone (e.g., this can be referred to as "non-verbal communication"). In some cases, predefined virtual avatars can be used to provide some of this context and / or tone. However, predefined content cannot cover all situations or provide fine-tuned context or tone. Therefore, there is a need for electronic devices that provide effective methods and interfaces for generating, sending, and receiving virtual avatars as part of a message. Such technology can reduce the cognitive burden on users sending and receiving messages, thereby improving productivity. In addition, such technology can reduce processor power and battery power that would otherwise be wasted on redundant user input.

[0061] under, Figure 1A to Figure 1B 、 Figure 2 、 Figure 3 、 Figures 4A to 4B as well as Figures 5A to 5B A description of an exemplary device for performing techniques for generating, sending, and receiving virtual avatars is provided. Figures 6A to 6MM and Figures 7A to 7J An exemplary user interface for receiving, generating, modifying, and sending a virtual avatar is shown. Figures 8A to 8B and Figures 9A to 9Bis a flow chart illustrating an exemplary method for receiving, generating, modifying, and sending a virtual avatar. Figures 6A to 6MM as well as Figures 7A to 7J The user interface in is used to illustrate the processes described below, which include Figures 8A to 8B and Figures 9A to 9B in the process. 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B An exemplary user interface for generating and modifying a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include 18A to 18B 、 Figure 19 、 Figure 20 、 Figure 21 、 Figure 22 、 Figure 23 、 Figure 24 and Figure 25 in the process. 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B The user interface and 18A to 18B 、 Figure 19 、 Figure 20 、 Figure 21 、 Figure 22 、 Figure 23 、 Figure 24 and Figure 25 The process can be used to generate Figures 6A to 6MM as well as Figures 7A to 7J The interface and Figures 8A to 8B as well as Figures 9A to 9B The virtual avatar used in the process.

[0062] Although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first touch may be named a second touch, and similarly, a second touch may be named a first touch, without departing from the scope of the various embodiments described. Both the first touch and the second touch are touches, but they are not the same touch.

[0063] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0064] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that," or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining," or "in response to determining," or "upon detecting [stated condition or event]," or "in response to detecting [stated condition or event]," depending on the context.

[0065] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod equipment, and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0066] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0067] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a rendering application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0068] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0069] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. Touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and external ports 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contacts on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0070] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the surrogate measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurement). In some embodiments, the surrogate measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functionality that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0071] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. While such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0072] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 100 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0073] Memory 102 optionally includes high-speed random access memory and optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0074] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0075] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with communication networks and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). RF circuitry 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as by a short-range communication radio component. The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0076] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0077] The I / O subsystem 106 couples input / output peripherals on the device 100, such as the touch screen 112 and other input control devices 116, to a peripherals interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, a depth controller 169, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in ).

[0078] A quick press of the push button optionally releases the lock on the touch screen 112 or optionally initiates the process of unlocking the device using gestures on the touch screen, as described in U.S. patent application Ser. No. 11 / 322,549, filed Dec. 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0079] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives and / or sends electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0080] The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact on the touch screen 112 (and any movement or interruption of that contact) and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0081] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® from Apple Inc. (Cupertino, California). and iPod The technology used in

[0082] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpads described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, whereas a touch-sensitive touchpad does not provide visual output.

[0083] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following patent applications: (1) U.S. patent application 11 / 381,313, entitled “Multipoint Touch Surface Controller,” filed on May 2, 2006; (2) U.S. patent application No. 10 / 840,862, entitled “Multipoint Touchscreen,” filed on May 6, 2004; (3) U.S. patent application No. 10 / 903,964, entitled “Gestures For Touch Sensitive Input Devices,” filed on July 30, 2004; (4) U.S. patent application 11 / 048,264, entitled “Gestures For Touch Sensitive Input Devices,” filed on January 31, 2005; and (5) U.S. patent application 11 / 086, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive and (9) U.S. patent application No. 11 / 367,749, filed on March 3, 2006, entitled “Multi-Functional Hand-Held Device.” All of these applications are hereby incorporated by reference in their entirety.

[0084] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or appendage, such as a stylus, a finger, or the like, to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the user's desired action.

[0085] In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0086] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0087] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor is shown coupled to the optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, facing away from the touch screen display 112 on the front of the device, enabling the touch screen display to be used as a viewfinder for still and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing the user to optionally capture an image of the user for video conferencing while viewing other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing a single optical sensor 164 to be used with the touch screen display for both video conferencing and still and / or video image acquisition.

[0088] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to force sensor controller 159 in I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite touch screen display 112 located on the front of device 100.

[0089] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: No. 11 / 241,839, entitled “Proximity Detector In Handheld Device”; No. 11 / 240,788, entitled “Proximity Detector In Handheld Device”; No. 11 / 620,702, entitled “Using Ambient Light Sensor To Augment Proximity Sensor Output”; No. 11 / 586,862, entitled “Automated Response To And Sensing Of User Activity In Portable Devices”; and No. 11 / 638,251, entitled “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entireties. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0090] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile output on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0091] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally implements as described in the following U.S. Patent Publication: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0092] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1AA depth camera sensor is shown coupled to a depth camera controller 169 in the I / O subsystem 106. The depth camera sensor 175 receives data projected by the sensor from the environment. In conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 is optionally used to determine a depth map for different portions of an image captured by the imaging module 143. In some embodiments, the depth camera sensor is located on the front of the device 100, allowing the user to optionally obtain an image of the user with depth information for video conferencing while viewing other video conference participants on the touch screen display, and to capture a selfie with depth map data. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), allowing the depth camera sensor 175 to be used with the touch screen display for both video conferencing and still image and / or video image acquisition.

[0093] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications, if any, are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state, which includes information obtained from the device's various sensors and input control devices 116; and position information regarding the device's position and / or posture.

[0094] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware components and software components.

[0095] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0096] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger up event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single-finger contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multiple-finger contact). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touchpad.

[0097] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, a mouse "click" threshold for a touchpad or touchscreen can be set to any one of a large range of predefined thresholds without changing the touchpad or touchscreen display hardware. In addition, in some embodiments, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0098] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0099] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0100] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0101] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0102] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0103] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as the weather widget, the local yellow pages widget, and the map / navigation widget).

[0104] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0105] Contacts module 137 (sometimes called address book or contact list);

[0106] Telephone module 138;

[0107] Video conferencing module 139;

[0108] Email client module 140;

[0109] Instant messaging (IM) module 141;

[0110] Fitness support module 142;

[0111] A camera module 143 for still and / or video images;

[0112] Image management module 144;

[0113] Video player module;

[0114] Music player module;

[0115] Browser module 147;

[0116] Calendar module 148;

[0117] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets acquired by the user, and user-created widgets 149-6;

[0118] A widget creator module 150 for forming user-created widgets 149-6;

[0119] Search module 151;

[0120] Video and music player module 152, which combines the video player module and the music player module;

[0121] Notepad module 153;

[0122] Map module 154; and / or

[0123] Online video module 155.

[0124] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0125] In combination with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0126] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a character sequence corresponding to a phone number, access one or more phone numbers in contacts module 137, modify an entered phone number, dial the corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0127] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0128] In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0129] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0130] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0131] In combination with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0132] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0133] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching for, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0134] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0135] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0136] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., converting a user-specified portion of a web page into a desktop widget).

[0137] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0138] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0139] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0140] In combination with the RF circuitry 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 is optionally used to receive, display, modify, and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0141] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for allowing a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video. Additional descriptions of online video applications can be found in U.S. provisional patent application 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. patent application 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0142] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as separate software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0143] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0144] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0145] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0146] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0147] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0148] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0149] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0150] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0151] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0152] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0153] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0154] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0155] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver 182.

[0156] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0157] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 is part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application's internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0158] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0159] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0160] The event comparator 184 compares the event information with a predefined event or sub-event definition and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, the sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0161] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0162] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0163] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0164] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0165] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag obtains the flag and performs a predefined process.

[0166] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0167] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0168] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0169] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0170] Figure 2A portable multifunction device 100 with a touch screen 112 is shown according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0171] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0172] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding it in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions via the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0173] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or contact intensity sensors (similar to the above referenced Figure 1A The one or more contact intensity sensors 165 described herein)). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores information related to portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0174] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the memory device mentioned previously.Each module in the above-mentioned modules corresponds to the instruction set for performing the above-mentioned functions.Above-mentioned modules or programs (for example, instruction sets) need not be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structures.In addition, memory 370 optionally stores other modules and data structures not described above.

[0175] Attention is now turned to an embodiment of a user interface, optionally implemented on, for example, portable multifunction device 100 .

[0176] Figure 4A An exemplary user interface for an applications menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0177] One or more signal strength indicators 402 of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0178] Time 404;

[0179] Bluetooth indicator 405;

[0180] Battery status indicator 406;

[0181] A tray 408 with icons for commonly used applications, such as:

[0182] o An icon 416 labeled “Phone” of the phone module 138 , which optionally includes an indicator 414 of the number of missed calls or voice messages;

[0183] o An icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0184] o An icon 420 labeled "Browser" of the browser module 147; and

[0185] o An icon 422 labeled “iPod” for the video and music player module 152 (also referred to as the iPod (trademark of Apple Inc.) module 152); and

[0186] Icons for other apps, such as:

[0187] o Icon 424 labeled “Messages” of IM module 141;

[0188] o Icon 426 labeled “Calendar” of calendar module 148;

[0189] o Icon 428 labeled “Photos” of the image management module 144;

[0190] o An icon 430 labeled “Camera” of the camera module 143;

[0191] o Icon 432 labeled “Online Video” of the online video module 155;

[0192] o Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0193] o Icon 436 labeled “Map” of the map module 154;

[0194] o Icon 438 labeled “Weather” of the weather widget 149-1;

[0195] o Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0196] o An icon 442 labeled “Fitness Support” of the fitness support module 142 ;

[0197] o Icon 444 labeled "Notepad" of the Notepad module 153; and

[0198] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .

[0199] It should be pointed out that Figure 4A The icon labels shown are exemplary only. For example, icon 422 for video and music player module 152 is labeled "Music" or "Music Player." Other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0200] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, touch screen display 112) that is separate from a display 450 (eg, touch screen display 112). Figure 3 tablet or touchpad 355) of the device (e.g., Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0201] Although some of the examples below will be given with reference to input on a touch screen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B , 460 corresponds to 468 and 462 corresponds to 470 ) at contact with touch-sensitive surface 451 (e.g., Figure 4B 460 and 462 in FIG. 4 ). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and the display of the multi-function device ( Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0202] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0203] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include a body 502 relative to the devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in . In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. As an alternative to or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of applied contact (e.g., touch). The one or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.

[0204] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0205] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0206] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a reference Figure 1A 、 Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. The I / O portion 514 can be connected to a display 504, which can have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, the I / O portion 514 can be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, the input mechanism 508 is optionally a button.

[0207] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to I / O portion 514.

[0208] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the following techniques, including processes 800, 900, 1800, 1900, 2000, 2100, 2200, 2300, 2400, and 2500 (corresponding to processes 800, 900, 1800, 1900, 2000, 2100, 2200, 2300, 2400, and 2500, respectively. Figures 8A to 8B 、 Figures 9A to 9B 、 18A to 18B 、 Figure 19 、 Figure 20 、 Figure 21 、 Figure 22 、 Figure 23 、 Figure 24 and Figure 25). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.

[0209] As used herein, the term "indicator" refers to an indication that is optionally provided on device 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figures 5A to 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0210] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in FIG, the particular user interface element is adjusted according to the detected input. In the case of a touch screen display (e.g., a touch screen display) that enables direct interaction with user interface elements on the touch screen display Figure 1A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG, 2 , a contact detected on the touch screen acts as a “focus selector” such that when input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0211] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0212] In some embodiments, a portion of a gesture is identified for determining the characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic strength.

[0213] The intensity of the contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector in accordance with the movement of the contact on the touch-sensitive surface, without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0214] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0215] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the intensity of the contact (or multiple contacts) to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact to above the press input intensity threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the press input intensity threshold, and the corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0216] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0217] For ease of explanation, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting any of the following: contact intensity increasing above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing below the press input intensity threshold, and / or contact intensity decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0218] Attention is now turned to an embodiment of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0219] Figures 6A to 6MM An exemplary user interface for generating and sending a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include Figures 8A to 8B as well as Figures 9A to 9B in the process. Figures 7A to 7J An exemplary user interface for receiving and playing a virtual avatar is shown according to some embodiments. Figures 6A to 6MM as well as Figures 7A to 7J A virtual avatar is used as a specific example of a virtual avatar.

[0220] Figure 6A A device 600 is depicted having a display 601 (which, in some cases, is a touch-sensitive display) and a camera 602 (which includes at least an image sensor capable of capturing data representing a portion of a light spectrum (e.g., visible light, infrared light, or ultraviolet light)). In some embodiments, camera 602 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, camera 602 is capable of capturing other types of data such as depth data. For example, in some embodiments, camera 602 also captures depth data using techniques based on speckle, time of flight, parallax, or focus. Image data captured by device 600 using camera 602 includes data corresponding to a portion of the light spectrum of a scene within the camera's field of view. Additionally, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes data sufficient to determine or generate depth data for the data for that portion of the light spectrum. In some embodiments, device 600 includes one or more features of device 100, 300, or 500.

[0221] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate the object during image capture by the visible light camera and the depth camera (e.g., an IR camera), and information from the depth camera and the visible light camera is used to determine a depth map of different parts of the object captured by the visible light camera. In some embodiments, the lighting effects described herein are displayed using parallax information from two cameras (e.g., two visible light cameras) for rear-facing images, and depth information from the depth camera is used in combination with image data from the visible light camera for front-facing images (e.g., selfie images). In some embodiments, the same user interface is used when determining depth information using two visible light cameras and when determining depth information using the depth camera, thereby providing a consistent experience for the user even when completely different technologies are used to determine the information used to produce the lighting effects. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects selection of a camera switching indicator and switches from a front-facing camera (e.g., a depth camera and a visible light camera) to a rear-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), while maintaining display of user interface controls to apply the lighting effect and replacing the display of the field of view of the front-facing camera with the field of view of the rear-facing camera (or vice versa).

[0222] exist Figure 6A, device 600 is displaying a home screen interface having multiple icons for various applications, including an icon 603 for a messaging application. In response to a gesture on icon 603 (e.g., a tap gesture 604), device 600 displays Figure 6B 603 corresponds to a user interface of a messaging application associated with icon 603 .

[0223] exist Figure 6B , device 600 is displaying messaging interface 608. Elements 605-1 through 605-6 correspond to previous messaging communications. Each element 605-1 through 605-6 represents a communication with one or more remote users, each of which is associated with its own electronic device. In response to a gesture on a particular element (e.g., tap gesture 606), device 600 updates messaging interface 608 to display a portion of the previous messaging communication with the one or more remote users that was part of the communication, such as Figure 6C Described in.

[0224] exist Figure 6C , device 600 is displaying a messaging interface 608 for messaging communications with a remote user named "John" (and having the initials or monograms "JA"). Messaging interface 608 includes a message area 609 that includes four previously exchanged messages 610-1 through 610-3 (message 610-3 was sent from the user of device 600 to "John," while the other two messages were received by device 600 from "John"). Messaging interface 608 also includes a message composition area 612 and message options icons to the left of message composition area 612 (including icon 614, e.g., an icon accessing an interface for selecting stickers and / or other multimedia elements for a message). In some embodiments, the message options icon allows different types of messages to be sent, including photos, emoticons, stickers, and other forms of non-text messages, such as those described below.

[0225] In response to device 600 detecting (e.g., via Figure 6C 616) to select the message writing area 612, such as Figure 6D Update the messaging interface 608 as depicted in . For example, Figure 6D , the message options icons are hidden (but can be displayed again by selecting button 618), suggested message responses 620 are displayed, and a virtual keyboard 622 is displayed. In some cases, the virtual keyboard 622 is used to enter a new message to be sent to the remote user.

[0226] exist Figure 6E, the message composition area 612 includes the text "Come later", for example, entered via the virtual keyboard 622 or other method such as voice input. In response to selection of the send button 621, the device 600 sends the text as part of a message to one or more participants associated with the communication in message 609. Figure 6E In the case of , device 600 sends the message to the user named "John". Figure 6F , device 600 has message area 609 updated to reflect the sending of the message by updating message area 612 to include message 610 - 4 .

[0227] In some cases, the message options icon is accessed to add or compose a new message (e.g., by adding non-text content to the message). For example, in response to device 600 detecting a selection of affordance 618 (e.g., via Figure 6F 624), the message options icon including icon 614 is displayed again, as shown in FIG. Figure 6G In response to selection of icon 614 (e.g., via a gesture such as Figure 6H ), device 600 updates messaging interface 608 (e.g., by replacing virtual keyboard 622 with multimedia items interface 628 that is currently displaying a recent items menu 629 (sometimes referred to as a "tray" of recent items). Figure 6I ), the multimedia item interface includes previously sent multimedia items (e.g., Figure 6I (The stickers 630-1 to 630-4 in the figure may also include other types of multimedia items, such as sounds, animations, or videos). Using this interface, the user can select a previously sent multimedia item to send again. For example, the user can select a previously sent multimedia item in the image via a tap gesture on the selected sticker. Figure 6I609 . In response to such a selection, the device 600 places the sticker in the message composition area 612 or sends the selected sticker to one or more remote users involved in the communication represented in the message area 609 . In some embodiments, a tap and drag gesture is used to place the selected sticker (or other multimedia item) in the message composition area 612 or the message area 609 (and in some cases, on a specific message). For example, a specific sticker is selected via a tap gesture. Without breaking contact with the touch-sensitive display 601, the sticker is dragged to the message composition area 612 or the message area 609 via a drag gesture. Once the sticker reaches the desired location, contact with the touch-sensitive display 601 is stopped, and the sticker is placed at the last contact location. If the last contact location is in the message area 609, the sticker is sent to the one or more remote users associated with the communication represented in the message area 609. Optionally, the sticker is sent to the remote user along with data associating the sticker with a specific message (e.g., the sticker is sent along with data indicating that the sticker is “stuck” at a specific location in a specific message). These techniques are not specific to selecting and sending stickers. They can also be applied to other types of multimedia items that can be selected from the recent items menu 629 or other locations.

[0228] exist Figure 6I , multimedia item interface 628 also includes a menu selection button 632 (which allows a menu or interface other than recent items menu 629 to be selected via displaying buttons or other selectable items corresponding to available menus) and a full screen button 634 (which allows multimedia item interface 628 to expand to occupy more of the display (or occupy the entire display)). Full screen button 634 is further described below.

[0229] In addition to using menu selection button 632 to switch between menus or interfaces, gestures are optionally used to switch between menus. For example, in response to a swipe gesture (e.g., Figure 6J and Figure 6K 6 (depicted in FIG), by movement of contact 636 across multimedia items interface 628 (represented by a swipe), the device updates multimedia items interface 628 to replace display of recent items menu 629 with virtual avatar menu 638. While recent items menu 629 is being replaced with virtual avatar menu 638, scroll indicator 639 provides feedback as to how many other menus are available in multimedia items interface 628.

[0230] exist Figure 6L , the virtual avatar menu 638 has completely replaced the display of the recent items menu 629. In response to the device 600 detecting (e.g., via a gesture such as Figure 6M The tap gesture 642) is used to select the option to continue indicating 640, such as Figure 6N 6. The virtual avatar interface 643 is displayed as depicted in FIG. This interface allows a user to generate a new virtual avatar that reflects the user's facial movements and expressions, as further described below. In some embodiments, the virtual avatar menu 638 is not displayed at all. Instead, the virtual avatar interface 643 is displayed without first displaying the virtual avatar menu 638.

[0231] Figure 6N The virtual avatar interface 643 includes avatar template representations 644-1 to 644-7 corresponding to different avatar frames (e.g., avatar characters with different appearances and behaviors). Each avatar template represents an avatar frame onto which detected facial movements and expressions can be mapped. Indicator 645 corresponds to the currently selected avatar template. The virtual avatar preview 646 is a "real-time" preview of the virtual avatar because it is updated to reflect the user's current facial movements and expressions. For example, in some embodiments, using camera 602, the device 600 continuously captures image data from the camera 602. The captured image data includes visible light data and depth data. The device 600 analyzes the captured image data to identify facial movements (e.g., muscle movements, head orientation, gaze direction, etc.) and / or facial expressions (e.g., smile, frown, angry expression, sad expression, confused expression, etc.). The device 600 then updates the avatar preview 646 to reflect the detected user features based on the parameters of the avatar frame currently associated with the virtual avatar preview 646. In some embodiments, the device 600 automatically begins continuously updating the virtual avatar preview 646 in response to the virtual avatar interface 643 being first executed or displayed at 646. Detecting selection of a representation of a different avatar template causes the device 600 to update the virtual avatar preview 646 based on the newly selected avatar template.

[0232] Figure 6O Several examples of the user's face in captured image data 650-1 to 650-5 and corresponding updates 651-1 to 651-5 to the virtual avatar preview are depicted. These are examples of the device 600 updating the emoji preview 646 to reflect the user's facial movements, expressions, and gestures. In captured image data 650-1, the device 600 detects (e.g., based on facial features, facial muscles, facial movements, and / or facial expressions) that the user is looking straight ahead, smiling, and / or happy. In response, in addition to updating the eyes of the virtual avatar preview to look straight ahead, the device 600 also updates the virtual avatar preview to reflect the user's smile and / or happy expression, as depicted in update 651-1. Although the user's physical features detected in the captured image data are sometimes updated so that the virtual avatar reflects the same physical features in the user's virtual avatar, in other cases, changes in the detected physical features of the user result in updating different types of physical features of the virtual avatar. For example, in Figure 6OIn the example of image data 650-3 and update 651-3, since monkeys lack eyebrows, changes in the user's eyebrows are mapped to the monkey's ears (or other features), as shown in 650-2 and 651-2. In this example, the user's mouth and eyes are mapped to the monkey's mouth and eyes. In the example of image data 650-3 and update 651-3, the user's unhappy expression and / or frown is reflected in the corresponding features of the virtual avatar preview. In some embodiments, if the user maintains a facial expression or facial gesture, as depicted in images 650-3 and 650-4, the virtual avatar preview is updated with additional features (such as tears in the case of update 651-4). This type of predefined update can also occur in response to insufficient movement being detected. In some embodiments, updates are also based on user movement detected in the image data. For example, device 600 detecting rotation of the user's head generates an update that causes the virtual avatar preview to rotate similarly. In some embodiments, updates are also based on a physical model of the features used for the virtual avatar. For example, in image data 650-5, device 600 detects that the user is shaking his head. In response, device 600 generates update 651-5 to reflect the head shaking.In addition, in update 651-5, the puppy's ears are also extended due to the physical model applied to the puppy's ears.

[0233] exist Figure 6P , device 600 detects selection of record button 652 via a gesture (e.g., a tap gesture represented by contact 653). In response, avatar interface 643 is updated to show that an animated avatar is being generated, as shown in FIG. Figure 6Q . For example, the record button 652 is replaced with a stop button 654, the avatar template representations 644-1 to 644-7 are no longer displayed, and a recording progress indicator 656 is displayed that indicates how long the animated emoticon has been recorded and the relative amount of time that the virtual avatar can still be recorded. Recording can be stopped by any number of methods, such as by the expiration of a predetermined amount of time (e.g., 15 seconds) or by selecting the stop button 654. In some embodiments, while recording, the device 600 is detecting and / or storing a series of data points for creating the animated virtual avatar. For example, in some embodiments, the device 600 records a time series of facial movements and / or facial expressions (e.g., as values ​​in a range of possible values, where each value in the range of possible values ​​corresponds to a predetermined movement or expression), and then maps the facial movements and / or facial expressions onto the avatar template to create the animated virtual avatar. Alternatively, the device 600 records the animated virtual avatar by creating a video recording of the virtual avatar preview as the device 600 updates the virtual avatar preview to reflect the user's facial movements and / or expressions. In some embodiments, the device 600 also records the sound captured using the microphone of the device 600, so that the recorded animated virtual avatar includes the sound that can be played back together with the recorded animation of the virtual avatar.

[0234] Figure 6R Depicted is a later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0235] Figure 6R-1 Depicted is a device 600 that detects that a user has changed their position relative to the device during avatar recording. Figure 6R-1 At the corresponding time point, the user's face is no longer in the camera's field of view. In response, the device 600 displays the virtual avatar at the edge of the avatar interface 643 (e.g., the edge corresponding to the last detected position of the user's face), displays frame corners 653 around the virtual avatar, and displays a message 655A ("Please put your face in view") to prompt the user to adjust its alignment relative to the device. In some embodiments, even after the user's face is no longer detected in the camera's field of view, the recording of the virtual avatar continues, but when the user's face is not detected, the virtual avatar will remain still (or assume a predetermined posture (e.g., a neutral posture)).

[0236] Figure 6R-2 The device 600 is depicted after the user has been outside the camera's field of view for more than a predetermined threshold time. In response to detecting that the user has been outside the camera's field of view for more than the predetermined time, the device 600 pauses recording the virtual avatar. Figure 6R-2 As shown, in accordance with the recording pause, the device 600 has replaced the stop button 654 with the record button 648. The device 600 also displays a message 655B ("Tap to Resume") indicating to the user that recording has been paused. In some embodiments, the user can resume recording by tapping anywhere in the avatar interface 643, including tapping the record button 648. The recording of the virtual avatar is paused when the user has been outside the camera's field of view for a period exceeding a predetermined threshold amount of time, and another input is requested to resume recording, which reduces energy usage and usage of the depth camera, thereby extending the battery life of battery-powered devices and extending the life of the depth camera.

[0237] Figure 6S Depicts yet another later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to further reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0238] exist Figure 6SIn the embodiment of the present invention, a gesture requesting to stop recording the animated virtual avatar (e.g., a tap gesture represented by contact 658) is received. In response, the device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, such as Figure 6T In other cases, in response to the expiration of a predetermined time period (e.g., 15 seconds), the device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, such as Figure 6T Described in.

[0239] Figures 6S-1 to 6S-3 Another embodiment of a virtual avatar interface 643 for recording (e.g., generating) a virtual avatar is depicted. Figure 6S-1 As shown, the device 600 displays a timer 659 (e.g., showing 10 seconds remaining) that indicates the time remaining in the current avatar recording session (e.g., the session initiated by activating the record button 648). Figure 6S-2 In , the same recording session is conducted for 4 seconds and the timer 659 currently shows that 6 seconds remain in the avatar recording session. Figure 6S-3 , the recording session has ended (i.e., Figure 6S-3 is better than Figure 6S-1 In response to the recording session ending, device 600 replaces timer 659 with a trash can indicator 660-1, which can be activated (e.g., by a tap gesture) to discard the completed recording session. In some embodiments, the functionality of trash can indicator 660 is similar to Figure 6T as well as Figure 6U to Figure 6U-1 The discard indication indicates 660.

[0240] exist Figure 6T In the example shown in FIG. 6 , the avatar interface 643 is currently playing the recorded animated avatar 659, rather than displaying a preview of the avatar, as depicted by the three snapshots of the playback of the animated avatar 659. In some embodiments, the recorded animated avatar is played in a loop (e.g., the recorded animated avatar is played at least twice without user input, as shown in FIG. 6 ). Figure 6T). The virtual avatar interface 643 also includes a discard button 660, a mute button 662, and a confirmation button 664 (the confirmation button is displayed in place of the record button 652). The discard button 660 discards the displayed recorded animated virtual avatar without saving the avatar or sending the avatar to the remote user. The mute button 662 allows the user to mute the playback of the recorded animated virtual avatar. The confirmation button 664 allows the recorded animated virtual avatar to be sent to the remote user (e.g., directly to one or more users associated with the communication displayed in the message area 609 in response to activation of the confirmation button 664 or moving to the message composition area 612 before the user sends the message). After the device 600 detects selection of the confirmation button 664, the virtual avatar interface 643 is updated to return to the state relative to Figure 6N In some embodiments, the confirmation button 664 includes a send glyph or icon that is displayed in the send button for sending the message in the message composition area (e.g., Figure 6V 670) in, which is used to indicate that the recorded animated virtual avatar can be sent to the remote user by selecting the confirmation button 664.

[0241] While the animated avatar 659 is playing, in response to a tap gesture 665 on a representation of an avatar template that is different from the currently selected template, the animated avatar is updated to reflect the new avatar template without having to re-record the animated avatar. Figure 6U Described in Figure 6U Depicts animated avatar 659 having been replaced by animated avatar 666. Figure 6T The facial muscles, facial movements, facial features and facial expressions of the animated avatar 659 in Figure 6U The newly selected avatar template in .

[0242] exist Figure 6U-1 , while the animated avatar 666 is playing, the device 600 detects a tap gesture 661 corresponding to a selection of the discard affordance 660. In response, the device 600 discards the captured animated avatar data (e.g., abandons adding the animated avatar to the message composition area 612) and transitions to Figure 6U-2 In the interface. Figure 6U-2 , the device 600 displays a pre-recorded avatar interface 643, which is similar to Figure 6P The interface you see in (including, for example, the record button). Figure 6P In contrast, the avatar template remains the avatar template of the robot (e.g., virtual avatar 666) instead of returning to the monkey virtual avatar 659. That is, the avatar template is retained. Figure 6T and Figure 6UDetection results of the change from avatar 659 to avatar 666 during playback as depicted in .

[0243] Re-reference Figure 6T In response to a gesture (e.g., a tap gesture represented by contact 667), device 600 adds a recorded animated virtual avatar 668 to message composition area 612 (see Figure 6V ) and return the virtual avatar interface 643 to Figure 6N The state described in (see Figure 6V ). The user may then send the message on device 600 (e.g., in response to a tap gesture represented by contact 672 on send affordance 670, as shown in FIG. Figure 6W Described in ) before the message (see Figure 6V ) to add more message content (e.g., text or other multimedia items). Alternatively, when the device 600 detects selection of the confirmation button 664, the device 600 sends the recorded animated virtual avatar to one or more remote users associated with the communication displayed in the message area 609, and then updates the message area to reflect that the animated virtual avatar 668 has been sent to one or more users associated with the communication included in the message area 609, as shown. Figure 6X Described in.

[0244] Figures 6Y to 6BB Depicts the response of the virtual avatar interface 643 to a user input of scrolling through a list of avatar templates. For example, in response to a swipe gesture (e.g., a representation of contact 676 moving vertically across an avatar template, such as Figures 6Y to 6AA ), the device 600 scrolls the avatar templates and changes the currently selected avatar template. The avatar templates present within the avatar template indicator 645 are updated based on the swipe gesture. In response to detecting that a new avatar template is being selected, the device 600 updates the virtual avatar preview. For example, Figure 6Z In FIG. 6 , when the device 600 detects selection of the avatar template representation 644 - 5 , a virtual avatar preview 678 is displayed (the virtual avatar preview is based on the avatar template corresponding to the representation 644 - 5 ), and in FIG. Figure 6AA , when the avatar template representation 644-8 is selected, a virtual avatar preview 680 is displayed (the virtual avatar preview is based on the avatar template corresponding to the representation 644-8).

[0245] In addition to generating recordings of animated puppet emoticons, the emoticon interface 643 also allows for the generation of static virtual avatars (e.g., stickers with expressions / appearances determined based on the state of the virtual avatar). Figure 6CCIn the embodiment, in response to user input on the virtual avatar preview 680 (e.g., a tap and hold gesture represented by contact 682), the device 600 generates a sticker corresponding to the state of the virtual avatar preview 680 at a time associated with the user input (e.g., when the input is received, when the user input ends, or at some other time associated with the user input). In an embodiment, the device 600 displays the sticker 683 ( Figure 6DD and Figure 6EE ) to indicate that a sticker has been generated and / or the user can place a sticker.

[0246] After the device 600 generates the sticker, the user can optionally select from several actions for the sticker. For example, the user can cause the device 600 to place the sticker in a recent menu or other similar interface that allows for later use. The user can also cause the device 600 to place the sticker in the message composition area 612 before the device 600 sends the message containing the sticker. The user can place the sticker in the message area 609 (and optionally) on a specific message to cause the device 600 to send the sticker to one or more users participating in the communication in the message area 609.

[0247] For example, in Figure 6FF , the device 600 has detected the lift-off of the contact 682 while the contact is still above the virtual avatar preview 680. In response, the device 600 has saved the generated sticker to the device 600, such as in a database or library in the memory of the device 600, which can be accessed through the recent items menu 629 ( Figure 6I ) accesses the database or library so that stickers can be optionally selected via the recent items menu 629 or via other interfaces on the device 600. The device 600 optionally displays that stickers are saved locally via animations with different graphic versions 684 and 686, such as Figures 6FF to 6GG As depicted, movement toward menu selection button 632 occurs.

[0248] For example, Figures 6HH to 6KK An example is shown where the device 600 sends the generated sticker to one or more users participating in the communication represented in the message area 609. Figure 6II , the device 600 detects user input on the virtual avatar preview 680 (e.g., by Figures 6HH to 6JJ 680 in the virtual avatar preview 680). When the device 600 detects that the user drags the sticker representation into the message area 609, the sticker representation 690 follows the contact 688. Once the device 600 detects that the contact 688 in the message area 609 is lifted off, the device 600 sends the sticker 691 to one or more remote users who are participants in the communication represented in the message area 609, such as Figure 6KK Described in.

[0249] exist Figure 6LL , device 600 responds to selection of full screen button 634 (e.g., via Figure 6LL ) updates the virtual avatar interface 643 to display more of the screen (or display it in full-screen mode). Figure 6MM Virtual avatar interface 643 is depicted after being enlarged to take up more of display 601. Button 692, when selected, causes device 600 to return virtual avatar interface 643 to its previous configuration.

[0250] Figures 7A to 7J Depicted is the messaging interface 608 after receiving an animated emoticon from a remote user. Figures 7A to 7J use Figures 6A to 6MM The device 600 is used as an example, but Figures 7A to 7J The user interface and functionality depicted in are also applicable to other devices (e.g., device 100, 300, or 500), including those devices to which stickers or animated virtual avatars have not been previously sent.

[0251] Figure 7A The message interface 608 is depicted after the animated virtual avatar 700 has been received from a remote user named "John" (and having the initials or monograms "JA") just before the animated virtual avatar was played. In some embodiments, after receiving the animated virtual avatar 700, the device 600 automatically plays the animated virtual avatar. When the mute button 702 is selected, the mute button silences any sound associated with the animated virtual avatar 700 of the device 600. In some embodiments, if the animated virtual avatar scrolls off the display, any sound also silences. In some embodiments, as with respect to Figures 6A to 6MM The virtual avatar interface 643 is displayed as described (eg, the virtual avatar interface 643 includes a preview virtual avatar based on the detected facial movements / expressions and the selected avatar template).

[0252] exist Figure 7B In response to playing the animated virtual avatar 700 once (e.g., playing it once from the beginning to the end), a still frame 703 of the animated virtual avatar 700 is displayed in place of the animated virtual avatar 700. A playback button 704 is also displayed in the message area 609 and allows playback to be performed by, for example, Figure 7C The tap feature represented by contact 706 plays the animated virtual avatar 700 again. Figure 7D Depicts a device 600 for replaying an animated virtual avatar 700 (relative to Figure 7A Describes the play animation emoji).

[0253] In some embodiments, when the animated virtual avatar 700 is playing, if the device 600 receives user input on the mute button 702 (e.g., by Figure 7E In some embodiments, in response to selection of the mute button 702 (or if the sound on the device 600 is turned off or the device 600 has an enabled accessibility feature), a text button 714 is displayed, such as Figure 7H In response to selection of the text button 714 (e.g., via Figure 7G 716 represents a tap gesture), a text book 718 for the sound of the animated virtual avatar 700 is displayed, such as Figure 7H The contents of the script 718 are generated locally on the device 600 or remotely (eg, using remote server computing resources).

[0254] In response to user input on the animated virtual avatar 700 (e.g., Figure 7I ), the device 600 displays a menu of options related to the animated virtual avatar 700, such as Figure 7J . For example, menu 722 includes several response buttons 723-1 to 723-6 that device 600 can send to one or more remote users participating in the communication represented in message area 609. In addition, menu 724 is also displayed, which has a copy button 726, a save button 728, and a more button 730. Copy button 726 copies animated virtual avatar 700 to the clipboard of device 600. Save button 728 saves animated virtual avatar 700 to device 600 (e.g., to a database or library that can be accessed later by an application installed on device 600). More button 730 displays additional operations that can be performed with respect to animated virtual avatar 700.

[0255] Figures 8A to 8B 800 is a flowchart illustrating a method of using an electronic device according to some embodiments. The method 800 is performed at a device having a display and a camera (e.g., 100, 300, 500, 600). Some operations in the method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0256] As described below, method 800 provides an intuitive way to generate and send emoticons, such as avatars. This method reduces the cognitive burden on users of generating and sending emoticons, thereby creating a more efficient human-computer interface. For battery-powered computing devices, this allows users to generate and send emoticons more quickly and efficiently, conserving power and increasing the time between battery charges.

[0257] An electronic device (e.g., 600) having a camera (e.g., 602) (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (802) a virtual avatar generation interface (e.g., Figure 6N 643) (e.g., an interface for selecting an emoticon (animated or static), generating a static sticker, and / or recording an animated virtual avatar). The electronic device displays (804) a preview of the virtual avatar in the virtual avatar generation interface (e.g., Figure 6N 646) (e.g., a 2D or 3D computer-generated graphic object that, in some cases, is intended to convey a non-verbal message, such as an emotion or reaction) (e.g., an animated avatar selected from a plurality of different available avatar templates). The preview of the avatar reacts to changes in the appearance of the face in the camera's field of view (e.g., Figure 6O ) (e.g., the animated virtual avatar will reflect the user's head movements, facial expressions, and orientation detected in image data from one or more image sensors in the camera). While displaying a preview of the virtual avatar, the electronic device detects (806) an input in the virtual avatar generation interface (e.g., contact 652, 682, or 690). In response to (808) detecting the input in the virtual avatar generation interface and based on determining that the input begins on the preview of the virtual avatar (e.g., 682 or 690) (e.g., a touch and hold input on the animated virtual avatar or a trackpad input to control a cursor), the electronic device generates (810) a static virtual avatar sticker (e.g., 683 or 691) (e.g., a still image of an animated emoticon that can be "sticked" to a specific location in the message area) that represents the expression of the face in the camera's field of view at the corresponding time. In some embodiments, the corresponding time is determined based on the timing of the input (e.g., when the input is first received, when the input ends, when a gesture corresponding to the input begins to move across the touch-sensitive surface, or at any other time related to the input). Based on determining that the input includes activation of a record-enabling representation (e.g., 648) in the virtual avatar generation interface (e.g., tapping the record-enabling representation), the electronic device generates (812) an animated virtual avatar (e.g., 668) that represents a sequence of changes in facial expressions in the camera field of view over a period of time (e.g., as shown in FIG. 1 ). Figures 6Q to 6SAs shown). In some embodiments, the time period is determined based on the timing of the input (e.g., a time period that begins when the start of the input is detected, when the end of the input is detected, when a certain type of movement of the input is detected (such as when the input is a gesture on a touch-sensitive surface), or based on some other time period of the input). In some embodiments, the virtual avatar is three-dimensional. In some embodiments, a preview of the virtual avatar (e.g., 646) or an animated virtual avatar (e.g., 659) is displayed in 3D. Disambiguating between user inputs for two possible styles of communication (e.g., an animated virtual avatar and a static virtual avatar) avoids the need for a separate interface to generate each type of message content. Reducing the number of inputs required to deliver a desired message enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing multiple types of multimedia communications from a single interface to help users achieve their intended communications), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0258] In some embodiments, the electronic device displays (814) a messaging interface (e.g., interface 608) (e.g., a messaging application such as Apple's Messages) that includes a message area (e.g., 609). The message area includes messages from a communication (e.g., Figure 6N communication in the message area 609 of the message area 609) (eg, a messaging thread) of two or more participants (eg, Figure 6C , remote user "John" and the user of device 600) (e.g., messages sent from the user of the electronic device and messages received from the remote user of a different electronic device). The virtual avatar generation interface is displayed simultaneously with the message transmission interface (e.g., Figure 6N ) (e.g., the virtual avatar generation interface is displayed in the lower half of the messaging interface). In some embodiments, the virtual avatar preview (e.g., 646) is automatically displayed as part of the initial display of the virtual avatar generation interface.

[0259] In some embodiments, the messaging interface includes a message composition area (e.g., 612) (e.g., a message input area for entering text, emoticons, and other content before sending a message to a recipient) and the input is a tap on a virtual avatar preview (e.g., 646). In response to detecting the input in the virtual avatar generation interface, the electronic device displays a static virtual avatar (e.g., 683 or 691) in the message composition area. In some embodiments, displaying the virtual avatar generation interface includes replacing display of a virtual keyboard (e.g., 622) of the messaging interface with display of the virtual avatar generation interface (e.g., replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with display of the virtual avatar generation interface (e.g., replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with display of the virtual avatar generation interface). Figures 6H to 6NThe multimedia content of a message displayed before it is sent reduces the likelihood of error messages and allows the user to add more content (e.g., via text or other content) before sending the message. Reducing the number of messages required to convey a desired message enhances device operability and makes the user-device interface more efficient (e.g., by helping the user achieve the desired communication while reducing the number of messages required for communication), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0260] In some embodiments, the avatar generation interface includes a static virtual avatar area (e.g., 629) (e.g., a tray of previously generated stickers displayed at the bottom of the avatar generation interface) that includes a set of one or more previously generated virtual avatars (e.g., 630-1 through 630-4). Figure 6CC 682), the electronic device adds (612) the generated virtual avatar to a collection of one or more previously generated virtual avatars (e.g., sends the avatar, favorites the virtual avatar, or otherwise marks the virtual avatar to include it in the collection of virtual avatars). In some embodiments, in response to user input (e.g., 626) (e.g., selection of a virtual avatar collection affordance in an avatar generation user interface or a messaging user interface), a collection of virtual avatars (e.g., stickers) is displayed (e.g., including miniature versions of the newly generated stickers in the tray). In some embodiments, the tray of previously generated stickers is hidden until input is received from the user requesting display of the tray (e.g., input 626) or until some other event detected on the electronic device indicates that the tray may be relevant to the current state of the messaging interface or avatar generation interface. In some embodiments, after adding the virtual avatar to the collection of virtual avatars, the electronic device receives a request from the user to share the collection of virtual avatars with a second user, and in response, the electronic device sends the collection of virtual avatars to the second user. Maintaining previously sent message multimedia content allows the user to add and reuse previous content when applicable to new messages. Not having to recreate content enhances device operability and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends device battery life by enabling the user to use the device more quickly and efficiently.

[0261] In some embodiments, the input begins (816) on the preview of the virtual avatar (e.g., 680) and ends at a location within the message area (e.g., see Figures 6HH to 6KK) (e.g., a gesture that begins with a finger contacting a preview of the virtual avatar (e.g., 680), continues with dragging the finger into the message area, and ends with lifting the finger from the message area (in some cases, the gesture can end on a specific message in the message area, and the sticker is associated with the specific message and optionally moves as the message moves in the conversation). The electronic device sends (818) the static virtual avatar (e.g., 691) to the participants associated with the communication (e.g., one or more remote users) (e.g., Figure 6KK In some embodiments, an animation is displayed in response to a gesture showing a static virtual avatar being peeled away from the virtual avatar preview ( Figures 6CC to 6FF ).

[0262] In some embodiments, a static virtual avatar (e.g., 691) has an appearance determined based on the expression of the face in the camera's field of view when an input (e.g., 688) is detected on the virtual avatar preview (e.g., 680). In some embodiments, in response to detecting the start of an input on the preview of the virtual avatar, the electronic device causes the preview of the virtual avatar to stop (820) in reaction to a change in the appearance of the face in the camera's field of view. This indicates to the user that a sticker has been generated and, by sending the sticker to the user, provides the user with a preview of what the sticker will look like when dragging the sticker to the communication displayed in message 609. By displaying a preview of the sticker that will be generated without the user having to perform additional interactions or complete a full sticker generation gesture before seeing the resulting sticker, the operability of the device is enhanced, which results in an improved and more efficient human-computer interface. This reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0263] In some embodiments, after the input moves away from the virtual avatar, the virtual avatar preview (e.g., 680) resumes reacting to changes (e.g., animation resumes when the static avatar is dragged toward the messaging conversation). In some embodiments, the virtual avatar preview (e.g., 680) stops reacting to changes in facial appearance until the input of dragging the static avatar ends. Resuming updating of the virtual avatar preview enables the user to compare the appearance of the generated sticker with other possible appearances of the virtual avatar, which could potentially be the basis for different / additional stickers. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback about other content that can be generated before the user sends the generated content to help the user achieve the desired result), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0264] In some embodiments, in response to the expiration of a time period for generating an animated virtual avatar (e.g., the expiration of a 5-second, 10-second, or 15-second timer, or user input that stops the time period), the electronic device displays a send or confirm enable indication (e.g., 664) to replace the record enable indication (e.g., the virtual record button is no longer displayed, but a virtual send button is displayed instead of the virtual record button). In response to receiving input selecting the send or confirm enable indication (e.g., a tap gesture on the send enable indication on the touch-sensitive display), the electronic device sends (824) the generated animated virtual avatar to the remote user (e.g., see Figure 6U and Figure 6X , instead of first sending the animated virtual avatar to the message composition area 612, such as Figure 6V and Figure 6W As shown) (e.g., sending an animated avatar to a remote user associated with a messaging thread or conversation without first placing the animated avatar in another area of ​​the messaging interface, such as a message composition area). Displaying a send or confirm button in place of a record button after recording of the animated avatar is complete enables more information to be displayed in the interface by reusing area occupied by buttons that are not applicable to the current state of the interface and by providing more context-relevant functionality to the user. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0265] In some embodiments, in response to the expiration of the time period for generating the animated virtual avatar, the electronic device displays (822) a confirmation affordance (e.g., 664) in place of the recording affordance. In response to receiving input (e.g., via contact 667) selecting the send affordance (e.g., a tap gesture on the send affordance on the touch-sensitive display), the electronic device displays (823) a confirmation affordance (e.g., 664) in place of the recording affordance. Figure 6V ) in a message composition area (e.g., 612) (e.g., an area of ​​the messaging interface where text typed on a keyboard will be displayed). In some embodiments, the time period is based on a predetermined amount of time (e.g., the amount of time represented by progress indicator 656). After generating the animated avatar, the electronic device stops displaying the preview of the avatar and displays a looped version of the animated avatar (e.g., Figure 6T). Displaying a looped version of the animated virtual avatar includes displaying the animation sequence two or more times (e.g., as described below with respect to method 900). Displaying a send or confirm button in place of the record button after the recording of the animated virtual avatar is completed enables more information to be displayed in the interface by reusing area occupied by buttons that are not applicable (or less applicable) to the current state of the interface. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0266] In some embodiments, a first virtual avatar template (e.g., the template represented by element 644-4, or the template represented by element 644-5) is used. Figure 6N The electronic device displays representations of multiple other virtual avatar templates (e.g., elements 644-1 to 644-7) (e.g., miniature universal versions of different virtual avatar templates, such as smiling faces, animals, robots, or other objects), including a representation of a second virtual avatar template (e.g., Figure 6Z 644-8 of ), wherein the second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the plurality of virtual avatar templates is based on an emoticon that can be sent via a messaging application. Before starting to display a looped version of an animated virtual avatar (e.g., Figure 6T ) and in response to receiving user input selecting the first virtual avatar template representation (e.g., 665), the electronic device updates the display of the looped version of the animated virtual avatar to reflect the second virtual avatar template (e.g., Figure 6U (e.g., changing the appearance of the animated avatar to reflect the new avatar template while the animated avatar is still based on a sequence of changes in facial expressions). For example, the animated avatar may change from a monkey to a robot, but it will still reflect the same sequence of changes in facial expressions over time in the same camera field of view (e.g., see the sequence from Figures 6T to 6U Updating the animated avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the user to select the style of the animated avatar after recording the movements and actions of the animated avatar. Not having to re-record the animated avatar to try out a new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0267] In some embodiments, the virtual avatar preview is automatically displayed in response to launching the virtual avatar generation interface (e.g., 646). In some embodiments, once the virtual avatar generation interface is displayed, the virtual avatar preview is displayed without user input.

[0268] In some embodiments, in response to detecting a characteristic (e.g., position, orientation, movement) of a first physical feature of a face in the camera field of view (e.g., a mouth smiling, tongue sticking out, ears wiggling, eyebrows raised, or any other movement of any other physical feature), the electronic device updates the first physical feature of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the first physical feature of the face (e.g., eyes, eyebrows, mouth, tongue, ears) is the same as the type of the first physical feature of the displayed preview. In some embodiments, if the user's mouth is open, the virtual avatar's mouth will open responsively (e.g., Figure 6O 650-1 and 651-1 of ). Similar results can be based on facial expressions. For example, if one or more movements of physical features or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be updated to reflect the predefined movement by updating the corresponding physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to similar physical features of the virtual avatar enables the user to provide movements, expressions, and gestures that provide input to the system, which are intuitively mapped to the virtual avatar without the need for cumbersome or time-consuming touch or key input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve desired results by mapping the user's features to the virtual avatar in a predictable manner, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0269] In some embodiments, in response to detecting a characteristic of a second physical feature of a face in the camera field of view, the electronic device updates the second physical feature of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the second physical feature of the face is different from the type of the second physical feature of the displayed preview (for example, in Figure 6O, movement of eyebrows 650-2, and movement of ears 651-2). In some embodiments, if the user is smiling, and this indicates that the user is happy, different features of the virtual avatar (such as a unicorn's horn or lights on a robot) can be changed to reflect the smile. Similar results can be based on facial expressions. For example, if one or more movements of physical features or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be optionally updated to reflect the predefined movement by updating different sets of physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to different physical features of the virtual avatar enables the user to provide movements, expressions, and gestures that provide input to the system, which are mapped to avatar features that the user could not otherwise easily control. This enhances the operability of the device and makes the user-device interface more efficient (for example, by mapping the user's features to the virtual avatar so that additional features of the virtual avatar can be controlled to help the user achieve desired results, and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0270] In some embodiments, in response to detecting movement of a face in the camera field of view, the electronic device updates a third physical feature (e.g., Figure 6O 650-5 and 651-5 in ). In some embodiments, for example, if the virtual avatar is based on a virtual avatar template for a puppy, when it is detected that the user's face is shaking, the face of the virtual avatar will shake, and even if the user's ears do not extend in response to the shaking, the ears of the virtual avatar can extend to reflect the physics of the shaking motion. In some embodiments, the same physical features of the displayed virtual avatar preview are updated based on the movement and physical model of the corresponding features of the face in the camera field of view (for example, the ears move based on the movement of the user's ears, but also move based on the physical model of the puppy's big ears). Updating the virtual avatar based on the physical model for the virtual avatar enables the user to create a realistic and interactive virtual avatar that can convey a wider range of non-verbal information. This enhances the operability of the device and makes the user-device interface more efficient (for example, by helping the user to convey a predetermined message using a more realistic virtual avatar movement), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0271] In some embodiments, the preview of the virtual avatar is based on a predefined virtual avatar template (e.g., Figure 6NThe electronic device updates the preview of the virtual avatar based on one or more predefined behaviors associated with the predefined virtual avatar template. In some embodiments, if no movement or change in facial expression is detected on the face in the camera field of view (e.g., 650-3 and 650-4), the preview of the virtual avatar displays a predefined response (e.g., 651-4), such as blinking, rotating the head, making a facial expression, or other action.

[0272] In some embodiments, in response to determining that a face is no longer detected in the camera's field of view (e.g., facial tracking fails because the face has moved out of the camera's field of view; the face is obscured and does not appear in the camera's field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device gradually fades out the display of the virtual avatar preview (e.g., the virtual avatar preview 646 will fade out). In some embodiments, the device makes other modifications to the virtual avatar preview to indicate that the user's face can no longer be tracked, such as downgrading the virtual avatar preview based on the last information detected by the device, including changing the size, rotation, movement, etc. of the virtual avatar preview. In some embodiments, in response to determining that a face is no longer detected in the camera's field of view (e.g., facial tracking fails because the face has moved out of the camera's field of view; the face is obscured and does not appear in the camera's field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device displays a message (in the ) indicating that the camera can no longer accurately detect the face. Figure 6N Above or instead of the virtual avatar 643 Figure 6NIn some embodiments, in response to determining that a face is no longer detected in the camera field of view (e.g., facial tracking fails because the face has moved out of the camera field of view; the face is obscured and does not appear in the camera field of view; or the face has been repositioned such that the device can no longer accurately track the movement of features on the face), the electronic device updates the display of the preview of the virtual avatar based on changes in facial appearance that occurred during a time period before the face could no longer be detected in the field of view (e.g., immediately before or shortly before the face could no longer be detected in the field of view) (e.g., the device 600 repeatedly displays a transition from a smile to a frown or eye movement). In some embodiments, updating the display of the preview of the virtual avatar based on changes in facial appearance that occurred during a time period before the face could no longer be detected in the camera field of view includes gradually slowing down the updating of the preview of the virtual avatar over time so that the updating of the virtual avatar gradually stops (e.g., a rotating avatar slowly stops rotating, eyes that are opening or closing slowly stop opening or closing, and a mouth that is opening or closing slowly stops opening or closing). Displaying feedback about whether the avatar preview is tracking the user's face enables the user to determine whether the device is being held correctly and the conditions for detecting the user's face are correct. Providing improved feedback to the user about the device's status enhances device operability and makes the user-device interface more efficient by providing better continuity of the user interface with an indication that the device is still attempting to track the user's face. This provides a better and more intuitive human-computer interface and enables the user to continue interacting with the device even when the device is unable to track the user's face.

[0273] Note that the process described above with respect to method 800 (e.g., Figures 8A to 8B ) also applies in a similar manner to the methods described below. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, the generation of stickers described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For another example, muting the sound before sending an animated virtual avatar (e.g., an animated virtual avatar) as described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For the sake of brevity, these details are not repeated below.

[0274] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 and Figure 5A In addition, the above reference Figure 8A and Figure 8BThe operation is optionally performed by Figure 1A to Figure 1B . For example, detecting input in the virtual avatar generation interface (806) is optionally implemented by the event classifier 170, the event recognizer 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and the event distributor module 174 delivers the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to Figure 1A to Figure 1B The components depicted in the figure implement other processes.

[0275] Figures 9A to 9B 9 is a flowchart illustrating a method 900 of using an electronic device according to some embodiments. Method 900 is performed at a device having a display and a camera (e.g., 100, 300, 500, 600). Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0276] As described below, method 900 provides an intuitive way to generate and send emoticons, such as avatars. This method reduces the cognitive burden on users of generating and sending emoticons, thereby creating a more efficient human-computer interface. For battery-powered computing devices, this allows users to generate and send emoticons more quickly and efficiently, conserving power and increasing the time between battery charges.

[0277] An electronic device (e.g., 600) having a camera (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (902) a virtual avatar generation interface (e.g., Figure 6N 643) (for example, an interface for selecting an emoticon, generating a static emoticon, and recording an animated emoticon). The electronic device displays (904) a preview of the virtual avatar in the virtual avatar generation interface (for example, Figure 6N646) (e.g., a 2D or 3D computer-generated graphic object that, in some cases, is intended to convey a non-verbal message, such as an emotion or reaction) (e.g., a moving emoticon selected from a plurality of different available emoticon styles or templates). The preview of the virtual avatar reacts to changes in the appearance of the face in the camera's field of view (e.g., Figure 6O ) (e.g., the animated emoticon will reflect the user's head movement, facial expression, and orientation detected in image data from one or more image sensors). The electronic device receives (906) a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera's field of view (e.g., contact 652). In response to receiving the request to generate the animated virtual avatar, the electronic device records (908) (e.g., Figures 6Q to 6S ) a facial expression sequence of a face in the camera's field of view (e.g., a sequence comprising a series of data points for providing a mapping of points that can be applied to a virtual avatar template to generate an animated virtual avatar). After recording the facial expression of the face in the camera's field of view, the electronic device displays (910) a looped version of the animated sequence (e.g., Figure 6T and 6U ) is provided with an animated virtual avatar (e.g., 659) that is based on a facial expression sequence recorded in response to a request to generate the animated virtual avatar (e.g., by sequentially mapping a series of data points representing the recorded facial expressions to a predefined animated virtual avatar template to display the animated virtual avatar). The electronic device displays a looped version of the animated virtual avatar two or more times. In some embodiments, recording the facial expression sequence includes recording a time series of values ​​of discrete mapping points for the predefined virtual avatar template. Displaying the looped version of the animated virtual avatar enables the user to subsequently review the animated virtual avatar content to understand whether the appropriate message is being conveyed. Providing improved visual feedback to the user enhances the operability of the device, reduces instances of error, and makes the user-device interface more efficient (e.g., by providing feedback indicating output that will cause the device to generate the expected result to help the user achieve the expected result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0278] In some embodiments, the electronic device stops recording the facial expression sequence in response to the expiration of a timer (e.g., represented by progress indicator 656) (e.g., a 5-second, 10-second, or 15-second timer). In some embodiments, the electronic device stops recording the facial expression sequence in response to receiving user input (e.g., contact 658) (e.g., a user tapping a virtual button displayed on a display). Limiting the time of animated virtual avatar recordings enables users to create animated virtual avatar recordings while limiting the impact on the device's computing resources (e.g., storage). This enhances the operability of the device by conserving the device's computing resources.

[0279] In some embodiments, the electronic device replaces (912) the display of the preview with the display of a looped version of the animated virtual avatar (e.g., see Figures 6S to 6T The system also provides a method for displaying a sequence of facial expressions in a loop (e.g., playing back the generated animated avatar to the user in a loop when the recording of the facial expression sequence is complete). Playing a looped version of the recorded animated emoticon automatically enables the user to review the animated emoticon before deciding whether to send, delete, or save the animated emoticon. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback indicating the result before the user commits to the result to help the user achieve the expected result, and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0280] In some embodiments, the request to generate an animated virtual avatar includes selecting a record enable indication (e.g., 648) displayed in the virtual avatar generation interface. After recording the facial expression of the face in the camera field of view, the electronic device replaces the display of the record enable indication with a send or confirm enable indication (e.g., 664). In some embodiments, the send enable indication operates as explained above with respect to method 800. The animated virtual avatar is recorded in response to the selection of the record enable indication, so that the user can use the virtual avatar preview to verify that the device is tracking the user and that the currently selected virtual avatar template is consistent with the message the user wishes to convey. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve the expected result by providing a preview of the expected result before the user generates the expected result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0281] In some embodiments, a looped version of the animated virtual avatar is displayed (918) using a first virtual avatar template (e.g., the avatar template corresponding to element 644-4). The electronic device displays representations of a plurality of other virtual avatar templates (e.g., elements 644-1 through 644-7) (e.g., miniature, generic versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), including a representation of a second virtual avatar template (e.g., Figure 6Z 644-8 of ). The second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the plurality of virtual avatar templates are based on emoticons that can be sent via a messaging application. After beginning to display the looping version of the animated virtual avatar and in response to receiving user input representing a selection of the first virtual avatar template, the electronic device updates (922) the display of the looping version of the animated virtual avatar to reflect the second virtual avatar template (e.g., while the animated virtual avatar is still based on the changing sequence of facial expressions, changing the appearance of the animated virtual avatar to reflect the new virtual avatar template). For example, the animated virtual avatar may change from a monkey to a robot, but it will still reflect the changing sequence of facial expressions in the same camera field of view over time (e.g., see from ). Figures 6T to 6U In some embodiments, the electronic device displays (914) multiple representations of the virtual avatar template (e.g., miniature generic versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), the multiple representations including a representation of the first virtual avatar template. In response to receiving user input corresponding to a selection of a representation of the first virtual avatar template, the electronic device updates (916) the display of a looped version of the animated virtual avatar to correspond to the first virtual avatar template (e.g., changing the animation to a robot animation based on the robot virtual avatar template instead of a puppy animation based on the puppy virtual avatar template without requiring the user to re-record any facial expressions). Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the style of the animated virtual avatar to be selected after recording the movements and actions of the animated virtual avatar. Not having to re-record the animated virtual avatar to preview the new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0282] In some embodiments, the preview virtual avatar is based on the second avatar template. In response to detecting a face in the camera field of view (e.g., Figure 6OThe electronic device detects a first characteristic of a first physical characteristic of a face in the camera field of view (e.g., movement of the user's eyebrows), and based on the detected first characteristic, the electronic device updates (920) the first physical characteristic of the displayed virtual avatar preview (e.g., causing the virtual avatar preview to move its eyebrows). The first physical characteristic of the displayed preview has a first feature type (e.g., eyebrows). After receiving user input corresponding to selection of the first graphical element (e.g., switching the avatar template from a puppy to a monkey) and in response to detecting a second characteristic of the first physical characteristic of a face in the camera field of view (e.g., movement of the user's eyebrows), the electronic device updates (922) the second physical characteristic of the displayed virtual avatar preview (e.g., Figure 6O 651-2) (e.g., moving a monkey's ear), wherein the second physical feature of the displayed preview has a second feature type (e.g., ear) that is different from the first feature type (e.g., eyebrow). Mapping the same physical feature of the user to different physical features of different avatar templates enables the user to have a wider range of options for conveying messages by generating multiple virtual avatars using the same input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing more options for conveying messages to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0283] In some embodiments, in response to receiving a Figures 6Y to 6AA In response to a user input corresponding to a request for multiple representations of a virtual avatar template (contact 676), the electronic device scrolls the display of the multiple representations of the virtual avatar template to display a second graphical element that is not part of the multiple representations of the virtual avatar template. In some embodiments, the scrolling is based on the speed of the user input corresponding to the request. Scrolling through the virtual avatar templates enables the user to quickly view different options for the virtual avatar. In addition, scrolling the display of the multiple representations of the virtual avatar template enables the user to view the previous virtual avatar template and the next virtual avatar template. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback indicating an output that will cause the device to generate the expected result to help the user achieve the expected result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0284] In some embodiments, after the end of the user input is detected, the scrolling speed is gradually reduced over time (e.g., the scrolling gradually stops, just as the multiple representations of the virtual avatar have inertia that is gradually slowed by friction). Gradually reducing the scrolling speed over time enables the user to continue to view different virtual avatar template options without providing additional input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback of possible results without requiring additional interaction to help the user achieve the desired result, and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0285] In some embodiments, in response to receiving user input corresponding to the request, the electronic device generates an auditory output and / or tactile output corresponding to the currently selected virtual avatar template that changes from one virtual avatar template to a different virtual avatar template. For example, when each of the multiple representations of the virtual avatar template scrolls past a position indicating the currently selected virtual avatar template, an auditory output and / or tactile output is generated. Generating auditory or tactile feedback enables the user to determine when a new selection occurs. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback indicating when a new selection is made to help the user achieve the desired results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] In some embodiments, in response to receiving user input on the animated virtual avatar, the input corresponding to a request to save the animated virtual avatar, the electronic device stores data for the animated virtual avatar in a database on the electronic device (e.g., Figures 6CC to 6GG ). For example, the electronic device stores the data representing the animated virtual avatar in a directory or library in a non-volatile storage device in the electronic device.

[0287] In some embodiments, the electronic device receives a request (e.g., contact 688) to send an animated virtual avatar to a remote user of the remote device (e.g., Figures 6HH to 6KK). Based on determining that the remote device meets the first set of criteria (e.g., the remote device has a required application for playing the required version of the first version of the animated virtual avatar), the electronic device sends the first version of the animated virtual avatar to the user of the remote device (e.g., sending non-graphic data representing the recorded sequence of facial expressions and an indication of the virtual avatar template so that the remote device can reproduce the animated virtual avatar). Based on determining that the remote device does not meet the first set of criteria (e.g., the remote device does not have a suitable messaging application or a suitable version of the messaging application for playing the first version of the animated virtual avatar), the electronic device sends a second version of the animated virtual avatar that is different from the first version (e.g., sending a video file representing the animated virtual avatar) to the user of the remote device. Determining which version of the animated virtual avatar to send to the remote user conserves resources of the device by sending only a minimum amount of compatible information to the remote user. In addition, doing so reduces the need for the user to resend information in a more compatible format (e.g., in response to the remote user indicating that the initial format is not visible). Efficient and effective data transmission enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0288] In some embodiments, while displaying the looped version of the animated virtual avatar, the electronic device plays (924) audio data based on the sounds recorded while recording the sequence of facial expressions and based on audio filters associated with the predefined avatar template. In some embodiments, while displaying the looped version of the animated virtual avatar, the electronic device plays audio data based on the sounds recorded while recording the sequence of facial expressions. In response to receiving an indication related to a mute affordance (e.g., Figure 6T and Figure 6U 662) selection, the electronic device stops playing the audio data. Filtered audio for the animated virtual avatar is played based on a filter specific to the avatar template for the virtual avatar, which enables the user to more efficiently transmit messages by providing more options on how to transmit messages and more attractive animated virtual avatars. This enhances the operability of the device and makes the user-device interface more efficient (for example, by providing the user with more options on how to transmit messages to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0289] In some embodiments, in response to receiving (926) a request to send an animated virtual avatar to a remote user and based on determining that a request to send the animated virtual avatar to the remote user was received while audio data associated with the display of the looping version of the animated virtual avatar was muted (e.g., contact 667), the electronic device sends (928) data representing the animated virtual avatar to the remote user without sending sound data for the animated virtual avatar. Based on determining that a request to send the animated virtual avatar to the remote user was received while audio data associated with the display of the looping version of the animated virtual avatar was not muted, the electronic device sends (930) the data representing the animated virtual avatar along with the sound data for the animated virtual avatar to the remote user. Sending the animated virtual avatar without sound when the user mutes the playback of the animated virtual avatar enables the user to effectively choose whether to include sound in a message sent to the remote user. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve the intended message with minimal interaction), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0290] In some embodiments, in response to receiving a request to generate an animated virtual avatar (e.g., contact 652), the electronic device records first facial movements of a face in the camera field of view, wherein displaying a looped version of the animated virtual avatar includes animating the virtual avatar based on a physical model for the animated virtual avatar and the first facial movements (e.g., see image data 650-5, and Figure 6O Update 651-5). Updating the animated avatar based on the physical model for the avatar enables users to create realistic and interactive avatars that can convey a wider range of non-verbal communications. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping users convey predetermined messages using more realistic avatar movements), thereby further reducing power usage and extending the battery life of the device by enabling users to use the device more quickly and efficiently.

[0291] In some embodiments, in response to detecting that a particular feature of a face in the camera's field of view is held in a particular pose for more than a threshold amount of time while recording a sequence of facial expressions (e.g., see Figure 6O 650-4 and 650-5), the electronic device adds a predefined animated expression corresponding to a specific facial posture to the animated virtual avatar (for example, see Figure 6O651-5). For example, if the face has a neutral expression for a predetermined time period, a predefined movement (such as a head turn or a blink) is added to the animated virtual avatar. For another example, if the face has an angry expression for a predetermined time period, one or more additional features representing anger (such as the ears changing color or steam coming out of the ears) are added to the animated virtual avatar. Updating the animated virtual avatar based on the device detecting that features of the user's face remain in a particular posture for a threshold amount of time enables the user to add more actions to the animated virtual avatar than could be achieved using only facial expressions, features, and movements. This enhances the operability of the device and makes the user-device interface more effective (e.g., by providing additional mechanisms for transmitting actions that do not otherwise correspond to readily available facial expressions, movements, or features to help the user achieve desired results), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0292] In some embodiments, while recording a sequence of facial expressions, in response to receiving user input via an input mechanism separate from the camera (e.g., a touch on a touch-sensitive surface, movement of the electronic device detected by a motion sensor, activation of a button, or other input), the electronic device adds a first facial expression to the sequence of facial expressions (e.g., recording a happy facial expression, a tongue-out facial expression, or any other facial expression that was not actually recorded as a facial expression made by a face in the camera's field of view when recording the facial expression for inclusion in the animated virtual avatar). The first facial expression is based on the user input received via the input mechanism. In some embodiments, while the animated virtual avatar is looping, the user can use touchscreen controls to add additional expressions to the animated virtual avatar, such that as the animated virtual avatar loops, the user can gradually add expressions to the animated virtual avatar, such that even if those changes in expression differ from the changes in expression recorded based on the facial expressions of the face in the camera's field of view (e.g., the user's face) when the animated virtual avatar was initially created, the animated virtual avatar includes the changes in expression selected by the user. Updating the animated virtual avatar based on user input rather than as captured by the camera enables the user to add more actions to the animated virtual avatar than would be possible using only facial expressions, features, and movements. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing additional mechanisms for conveying actions that do not otherwise correspond to readily available facial expressions, movements, or features to help the user achieve desired results), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0293] Note that the process described above with respect to method 900 (e.g., Figures 9A to 9B) can also be applied in a similar manner to the methods described above. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, displaying an animated virtual avatar preview based on a sequence of recorded facial features, movements, and / or expressions and based on a frame associated with the avatar template described with respect to method 900 can be applied to the sticker and animated virtual avatar interfaces described above with respect to method 800.

[0294] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A 、 Figure 3 and Figure 5A In addition, the above reference Figure 9A and Figure 9B The operation is optionally performed by Figure 1A to Figure 1B . For example, receiving a request to generate an animated virtual avatar (906) is optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and an event distributor module 174 delivers the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as the selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls a data updater 176 or an object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know how to Figure 1A to Figure 1B The components depicted in the figure implement other processes.

[0295] 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B An exemplary user interface for generating and modifying a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include 18A to 18B 、 Figure 19 、 Figure 20 、 Figure 21 、 Figure 22 、 Figure 23 、 Figure 24 and Figure 25 in the process.

[0296] In some embodiments, a virtual avatar is a representation of a user that can be graphically depicted. In some embodiments, the virtual avatar is non-photorealistic (e.g., cartoon-like). In some embodiments, the avatar is an anthropomorphic construct such as a stylized animal (e.g., avatars 1100, 1300, 1500, 1600, and 1700), a stylized robot (e.g., avatar 1400), or a stylization of a generally inanimate object (e.g., avatar 1000). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., map) to one or more physical features of the user's face such that detected movement of the user's physical features affects the avatar features (e.g., affects the graphical representation of the features).

[0297] In some examples, the user can use a camera sensor (e.g., camera module 143, optical sensor 164) to manipulate the characteristics or features of the virtual avatar. When the user's physical characteristics (such as facial features) and position (such as head position or head tilt) change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical characteristics and position. In some embodiments, the changes to the user's physical characteristics and position indicate various expressions, emotions, context, tone, or other non-verbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, context, tone, or other non-verbal communication.

[0298] 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B Exemplary user inputs and corresponding changes to exemplary virtual avatars (e.g., a poop avatar, a bear avatar, an alien avatar, a rabbit avatar, a robot avatar, a unicorn avatar, a chicken avatar, and a pig avatar) displayed on an electronic device are shown. In some embodiments, the electronic device includes one or more elements and / or features of devices 100, 300, and 500. 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B The image on the left represents an image of the user detected by the electronic device when the user is within the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In other words, the image of the user comes from the perspective of a camera (e.g., camera module 143, optical sensor 164), which in some embodiments can be located on the electronic device (e.g., devices 100, 300, and 500) and in other embodiments can be located separately from the electronic device (e.g., an external camera or sensor that transmits data to the electronic device). In some embodiments, 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B The boundaries of the image on the left represent the boundaries of the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In some embodiments, the user's image is displayed on the display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504). In some embodiments, the user's image is transmitted to an external electronic device for display. In some embodiments, the external electronic device includes one or more elements and / or features of devices 100, 300, and 500. In some embodiments, the user's image data is collected and processed by the device, but is not immediately displayed on the electronic device or transmitted to the external device.

[0299] 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B Each of the images on the left side shows a virtual avatar (e.g., a poop avatar) in a state presented (e.g., displayed after being modified) based on a detected corresponding user image located on the left side of the figure. In some embodiments, the virtual avatar is displayed from the perspective of the user observing the virtual avatar. In some embodiments, the virtual avatar is displayed on a display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504). In some embodiments, the virtual avatar is transmitted to an external electronic device for display. In some embodiments, 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B The image on the right side represents the position of the virtual avatar within the display area of ​​the display of the electronic device (eg, touch screen 112, display 340, display 450, display 504), and 10A to 10I 、 Figures 11A to 11C 、 12A to 12C 、 Figure 13 、 14A to 14D 、 FIG. 15A to FIG. 15B 、 16A to 16B as well as 17A to 17B The boundaries of the image on the right represent the boundaries of the display area including the virtual avatar. In some embodiments, the display area represented on the right corresponds to the avatar display area of ​​the application user interface, such as the virtual avatar interface 643, the message composition area 612, the message area 609 (or a portion thereof) discussed above.

[0300] In some embodiments, the magnitude of the reaction of an avatar feature (e.g., a discrete element of an avatar that can be discretely moved or modified relative to other avatar features) corresponds to the magnitude of the change in a physical feature of the user (e.g., a detected or tracked feature, such as a muscle, muscle group, or anatomical feature such as an eye). For example, in some embodiments, the magnitude of the change in a physical feature is determined based on the potential range of motion of the physical feature, where the magnitude represents the relative position of the physical feature within the range of motion of the physical feature (e.g., a predicted or modeled range of motion). In such embodiments, the magnitude of the reaction of the avatar feature is similarly the relative position of the avatar feature within the range of motion of the avatar feature. In some embodiments, the magnitude of the change is determined based on a comparison or measurement (e.g., a distance) of the starting and ending positions of the change in the physical feature. In such embodiments, the change in the physical feature can be converted into a modification to the first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly, or as a scaled or adjusted value).

[0301] In some embodiments, the modification to the avatar feature has a magnitude component and a direction component, wherein the direction component of the modification to the avatar feature is based on a direction component of the change in one or more physical features to which the avatar feature reacts. In some embodiments, the direction of the avatar feature's reaction corresponds to (e.g., directly corresponds to or inversely corresponds to) a relative direction of the change in the user's physical feature, wherein the relative direction of the change in the physical feature is determined based on the direction of movement of the physical feature from an initial position (e.g., a neutral position, a rest position of the physical feature, or, in some embodiments, a position of the physical feature initially detected by the device). In some embodiments, the direction of the avatar feature's reaction corresponds directly to the relative direction of the change in the physical feature (e.g., if the physical feature moves upward, the avatar feature also moves upward). In other embodiments, the direction of the avatar feature's reaction corresponds inversely to the relative direction of the change in the physical feature (e.g., if the physical feature moves upward, the avatar feature moves downward).

[0302] In some embodiments, the directional component of the change of the avatar feature is mirrored relative to the directional component of the change of the physical feature. For example, when the physical feature (e.g., the user's mouth) moves to the left, the avatar feature (e.g., the avatar's mouth) moves to the right. In some embodiments, for movement along the vertical axis, the directional component of the change of the avatar feature is the same as the directional component of the change of the physical feature, and for movement along the horizontal axis, the directional component of the change of the avatar feature is mirrored to the directional component of the change of the physical feature, similar to the effect seen when looking in a mirror. In some embodiments, the neutral resting position of the user's iris is determined to be a specific position relative to the periphery of the user's eyeball (e.g., a centered position).

[0303] Figure 10A An exemplary embodiment is shown in which an electronic device modifies a poop avatar 1000 in response to detecting a change in a user's facial features. The poop avatar is shown as having four display states (1011A, 1011B, 1011C, and 1011D), wherein each of the four display states of the poop avatar corresponds to the user's four detection states (1001A, 1001B, 1001C, and 1001D). Specifically, in Figure 10A In each display state of the user, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 10A In the illustrated embodiment, the detected facial features of the user include the user's mouth 1020 (with corners 1020A and 1020B) and the user's eyebrows 1022. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 10A In an embodiment of the present invention, the features of the poop avatar include an avatar face 1000C, a mouth 1030, avatar eyes 1032, an upper portion of the avatar 1034, a lower portion of the avatar 1036, and (in some cases) avatar eyebrows 1038.

[0304] As shown in 1001A, the electronic device detects a neutral facial expression of the user. For example, the electronic device detects that the user's mouth 1020 and eyebrows 1022 are positioned in a relaxed, neutral state, rather than in positions associated with a particular facial expression (e.g., a smile or a frown). In response to detecting the user's neutral facial expression, the electronic device displays a poop avatar 1000 with a neutral expression (e.g., a neutral state) in 1011A. Specifically, the electronic device displays the poop avatar with the avatar's mouth 1030 in a relaxed, neutral state, rather than in a position typically associated with a particular facial expression (e.g., a smile or a frown). In some embodiments, the avatar's neutral position corresponds to other representations of the relevant image, such as a static poop emoji that can be found in a messaging application. In addition, the electronic device displays the poop avatar's face 1000C without eyebrows 1038 and displays the avatar's eyes 1032 looking in a direction perpendicular to the display (or the focal plane of a camera (e.g., camera 143, optical sensor 164)). The electronic device also displays an upper portion 1034 of the poop avatar at a neutral vertical position above a lower portion 1036 .

[0305] As shown in 1001B, the electronic device detects the position of the user's mouth 1020 for forming a smiling facial expression (e.g., one or both of the corners 1020A and 1020B of the user's mouth are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression). In response to detecting the position of the user's mouth 1020, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, such that the avatar's mouth 1030 has an open smiling position, as shown in 1011B. As shown in 1011B, the eyes 1032, upper portion 1034, and lower portion 1036 remain unchanged relative to their original positions in 1011A.

[0306] As shown in 1001C, the electronic device detects a frowning facial expression of the user. In some embodiments, the electronic device detects a frowning facial expression by detecting that one or both of the corners 1020A and 1020B of the user's mouth are positioned in a downward posture (e.g., a downward position) and that the user's eyebrows 1022 are in a downward position (e.g., the user's eyebrows are wrinkled on the user's face or positioned at a lower position on the user's face compared to the position of the eyebrows 1022 in the relaxed neutral state in 1001A and 1001B). In response to detecting the frowning facial expression of the user, the electronic device modifies the poop avatar to have a frowning and drooping face, as shown in 1011C. For example, the electronic device modifies the poop avatar so that the corners of the avatar's mouth 1030 are rotated downward so that the mouth 1030 has a slightly open posture, and the lower portion 1036 of the poop avatar is bent downward, similar to the downward rotated posture of the poop avatar's mouth 1030. Figure 10A In an embodiment, when the electronic device detects a downward position of the user's eyebrows 1022, the anatomical counterpart of the poop avatar above the avatar's eyes 1032 is not modified.

[0307] In some embodiments, the electronic device modifies the poop avatar by displaying an animation of the mouth 1030 rotating downward and the lower portion 1036 bending downward as the mouth 1030 is moving to the downwardly rotated position, so that the poop avatar has a drooping face as shown in 1011C. In some embodiments, the electronic device further modifies the poop avatar so that when the poop avatar presents a drooping face, the poop avatar's tip 1040 collapses or tilts downward. In some embodiments, the position of the tip 1040 is specifically based on the position of the user's eyebrows 1022 (a physical feature that does not anatomically correspond to the tip 1040). In some embodiments, when the user no longer makes a frowning expression, the electronic device modifies the poop avatar to return it to a neutral position. In such embodiments, the electronic device modifies the poop avatar by displaying an animation of the mouth 1030 moving to a neutral position and the lower portion 1036 moving back to its neutral position, so that the poop avatar returns to the neutral state shown in 1011A. In some embodiments, returning the poop avatar from the droopy face to the neutral state includes the electronic device displaying the tip portion 1040 of the poop avatar as straightened to its neutral position.

[0308] As shown in 1001D, the electronic device detects that the corners 1020A and 1020B of the user's mouth are slightly raised and the user's eyebrows 1022 are in a raised position (e.g., higher on the user's face compared to the position of eyebrows 1022 when in the relaxed neutral state shown in 1001A and 1001B). In response to detecting the position of corners 1020A and 1020B of the user's mouth 1020, the electronic device modifies the display of the poop avatar so that the corners of the avatar's mouth 1030 are slightly raised to match the position of corners 1020A and 1020B of the user's mouth 1020. In response to detecting the raised position of the user's eyebrow 1022, the electronic device modifies the poop avatar by introducing an eyebrow 1038 located above the poop avatar's eye 1032 into a raised position (e.g., to convey the expression that the poop avatar 1000 is raising its eyebrow 1038) and extending the poop avatar's upper portion 1034 in an upward direction (e.g., by extending the poop avatar's tip 1040 while maintaining the original position of the poop avatar's lower portion 1036). In the embodiments shown in 1001D and 1011D, when the user's eyebrow 1022 is raised, the electronic device introduces the avatar's eyebrow 1038 and extends the poop avatar's tip 1040. In some embodiments, when the user's eyebrow 1022 returns to its neutral position, the electronic device removes the avatar's eyebrow 1038 and relaxes the tip 1040. In some embodiments, the electronic device removes the poop avatar's eyebrows 1038 by animating the eyebrows 1038 moving downward toward the poop avatar's eyes 1032 and disappearing into the poop avatar's face 1000C above the eyes 1032 .

[0309] Figure 10B An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's facial features, wherein the modification of the poop avatar includes moving the avatar features in an exaggerated manner. In some embodiments, exaggerating the features of the virtual avatar allows the user to effect the maximum change to the avatar features without uncomfortably changing corresponding features of their face. For example, Figure 10B As shown, the user can cause the avatar to open their mouth as wide as possible (e.g., in a surprised expression) without causing the user to open their mouth uncomfortably (e.g., without causing the user to open their mouth to the maximum extent of the predicted or determined range of motion of the user's mouth).

[0310] Figure 10B The poop avatar is shown as having three display states (1012A, 1012B and 1012C), wherein each of the three display states of the poop avatar corresponds to the three detection states (1002A, 1002B and 1002C) of the user. Figure 10BIn each display state of the user, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 10B In the illustrated embodiment, the detected facial features of the user include the user's mouth 1020. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 10B In the embodiment of the present invention, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, an upper portion 1034 of the avatar, and a lower portion 1036 of the avatar.

[0311] As shown in 1002A, the electronic device detects a first state of the user, in which the user's mouth 1020 is slightly open (e.g., 10% of the maximum range of the predicted or determined range of motion of the user's mouth). In response to detecting the slightly open user's mouth 1020, the electronic device modifies the poop avatar, as shown in 1012A, so that the poop avatar's mouth 1030 has an open position (20% of the maximum range of the simulated range of motion of the avatar's mouth, the degree of openness being greater than (e.g., within its corresponding range of motion) the degree of openness of the user's mouth 1020), while keeping other features of the avatar unchanged (such as the poop avatar's eyes 1032, upper portion 1034, and lower portion 1036).

[0312] As shown in 1002B, the electronic device detects a change in the user's facial features, in which the user's mouth 1020 is opened wider than in state 1002A (e.g., opened 25% of the maximum range of the predicted or determined range of motion of the user's mouth). In response to detecting that the user's mouth 1020 transitions from the slightly open position in 1002A to the wider open position in 1002B, the electronic device modifies the poop avatar's mouth 1030 to increase in size, as shown in 1012B, so that the mouth 1030 has a larger open position than the position shown in 1012A (e.g., 50% of the maximum range of the simulated range of motion of the avatar's mouth), while still maintaining the positioning of other avatar features (including the poop avatar's eyes 1032, upper portion 1034, and lower portion 1036).

[0313] As shown in 1002C, the electronic device detects another change in the user's facial features, in which the user's mouth 1020 is opened even wider than in 1002B (e.g., opened to 50% of the maximum range of motion of the predicted or determined range of motion of the user's mouth). In response to detecting the transition of the user's mouth 1020 from the open position in 1002B to the wider open position in 1002C, the electronic device modifies the poop avatar's mouth 1030 to further increase in size, as shown in 1012C, such that mouth 1030 has an open position that is even wider than the position shown in 1012B (e.g., 100% of the maximum range of motion of the avatar's mouth). However, in 1012C, the opening of mouth 1030 is greater than the height of lower portion 1036. Therefore, the electronic device causes lower portion 1036 of poop avatar 1000 to expand at 1042 in response to the user's open mouth 1020 to maintain the structural integrity of poop avatar 1000. In other words, in order to maintain a consistent positioning of the avatar and its features in response to the user's open mouth 1020, the device modifies one or more interconnected portions of the virtual avatar (e.g., lower portion 1036). For example, as shown in 1012C, the electronic device expands the lower portion 1036 of the poop avatar, adjacent to the avatar's mouth 1030, at area 1042 to accommodate the increased size of the enlarged avatar's mouth 1030. If the electronic device did not modify the lower portion 1036 in this manner, the enlarged mouth 1030 would extend beyond the structure of the virtual avatar, potentially obstructing the context and / or tone that the user intended to convey using the virtual avatar.

[0314] In some embodiments, the increase in size of the avatar's mouth 1030 (e.g., from the position shown in 1012A to the position shown in 1012B, or from the position shown in 1012B to the position shown in 1012C) is not proportional to the increase in size of the user's mouth 1020 (e.g., from the position shown in 1002A to the position shown in 1002B, or from the position shown in 1002B to the position shown in 1002C), but is scaled to provide an exaggerated rate of change in size. For example, in some embodiments, the scaling ratio is a multiplication factor of 2, so that the relative open position of the avatar's mouth is twice the relative open position of the user's mouth. For example, if the user's mouth is opened 10% of the maximum range of motion of the user's mouth, the electronic device displays the avatar's mouth opening 20% ​​of the maximum range of simulated motion of the avatar's mouth.

[0315] Figure 10CAn exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics, wherein the modification of the poop avatar includes rotating (e.g., tilting) the upper portion of the poop avatar forward and backward while keeping the lower portion of the poop avatar still. The electronic device displays the poop avatar as having three display states (1013A, 1013B, and 1013C), wherein each of the three display states of the poop avatar corresponds to the three detection states of the user (1003A, 1003B, and 1003C). Specifically, in Figure 10C In each display state of the user, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 10C In the embodiment shown in , the user's physical features include the user's face 1024, chin 1026, and head 1028. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 10C In the embodiment of the present invention, the features of the poop avatar include a mouth 1030 of the avatar, eyes 1032 of the avatar, an upper portion 1034 of the avatar (including a tip 1040 of the poop avatar), and a lower portion 1036 of the avatar.

[0316] As shown in 1003A, the electronic device detects a user's face 1024 rotated in an upward position. For example, the user raises their chin 1026 and tilts their head 1028 backward (e.g., away from the camera's (e.g., camera 143, optical sensor 164)) to position their face 1024 in an upward position. In response to detecting the upward position of the user's face 1024, the electronic device modifies the poop avatar to an upward-looking state by tilting the upper portion 1034 of the poop avatar in a direction away from the display (e.g., backward from the camera's (e.g., camera 143, optical sensor 164)) focal plane, as shown in 1013A. In this backward-tilted position, the electronic device displays the tip 1040 of the poop avatar positioned toward the rear of the poop avatar 1000 to illustrate the offset position of the tip 1040 when the poop avatar is modified to an upward-looking state. Additionally, the electronic device modifies the poop avatar's eyes 1032 to look upward (e.g., by shifting the poop avatar's pupils or irises 1032A toward the top of the avatar's eyes 1032), as shown in 1013A. While the electronic device tilts the poop avatar's upper portion 1034 and modifies the poop avatar's eyes 1032 to look upward, the electronic device leaves other features of the poop avatar unchanged. For example, the electronic device maintains the position of the avatar's mouth 1030 and fixes the position of the poop avatar's lower portion 1036, such that the electronic device shows the upper portion 1034 rotated backward about an axis (e.g., an x-axis) that extends along the width of the lower portion 1036.

[0317] As shown in 1003B, the electronic device detects a user's face 1024 rotated in a downward direction. For example, the user lowers (tucks) their chin 1026 and tilts their head 1028 forward (e.g., toward the focal plane of a camera (e.g., camera 143, optical sensor 164)) to position their face 1024 in a downward direction. In response to detecting the downward position of the user's face 1024, the electronic device modifies the poop avatar to a downward-looking state by tilting the upper portion 1034 of the poop avatar in a direction toward the display (e.g., toward the forward focal plane of the camera (e.g., camera 143, optical sensor 164)), as shown in 1013B. In this forward-tilted position, the electronic device displays the tip 1040 of the poop avatar positioned toward the front of the poop avatar to illustrate the offset position of the tip 1040 when the poop avatar is modified to the downward-looking state. Additionally, the electronic device modifies the poop avatar's eyes 1032 to look downward (e.g., by shifting the poop avatar's pupils or irises 1032A toward the bottom of the avatar's eyes 1032), as shown in 1013B. While the electronic device tilts the poop avatar's upper portion 1034 and modifies the poop avatar's eyes 1032 to look downward, the electronic device leaves other features of the poop avatar unchanged. For example, the electronic device maintains the position of the avatar's mouth 1030 and fixes the position of the poop avatar's lower portion 1036, such that the electronic device shows the upper portion 1034 rotated forward about an axis (e.g., an x-axis) that extends along the width of the lower portion 1036.

[0318] As shown in 1003C, the electronic device detects that the user's position (specifically, the position of the user's face 1024 and head 1028) is in the downwardly rotated position shown in 1003B, but the user's position is also offset downward from the position shown in 1003B within the field of view of the camera (e.g., camera 143, optical sensor 164). In response, the electronic device displays the poop avatar 1000 in the forward-tilted position shown in 1013B, but the poop avatar is also offset downward within the display area in 1013C to mirror the downward direction of the user's offset within the camera's field of view.

[0319] Figure 10D An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics, wherein the modification of the poop avatar includes rotating the upper portion of the poop avatar while keeping the lower portion of the poop avatar stationary. The poop avatar is shown as having four display states (1014A, 1014B, 1014C, and 1014D), wherein each of the four display states of the poop avatar corresponds to the user's four detection states (1004A, 1004B, 1004C, and 1004D). Figure 10DIn each display state of the user, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 10D In the embodiment shown in , the user's physical features include the user's mouth 1020, face 1024, head 1028, and shoulders 1021. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 10D In the embodiment of the present invention, the features of the poop avatar include a mouth 1030 of the avatar, eyes 1032 of the avatar, an upper portion 1034 of the avatar (including a tip 1040 of the poop avatar), and a lower portion 1036 of the avatar.

[0320] As shown in 1004A, the electronic device detects that the user's head 1028 and, optionally, the user's face 1024 (or various physical features including face 1024) are rotated to the user's right, while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the right (while keeping the lower portion 1036 stationary), as shown in 1014A, such that the twisting motion of the avatar mirrors the rightward rotational movement of the user's head 1028 and face 1024. The electronic device also detects a smiling gesture of the user's mouth 1020 and modifies the avatar's mouth 1030 to a smiling state. As shown in 1014A, the electronic device twists the upper portion 1034 of the poop avatar about an axis 1051 (e.g., a y-axis) that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the right, the electronic device also shifts the tip 1040 of the poop avatar to the left, moves the poop avatar's eyes 1032 to the right, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, thereby giving the poop avatar a distorted appearance including a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnected relationship between the upper portion 1034 and the lower portion 1036. Even though parts of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the user's physical features, these modifications to the poop avatar provide an animation effect that mimics the user's physical movements.

[0321] 1004B and 1014B illustrate similar effects, where the electronic device detects that the user's head 1028 and, optionally, the user's face 1024 (or various physical features including the face 1024) are rotated to the user's left, while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the left (while keeping the lower portion 1036 stationary), as shown in 1014B, such that the twisting motion of the virtual avatar mirrors the leftward rotational movement of the user's head 1028 and face 1024. As shown in 1014B, the electronic device twists the upper portion 1034 of the poop avatar about an axis 1051 that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the left, the electronic device also shifts the tip 1040 of the poop avatar to the right, moves the poop avatar's eye 1032 to the left, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, thereby giving the poop avatar a distorted appearance including a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnected relationship between the upper portion 1034 and the lower portion 1036.

[0322] In some embodiments, the electronic device does not track the movement (e.g., rotational movement) or positioning of the user's shoulder 1021, so that the user can affect changes to the virtual avatar without having to maintain a fixed orientation or position in front of a camera (e.g., camera 143, optical sensor 164). For example, as shown in 1004C, the user's shoulder 1021 tilts or rotates to the user's right, but the lower portion 1036 of the poop avatar remains fixed, as shown in 1014C. However, the electronic device detects that the user's head 1028 and optionally the user's face 1024 rotate to the user's right. Therefore, as shown in 1014C, the electronic device modifies the poop avatar accordingly as discussed above with respect to 1014A, without further modifying the poop avatar in response to the user rotating their shoulder 1021. A similar effect is shown in 1004D and 1014D, where the user's shoulder 1021 tilts or rotates to the user's left along with the user's head 1028, and the electronic device modifies the poop avatar as discussed above with respect to 1014B without further modifying the poop avatar (e.g., lower portion 1036) in response to the user rotating their shoulder 1021.

[0323] Figure 10EAn exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical features, wherein the modification to the poop avatar includes tilting the upper portion of the poop avatar while keeping the lower portion of the poop avatar still. The poop avatar is shown as having four display states (1015A, 1015B, 1015C, and 1015D), wherein each of the four display states of the poop avatar corresponds to four detection states of the user (1005A, 1005B, 1005C, and 1005D), respectively. In each display state, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 10E In the embodiment shown in , the user's physical features include the user's mouth 1020, face 1024, head 1028, shoulders 1021, eyes 1023, and neck 1025. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 10E In the embodiment of the present invention, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, an upper portion 1034 of the avatar (including a tip 1040 and a middle portion 1031 of the poop avatar), and a lower portion 1036 of the avatar.

[0324] As shown in 1005A, the electronic device detects that the user's head 1028 and, optionally, the user's face 1024 (or various physical features including face 1024) are tilted to the user's right while the user's shoulders 1021 remain positioned forward. The electronic device also detects that the user's neck 1025 is tilted slightly to the user's right. In response, the electronic device modifies the poop avatar by tilting the upper portion 1034 of the poop avatar to the right (the upper portion 1034 of the poop avatar includes the tip 1040 and the middle portion 1031) while the lower portion 1036 remains stationary, as shown in 1015A, so that the tilting motion of the virtual avatar mirrors the rightward tilt of the user's head 1028 (and / or face 1024) and neck 1025.

[0325] In addition to mirroring the tilt direction, the electronic device also modifies the virtual avatar to account for the different degrees of tilt present in the various physical features of the user shown in 1005A. For example, the upper part of the user (e.g., the user's head 1028) is tilted to a greater extent than the lower part of the user (e.g., the user's neck 1025). Therefore, as shown in 1015A, the electronic device modifies the virtual avatar so that the movement or tilt is greatest at the top of the virtual avatar (e.g., at the tip 1040) and the movement or tilt is smallest at the bottom of the virtual avatar (e.g., at the bottom 1036). In other words, the tilt gradually decreases from the top of the virtual avatar to the bottom of the virtual avatar, which is consistent with the change in tilt shown in the user in 1005A. This is demonstrated by the tip 1040 of the poop avatar in 1015A having a large tilt, the middle part 1031 having a smaller tilt than the tip 1040, and the lower part 1036 not tilted. Even though portions of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the user's physical features, these modifications to the poop avatar provide an animation effect that mimics the user's physical movements. Furthermore, for example, when the user tilts their head 1028 and neck 1025, the electronic device modifies the virtual avatar to have different degrees of tilt to mimic the reduced range of motion of the user's physical features.

[0326] 1005B and 1015B illustrate a similar effect, where the electronic device detects that the user's head 1028 and neck 1025 are tilted to the user's left. In response, the electronic device modifies the poop avatar by tilting the upper portion 1034 of the poop avatar to the left at different degrees of tilt (e.g., the tip 1040 is tilted to a greater degree than the middle portion 1031), while keeping the lower portion 1036 stationary, as shown in 1015B, so that the tilting motion of the virtual avatar mirrors the leftward tilt of the user's head 1028 and neck 1025, as discussed in more detail with respect to 1005A and 1015A.

[0327] As shown in 1005C, the electronic device detects the user's eye 1023 (e.g., the iris or pupil of the user's eye) shifted to the user's right and a smiling facial expression formed by the corners 1020A and 1020B of the user's mouth 1020 positioned in an upward posture. In response to detecting the rightward shift of the user's eye 1023, the electronic device modifies the poop avatar's eye 1032 to a rightward-looking state (e.g., by shifting the poop avatar's pupil or iris 1032A toward the right side of the avatar's eye 1032), as shown in 1015C. In response to detecting one or both of the corners 1020A and 1020B of the user's mouth positioned in an upward posture, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, wherein the avatar's mouth 1030 has an open, smiling position, as shown in 1015C. As shown in 1015C, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A).

[0328] 1005D and 1015D illustrate similar effects, where the electronic device detects a smiling facial expression of the user and the user's eye 1023 (e.g., the iris or pupil of the user's eye) shifted to the user's left. In response to detecting the leftward shift of the user's eye 1023, the electronic device modifies the poop avatar's eye 1032 to a leftward-looking state (e.g., by shifting the poop avatar's pupil or iris 1032A toward the left of the avatar's eye 1032), as shown in 1015D. In response to detecting one or both of the corners 1020A and 1020B in an upward pose, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, as discussed above with respect to 1015C. Similarly, in 1015D, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A and 1015C).

[0329] Figure 10F An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a shift in the position of a user within the field of view of a camera (e.g., camera 143, optical sensor 164). Modifying the poop avatar includes shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user within the field of view of the camera. The poop avatar is shown as having four display states (1016A, 1016B, 1016C, and 1016D) in four display areas, wherein each of the four display states of the poop avatar corresponds to the four detection states (1006A, 1006B, 1006C, and 1006D) of the user, respectively. Figure 10FIn each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting a position or change in position of the user detected within the camera's field of view displayed in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., boundaries 1016A, 1016B, 1016C, and 1016D) represent boundaries of the display area that includes the virtual avatar.

[0330] As shown in 1006A, the electronic device detects that the user's position is horizontally centered within the field of view of a camera (e.g., camera 143, optical sensor 164). In response to detecting the user's horizontally centered position within the camera's field of view, the electronic device displays a poop avatar with a horizontally centered position within the display area as shown in 1016A.

[0331] In 1006B, the electronic device detects that the user's position is off-center (e.g., offset or translated) in a rightward direction within the field of view of a camera (e.g., camera 143, optical sensor 164). In other words, the user is offset to the user's left (e.g., offset to the right relative to the camera's field of view), but remains fully visible within the camera's field of view. In response to detecting the user's offset position in 1006B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the poop avatar is displayed at a position offset to the left, as shown in 1016B, in order to mirror the user's directional offset to the user's left direction. As shown in 1006B, the user is offset to their left, with their left shoulder 1021A near the right edge of the field of view. As a result, the electronic device displays the poop avatar near the left edge of the display area 1016B, which mirrors the direction of the user's offset position within the camera's field of view. In some embodiments, the user's offset position is mirrored by shifting the virtual avatar in both direction and magnitude. In some embodiments, the user's offset position is mirrored by shifting the virtual avatar only in direction, and the magnitude of the virtual avatar's offset is adjusted (e.g., attenuated) to maintain the virtual avatar's position within the boundaries of the display area. Examples of such embodiments are discussed below with respect to 1006C, 1016C, 1006D, and 1016D.

[0332] In 1006C, the electronic device detects that the user's position is off-center (e.g., offset or translated) in the rightmost direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is offset far to the user's left (e.g., offset to the right relative to the camera's field of view), but to such an extent that the user's left shoulder 1021A is no longer within the camera's field of view. In response to detecting the user's significantly offset position in 1006C, the electronic device offsets the horizontal position of the poop avatar so that the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) is displayed in a fully leftwardly offset position, wherein the outermost edge of the poop avatar (e.g., edge 1036A of the lower portion 1036) is positioned against the left boundary of the display area shown in 1016C. The offset display of the poop avatar in 1016C mirrors the direction of the user's offset to the left of the user, but rather than offsetting the poop avatar so that a portion of the avatar extends beyond the display area in 1016C (as the user did in 1006C), the device positions the poop avatar at the edge of the display area 1016C. By maintaining the position of the virtual avatar within the display area (e.g., 1016A, 1016B, 1016C, and 1016D) even when a portion of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect changes to the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0333] A similar effect is shown in 1006D and 1016D. In 1006D, the electronic device detects that the user has shifted to the right (e.g., to the left in the camera's field of view), such that the user's right shoulder is no longer within the camera's field of view. In response to detecting the user's far-off position, the electronic device shifts the horizontal position of the poop avatar so that the entire poop avatar (including both upper portion 1034 and lower portion 1036) is displayed in a fully right-offset position, with the outermost edge of the poop avatar (e.g., edge 1036B of lower portion 1036) positioned against the right boundary of the display area shown in 1016D. As described above, by maintaining the position of the virtual avatar within display area 1016D even when a portion of the user is outside the field of view of a camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect changes to the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0334] Figure 10G An exemplary embodiment is shown in which the electronic device modifies the poop avatar in response to detecting a shift in the position of a user's physical feature within the field of view of a camera (e.g., camera 143, optical sensor 164). Modifying the poop avatar includes shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user's physical feature within the field of view of the camera. Figure 10G The embodiment shown in is similar to the embodiment described above with respect to Figures 10D to 10F In the embodiments discussed, the electronic device tracks the movement and position (e.g., rotational movement and / or translational movement) of the user's head 1028, but does not track the movement or position of the user's shoulders 1021 and optionally the user's neck 1025. Figure 10G The embodiment shown is similar to Figure 10F Embodiments in which modifications to the virtual avatar mirror the user's movement in direction, but not necessarily in magnitude. By implementing these techniques, the electronic device allows the user to affect changes in the virtual avatar without having to maintain a fixed orientation or position in front of a camera (e.g., camera 143, optical sensor 164).

[0335] The poop avatar is shown with two display states (1017A and 1017B) in two display areas, wherein each of the two display states of the poop avatar corresponds to the two detection states of the user (1007A and 1007B). Figure 10G In each of the two display states, the device positions or modifies the poop avatar in response to detecting a position or change in position of a physical feature of the user detected within the camera's field of view that is displayed in the user's corresponding state. In each of the two display states, the boundaries of the displayed state (e.g., the boundaries of 1017A and 1017B) represent the boundaries of the display area that includes the virtual avatar.

[0336] In 1007A, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset (e.g., translated) in a leftward direction within the field of view of a camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset in a rightward direction for the user (e.g., offset to the left relative to the field of view of the camera). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007A, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the poop avatar is displayed in a rightwardly offset position, as shown in 1017A, to mirror the offset of the user's head to the user's rightward direction.

[0337] A similar effect is shown in 1007B and 1017B. In 1007B, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset (e.g., translated) in a rightward direction within the field of view of a camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset to the user's left (e.g., offset to the right relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the poop avatar is displayed at a position offset to the left, as shown in 1017B, to mirror the offset of the user's head to the user's left.

[0338] Figure 10H An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in the position of a user's physical features within the field of view of a camera (e.g., camera 143, optical sensor 164). Modifications to the poop avatar include increasing or decreasing the size of the poop avatar, and shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user's physical features within the field of view of the camera. The poop avatar is shown as having four display states (1018A, 1018B, 1018C, and 1018D) in four display areas, wherein each of the four display states of the poop avatar corresponds to four detection states (1008A, 1008B, 1008C, and 1008D) of the user, respectively. In Figure 10H In each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting a position or change in position of a physical feature of the user detected within the camera's field of view that is displayed in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., boundaries 1018A, 1018B, 1018C, and 1018D) represent boundaries of a display area that includes the virtual avatar.

[0339] In 1008A, the electronic device detects that the user's head 1028 is offset (e.g., translated) in an upward direction relative to the user's shoulder 1021 in the field of view of a camera (e.g., camera 143, optical sensor 164) (e.g., the user is stretching their neck 1025 upward). In response to detecting the upward offset position of the user's head 1028 in 1008A, the electronic device offsets the vertical position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the entire poop avatar is displayed in an upward offset position, wherein the uppermost edge of the poop avatar (e.g., edge 1040A of the tip 1040) is positioned near the upper boundary of the display area shown in 1018A to mirror the offset of the user's head 1028 in the upward direction shown in 1008A.

[0340] In 1008B, the electronic device detects that the user's head 1028 is offset (e.g., translated) in a downward direction relative to the user's shoulder 1021 in the field of view of a camera (e.g., camera 143, optical sensor 164) (e.g., the user is lowering their head 1028). In response to detecting the downward offset position of the user's head 1028 in 1008B, the electronic device offsets the vertical position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the entire poop avatar is displayed in a downward offset position, wherein the lowermost edge of the poop avatar (e.g., edge 1036C of the lower portion 1036) is positioned near the lower boundary of the display area shown in 1018B to mirror the offset of the user's head 1028 in the downward direction shown in 1008B.

[0341] In 1008C, the electronic device detects that the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) increases, for example, when the user's head 1028 is positioned closer to the camera. In response to detecting the increase in size of the user's head 1028 in 1008C, the electronic device increases the size of the entire poop avatar. In some embodiments, the electronic device changes the size of the user's head 1028 from one detected state (e.g., Figure 10A The electronic device increases the size of the poop avatar by changing its size from the neutral state in 1001A of the electronic device (e.g., the detected state 1008C). In 1018C, the electronic device increases the size of the poop avatar to fill the display area without causing a portion of the poop avatar to extend beyond the boundaries of the display area. In some embodiments, the electronic device increases the size of the virtual avatar to provide the impression that the avatar is positioned very close to the display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504).

[0342] For example, in 1018C, the electronic device increases the size of the poop avatar so that the tip 1040 of the poop avatar is adjacent to the upper border of the display area at 1040A, the lower portion 1036 of the poop avatar is adjacent to the lower border of the display area at 1036C, the left edge of the lower portion 1036 is adjacent to the left border of the display area at 1036A, and the right edge of the lower portion 1036 is adjacent to the right border of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018C, the electronic device proportionally increases the size of the poop avatar so that the relative positions of various avatar features (e.g., the avatar's eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) are not distorted relative to the shape of the poop avatar. For example, when the electronic device increases the size of the poop avatar in 1018C, the sizes of the avatar's eyes 1032, mouth 1030, upper portion 1034 (including tip 1040), and lower portion 1036 also increase, but the other parts remain unchanged.

[0343] In 1008D, the electronic device detects that the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) decreases, for example, when the user's head 1028 is positioned away from the camera. In response to detecting the size reduction of the user's head 1028 in 1008D, the electronic device reduces the size of the entire poop avatar. In some embodiments, the electronic device changes the size of the user's head 1028 from one detected state (e.g., Figure 10A In some embodiments, such as in 1018D, the electronic device reduces the size of the poop avatar to provide the impression that the virtual avatar is positioned away from the display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504).

[0344] For example, in 1018D, the electronic device reduces the size of the poop avatar so that the tip 1040 of the poop avatar is positioned away from the upper boundary of the display area at 1040A, the lower portion 1036 of the poop avatar is positioned away from the lower boundary of the display area at 1036C, the left edge of the lower portion 1036 is positioned away from the left boundary of the display area at 1036A, and the right edge of the lower portion 1036 is positioned away from the right boundary of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018D, the electronic device proportionally reduces the size of the poop avatar so that the relative positions of various avatar features (e.g., the avatar's eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) are not distorted relative to the shape of the poop avatar. For example, when the electronic device reduces the size of the poop avatar in 1018D, the sizes of the avatar's eyes 1032, mouth 1030, upper portion 1034 (including tip 1040), and lower portion 1036 are also reduced, but the other parts remain unchanged.

[0345] Figure 10I An exemplary embodiment is shown illustrating that an electronic device modifies a poop avatar 1000 in response to detecting a change in a user's physical features, such as facial features. The poop avatar is shown as having one display state 1019 corresponding to the user's detection state 1009. As shown in 1009 and 1019, the electronic device detects that the user is making a pouting expression, and in response, the electronic device modifies the poop avatar 1000 by replacing the poop avatar's mouth 1030 with a set of pouting lips 1050. In some embodiments, such as the embodiment shown in 1009, the electronic device determines that the user is making a pouting facial expression by detecting that the user's jaw 1027 is in a closed position, and detecting that the corners 1020A and 1020B of the user's mouth 1020 move toward each other so that the user's lips 1029 (e.g., the user's upper lip 1029A and lower lip 1029B) extend outward from the user's mouth 1020 in a pouting posture. Although not shown in FIG. Figure 10I However, in some embodiments, the electronic device modifies the poop avatar so that it appears similar to Figure 16B Hearts are emitted from the pouty lips 1050 in the manner shown in and discussed in more detail below.

[0346] Figure 11A An exemplary embodiment is shown illustrating that an electronic device modifies a bear avatar 1100 in response to detecting a change in a user's facial features. The bear avatar is shown as having four display states (1111A, 1111B, 1111C, and 1111D), wherein each of the four display states of the bear avatar corresponds to the four detection states of the user (1101A, 1101B, 1101C, and 1101D). Figure 11AIn each display state of the avatar, the electronic device positions or modifies features of the bear avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Figure 11A In the illustrated embodiment, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 11A In the embodiment of the present invention, the features of the bear avatar include an avatar mouth 1130, an avatar eyes 1132, an avatar ears 1133, an avatar nose 1137, an avatar head 1135, and (in some cases) avatar eyebrows 1138.

[0347] As shown in 1101A, the electronic device detects a neutral facial expression of the user. For example, the device detects that the user's mouth 1120 and eyebrows 1122 are positioned in a relaxed, neutral state, rather than in positions associated with a particular facial expression (such as a smile or a frown). In response to detecting the user's neutral facial expression, the electronic device displays a bear avatar 1100 in 1111A with a neutral expression or state. Specifically, the electronic device displays the bear avatar with the avatar's mouth 1130 in a relaxed, neutral state, rather than in a position typically associated with a particular facial expression (e.g., a smile or a frown). In some embodiments, the avatar's neutral position corresponds to other representations of related imagery, such as a static bear emoji that can be found in messaging applications. In 1111A, the neutral state of the bear's mouth 1130 is indicated by lines 1130A and 1130B, which extend horizontally from the outside of the bear's mouth 1130 or "muzzle" area and then curve slightly upward at the base of the bear's nose 1137. The electronic device also displays a bear head avatar having a nose 1137 and ears 1133, wherein the nose is located above a relaxed mouth 1130, and the ears are in a relaxed neutral state, positioned along the side of the bear's head 1135, and are not curled or stretched. In addition, the electronic device displays the bear head avatar without eyebrows 1138 and displays the avatar's eyes 1132 looking in a direction perpendicular to the display (or the focal plane of a camera (e.g., camera 143, optical sensor 164)).

[0348] As shown in 1101B and 1111B, the electronic device detects the position of the user's mouth 1120 (e.g., detects that one or both of the corners 1120A and 1120B of the user's mouth are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression), and in response, the electronic device modifies two features of the bear avatar. For example, in response to detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture, the electronic device modifies both the bear ears 1133 and the bear's mouth 1130. The electronic device modifies the bear's mouth 1130 to have a smiling expression by rotating lines 1130A and 1130B upward, with the upward rotation of lines 1130A and 1130B indicating the smiling expression of the bear's mouth 1130, as shown in 1111B. The electronic device modifies the bear's ears 1133 to be "perched up" or extended in an upward direction. In some embodiments, such as the embodiment shown in 1111B, the modification of the bear ears 1133 also includes slightly narrowing the width of each ear 1133 and changing the vertical position of the ears 1133 on the side of the bear's head 1135, so that when the ears are raised, the ears 1133 are positioned higher on the bear's head 1135 (compared to the vertical position of the ears 1133 on the side of the bear's head 1135 in the neutral state in 1111A). As shown in 1111B, the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user smiles, the electronic device changes the bear's mouth 1130 to the smiling state and causes the ears 1133 to rise upward. When the user stops smiling, the electronic device returns the mouth 1130 and ears 1133 to their neutral positions.

[0349] As shown in 1101C, the electronic device detects that the user's eyebrows 1122 are in a raised position (e.g., positioned higher on the user's face than the position of eyebrows 1122 when in the relaxed neutral state shown in 1101A and 1101B). In response to detecting the raised eyebrows 1122, the electronic device modifies two features of the bear avatar. For example, the electronic device modifies the bear's ears 1133 to point upward, and modifies the bear avatar to introduce eyebrows 1138 located above the bear's eyes 1132, thereby conveying the impression that the bear avatar 1100 is raising its eyebrows 1138, as shown in 1111C. As shown in 1111B, the bear's mouth 1130 returns to its neutral position, and the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's eyebrows 1122 are raised, the electronic device introduces the avatar's eyebrows 1138 and causes the ears 1133 to point upward. Thus, when the user's eyebrows 1122 return to their neutral position, the electronic device removes the avatar's eyebrows 1138 and relaxes the ears 1133.

[0350] In 1101D, the electronic device detects that the user's eyebrows 1122 are raised and one or both corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture. In response, the electronic device modifies the bear avatar 1100 to achieve a significantly erected ear, introduces a raised eyebrow 1138, and causes the bear to smile, as shown in 1111D. The electronic device modifies the ears 1133 to achieve the significantly erected ear position by significantly reducing the width of each ear 1133, extending the ear in an upward direction, and changing the vertical position of the ear 1133 on the side of the bear's head 1135 so that the ear 1133 is positioned even higher on the bear's head 1135 than in the erected position shown in 1111B and 1111C. This combination of modifications to the bear's ears 1133 results in the appearance of the ears 1133 being significantly stretched in an upward direction, a degree of stretching greater than the erected ear position shown in 1111B and 1111C. It should be understood that in this embodiment, when the user raises their eyebrows 1122 and smiles (e.g., one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture), the electronic device introduces the avatar's eyebrows 1138, modifies the bear's mouth 1130 into a smiling state, and greatly erects the ears 1133. Therefore, when the user's eyebrows 1122 and mouth 1120 return to their neutral positions, the electronic device removes the avatar's eyebrows 1138 and relaxes the ears 1133 and mouth 1130.

[0351] Figure 11B An exemplary embodiment illustrating that an electronic device modifies a bear avatar 1100 in response to detecting a change in a user's facial features is shown. The bear avatar is shown as having four display states (1112A, 1112B, 1112C, and 1112D), wherein each of the four display states of the bear avatar corresponds to the user's four detection states (1102A, 1102B, 1102C, and 1102D). Figure 11B In each of the four display states, the electronic device positions or modifies features of the bear avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., boundaries of 1112A, 1112B, 1112C, and 1112D) represent boundaries of a display area that includes the virtual avatar.

[0352] exist Figure 11BIn the illustrated embodiment, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Figure 11B In the embodiment of the present invention, the features of the bear avatar include an avatar mouth 1130, an avatar eyes 1132, an avatar ears 1133, an avatar nose 1137, an avatar head 1135, and (in some cases) avatar eyebrows 1138.

[0353] As shown in 1102A, the electronic device detects that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture (e.g., a downward position), and in response, the electronic device modifies two features of the bear avatar. In some embodiments, the electronic device determines that the user is making a sad facial expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth are in a downward position, and optionally detecting that one or both of the user's eyebrows 1122 are not in a lower position (e.g., eyebrows 1122 are raised or in their neutral position). In response to detecting the downward posture or position of the corners 1120A and 1120B of the user's mouth 1120, the electronic device modifies both the bear mouth 1130 and the bear ears 1133, as shown in 1112A.

[0354] The electronic device modifies the bear's mouth 1130 to have a sad expression by rotating lines 1130A and 1130B downward, which forms the sad expression of the bear's mouth 1130, as shown in 1112A. The electronic device modifies the bear's ears 1133 to "curl" or "droop" in a downward direction by rotating the outer edges 1133A and 1133B of the bear's ears 1133 downward (e.g., folding, rotating, or tilting). In some embodiments, curling the bear's ears 1133 also includes changing the vertical position of the ears 1133 on the sides of the bear's head 1135, so that when the ears are curled, the ears 1133 are positioned lower on the bear's head 1135 (compared to being in a downward position). Figure 11A (Compared to the vertical position of the ears 1133 on the side of the bear's head 1135 in the neutral state in 1111A.) As shown in 1112A, the bear's eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's mouth 1120 forms a sad facial expression, the electronic device changes the bear's mouth 1130 to a sad expression and curls up the bear's ears 1133. When the user no longer makes the sad facial expression, the electronic device returns the bear's mouth 1130 and ears 1133 to their neutral positions.

[0355] In 1102B and 1112B, the electronic device detects that the user has maintained a sad facial expression for a threshold amount of time, and in response, the electronic device modifies the bear avatar 1100 by introducing tears 1140 emanating from the bear's eyes 1132 to display the bear avatar as crying when the bear avatar makes a sad facial expression with curled ears 1133. In some embodiments, when the user maintains the sad expression for a first threshold amount of time (e.g., 0.3 seconds), the electronic device modifies the bear avatar to introduce only a single tear 1140 emanating from one or both of the bear's eyes 1132, and then, after the user continues to maintain the sad expression for a second threshold amount of time (e.g., 0.6 seconds), the electronic device modifies the bear avatar to introduce additional tears 1140 emanating from the eyes 1132. The electronic device displays the tears 1140 forming from the bear's eyes 1132, flowing down the bear's face, dripping from the bear's head 1135, and disappearing from view at the bottom edge of the display area in 1112B. In some embodiments, the electronic device displays tears 1140 intermittently and optionally at random intervals. In some embodiments, tears 1140 continue to appear until the user stops making the sad facial expression. As shown in 1112B, the electronic device continues to display the curled bear ears 1133 and continues to display the bear's mouth 1130 making a sad expression.

[0356] In 1102C and 1112C, the electronic device detects that the user has maintained a frowning facial expression for a threshold amount of time, and in response, the electronic device modifies the bear avatar 1100 to have a frowning expression and storm clouds 1142 located above the bear's head 1135. The electronic device detects the user's frowning expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture and detecting that the user's eyebrows 1122 are in a downward position (e.g., the user's eyebrows are wrinkled or positioned lower on the user's face compared to the position of eyebrows 1122 in the relaxed neutral state in 1101A). In response to detecting the user's frowning facial expression, the electronic device modifies the bear avatar by rotating the bear's mouth 1130 downward (e.g., by bending lines 1130A and 1130B downward, as shown in 1112C) and introducing the bear's eyebrows 1138 in a wrinkled state to form a frowning eyebrow posture (as shown in 1112C). In the embodiment shown in 1112B, the electronic device does not immediately introduce storm clouds 1142 in response to detecting the user's frowning facial expression, nor does the electronic device modify the bear's ears 1133 or nose 11311.

[0357] After the electronic device detects that the user maintains a frowning facial expression for a threshold amount of time, the electronic device modifies the frowning bear avatar 1100 to introduce storm clouds 1142 located above the bear's head 1135. In some embodiments, the storm clouds 1142 are animated and dynamic. For example, the electronic device can modify the storm clouds 1142 to introduce lightning 1144 and / or rain. In some embodiments, the electronic device displays rain falling from the storm clouds 1142 and landing on the bear avatar's head 1135. In some embodiments, the falling rain flows down the bear's face, falls from the bear's head 1135, and disappears from view at the bottom edge of the display area in 1112C. In some embodiments, the electronic device displays lightning 1144 and rain intermittently and optionally at random ...

Claims

1. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different from the first range of change of the second physical feature; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature, modifying the first avatar feature to reflect the change in the first physical feature; and Based on determining that the first change in the corresponding physical feature detected is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or the second change range.

2. The method according to claim 1, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and An appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to an appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

3. The method according to claim 1, further comprising: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

4. The method according to claim 1, further comprising: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in the possible variation range of the second physical characteristic; the first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

5. The method according to claim 1, further comprising: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

6. The method of claim 5, wherein the first manner of changing the appearance of the second avatar feature is different from the second manner of changing the appearance of the second avatar feature.

7. The method according to claim 5, further comprising: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

8. The method according to claim 5, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes the avatar's eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the change in the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

9. The method according to claim 5, wherein: Modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic includes modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

10. The method according to claim 5, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical characteristic.

11. An electronic device comprising: one or more cameras; display device; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face, the manner depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different from the first range of change of the second physical feature; detecting, when displaying the virtual avatar, a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature, modifying the first avatar feature to reflect the change in the first physical feature; and Based on determining that the first change in the corresponding physical feature detected is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or the second change range.

12. The electronic device according to claim 11, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and An appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to an appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

13. The electronic device of claim 11, wherein the one or more programs further comprise instructions for performing the following operations: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

14. The electronic device according to claim 11, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in the possible variation range of the second physical characteristic; the first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

15. The electronic device of claim 11, wherein the one or more programs further comprise instructions for performing the following operations: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

16. The electronic device of claim 15, wherein the first manner of changing the appearance of the second avatar feature is different from the second manner of changing the appearance of the second avatar feature.

17. The electronic device according to claim 15, wherein: Modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic includes modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

18. The electronic device according to claim 15, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical characteristic.

19. The electronic device of claim 15, wherein the one or more programs further comprise instructions for performing the following operations: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

20. The electronic device according to claim 19, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes the avatar's eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the change in the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

21. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different from the first range of change of the second physical feature; detecting, while displaying the virtual avatar, a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature, modifying the first avatar feature to reflect the change in the first physical feature; and Based on determining that the first change in the corresponding physical feature detected is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or the second change range.

22. The non-transitory computer-readable storage medium of claim 21, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and An appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to an appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

23. The non-transitory computer-readable storage medium of claim 21 , wherein the one or more programs further comprise instructions for: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

24. The non-transitory computer-readable storage medium of claim 21, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in the possible variation range of the second physical characteristic; the first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

25. The non-transitory computer-readable storage medium of claim 21, wherein the one or more programs further comprise instructions for: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

26. The non-transitory computer-readable storage medium of claim 25, wherein the first manner of changing the appearance of the second avatar feature is different from the second manner of changing the appearance of the second avatar feature.

27. The non-transitory computer-readable storage medium of claim 25, wherein: Modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic includes modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

28. The non-transitory computer-readable storage medium of claim 25, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical characteristic; and Changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical characteristic includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical characteristic.

29. The non-transitory computer-readable storage medium of claim 25, wherein the one or more programs further comprise instructions for: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

30. The non-transitory computer-readable storage medium of claim 29, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes the avatar's eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the change in the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

31. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and upon determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the avatar cheek, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature includes modifying the avatar cheek based on the change magnitude of the user's cheek.

32. An electronic device comprising: one or more cameras; display device; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and upon determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the avatar cheek, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature includes modifying the avatar cheek based on the change magnitude of the user's cheek.

33. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; detecting, while displaying the virtual avatar, a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the first change range, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical characteristic; and Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical feature; wherein changing the appearance of the second avatar feature in the second manner within the second change range based on the change in the second physical feature includes introducing an exaggerated avatar effect at the avatar cheeks, and changing the appearance of the second avatar feature in the first manner within the first change range based on the change in the second physical feature includes modifying the avatar cheeks based on the change magnitude of the user's cheeks.

34. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and upon determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

35. An electronic device comprising: one or more cameras; display device; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and upon determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

36. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs comprising instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar characteristic responsive to changes in a first physical characteristic of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; detecting, while displaying the virtual avatar, a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the first change range, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical characteristic; and upon determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, forgoing changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

Images