Emoticon recording and sending

Abstract

The invention relates to emoticon recording and sending. The present disclosure generally relates to generating and modifying a virtual avatar. An electronic device having a camera and a display device displays a virtual avatar whose appearance changes in response to a change in a face in a field of view of the camera. In response to detecting a change in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of January 23, 2018, the national application number of 201880004632.1, and the invention name of “Emoji Recording and Sending”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application claims priority to U.S. Provisional Application No. 62 / 507,177, filed on May 16, 2017, entitled “Emoji Recording and Sending,” U.S. Provisional Application No. 62 / 556,412, filed on September 9, 2017, entitled “Emoji Recording and Sending,” and U.S. Provisional Application No. 62 / 557,121, filed on September 11, 2017. This patent application also claims priority to Danish Patent Application No. PA201770393, filed on May 29, 2017, entitled “Emoji Recording and Sending,” Danish Patent Application No. PA201770720, filed on September 22, 2017, entitled “Emoji Recording and Sending,” and Danish Patent Application No. PA201770721, filed on September 22, 2017, entitled “Emoji Recording and Sending.” The contents of these patent applications are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to computer user interfaces, and more particularly to techniques for generating, recording, and sending emoticons and avatars. Background Art

[0005] Multimedia content (such as emojis and avatars) is sometimes sent as part of a messaging communication. Emojis and avatars represent a variety of predefined people, objects, actions, and / or other things. Some messaging applications allow users to select from a library of predefined emojis and avatars and send them as part of a message that may contain other content (e.g., other multimedia and / or text content). Stickers are another type of multimedia content that is sometimes sent with messaging applications. In some respects, stickers are similar to emojis and avatars in that they can represent people, objects, actions, and / or other things. Some sticker and / or messaging applications allow stickers to be associated with previously sent or received messages. Summary of the invention

[0006] However, some techniques for generating, sending and receiving emojis and virtual avatars using electronic devices are generally cumbersome and inefficient. For example, some prior art techniques use complex and time-consuming user interfaces that may include multiple keystrokes or keystrokes. Some other prior art techniques use complex and time-consuming methods to manipulate and generate emojis and virtual avatars, including requiring the user to provide a large amount of input to achieve a desired emoji (e.g., a desired animated emoji or dynamic emoji). Prior art techniques require more time than required, which results in a waste of user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, the present technology provides a faster and more efficient method and interface for electronic devices to send and receive emoticons and virtual avatars. Such methods and interfaces optionally supplement or replace other methods for sending and receiving emoticons. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0008] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording affordance in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera field of view over a time period, wherein the time period is determined based on the timing of the input.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera field of view over a time period, wherein the time period is determined based on the timing of the input.

[0010] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera field of view over a time period, wherein the time period is determined based on the timing of the input.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; a camera; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in facial appearance in the camera field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera field of view within a time period, wherein the time period is determined based on the timing of the input.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera field of view; a device for the following operations: while displaying a preview of the virtual avatar, detecting input in the virtual avatar generation interface; and a device for the following operations: in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing facial expressions in the camera field of view at a corresponding time based on determining that input begins on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the animated virtual avatar represents a sequence of changes in facial expressions in the camera field of view over a time period, wherein the time period is determined based on the timing of the input.

[0013] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar reacts to changes in the appearance of the face in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence twice or more.

[0014] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of the face in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence twice or more.

[0015] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of the face in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence twice or more.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of the face in the camera field of view; receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; after recording the facial expressions of the face in the camera field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence twice or more.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the facial appearance in the camera field of view; a device for the following operations: receiving a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view; a device for the following operations: in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera field of view; a device for the following operations: after recording the facial expression of the face in the camera field of view, causing a looped version of the animated virtual avatar to be displayed, the looped version including an animation sequence based on the facial expression sequence recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence twice or more.

[0018] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; while displaying the preview of the virtual avatar, detecting input in the virtual avatar generation interface; in response to detecting input in the virtual avatar generation interface: based on determining that input begins on the preview of the virtual avatar, generating a static virtual avatar representing facial expressions in the camera's field of view at a corresponding time, wherein the corresponding time is determined based on the timing of the input; and based on determining that the input includes activation of a recording enable indication in the virtual avatar generation interface, generating an animated virtual avatar, the animated virtual avatar representing a sequence of changes in facial expressions in the camera's field of view over a time period, wherein the time period is determined based on the timing of the input.

[0019] In some embodiments, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of the virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; receiving a request to generate an animated virtual avatar based on changing facial expressions of a face in the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0020] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, the virtual avatar changing appearance in response to changes in a face in a field of view of the one or more cameras, wherein the virtual avatar includes: a first part and a second part different from the first part; while displaying the virtual avatar via the display device, detecting changes in facial posture within the field of view of the one or more cameras; in response to detecting the changes in facial posture, changing the appearance of the virtual avatar, which includes: based on determining that the change in facial posture includes a first type of change in facial posture, changing the appearance of the virtual avatar includes moving the first part of the virtual avatar relative to the second part of the virtual avatar according to a magnitude of the first type of change in facial posture; and based on determining that the change in facial posture includes a second type of change in facial posture, changing the appearance of the virtual avatar includes moving the first part of the virtual avatar and the second part of the virtual avatar based on the magnitude of the second type of change in facial posture.

[0021] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responding to changes in a first physical feature of a face in a field of view of the one or more cameras and a second physical feature of a face in a field of view of the one or more cameras; and a second avatar feature; while displaying the virtual avatar via the display device, detecting changes in one or more physical features of a face in a field of view of the one or more cameras; based on determining that the changes include changes in the first physical feature: modifying the first avatar feature of the virtual avatar based on the changes in the first physical feature, and abandoning modifying the second avatar feature based on the changes in the first physical feature; and based on determining that the changes include changes in the second physical feature: modifying the first avatar feature based on the changes in the second physical feature, and abandoning modifying the second avatar feature based on the changes in the second physical feature;

[0022] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responds to a change in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature, the second avatar feature responds to a change in the first physical feature; and a third avatar feature, the third avatar feature does not primarily respond to a change in the first physical feature; while displaying the virtual avatar, detecting a change in the first physical feature; and in response to detecting the change in the first physical feature: modifying the first avatar feature based on the detected change in the first physical feature; modifying the second avatar feature based on the detected change in the first physical feature; and abandoning modification of the third avatar feature based on the detected change in the first physical feature.

[0023] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responding to a change in a first physical feature of the face within a field of view of the one or more cameras; a second avatar feature, the second avatar feature responding to a change in a second physical feature of the face in a different manner, the manner depending on whether the change in the second physical feature of the face occurs within a first change range of the second physical feature or within a second change range of the second physical feature, the second change range of the second physical feature being different from the first change range of the second physical feature; while displaying the virtual avatar, detecting the one or more cameras A first change in a corresponding physical feature of a face within a field of view; and in response to detecting the first change in the corresponding physical feature, modifying the virtual avatar, which includes: based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature, modifying the first avatar feature to reflect the change in the first physical feature; and based on determining that the detected first change is a change in the second physical feature and the change in the second physical feature is within a first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; based on determining that the detected first change is a change in the second physical feature and the change in the second physical feature is within a second range of change, abandoning changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature.

[0024] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes multiple avatar features that react to changes in one or more physical features of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting changes in multiple physical features of the face, the multiple physical features of the face including a first physical feature corresponding to one or more of the multiple avatar features and a second physical feature that does not correspond to any of the multiple avatar features; and in response to detecting changes in multiple physical features of the face: changing the appearance of corresponding avatar features of the multiple avatar features, wherein the magnitude and / or direction of the change in the corresponding avatar feature is based on the magnitude or direction of the change in the first physical feature; and before detecting changes in one or more physical features of the face, deforming a portion of the virtual avatar that does not include the avatar features, wherein the magnitude and / or direction of the deformation of a portion of the avatar features is based on the magnitude and / or direction of the change in the second physical feature.

[0025] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature, the first avatar feature responding to a change in a first physical feature of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting a first physical feature change having a first physical feature change magnitude; in response to detecting the change in the first physical feature: changing the first avatar feature by a first avatar feature change magnitude based on the first physical feature change magnitude based on a determination that the change in the first physical feature is within a first physical feature value range; and changing the first avatar feature by a second avatar feature change magnitude based on a determination that the change in the first physical feature is within a second physical feature value range different from the first physical feature value range, the second avatar feature change magnitude being different from the first avatar feature change magnitude and being based on the first physical feature change magnitude.

[0026] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar has a corresponding spatial position within a reference system, wherein the corresponding spatial position is based on the position of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting a corresponding quantitative change in the position of the face within the field of view of the one or more cameras; in response to detecting a change in the position of the face within the field of view of the one or more cameras: modifying the spatial position of the virtual avatar within the reference system based on the magnitude of the first change component and a first modification factor based on determining that the change in the position of the face includes a first change component in a first direction; and modifying the spatial position of the virtual avatar within the reference system based on the magnitude of the second change component and a second modification factor based on determining that the change in the position includes a second change component in a second direction different from the first direction, the second modification factor being different from the first modification factor.

[0027] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar responds to changes in one or more physical features of a face within a field of view of the one or more cameras; while displaying the virtual avatar, detecting a first configuration of one or more physical features of the face; while detecting the first configuration of one or more physical features of the face: based on determining that the first configuration of the one or more physical features satisfies an animation standard, modifying the virtual avatar to include a first animation effect, the animation standard including requiring that the first configuration be maintained for at least a first threshold amount of time in order for the animation standard to be satisfied; and based on the first configuration of the one or more physical features not satisfying the animation standard, abandoning modifying the virtual avatar to include the first animation effect.

[0028] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0029] Thus, faster, more efficient methods and interfaces are provided for devices for generating, sending, and receiving emoticons, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace other methods for sending and receiving emoticons. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0031] Figure 1A is a block diagram showing a portable multifunction device with a touch-sensitive display according to some embodiments.

[0032] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0033] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0034] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0035] Figure 4A An exemplary user interface for a menu of applications on a portable multifunction device is shown according to some embodiments.

[0036] Figure 4B An exemplary user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0037] Figure 5A A personal electronic device according to some embodiments is shown.

[0038] Figure 5B is a block diagram illustrating a personal electronic device according to some embodiments.

[0039] Figures 6A to 6MM An exemplary user interface for generating and sending emojis, stickers, avatars, and / or other multimedia content is shown.

[0040] 7A to 7J An exemplary user interface for receiving emojis, stickers, avatars, and / or other multimedia content is shown.

[0041] FIG. 8A to FIG. 8B is a flow chart illustrating a method for generating and sending emoticons, stickers, avatars, and / or other multimedia content.

[0042] FIG. 9A to FIG. 9B is a flow chart illustrating a method for generating and sending emoticons, stickers, avatars, and / or other multimedia content.

[0043] FIG. 10A to FIG. 10I An exemplary user interface for generating and modifying a poop avatar is shown.

[0044] FIG. 11A to FIG. 11C An exemplary user interface for generating and modifying a bear avatar is shown.

[0045] FIG. 12A to FIG. 12C An exemplary user interface for generating and modifying an alien avatar is shown.

[0046] Fig.13 An exemplary user interface for generating and modifying a rabbit avatar is shown.

[0047] FIG. 14A to FIG. 14D An exemplary user interface for generating and modifying a robotic avatar is shown.

[0048] FIG. 15A to FIG. 15B An exemplary user interface for generating and modifying a unicorn avatar is shown.

[0049] FIG. 16A to FIG. 16B An exemplary user interface for generating and modifying a chicken avatar is shown.

[0050] FIG. 17A to FIG. 17B An exemplary user interface for generating and modifying a pig avatar is shown.

[0051] FIG. 18A to FIG. 18B is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0052] Fig.19 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0053] Fig. 20 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0054] Fig.21is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0055] Fig. 22 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0056] Fig.23 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0057] Fig.24 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras.

[0058] Fig.25 is a flow chart illustrating a method for generating and modifying a virtual avatar based on faces detected by one or more cameras. DETAILED DESCRIPTION

[0059] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0060] Sending a message with multimedia content and text content or with only multimedia content is likely to better convey the sender's message. For example, multimedia content such as a virtual avatar (e.g., an animated or static emoticon or sticker) can provide context and / or tone that is difficult or impossible to convey with text alone (e.g., it can be referred to as "non-verbal communication"). In some cases, predefined virtual avatars can be used to provide some of the context and / or tone. However, predefined content cannot cover all situations or provide fine-tuned context or tone. Therefore, it is necessary to provide an electronic device for generating, sending and receiving virtual avatars as part of a message. An effective method and interface. Such technology can reduce the cognitive burden of users who send and receive messages, thereby improving productivity. In addition, such technology can reduce processor power and battery power that would otherwise be wasted on redundant user input.

[0061] under, Figure 1A to Figure 1B , Figure 2 , Figure 3 , FIG. 4A to FIG. 4B as well as FIG. 5A to FIG. 5B A description of an exemplary device for performing techniques for generating, sending, and receiving virtual avatars is provided. Figures 6A to 6MM and 7A to 7J An exemplary user interface for receiving, generating, modifying, and sending a virtual avatar is shown. FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9Bis a flow chart illustrating an exemplary method for receiving, generating, modifying, and sending a virtual avatar. Figures 6A to 6MM as well as 7A to 7J The user interface in is used to illustrate the processes described below, which include FIG. 8A to FIG. 8B and FIG. 9A to FIG. 9B process. FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B An exemplary user interface for generating and modifying a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25 process. FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B The user interface and FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25 The process can be used to generate Figures 6A to 6MM as well as 7A to 7J The interface and FIG. 8A to FIG. 8B as well as FIG. 9A to FIG. 9B A virtual avatar used in the process.

[0062] Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element. For example, a first touch can be named a second touch and similarly a second touch can be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0063] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0064] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.

[0065] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0066] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0067] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0068] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0069] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A is a block diagram showing a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes referred to as or referred to as a "touch-sensitive display system." The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0070] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with limited real estate, which is used to display an enable indication (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control, such as a knob or button).

[0071] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0072] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the EMBODIMENTS 2000 are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0073] Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0074] The peripheral device interface 118 can be used to couple the input peripheral devices and output peripheral devices of the device to the CPU 120 and the memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as the chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0075] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 108 optionally communicates with networks and other devices through wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuits for detecting near field communication (NFC) fields, such as by detecting through a short-range communication radio component. Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE802.11n and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communications protocol including communications protocols that have not been developed as of the filing date of this document.

[0076] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0077] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch screen 112 and other input control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, a depth controller 169, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an increase / decrease button for volume control of the speaker 111 and / or the microphone 113. One or more buttons optionally include a push button (e.g., Figure 2 206).

[0078] A quick press of the push button optionally disengages the lock of the touch screen 112 or optionally begins the process of unlocking the device using gestures on the touch screen, as described in U.S. Patent Application 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press of the push button (e.g., 206) optionally turns the device 100 on or off. The functions of one or more buttons are optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0079] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen 112 and / or sends electrical signals to the touch screen 112. The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0080] The touch screen 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch screen 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of the contact) on the touch screen 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen 112. In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0081] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the Apple ® from Apple Inc. (Cupertino, California). and iPod The technology used in.

[0082] The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive trackpad described in the following U.S. Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.), and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0083] In some embodiments, the touch-sensitive display of the touch screen 112 is as described in the following patent applications: (1) U.S. patent application 11 / 381,313, entitled “Multipoint Touch Surface Controller,” filed on May 2, 2006; (2) U.S. patent application No. 10 / 840,862, entitled “Multipoint Touchscreen,” filed on May 6, 2004; (3) U.S. patent application No. 10 / 903,964, entitled “Gestures For Touch Sensitive Input Devices,” filed on July 30, 2004; (4) U.S. patent application 11 / 048,264, entitled “Gestures For Touch Sensitive Input Devices,” filed on January 31, 2005; (5) U.S. patent application 11 / 086, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive and (9) U.S. patent application No. 11 / 367,749, filed on March 3, 2006, entitled “Multi-Functional Hand-Held Device.” All of these applications are hereby incorporated by reference in their entirety.

[0084] The touch screen 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch screen 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0085] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0086] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0087] Device 100 optionally also includes one or more optical sensors 164 . Figure 1AAn optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the optical sensor 164 optionally captures a static image or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to the touch screen display 112 on the front of the device, so that the touch screen display can be used as a viewfinder for static images and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device so that the user's image is optionally obtained for video conferencing while the user views other video conference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that a single optical sensor 164 is used with the touch screen display for both video conferencing and static image and / or video image acquisition.

[0088] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. Contact force sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). Contact force sensor 165 receives contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the back of device 100, opposite to touch screen display 112 located on the front of device 100.

[0089] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1AA proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is optionally coupled to the input controller 160 in the I / O subsystem 106. The proximity sensor 166 is optionally implemented as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; No. 11 / 240,788, entitled "Proximity Detector In Handheld Device"; No. 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch screen 112.

[0090] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact force sensor 165 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in parallel or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0091] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 is optionally implemented as described in the following U.S. Patent Publication: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from one or more accelerometers. The device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0092] Device 100 optionally also includes one or more depth camera sensors 175 . Figure 1AA depth camera sensor coupled to a depth camera controller 169 in the I / O subsystem 106 is shown. The depth camera sensor 175 receives data projected by the sensor from the environment. In conjunction with the imaging module 143 (also referred to as a camera module), the depth camera sensor 175 camera is optionally used to determine a depth map of different portions of the image captured by the imaging module 143. In some embodiments, the depth camera sensor is located at the front of the device 100, so that the user's image with depth information is optionally obtained for video conferencing while the user views other video conference participants on the touch screen display, and a selfie with depth map data is captured. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) so that the depth camera sensor 175 is used with the touch screen display for both video conferencing and still image and / or video image acquisition.

[0093] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 ( Figure 1A ) or 370( Figure 3 ) storage device / global internal state 157, such as Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch screen display 112; a sensor state, including information obtained from the device's various sensors and input control devices 116; and position information about the device's position and / or posture.

[0094] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitating communication between various hardware components and software components.

[0095] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to (trademark of Apple Inc.) devices.

[0096] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the contact strength (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and display controller 156 detect contact on the touch pad.

[0097] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0098] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event.

[0099] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0100] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0101] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 167 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0102] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0103] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services, such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0104] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0105] Contacts module 137 (sometimes called address book or contact list);

[0106] Telephone module 138;

[0107] Video conferencing module 139;

[0108] Email client module 140;

[0109] Instant messaging (IM) module 141;

[0110] Fitness support module 142;

[0111] A camera module 143 for still images and / or video images;

[0112] Image management module 144;

[0113] Video player module;

[0114] Music player module;

[0115] Browser module 147;

[0116] Calendar module 148;

[0117] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0118] A widget creator module 150 for forming a user-created widget 149-6;

[0119] Search module 151;

[0120] Video and music player module 152, which merges the video player module and the music player module;

[0121] Notepad module 153;

[0122] Map module 154; and / or

[0123] Online video module 155.

[0124] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0125] In conjunction with the touch screen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and the like.

[0126] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, phone module 138 is optionally used to enter a sequence of characters corresponding to a phone number, access one or more phone numbers in contact module 137, modify an entered phone number, dial a corresponding phone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0127] In combination with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch screen 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the contact / motion module 130, the graphics module 132, the text input module 134, the contact module 137 and the telephone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0128] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0129] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0130] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154 and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., with time, distance and / or calorie burn goals); communicating with fitness sensors (sports equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for fitness; and displaying, storing and transmitting fitness data.

[0131] In conjunction with the touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132 and image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or videos (including video streams) and storing them in the memory 102, modifying the characteristics of the still images or videos, or deleting the still images or videos from the memory 102.

[0132] In conjunction with touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0133] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching, linking to, receiving and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0134] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0135] In conjunction with the RF circuit 108, the touch screen 112, the display controller 156, the contact / motion module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5) or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0136] In combination with the RF circuit 108, touch screen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134 and browser module 147, the desktop widget creator module 150 is optionally used by a user to create a desktop widget (e.g., converting a user-specified portion of a web page into a desktop widget).

[0137] In combination with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0138] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0139] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphics module 132 and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0140] In combination with the RF circuit 108, the touch screen 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the GPS module 135 and the browser module 147, the map module 154 is optionally used to receive, display, modify and store maps and data associated with the maps (e.g., driving directions, data relating to stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0141] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions for performing the following operations: allowing a user to access, browse, receive (e.g., by streaming and / or downloading), playback (e.g., on the touch screen or on an external display connected via external port 124), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a specific online video. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed on June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed on December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are hereby incorporated by reference in their entirety.

[0142] Each of the modules and applications described above corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (e.g., Figure 1A In some embodiments, the memory 102 optionally stores a subset of the above modules and data structures. In addition, the memory 102 optionally stores additional modules and data structures not described above.

[0143] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0144] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0145] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0146] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0147] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0148] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

[0149] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0150] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0151] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0152] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0153] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0154] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0155] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.

[0156] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0157] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0158] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0159] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0160] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift-off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0161] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0162] In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0163] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0164] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0165] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0166] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0167] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0168] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0169] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0170] Figure 2A portable multifunction device 100 with a touch screen 112 according to some embodiments is shown. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more of these graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics will occur when the user interrupts contact with the one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0171] The device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen 112.

[0172] In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch screen 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0173] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as chipsets) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or contact intensity sensors (similar to the above referenced Figure 1A The memory 370 may include one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0174] Figure 3Each element in the above-mentioned elements in is optionally stored in one or more memory devices of the previously mentioned memory device. Each module in the above-mentioned modules corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned modules or programs (e.g., instruction sets) do not have to be implemented as independent software programs, processes or modules, and therefore the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores other modules and data structures not described above.

[0175] Attention is now turned to an embodiment of a user interface that is optionally implemented on, for example, portable multifunction device 100.

[0176] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0177] One or more signal strength indicators 402 of one or more wireless communications such as cellular signals and Wi-Fi signals;

[0178] Time 404;

[0179] Bluetooth indicator 405;

[0180] Battery status indicator 406;

[0181] A tray 408 with icons for commonly used applications, such as:

[0182] o an icon 416 of the phone module 138 labeled “Phone”, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0183] o an icon 418 of the email client module 140 labeled “Mail”, the icon 418 optionally including an indicator 410 of the number of unread emails;

[0184] o an icon 420 labeled "Browser" of the browser module 147; and

[0185] o an icon 422 labeled “iPod” of a video and music player module 152 (also referred to as an iPod (trademark of Apple Inc.) module 152); and

[0186] Icons for other apps, such as:

[0187] o Icon 424 labeled “Messages” of IM module 141;

[0188] o Icon 426 labeled “Calendar” of calendar module 148;

[0189] o Icon 428 labeled “Photos” of the image management module 144;

[0190] o Icon 430 labeled “Camera” of camera module 143;

[0191] o Icon 432 labeled “Online Video” of the online video module 155;

[0192] o Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0193] o an icon 436 labeled “MAP” of the map module 154;

[0194] o Icon 438 labeled “Weather” of the Weather widget 149-1;

[0195] o Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0196] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0197] o Icon 444 labeled "Notepad" of the Notepad module 153; and

[0198] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .

[0199] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of a corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0200] Figure 4B A touch-sensitive surface 451 (eg, touch screen display 112) is shown having a touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contacts on touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0201] Although some of the examples below are given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the principal axis (for example, Figure 4B According to these embodiments, the device detects a position corresponding to a corresponding position on the display (e.g., Figure 4B 460 corresponds to 468 and 462 corresponds to 470) in contact with touch-sensitive surface 451 (e.g., Figure 4B 460 and 462 in FIG. 4 ). Thus, on a touch-sensitive surface (eg, Figure 4B 451) and the display of a multi-function device ( Figure 4B When the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is separated, the device is used to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0202] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0203] Figure 5AAn exemplary personal electronic device 500 is shown. Device 500 includes a body 502. In some embodiments, device 500 may include a body 502 relative to devices 100 and 300 (e.g., Figures 1A to 4B ) some or all of the features described in the foregoing. In some embodiments, device 500 has a touch-sensitive display screen 504, referred to hereinafter as touch screen 504. As an alternative or in addition to touch screen 504, device 500 has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more strength sensors for detecting the intensity of contact (e.g., touch) applied. One or more strength sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of touch. The user interface of device 500 can respond to touch based on the intensity of the touch, which means that touches of different intensities can invoke different user interface operations on device 500.

[0204] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International patent application serial number PCT / US2013 / 040061, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed on May 8, 2013, published as WIPO patent publication number WO / 2013 / 169849; and International patent application serial number PCT / US2013 / 069483, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed on November 11, 2013, published as WIPO patent publication number WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0205] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 (if included) can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) can allow the device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0206] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, the device 500 may include a reference Figure 1A , Figure 1B and Figure 3 Some or all of the components described. Device 500 has a bus 512 that operatively couples an I / O portion 514 to one or more computer processors 516 and a memory 518. I / O portion 514 may be connected to a display 504, which may have a touch-sensitive component 522 and optionally a strength sensor 524 (e.g., a contact strength sensor). In addition, I / O portion 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 is optionally a rotatable input device or a depressible input device and a rotatable input device. In some examples, input mechanism 508 is optionally a button.

[0207] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, orientation sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and / or combinations thereof, all of which are operably connected to I / O portion 514.

[0208] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the following techniques, including processes 800, 900, 1800, 1900, 2000, 2100, 2200, 2300, 2400, and 2500 (corresponding to processes 800, ... FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25). Computer-readable storage media can be any medium that can tangibly contain or store computer-executable instructions for use by or in conjunction with instruction execution systems, devices, and apparatuses. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B components and configurations, but may include other components or additional components in a variety of configurations.

[0209] As used herein, the term "indicative representation" refers to an optional feature of the apparatus 100, 300, and / or 500 ( Figure 1A , Figure 3 and FIG. 5A to FIG. 5B ) is a user-interactive graphical user interface object displayed on a display screen of a computer. For example, an image (e.g., an icon), a button, and text (e.g., a hyperlink) optionally each constitute an affordance.

[0210] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that a touch-screen display (e.g., Figure 1A A touch-sensitive display system 112 or Figure 4AIn some implementations of the touch screen 112 in FIG. 1 , a contact detected on the touch screen acts as a “focus selector” such that when an input (e.g., a press input by the contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted in accordance with the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to deliver the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user intends to activate the corresponding button (rather than other user interface elements shown on the device display).

[0211] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half-maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to abandon the corresponding operation) rather than to determine whether to perform the first operation or the second operation.

[0212] In some embodiments, a portion of a gesture is identified for determining characteristic strength. For example, the touch-sensitive surface optionally receives a continuous swipe contact that transitions from a starting position and reaches an end position, where the contact strength increases. In this example, the characteristic strength of the contact at the end position is optionally based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact before determining the characteristic strength of the contact. For example, a smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or depressions in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0213] The intensity of contact on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact having a characteristic intensity lower than the light press intensity threshold (e.g., and higher than a nominal contact detection intensity threshold, contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0214] An increase in contact feature intensity from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact feature intensity from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact feature intensity from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting contact on the touch surface. A decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting contact lifted from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0215] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "up stroke" of the corresponding press input).

[0216] In some embodiments, the device employs intensity hysteresis to avoid unexpected inputs sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below the hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., an "upward stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and, optionally, that the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact intensity or a decrease in contact intensity, depending on the circumstances).

[0217] For ease of explanation, optionally, a description of an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is triggered in response to detecting any of the following: contact intensity increasing to above the press input intensity threshold, contact intensity increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, contact intensity decreasing to below the press input intensity threshold, and / or contact intensity decreasing to below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operation is optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold.

[0218] Attention is now turned to embodiments of a user interface ("UI") and associated processes implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.

[0219] Figures 6A to 6MM An exemplary user interface for generating and sending a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include FIG. 8A to FIG. 8B as well as FIG. 9A to FIG. 9B process. 7A to 7J An exemplary user interface for receiving and playing a virtual avatar according to some embodiments is shown. Figures 6A to 6MM as well as 7A to 7J A virtual avatar is used as a specific example of a virtual avatar.

[0220] Fig. 6A A device 600 is depicted having a display 601 (in some cases, the display is a touch-sensitive display) and a camera 602 (the camera includes at least an image sensor capable of capturing data representing a portion of a light spectrum (e.g., visible light, infrared light, or ultraviolet light). In some embodiments, the camera 602 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, the camera 602 is capable of capturing other types of data such as depth data. For example, in some embodiments, the camera 602 also captures depth data using techniques based on speckle, time of flight, parallax, or focus. The image data captured by the device 600 using the camera 602 includes data corresponding to a portion of the light spectrum of a scene within the camera's field of view. In addition, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes data sufficient to determine or generate depth data for the data for that portion of the light spectrum. In some embodiments, the device 600 includes one or more features of the device 100, 300, or 500.

[0221] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate the object during the capture of the image by the visible light camera and the depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine the depth map of different parts of the object captured by the visible light camera. In some embodiments, the lighting effects described herein are displayed using parallax information from two cameras (e.g., two visible light cameras) for backward images, and the depth information from the depth camera is combined with the image data from the visible light camera for forward images (e.g., selfie images). In some embodiments, the same user interface is used when the depth information is determined using two visible light cameras and when the depth information is determined using a depth camera, thereby providing a consistent experience for the user even when the information used to produce the lighting effect is determined using completely different technologies. In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects selection of a camera switching indicator and switches from a front-facing camera (e.g., a depth camera and a visible light camera) to a rearward-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa) while maintaining display of user interface controls for applying the lighting effect and replacing the field of view display of the front-facing camera with the field of view of the rearward-facing camera (or vice versa).

[0222] exist Fig. 6A, device 600 is displaying a home screen interface having multiple icons for various applications, including an icon 603 for a messaging application. In response to a gesture on icon 603 (e.g., a tap gesture 604), device 600 displays Figure 6B The one corresponding to the user interface of the messaging application associated with icon 603.

[0223] exist Figure 6B , device 600 is displaying messaging interface 608. Elements 605-1 through 605-6 correspond to previous messaging communications. Each element 605-1 through 605-6 represents a communication with one or more remote users, each of which is associated with their own electronic device. In response to a gesture on a particular element (e.g., tap gesture 606), device 600 updates messaging interface 608 to display a portion of a previous messaging communication with one or more remote users as part of the communication, such as Figure 6C Described in.

[0224] exist Figure 6C , device 600 is displaying a messaging interface 608 for messaging communications with a remote user named "John" (and having the initials or monogram of "JA"). Messaging interface 608 includes a message area 609 that includes four previously exchanged messages 610-1 to 610-3 (message 610-3 is sent from the user of device 600 to "John", and the other two messages are received by device 600 from "John"). Messaging interface 608 also includes a message composition area 612 and message options icons to the left of message composition area 612 (including icon 614, e.g., an icon for accessing an interface for selecting stickers and / or other multimedia elements for a message). In some embodiments, the message options icon allows different types of messages to be sent, including photos, emoticons, stickers, and other forms of non-text messages, such as those described below.

[0225] In response to device 600 detecting (e.g., via Figure 6C 616) to select the message writing area 612, such as Fig.6D Update the message transmission interface 608 as depicted in . For example, Fig.6D , the message options icons are hidden (but can be displayed again by selecting button 618), suggested message responses 620 are displayed, and a virtual keyboard 622 is displayed. In some cases, the virtual keyboard 622 is used to enter a new message to be sent to the remote user.

[0226] exist Fig. 6E, message composition area 612 includes the text "Come later", for example, entered via virtual keyboard 622 or other methods such as voice input. In response to selection of send button 621, device 600 sends the text as part of a message to one or more participants associated with the communication in message 609. Fig. 6E In the case of , device 600 sends the message to a user named "John". Fig. 6F , device 600 has message area 609 updated to reflect the sending of the message by updating message area 612 to include message 610-4.

[0227] In some cases, the message options icon is accessed to add or compose a new message (e.g., by adding non-text content to the message). For example, in response to device 600 detecting a selection of affordance 618 (e.g., via Fig. 6F ), the message option icons including icon 614 are displayed again, such as Figure 6G In response to selection of icon 614 (e.g., via a gesture such as Figure 6H ), device 600 updates messaging interface 608 (e.g., by replacing virtual keyboard 622 with multimedia items interface 628 that is currently displaying recent items menu 629 (sometimes referred to as a "tray" of recent items). Fig.6I ), the multimedia item interface includes previously sent multimedia items (e.g., Fig.6I 630-1 to 630-4 in the figure, but may also include other types of multimedia items, such as sound, animation, or video). Using this interface, the user can select a previously sent multimedia item to send again. For example, the user can select a previously sent multimedia item in the image via a tap gesture on the selected sticker. Fig.6I609. In response to such a selection, the device 600 places the sticker in the message writing area 612, or sends the selected sticker to one or more remote users involved in the communication represented in the message area 609. In some embodiments, a tap and drag gesture is used to place the selected sticker (or other multimedia item) in the message writing area 612 or the message area 609 (and in some cases, on a specific message). For example, a specific sticker is selected via a tap gesture. Without breaking contact with the touch-sensitive display 601, the sticker is dragged to the message writing area 612 or the message area 609 via a drag gesture. Once the sticker reaches the desired position, the contact with the touch-sensitive display 601 is stopped, and the sticker is placed at the last contact position. If the last contact position is in the message area 609, the sticker is sent to one or more remote users associated with the communication represented in the message area 609. Optionally, the sticker is sent to the remote user together with data associating the sticker with a specific message (e.g., the sticker is sent together with data indicating that the sticker is "pasted" at a specific position of a specific message). These techniques are not specific to selecting and sending stickers. It can also be applied to other types of multimedia items that can be selected from the recent items menu 629 or other locations.

[0228] exist Fig.6I In the embodiment, multimedia item interface 628 also includes menu selection button 632 (which allows menus or interfaces other than recent item menu 629 to be selected via displaying buttons or other selectable items corresponding to available menus) and full screen button 634 (which allows multimedia item interface 628 to expand to occupy more of the display (or occupy the entire display)). Full screen button 634 is further described below.

[0229] In addition to using menu selection button 632 to switch between menus or interfaces, gestures are optionally used to switch between menus. For example, in response to a swipe gesture (e.g., Figure 6J and Figure 6K 6 (depicted in FIG. 6 ), the device updates multimedia item interface 628 to replace display of recent items menu 629 with virtual avatar menu 638 via contact 636's movement across multimedia item interface 628. While recent items menu 629 is being replaced with virtual avatar menu 638, scroll indicator 639 provides feedback about how many other menus are available in multimedia item interface 628.

[0230] exist Figure 6L , the virtual avatar menu 638 has completely replaced the display of the recent items menu 629. In response to the device 600 detecting (e.g., via a gesture such as Figure 6M The tap gesture 642) is used to indicate the selection of continuing to indicate 640, such as Figure 6N 643 is displayed as depicted in FIG. The interface allows a user to generate a new virtual avatar that reflects the user's facial movements and expressions, as described further below. In some embodiments, the virtual avatar menu 638 is not displayed at all. Instead, the virtual avatar interface 643 is displayed without first displaying the virtual avatar menu 638.

[0231] Figure 6N The virtual avatar interface 643 includes avatar template representations 644-1 to 644-7 corresponding to different avatar frames (e.g., avatar characters with different appearances and behaviors). Each avatar template represents an avatar frame to which detected facial movements and expressions can be mapped. Indicator 645 corresponds to the currently selected avatar template. Virtual avatar preview 646 is a "real-time" preview of the virtual avatar because it is updated to reflect the user's current facial movements and expressions. For example, in some embodiments, using camera 602, device 600 continuously captures image data from camera 602. The captured image data includes visible light data and depth data. Device 600 analyzes the captured image data to identify facial movements (e.g., muscle movements, head orientation, gaze direction, etc.) and / or facial expressions (e.g., smiles, frowns, angry expressions, sad expressions, confused expressions, etc.). Then, device 600 updates avatar preview 646 to reflect detected user features according to the parameters of the avatar frame currently associated with virtual avatar preview 646. In some embodiments, the device 600 automatically begins continuously updating the virtual avatar preview 646 in response to the virtual avatar interface 643 being first executed or displayed at 646. Detecting the selection of a representation of a different avatar template will cause the device 600 to update the virtual avatar preview 646 based on the newly selected avatar template.

[0232] Fig.6O Several examples of a user's face in captured image data 650-1 to 650-5 and corresponding updates 651-1 to 651-5 to a virtual avatar preview are depicted. These are examples of the device 600 updating the emoji preview 646 to reflect the user's facial movements, expressions, and gestures. In captured image data 650-1, the device 600 detects (e.g., based on facial features, facial muscles, facial movements, and / or facial expressions) that the user is looking straight ahead, smiling, and / or happy. In response, in addition to updating the eyes of the virtual avatar preview to look straight ahead, the device 600 also updates the virtual avatar preview to reflect the user's smile and / or happy expression, as depicted in update 651-1. Although physical features of a user detected in the captured image data are sometimes updated so that the virtual avatar reflects the same physical features in the user's virtual avatar, in other cases, changes in the detected physical features of the user result in updating different types of physical features of the virtual avatar. For example, in Fig.6OIn the example of image data 650-3 and update 651-3, the user's unhappy expression and / or frown is reflected in the corresponding features of the virtual avatar preview. In some embodiments, if the user maintains a facial expression or facial gesture, as depicted in images 650-3 and 650-4, the virtual avatar preview is updated with additional features (such as tears in the case of update 651-4). This type of predefined update can also occur in response to a detected lack of movement. In some embodiments, the update is also based on the user movement detected in the image data. For example, the device 600 that detects the rotation of the user's head generates an update that causes the virtual avatar preview to rotate similarly. In some embodiments, the update is also based on a physical model of the features used for the virtual avatar. For example, in image data 650-5, the device 600 detects that the user is shaking his head. In response, device 600 generates update 651-5 to reflect the shaking head. In addition, in update 651-5, the puppy's ears are also extended due to the physical model applied to the puppy's ears.

[0233] exist Figure 6P , device 600 detects selection of record button 652 via a gesture (e.g., a tap gesture represented by contact 653). In response, virtual avatar interface 643 is updated to show that an animated virtual avatar is being generated, such as Figure 6Q . For example, the record button 652 is replaced with a stop button 654, the avatar template representations 644-1 to 644-7 are no longer displayed, and a recording progress indicator 656 is displayed, which indicates how long the animated emoticon has been recorded and the relative amount of time that the virtual avatar can still be recorded. The recording can be stopped by any number of methods, such as by the expiration of a predetermined amount of time (e.g., 15 seconds) or by selecting the stop button 654. In some embodiments, while recording, the device 600 is detecting and / or storing a series of data points for creating an animated virtual avatar. For example, in some embodiments, the device 600 records a time series of facial movements and / or facial expressions (e.g., as values ​​in a range of possible values, where each value in the range of possible values ​​corresponds to a predetermined movement or expression), and then maps the facial movements and / or facial expressions to the avatar template to create an animated virtual avatar. Alternatively, the device 600 records the animated virtual avatar by creating a video recording of the virtual avatar preview as the device 600 updates the virtual avatar preview to reflect the user's facial movements and / or expressions. In some embodiments, the device 600 also records the sound captured using the microphone of the device 600, so that the recorded animated virtual avatar includes the sound that can be played back together with the recorded animation of the virtual avatar.

[0234] Figure 6R Depicted is a later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0235] Figure 6R-1 A device 600 is depicted that detects that a user has changed their position relative to the device during avatar recording. Figure 6R-1 At the corresponding time point, the user's face is no longer in the camera's field of view. In response, the device 600 displays the virtual avatar at the edge of the avatar interface 643 (e.g., the edge corresponding to the last detected position of the user's face), displays frame corners 653 around the virtual avatar, and displays a message 655A ("Please put your face in view") to prompt the user to adjust its alignment relative to the device. In some embodiments, the recording of the virtual avatar continues even after the user's face is no longer detected in the camera's field of view, but when the user's face is not detected, the virtual avatar will remain still (or present a predetermined posture (e.g., a neutral posture)).

[0236] Figure 6R-2 The device 600 is depicted after the user has been outside the camera's field of view for more than a predetermined threshold time. In response to detecting that the user has been outside the camera's field of view for more than the predetermined time, the device 600 pauses recording the virtual avatar. Figure 6R-2 As shown, in accordance with the recording pause, the device 600 has replaced the stop button 654 with the record button 648. The device 600 also displays a message 655B ("Tap to Resume") indicating to the user that the recording has been paused. In some embodiments, the user can resume recording by tapping anywhere in the avatar interface 643 (including tapping the record button 648). The recording virtual avatar is paused when the user has been outside the camera field of view for a time exceeding a predetermined threshold amount of time, and another input is requested to resume recording, which reduces energy usage and the use of the depth camera, thereby extending the battery life of battery-powered devices and extending the life of the depth camera.

[0237] Figure 6S Depicts another later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to further reflect the newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect the further progress of recording the animated virtual avatar.

[0238] exist Figure 6SIn the embodiment of the present invention, a gesture requesting to stop recording the animated virtual avatar (e.g., a tap gesture represented by contact 658) is received. In response, device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, such as Figure 6T In other cases, in response to the expiration of a predetermined time period (e.g., 15 seconds), the device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, such as Figure 6T Described in.

[0239] Figure 6S-1 to Figure 6S-3 Another embodiment of a virtual avatar interface 643 for recording (e.g., generating) a virtual avatar is depicted. Figure 6S-1 As shown, device 600 displays a timer 659 (e.g., showing 10 seconds remaining) indicating the time remaining in the current avatar recording session (e.g., a session initiated by activating the record button 648). Figure 6S-2 In , the same recording session is conducted for 4 seconds and timer 659 currently shows that 6 seconds remain in the avatar recording session. Figure 6S-3 , the recording session has ended (i.e., Figure 6S-3 is Figure 6S-1 In response to the recording session ending, device 600 replaces timer 659 with a trash can indicator 660-1, which can be activated (e.g., by a tap gesture) to discard the completed recording session. In some embodiments, the functionality of trash can indicator 660 is similar to Figure 6T as well as Figure 6U to Figure 6U-1 The discard indication indicates 660.

[0240] exist Figure 6T , the virtual avatar interface 643 is currently playing the recorded animated virtual avatar 659, rather than displaying a preview of the virtual avatar, as depicted by the three snapshots of the playback of the animated virtual avatar 659. In some embodiments, the recorded animated virtual avatar is played in a loop (e.g., the recorded animated virtual avatar is played at least twice without user input, such as Figure 6T). The virtual avatar interface 643 also includes a discard button 660, a mute button 662, and a confirmation button 664 (the confirmation button is displayed in place of the record button 652). The discard button 660 discards the displayed recorded animated virtual avatar without saving the avatar or sending the avatar to the remote user. The mute button 662 allows the user to mute the playback of the recorded animated virtual avatar. The confirmation button 664 allows the recorded animated virtual avatar to be sent to the remote user (for example, directly to one or more users associated with the communication displayed in the message area 609 in response to activation of the confirmation button 664 or moving to the message composition area 612 before the user sends the message). After the device 600 detects the selection of the confirmation button 664, the virtual avatar interface 643 is updated to return to the state relative to the remote user. Figure 6N In some embodiments, confirmation button 664 includes a send glyph or icon (e.g., Figure 6V 670) in the figure, which is used to indicate that the recorded animated virtual avatar can be sent to the remote user by selecting the confirmation button 664.

[0241] While the animated virtual avatar 659 is playing, in response to a tap gesture 665 on a representation of an avatar template that is different from the currently selected template, the animated virtual avatar is updated to reflect the new avatar template without having to re-record the animated virtual avatar. Figure 6U Described in Figure 6U Depicts animated avatar 659 having been replaced by animated avatar 666. Figure 6T The facial muscles, facial movements, facial features and facial expressions of the animated avatar 659 in Figure 6U The newly selected avatar template in .

[0242] exist Figure 6U-1 , while animated avatar 666 is playing, device 600 detects a tap gesture 661 corresponding to a selection of discard affordance 660. In response, device 600 discards the captured animated avatar data (e.g., abandons adding the animated avatar to message composition area 612) and transitions to Figure 6U-2 In the interface. Figure 6U-2 In the embodiment, the device 600 displays a pre-recorded avatar interface 643, which is similar to Figure 6P The interface you see in (including, for example, the record button). Figure 6P In contrast, the avatar template remains the avatar template of the robot (e.g., virtual avatar 666) instead of returning to the monkey virtual avatar 659. That is, the Figure 6T and Figure 6UDetection results of the change from avatar 659 to avatar 666 during playback as depicted in .

[0243] Re-reference Figure 6T In response to a gesture (e.g., a tap gesture represented by contact 667), device 600 adds a recorded animated virtual avatar 668 to message composition area 612 (see Figure 6V ) and return the virtual avatar interface 643 to Figure 6N The state described in (see Figure 6V ). The user may then send the message on device 600 (e.g., in response to a tap gesture represented by contact 672 on send affordance 670, as shown in FIG. Figure 6W ) before sending a message (see Figure 6V ) to add more message content (e.g., text or other multimedia items). Alternatively, when the device 600 detects the selection of the confirmation button 664, the device 600 sends the recorded animated virtual avatar to one or more remote users associated with the communication displayed in the message area 609, and then updates the message area to reflect that the animated virtual avatar 668 has been sent to one or more users associated with the communication included in the message area 609, such as Figure 6X Described in.

[0244] Figures 6Y to 6BB Depicted is the response of the virtual avatar interface 643 to a user input of scrolling through a list of avatar templates. For example, in response to a swipe gesture (e.g., a representation of contact 676 moving vertically across the avatar templates, such as Figure 6Y to Figure 6AA ), the device 600 scrolls the avatar templates and changes the currently selected avatar template. The avatar templates present within the avatar template indicator 645 are updated based on the swipe gesture. In response to detecting that a new avatar template is being selected, the device 600 updates the virtual avatar preview. For example, in Figure 6Z In FIG. 6 , when device 600 detects selection of avatar template representation 644-5, a virtual avatar preview 678 is displayed (the virtual avatar preview is based on the avatar template corresponding to representation 644-5), and in FIG. Figure 6AA , when the avatar template representation 644-8 is selected, a virtual avatar preview 680 is displayed (the virtual avatar preview is based on the avatar template corresponding to the representation 644-8).

[0245] In addition to generating recordings of animated puppet emoticons, emoticon interface 643 also allows for the generation of static virtual avatars (e.g., stickers with expressions / appearances determined based on the state of the virtual avatar). Figure 6CCIn the embodiment, in response to user input on the virtual avatar preview 680 (e.g., a tap and hold gesture represented by contact 682), the device 600 generates a sticker corresponding to the state of the virtual avatar preview 680 at a time associated with the user input (e.g., when the input is received, when the user input ends, or at some other time associated with the user input). In an embodiment, the device 600 displays a sticker 683 ( Figure 6DD and Figure 6EE ) to indicate that a sticker has been generated and / or that the user can place a sticker.

[0246] After the device 600 generates the sticker, the user optionally selects from several actions for the sticker. For example, the user can cause the device 600 to place the sticker in a recent menu or other similar interface that allows for later use. The user can also cause the device 600 to place the sticker in the message composition area 612 before the device 600 sends the message containing the sticker, and the user can place the sticker in the message area 609 (and optionally) on a specific message to cause the device 600 to send the sticker to one or more users participating in the communication in the message area 609.

[0247] For example, in Figure 6FF , device 600 has detected the lift-off of contact 682 while the contact is still above virtual avatar preview 680. In response, device 600 has saved the generated sticker to device 600, such as in a database or library in the memory of device 600, which can be accessed through recent items menu 629 ( Fig.6I ) accesses the database or library so that stickers can be optionally selected through the recent items menu 629 or via other interfaces on the device 600. The device 600 optionally displays that the stickers are saved locally via animations with different graphic versions 684 and 686, such as Figures 6FF to 6GG As depicted in , move toward menu selection button 632.

[0248] Another example: Figures 6HH to 6KK An example is shown where the device 600 sends the generated sticker to one or more users participating in the communication represented in the message area 609. Fig. 6II In the embodiment, the device 600 detects a user input on the virtual avatar preview 680 (e.g., by Figures 6HH to 6JJ 688 in the message area 609). When the device 600 detects that the user drags the sticker representation into the message area 609, the sticker representation 690 follows the contact 688. Once the device 600 detects that the contact 688 in the message area 609 is lifted off, the device 600 sends the sticker 691 to one or more remote users who are participants in the communication represented in the message area 609, such as Figure 6KK Described in.

[0249] exist Figure 6LL , device 600 responds to selection of full screen button 634 (e.g., via Figure 6LL The virtual avatar interface 643 may be updated to display more of the screen (or displayed in full screen mode) using a tap gesture represented by contact 692 in FIG. 6A . Figure 6MM Virtual avatar interface 643 is depicted after being enlarged to take up more of display 601. Button 692, when selected, causes device 600 to return virtual avatar interface 643 to its previous configuration.

[0250] 7A to 7J Depicted is messaging interface 608 after receiving an animated emoticon from a remote user. 7A to 7J use Figures 6A to 6MM The device 600 is taken as an example, but 7A to 7J The user interfaces and functionality depicted in are also applicable to other devices (e.g., devices 100, 300, or 500), including those devices to which stickers or animated virtual avatars have not been previously sent.

[0251] Fig. 7A The message interface 608 is depicted after the animated virtual avatar 700 has been received from a remote user named "John" (and having the initials or letter combination of "JA") just before the animated virtual avatar was played. In some embodiments, after receiving the animated virtual avatar 700, the device 600 automatically plays the animated virtual avatar. When the mute button 702 is selected, the mute button silences any sound associated with the animated virtual avatar 700 by the device 600. In some embodiments, if the animated virtual avatar scrolls off the display, any sound also dies. In some embodiments, such as with respect to Figures 6A to 6MM The virtual avatar interface 643 is displayed as described (eg, the virtual avatar interface 643 includes a preview virtual avatar based on the detected facial movements / expressions and the selected avatar template).

[0252] exist Figure 7B In the example, in response to playing the animated virtual avatar 700 once (e.g., playing once from the beginning to the end), a still frame 703 of the animated virtual avatar 700 is displayed to replace the animated virtual avatar 700. A playback button 704 is also displayed in the message area 609 and allows playback to be performed by, for example, Figure 7C The tap feature represented by contact 706 plays the animated virtual avatar 700 again. Fig.7D The device 600 (relative to Fig. 7A Describes the play animation emoji).

[0253] In some embodiments, when the animated virtual avatar 700 is playing, if the device 600 receives user input on the mute button 702 (e.g., by Fig. 7E In some embodiments, in response to selection of the mute button 702 (or if the sound on the device 600 is turned off or the device 600 has an enabled accessibility feature), a text button 714 is displayed, such as Figure 7H In response to selection of the text button 714 (e.g., via Figure 7G 716 represents a tap gesture), a text book 718 for the sound of the animated virtual avatar 700 is displayed, such as Figure 7H The contents of the text book 718 are generated locally on the device 600 or remotely (eg, using remote server computing resources).

[0254] In response to user input on the animated virtual avatar 700 (e.g., Fig.7I ), the device 600 displays an option menu related to the animated virtual avatar 700, such as Figure 7J 6. For example, menu 722 includes several response buttons 723-1 to 723-6 that device 600 can send to one or more remote users participating in the communication represented in message area 609. In addition, menu 724 is also displayed, which has a copy button 726, a save button 728, and a more button 730. Copy button 726 copies animated virtual avatar 700 to the clipboard of device 600. Save button 728 saves animated virtual avatar 700 to device 600 (e.g., to a database or library that can be accessed later by an application installed on device 600). More button 730 displays additional operations that can be performed with respect to animated virtual avatar 700.

[0255] FIG. 8A to FIG. 8B 800 is a flowchart illustrating a method of using an electronic device according to some embodiments. The method 800 is performed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in the method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0256] As described below, method 800 provides an intuitive way to generate and send emoticons such as virtual avatars. The method reduces the cognitive burden of users generating and sending emoticons, thereby creating a more effective human-computer interface. For battery-powered computing devices, users are enabled to generate and send emoticons more quickly and more efficiently, saving power and increasing the time interval between battery charges.

[0257] An electronic device (e.g., 600) having a camera (e.g., 602) (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (802) a virtual avatar generation interface (e.g., Figure 6N 643) (e.g., an interface for selecting an emoticon (animated or static), generating static stickers, and / or recording an animated virtual avatar). The electronic device displays (804) a preview of the virtual avatar in the virtual avatar generation interface (e.g., Figure 6N 646) (e.g., a 2D or 3D computer-generated graphic object that, in some cases, is intended to convey a non-verbal message, such as an emotion or reaction) (e.g., an animated avatar selected from a plurality of different available avatar templates). The preview of the avatar reacts to changes in the appearance of the face in the camera's field of view (e.g., Fig.6O ) (e.g., the animated virtual avatar will reflect the user's head movements, facial expressions, and orientation detected in image data from one or more image sensors in the camera). While displaying a preview of the virtual avatar, the electronic device detects (806) an input in the virtual avatar generation interface (e.g., contact 652, 682, or 690). In response to (808) detecting the input in the virtual avatar generation interface and based on determining that the input begins on the preview of the virtual avatar (e.g., 682 or 690) (e.g., a touch and hold input on the animated virtual avatar or a touchpad input that controls a cursor), the electronic device generates (810) a static virtual avatar sticker (e.g., 683 or 691) (e.g., a still image of an animated emoticon that can be "sticked" to a specific location in the message area), which represents the expression of the face in the camera's field of view at the corresponding time. In some embodiments, the corresponding time is determined based on the timing of the input (e.g., when the input is first received, when the input ends, when a gesture corresponding to the input begins to move across the touch-sensitive surface, or at any other time related to the input). Based on determining that the input includes activation of a recording enable representation (e.g., 648) in the virtual avatar generation interface (e.g., tapping the recording enable representation), the electronic device generates (812) an animated virtual avatar (e.g., 668) that represents a sequence of changes in facial expressions in the camera field of view over a period of time (e.g., as shown in FIG. 1 ). Figures 6Q to 6S). In some embodiments, the time period is determined based on the timing of the input (e.g., a time period that starts when the start of the input is detected, when the end of the input is detected, when a certain type of movement of the input is detected (such as when the input is a gesture on a touch-sensitive surface), or based on some other time period of the input). In some embodiments, the virtual avatar is three-dimensional. In some embodiments, a preview of the virtual avatar (e.g., 646) or an animated virtual avatar (e.g., 659) is displayed in 3D. Disambiguation between user inputs for two possible styles of communication (e.g., an animated virtual avatar and a static virtual avatar) avoids the need for a separate interface to generate each type of message content. Reducing the number of inputs required to deliver a desired message enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing multiple types of multimedia communications from a single interface to help users achieve desired communications), which in turn reduces power usage and extends the battery life of the device by enabling users to use the device more quickly and efficiently.

[0258] In some embodiments, the electronic device displays (814) a messaging interface (e.g., interface 608) (e.g., a messaging application such as Apple's Messages) that includes a message area (e.g., 609). The message area includes messages from a communication (e.g., Figure 6N communication in message area 609) (e.g., messaging thread) between two or more participants (e.g., Figure 6C , remote user "John" and the user of device 600) (e.g., messages sent from a user of an electronic device and messages received from a remote user of a different electronic device). The virtual avatar generation interface is displayed simultaneously with the message transmission interface (e.g., Figure 6N ) (e.g., the virtual avatar generation interface is displayed in the lower half of the messaging interface). In some embodiments, the virtual avatar preview (e.g., 646) is automatically displayed as part of the initial display of the virtual avatar generation interface.

[0259] In some embodiments, the messaging interface includes a message writing area (e.g., 612) (e.g., a message input area for entering text, emoticons, and other content before sending a message to a recipient) and the input is a tap on a virtual avatar preview (e.g., 646). In response to detecting the input in the virtual avatar generation interface, the electronic device displays a static virtual avatar (e.g., 683 or 691) in the message writing area. In some embodiments, displaying the virtual avatar generation interface includes replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with the display of the virtual avatar generation interface (e.g., replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with the display of the virtual avatar generation interface. Figure 6H to Figure 6NThe multimedia content of a message displayed before the message is sent reduces the likelihood of error messages and allows the user to add more content (e.g., via text or other content) before sending the message. Reducing the number of messages required to deliver a desired message enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve the desired communication while reducing the number of messages required for communication), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0260] In some embodiments, the avatar generation interface includes a static virtual avatar area (e.g., 629) (e.g., a tray of previously generated stickers displayed at the bottom of the avatar generation interface) that includes a collection of one or more previously generated virtual avatars (e.g., 630-1 to 630-4). Figure 6CC 682), the electronic device adds (612) the generated virtual avatar to a collection of one or more previously generated virtual avatars (e.g., sends the avatar, favorites the virtual avatar, or otherwise marks the virtual avatar to include it in the collection of virtual avatars). In some embodiments, in response to user input (e.g., 626) (e.g., selection of a virtual avatar collection affordance in an avatar generation user interface or a messaging user interface), a collection of virtual avatars (e.g., stickers) is displayed (e.g., including miniature versions of the newly generated stickers in the tray). In some embodiments, the tray of previously generated stickers is hidden until input (e.g., input 626) is received from the user requesting display of the tray or until some other event detected on the electronic device indicates that the tray may be related to the current state of the messaging interface or avatar generation interface. In some embodiments, after adding the virtual avatar to the collection of virtual avatars, the electronic device receives a request from the user to share the collection of virtual avatars with a second user, and in response, the electronic device sends the collection of virtual avatars to the second user. Maintaining previously sent message multimedia content allows the user to add and reuse previous content when applicable to new messages. Not having to recreate content enhances device operability and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends device battery life by enabling the user to use the device more quickly and efficiently.

[0261] In some embodiments, the input begins (816) at the preview of the virtual avatar (e.g., 680) and ends at a location within the message area (e.g., see Figures 6HH to 6KK)(e.g., a gesture that begins with a finger contacting a preview of the virtual avatar (e.g., 680), continues with the finger dragging to the message area, and ends with the finger lifting off the message area (in some cases, the gesture can end on a specific message in the message area, and the sticker is associated with the specific message and optionally moves as the message moves in the conversation)). The electronic device sends (818) the static virtual avatar (e.g., 691) to participants associated with the communication (e.g., one or more remote users) (e.g., Figure 6KK In some embodiments, in response to a gesture showing a static virtual avatar being peeled away from the virtual avatar preview, an animation is displayed ( Figure 6CC to Figure 6FF ).

[0262] In some embodiments, a static virtual avatar (e.g., 691) has an appearance determined based on the expression of a face in the camera field of view when an input (e.g., 688) is detected on the virtual avatar preview (e.g., 680). In some embodiments, in response to detecting the start of an input on the preview of the virtual avatar, the electronic device causes the preview of the virtual avatar to stop (820) in response to a change in the appearance of the face in the camera field of view. This indicates to the user that a sticker has been generated, and by sending the sticker to the user, the user is previewed to see the appearance of the sticker that will be sent when the sticker is dragged to the communication displayed in the message 609. By displaying a preview of the sticker that will be generated without the user having to perform additional interactions or complete a full sticker generation gesture before seeing the resulting sticker, the operability of the device is enhanced, which makes the human-computer interface improved and more efficient. This reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0263] In some embodiments, after the input moves away from the virtual avatar, the virtual avatar preview (e.g., 680) resumes reacting to changes (e.g., when the static avatar is dragged toward the message conversation, the animation resumes). In some embodiments, the virtual avatar preview (e.g., 680) stops reacting to changes in facial appearance until the input of dragging the static avatar ends. Resuming updating of the virtual avatar preview enables the user to compare the appearance of the generated sticker with other possible appearances of the virtual avatar, which may potentially be the basis for different / additional stickers. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback about other content that can be generated before the user sends the generated content to help the user achieve the desired results), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0264] In some embodiments, in response to the expiration of a time period for generating an animated virtual avatar (e.g., a timer of 5 seconds, 10 seconds, or 15 seconds expires, or a user input stops the time period), the electronic device displays a send or confirm enable indication (e.g., 664) to replace the recording enable indication (e.g., the virtual record button is no longer displayed, but a virtual send button is displayed instead of the virtual record button). In response to receiving input selecting a send or confirm enable indication (e.g., a tap gesture on the send enable indication on the touch-sensitive display), the electronic device sends (824) the generated animated virtual avatar to the remote user (e.g., see Figure 6U and Figure 6X , instead of first sending the animated virtual avatar to the message composition area 612, such as Figure 6V and Figure 6W As shown) (e.g., sending an animated avatar to a remote user associated with a messaging thread or conversation without first placing the animated avatar in another area of ​​the messaging interface, such as a message composition area). Displaying a send or confirmation button in place of a record button after the recording of the animated avatar is completed enables more information to be displayed in the interface by reusing area occupied by buttons that are not applicable to the current state of the interface and by providing the user with more context-related functionality. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0265] In some embodiments, in response to the expiration of the time period for generating the animated virtual avatar, the electronic device displays (822) a confirmation enable indication (e.g., 664) in place of the recording enable indication. In response to receiving input (e.g., via contact 667) selecting the send enable indication (e.g., a tap gesture on the send enable indication on the touch-sensitive display), the electronic device displays (823) a confirmation enable indication (e.g., 664) in place of the recording enable indication. Figure 6V ) in a message composition area (e.g., 612) (e.g., an area of ​​the messaging interface where text typed on a keyboard will be displayed). In some embodiments, the time period is based on a predetermined amount of time (e.g., the amount of time represented by progress indicator 656). After the animated virtual avatar is generated, the electronic device stops displaying a preview of the virtual avatar and displays a looped version of the animated virtual avatar (e.g., Figure 6T). Displaying a looped version of the animated virtual avatar includes displaying the animation sequence two or more times (e.g., as described below with respect to method 900). Displaying a send or confirm button in place of a record button after recording of the animated virtual avatar is complete enables more information to be displayed in the interface by reusing area occupied by buttons that are not applicable (or less applicable) to the current state of the interface. This enhances the operability of the device and makes the user device interface more efficient (e.g., by displaying more information / options on the display without cluttering the display with unused elements), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0266] In some embodiments, a first virtual avatar template (e.g., the template represented by element 644-4, or the template represented by element 644-5) is used. Figure 6N The electronic device displays representations of multiple other virtual avatar templates (e.g., elements 644-1 to 644-7) (e.g., miniature universal versions of different virtual avatar templates, such as smiling faces, animals, robots, or other objects), including a representation of a second virtual avatar template (e.g., Figure 6Z 644-8), wherein the second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the plurality of virtual avatar templates are based on emoticons that can be sent via a messaging application. Before starting to display a looped version of an animated virtual avatar (e.g., Figure 6T ) and in response to receiving a user input selecting the first virtual avatar template representation (e.g., 665), the electronic device updates the display of the looped version of the animated virtual avatar to reflect the second virtual avatar template (e.g., Figure 6U ) (e.g., when the animated avatar is still based on a sequence of changes in facial expressions, changing the appearance of the animated avatar to reflect the new avatar template). For example, the animated avatar may change from a monkey to a robot, but it will still reflect the same sequence of changes in facial expressions over time in the camera field of view (e.g., see the sequence from the camera field of view to the robot field of view). Figure 6T to Figure 6U Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the selection of the style of the animated virtual avatar after recording the movements and actions of the animated virtual avatar. Not having to re-record the animated virtual avatar to try out the new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0267] In some embodiments, the virtual avatar preview is automatically displayed in response to launching the virtual avatar generation interface (e.g., 646). In some embodiments, once the virtual avatar generation interface is displayed, the virtual avatar preview is displayed without user input.

[0268] In some embodiments, in response to detecting a characteristic (e.g., position, orientation, movement) of a first physical feature of a face in the camera field of view (e.g., a mouth smiling, a tongue sticking out, an ear wiggling, an eyebrow raising, or any other movement of any other physical feature), the electronic device updates the first physical feature of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the first physical feature of the face (e.g., eyes, eyebrows, mouth, tongue, ears) is the same as the type of the first physical feature of the displayed preview. In some embodiments, if the user's mouth is open, the virtual avatar's mouth will open in response (e.g., Fig.6O 650-1 and 651-1 of ). Similar results can be based on facial expressions. For example, if one or more movements of physical features or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be updated to reflect the predefined movement by updating the corresponding physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to similar physical features of the virtual avatar enables the user to provide movements, expressions, and gestures that provide input to the system, which are intuitively mapped to the virtual avatar without the need for cumbersome or time-consuming touch or key input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by mapping the user's features to the virtual avatar in a predictable manner to help the user achieve the desired results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0269] In some embodiments, in response to detecting a characteristic of a second physical feature of a face in the camera field of view, the electronic device updates the second physical feature of the displayed virtual avatar preview based on the detected characteristic, wherein the type of the second physical feature of the face is different from the type of the second physical feature of the displayed preview (for example, in Fig.6O, the movement of eyebrows 650-2, and the movement of ears 651-2). In some embodiments, if the user is smiling, and this indicates that the user is happy, different features of the virtual avatar (such as the horn of a unicorn or the lights on a robot) can be changed to reflect the smile. Similar results can be based on facial expressions. For example, if one or more movements of physical features or characteristics of the face are detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be optionally updated to reflect the predefined movement by updating different groups of physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to different physical features of the virtual avatar enables the user to provide movement, expression, and gestures that provide input to the system, which are mapped to avatar features that the user could not easily control. This enhances the operability of the device and makes the user-device interface more effective (for example, by mapping the user's features to the virtual avatar so that additional features of the virtual avatar are controlled to help the user achieve the desired results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0270] In some embodiments, in response to detecting movement of a face in the camera field of view, the electronic device updates a third physical feature (e.g., Fig.6O 650-5 and 651-5 in ). In some embodiments, for example, if the virtual avatar is based on a virtual avatar template for a puppy, when it is detected that the user's face is shaking, the face of the virtual avatar will shake, and even if the user's ears do not extend in response to the shaking, the ears of the virtual avatar can also extend to reflect the physics of the shaking motion. In some embodiments, the same physical features of the displayed virtual avatar preview are updated based on the movement and physical model of the corresponding features of the face in the camera field of view (for example, the ears move based on the movement of the user's ears, but also move based on the physical model of the puppy's big ears). Updating the virtual avatar based on the physical model for the virtual avatar enables the user to create a realistic and interactive virtual avatar that can convey a wider range of non-verbal information. This enhances the operability of the device and makes the user-device interface more efficient (for example, by helping the user to convey a predetermined message using a more realistic virtual avatar movement), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0271] In some embodiments, the preview of the virtual avatar is based on a predefined virtual avatar template (e.g., Figure 6NThe electronic device updates the preview of the virtual avatar based on one or more predefined behaviors associated with the predefined virtual avatar template. In some embodiments, if movement or change in facial expression is not detected on the face in the camera field of view (e.g., 650-3 and 650-4), the preview of the virtual avatar displays a predefined response (e.g., 651-4), such as blinking, rotating the head, making facial expressions, or other actions.

[0272] In some embodiments, in response to determining that a face is no longer detected in the camera's field of view (e.g., facial tracking fails because the face has moved out of the camera's field of view; the face is obscured and does not appear in the camera's field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device gradually fades out the display of the virtual avatar preview (e.g., the virtual avatar preview 646 will fade out). In some embodiments, the device makes other modifications to the virtual avatar preview to indicate that the user's face can no longer be tracked, such as downgrading the virtual avatar preview based on the last information detected by the device, including changing the size, rotation, movement, etc. of the virtual avatar preview. In some embodiments, in response to determining that a face is no longer detected in the camera's field of view (e.g., facial tracking fails because the face has moved out of the camera's field of view; the face is obscured and does not appear in the camera's field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device displays a message (in the Figure 6N Above or instead of the virtual avatar 643 Figure 6NThe virtual avatar preview 646 displays a display prompt). In some embodiments, in response to determining that the face is no longer detected in the camera field of view (e.g., facial tracking fails because the face has moved out of the camera field of view; the face is obscured and does not appear in the camera field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device updates the display of the preview of the virtual avatar based on changes in facial appearance that occurred during a time period before the face could no longer be detected in the field of view (e.g., immediately before or shortly before the face could no longer be detected in the field of view) (e.g., the device 600 repeatedly displays a transition from a smile to a frown or eye movement). In some embodiments, updating the display of the preview of the virtual avatar based on changes in facial appearance that occurred during a time period before the face could no longer be detected in the camera field of view includes: gradually slowing down the update of the preview of the virtual avatar over time so that the update of the virtual avatar gradually stops (e.g., a rotating avatar slowly stops rotating, eyes that are opening or closing slowly stop opening or closing, and a mouth that is opening or closing slowly stops opening or closing). Displaying feedback about whether the virtual avatar preview is tracking the user's face enables the user to determine whether the device is being held correctly and whether the conditions for detecting the user's face are correct. Providing improved feedback to the user of the device status enhances the operability of the device and makes the user-device interface more efficient by providing better continuity of the user interface with an indication that the device is still trying to track the user's face. This provides a better and more intuitive human-computer interface and enables the user to continue to interact with the device even when the device cannot track the user's face.

[0273] Note that the process described above with respect to method 800 (eg, FIG. 8A to FIG. 8B ) also applies in a similar manner to the methods described below. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, the generation of stickers described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For another example, muting the sound before sending an animated virtual avatar (e.g., an animated virtual avatar) as described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For the sake of brevity, these details are not repeated below.

[0274] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A , Figure 3 and Figure 5A In addition, the above reference Fig. 8A and Figure 8BThe operation is optionally performed by Figure 1A to Figure 1B . For example, detecting input in the virtual avatar generation interface (806) is optionally implemented by the event classifier 170, the event identifier 180, and the event handler 190. The event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and the event distributor module 174 delivers the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When the corresponding predefined event or sub-event is detected, the event identifier 180 activates the event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls the data updater 176 or the object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, a person skilled in the art will clearly know how to Figure 1A to Figure 1B The components depicted in the figure are used to implement other processes.

[0275] FIG. 9A to FIG. 9B 900 is a flowchart illustrating a method of using an electronic device according to some embodiments. The method 900 is performed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in the method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0276] As described below, method 900 provides an intuitive way to generate and send emoticons such as virtual avatars. The method reduces the cognitive burden of users generating and sending emoticons, thereby creating a more effective human-computer interface. For battery-powered computing devices, users are able to generate and send emoticons more quickly and more efficiently, saving power and increasing the time interval between battery charges.

[0277] An electronic device (e.g., 600) having a camera (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (902) a virtual avatar generation interface (e.g., Figure 6N 643) (e.g., an interface for selecting an emoticon, generating a static emoticon, and recording an animated emoticon). The electronic device displays (904) a preview of the virtual avatar in the virtual avatar generation interface (e.g., Figure 6N646) (e.g., a 2D or 3D computer-generated graphic object that, in some cases, is intended to convey a non-verbal message, such as an emotion or reaction) (e.g., a moving emoticon selected from a plurality of different available emoticon styles or templates). The preview of the virtual avatar reacts to changes in the appearance of the face in the camera's field of view (e.g., Fig.6O ) (e.g., the animated emoticon will reflect the user's head movement, facial expression, and orientation detected in image data from one or more image sensors). The electronic device receives (906) a request to generate an animated virtual avatar based on changing the facial expression of the face in the camera field of view (e.g., contact 652). In response to receiving the request to generate an animated virtual avatar, the electronic device records (908) (e.g., Figures 6Q to 6S ) a facial expression sequence of a face in the camera field of view (e.g., a sequence including a series of data points for providing a mapping of points that can be applied to a virtual avatar template to generate an animated virtual avatar). After recording the facial expression of the face in the camera field of view, the electronic device displays (910) a looped version of the animation sequence (e.g., Figure 6T and 6U ) is provided with an animated virtual avatar (e.g., 659) that is based on a facial expression sequence recorded in response to a request to generate an animated virtual avatar (e.g., by sequentially mapping a series of data points representing the recorded facial expressions to a predefined animated virtual avatar template to display the animated virtual avatar). The electronic device displays a looped version of the animated virtual avatar twice or more. In some embodiments, recording the facial expression sequence includes recording a time series of values ​​of discrete mapping points for a predefined virtual avatar template. Displaying a looped version of the animated virtual avatar enables the user to subsequently view the animated virtual avatar content to understand whether the appropriate message is being conveyed. Providing improved visual feedback to the user enhances the operability of the device, reduces instances of errors, and makes the user-device interface more efficient (e.g., by providing feedback indicating outputs that will cause the device to generate the expected results to help the user achieve the expected results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0278] In some embodiments, the electronic device stops recording the facial expression sequence in response to the expiration of a timer (e.g., represented by a progress indicator 656) (e.g., a 5 second, 10 second, or 15 second timer). In some embodiments, the electronic device stops recording the facial expression sequence in response to receiving a user input (e.g., contact 658) (e.g., a user tapping a virtual button displayed on a display). Limiting the time of an animated virtual avatar recording enables a user to create an animated virtual avatar recording while limiting the impact on the computing resources (e.g., storage) of the device. This enhances the operability of the device by conserving the computing resources of the device.

[0279] In some embodiments, the electronic device replaces (912) the display of the preview with the display of a looped version of the animated virtual avatar (e.g., see Figure 6S to Figure 6T The device also includes a plurality of recording modes, such as the one described in the embodiment of the present invention, wherein the user can select a recording mode, select a recording mode, and select a recording mode. The recording mode of the plurality of recording modes is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode. The recording mode is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode. The recording mode is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode. The recording mode is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode. The recording mode is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode. The recording mode is a plurality of recording modes, wherein the user selects a recording mode, and selects a recording mode.

[0280] In some embodiments, the request to generate an animated virtual avatar includes selecting a recording enable representation (e.g., 648) displayed in the virtual avatar generation interface. After recording the facial expression of the face in the camera field of view, the electronic device replaces the display of the recording enable representation with a sending or confirming enable representation (e.g., 664). In some embodiments, the sending enable representation is operated as explained above with respect to method 800. The animated virtual avatar is recorded in response to the selection of the recording enable representation, so that the user can use the virtual avatar preview to verify that the device is tracking the user and the currently selected virtual avatar template is consistent with the message the user wants to convey. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing a preview of the expected result before the user generates the expected result to help the user achieve the expected result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0281] In some embodiments, a looped version of the animated virtual avatar is displayed (918) using a first virtual avatar template (e.g., the avatar template corresponding to element 644-4). The electronic device displays representations of multiple other virtual avatar templates (e.g., elements 644-1 to 644-7) (e.g., miniature generic versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), including a representation of a second virtual avatar template (e.g., Figure 6Z 644-8 of element 644-8). The second virtual avatar template is different from the first virtual avatar template. In some embodiments, one or more of the multiple virtual avatar templates are based on emoticons that can be sent via a messaging application. After beginning to display the looping version of the animated virtual avatar and in response to receiving user input representing a selection of the first virtual avatar template, the electronic device updates (922) the display of the looping version of the animated virtual avatar to reflect the second virtual avatar template (e.g., while the animated virtual avatar is still based on the changing sequence of facial expressions, changing the appearance of the animated virtual avatar to reflect the new virtual avatar template). For example, the animated virtual avatar may change from a monkey to a robot, but it will still reflect the same sequence of changes in facial expressions in the same camera field of view over time (e.g., see from Figure 6T to Figure 6U In some embodiments, the electronic device displays (914) multiple representations of the virtual avatar template (e.g., miniature generic versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), the multiple representations including a representation of the first virtual avatar template. In response to receiving user input corresponding to a selection of a representation of the first virtual avatar template, the electronic device updates (916) the display of a looped version of the animated virtual avatar to correspond to the first virtual avatar template (e.g., changing the animation to a robot animation based on a robot virtual avatar template instead of a puppy animation based on a puppy virtual avatar template without requiring the user to re-record any facial expressions). Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the style of the animated virtual avatar to be selected after recording the movements and actions of the animated virtual avatar. Not having to re-record the animated virtual avatar to preview the new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding repeated generation of content), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0282] In some embodiments, the preview virtual avatar is based on the second avatar template. In response to detecting a face in the camera field of view (e.g., Fig.6OThe electronic device detects a first characteristic of a first physical characteristic of a face in the camera field of view (e.g., movement of a user's eyebrow), and based on the detected first characteristic, the electronic device updates (920) the first physical characteristic of the displayed virtual avatar preview (e.g., causing the virtual avatar preview to move its eyebrow). The first physical characteristic of the displayed preview has a first characteristic type (e.g., eyebrow). After receiving user input corresponding to selection of the first graphical element (e.g., switching the avatar template from a puppy to a monkey) and in response to detecting a second characteristic of the first physical characteristic of a face in the camera field of view (e.g., movement of a user's eyebrow), the electronic device updates (922) the second physical characteristic of the displayed virtual avatar preview (e.g., causing the virtual avatar preview to move its eyebrow) based on the detected second characteristic. Fig.6O 651-2) (e.g., moving a monkey's ear), where the second physical feature of the displayed preview has a second feature type (e.g., ear) that is different from the first feature type (e.g., eyebrow). Mapping the same physical feature of the user to different physical features of different avatar templates enables the user to have a wider range of options for conveying messages by generating multiple virtual avatars using the same input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing more options for conveying messages to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0283] In some embodiments, in response to receiving a signal related to scrolling (e.g., via Figure 6Y to Figure 6AA In response to a user input corresponding to a request for multiple representations of a virtual avatar template (contact 676), the electronic device scrolls the display of the multiple representations of the virtual avatar template to display a second graphical element that is not part of the multiple representations of the virtual avatar template. In some embodiments, the scrolling is based on the speed of the user input corresponding to the request. Scrolling through the virtual avatar template enables the user to quickly view different options for the virtual avatar. In addition, scrolling the display of the multiple representations of the virtual avatar template enables the user to view the previous virtual avatar template and the next virtual avatar template. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback indicating the output that will cause the device to generate the expected result to help the user achieve the expected result, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0284] In some embodiments, after the end of the user input is detected, the scrolling speed is gradually reduced over time (e.g., the scrolling gradually stops, just as the multiple representations of the virtual avatar have inertia that is gradually slowed by friction). Gradually reducing the scrolling speed over time enables the user to continue to view different virtual avatar template options without providing additional input. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback of possible results without the need for additional interaction to help the user achieve the desired results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0285] In some embodiments, in response to receiving a user input corresponding to a request, the electronic device generates an auditory output and / or tactile output corresponding to the currently selected virtual avatar template that changes from one virtual avatar template to a different virtual avatar template. For example, when each of the multiple representations of the virtual avatar template scrolls past a position indicating the currently selected virtual avatar template, an auditory output and / or tactile output is generated. Generating auditory or tactile feedback enables the user to determine when a new selection occurs. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by providing feedback indicating when a new selection is made to help the user achieve the desired results, and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] In some embodiments, in response to receiving user input on the animated virtual avatar, the input corresponding to a request to save the animated virtual avatar, the electronic device stores data for the animated virtual avatar in a database on the electronic device (e.g., Figure 6CC to Figure 6GG ). For example, the electronic device stores the data representing the animated virtual avatar in a directory or library in a non-volatile storage device in the electronic device.

[0287] In some embodiments, the electronic device receives a request (e.g., contact 688) to send an animated virtual avatar to a remote user of the remote device (e.g., Figures 6HH to 6KK). Based on determining that the remote device meets the first set of criteria (e.g., the remote device has a desired application for playing the desired version of the first version of the animated virtual avatar), the electronic device sends the first version of the animated virtual avatar to the user of the remote device (e.g., sending non-graphic data representing the recorded sequence of facial expressions and an indication of the virtual avatar template so that the remote device can reproduce the animated virtual avatar). Based on determining that the remote device does not meet the first set of criteria (e.g., the remote device does not have a suitable messaging application or a suitable version of the messaging application for playing the first version of the animated virtual avatar), the electronic device sends a second version of the animated virtual avatar that is different from the first version (e.g., sending a video file representing the animated virtual avatar) to the user of the remote device. Determining which version of the animated virtual avatar to send to the remote user conserves resources of the device by sending only a minimum amount of compatible information to the remote user. In addition, doing so reduces the need for the user to resend information in a more compatible format (e.g., in response to the remote user indicating that the initial format is not visible). Efficient and effective data transmission enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0288] In some embodiments, while displaying a looped version of the animated virtual avatar, the electronic device plays (924) audio data based on the sounds recorded while recording the sequence of facial expressions and based on an audio filter associated with a predefined avatar template. In some embodiments, while displaying a looped version of the animated virtual avatar, the electronic device plays audio data based on the sounds recorded while recording the sequence of facial expressions. In response to receiving an indication related to a mute affordance (e.g., Figure 6T and Figure 6U 662), the electronic device stops playing the audio data. The filtered audio for the animated virtual avatar is played based on a filter specific to the avatar template for the virtual avatar, which enables the user to more efficiently transmit messages by providing more options on how to transmit messages and more attractive animated virtual avatars. This enhances the operability of the device and makes the user device interface more effective (for example, by providing the user with more options on how to transmit messages to help the user achieve the intended message), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0289] In some embodiments, in response to receiving (926) a request to send an animated virtual avatar to a remote user and based on determining that a request to send the animated virtual avatar to the remote user is received while the audio data associated with the display of the looping version of the animated virtual avatar is muted (e.g., contact 667), the electronic device sends (928) data representing the animated virtual avatar to the remote user without sending sound data for the animated virtual avatar. Based on determining that a request to send the animated virtual avatar to the remote user is received while the audio data associated with the display of the looping version of the animated virtual avatar is not muted, the electronic device sends (930) the data representing the animated virtual avatar together with the sound data for the animated virtual avatar to the remote user. Sending the animated virtual avatar without sound when the user mutes the playback of the animated virtual avatar enables the user to effectively select whether to include sound in the message sent to the remote user. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user achieve the intended message with minimal interaction), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0290] In some embodiments, in response to receiving a request to generate an animated virtual avatar (e.g., contact 652), the electronic device records a first facial movement of a face in the camera field of view, wherein displaying a looped version of the animated virtual avatar includes animating the virtual avatar based on a physical model for the animated virtual avatar and the first facial movement (e.g., see image data 650-5, and Fig.6O Update 651-5). Updating the animated avatar based on the physical model for the avatar enables the user to create a realistic and interactive avatar that can convey a wider range of non-verbal communications. This enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user to convey a predetermined message using more realistic avatar movements), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0291] In some embodiments, in response to detecting that a particular feature of a face in the camera's field of view is held in a particular pose for more than a threshold amount of time while recording a sequence of facial expressions (e.g., see Fig.6O 650-4 and 650-5), the electronic device adds a predefined animated expression corresponding to a specific facial posture to the animated virtual avatar (for example, see Fig.6O651-5). For example, if the face has a neutral expression within a predetermined time period, a predefined movement (such as a head turn or a blink) is added to the animated virtual avatar. For another example, if the face has an angry expression within a predetermined time period, one or more additional features representing anger (such as the ears changing color or steam coming out of the ears) are added to the animated virtual avatar. Updating the animated virtual avatar based on the device detecting that features of the user's face remain in a particular posture for a threshold amount of time enables the user to add more actions to the animated virtual avatar than can be achieved using only facial expressions, features, and movements. This enhances the operability of the device and makes the user-device interface more effective (e.g., by providing additional mechanisms for transmitting actions that do not otherwise correspond to readily available facial expressions, movements, or features to help users achieve desired results), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0292] In some embodiments, while recording a sequence of facial expressions, in response to receiving user input via an input mechanism separate from the camera (e.g., a touch on a touch-sensitive surface, movement of the electronic device detected by a motion sensor, activation of a button, or other input), the electronic device adds a first facial expression to the sequence of facial expressions (e.g., recording a happy facial expression, a tongue-out facial expression, or any other facial expression that was not actually recorded as a facial expression made by a face in the camera's field of view when recording the facial expression for inclusion in the animated virtual avatar). The first facial expression is based on the user input received via the input mechanism. In some embodiments, while the animated virtual avatar is looping, the user can use a touch screen control to add additional expressions to the animated virtual avatar, so that as the animated virtual avatar loops, the user can gradually add expressions to the animated virtual avatar, so that even if those changes in expression are different from the changes in expression recorded based on the facial expressions of the face in the camera's field of view (e.g., the user's face) when the animated virtual avatar was initially created, the animated virtual avatar includes changes in the expression selected by the user. Updating the animated virtual avatar based on user input rather than as captured by the camera enables the user to add more actions to the animated virtual avatar than is possible using only facial expressions, features, and movements. This enhances the operability of the device and makes the user-device interface more effective (e.g., by providing additional mechanisms for delivering actions that do not otherwise correspond to readily available facial expressions, movements, or features to help the user achieve desired results), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0293] Note that the process described above with respect to method 900 (eg, FIG. 9A to FIG. 9B) may also be applied in a similar manner to the methods described above. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, displaying an animated virtual avatar preview based on a sequence of recorded facial features, movements, and / or expressions and based on a frame associated with an avatar template described with respect to method 900 may be applied to the sticker and animated virtual avatar interface described above with respect to method 800.

[0294] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., Figure 1A , Figure 3 and Figure 5A In addition, the above reference Fig.9A and Fig. 9B The operation is optionally performed by Figure 1A to Figure 1B . For example, receiving a request to generate an animated virtual avatar (906) is optionally implemented by an event classifier 170, an event identifier 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and an event distributor module 174 delivers the event information to the application 136-1. The corresponding event identifier 180 of the application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on a user interface. When a corresponding predefined event or sub-event is detected, the event identifier 180 activates an event handler 190 associated with the detection of the event or sub-event. The event handler 190 optionally utilizes or calls a data updater 176 or an object updater 177 to update the application internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, a person skilled in the art will clearly know how to Figure 1A to Figure 1B The components depicted in the figure are used to implement other processes.

[0295] FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B An exemplary user interface for generating and modifying a virtual avatar according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, which include FIG. 18A to FIG. 18B , Fig.19 , Fig. 20 , Fig.21 , Fig. 22 , Fig.23 , Fig.24 and Fig.25 process.

[0296] In some embodiments, a virtual avatar is a representation of a user that can be graphically depicted. In some embodiments, a virtual avatar is non-realistic (e.g., cartoonish). In some embodiments, an avatar is an anthropomorphic construct such as a stylized animal (e.g., avatars 1100, 1300, 1500, 1600, and 1700), a stylized robot (e.g., avatar 1400), or a stylization of a generally inanimate object (e.g., avatar 1000). In some embodiments, a virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond (e.g., map) to one or more physical features of the user's face, such that movement of the detected user's physical features affects the avatar features (e.g., affects the graphical representation of the features).

[0297] In some examples, the user can manipulate the characteristics or features of the virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164). When the user's physical features (such as facial features) and position (such as head position or head tilt) change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical features and position. In some embodiments, the changes to the user's physical features and position indicate various expressions, emotions, contexts, tones, or other non-verbal communications. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these expressions, emotions, contexts, tones, or other non-verbal communications.

[0298] FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B Exemplary user inputs and corresponding changes of exemplary virtual avatars (e.g., poop avatars, bear avatars, alien avatars, rabbit avatars, robot avatars, unicorn avatars, chicken avatars, and pig avatars) displayed on electronic devices are shown. In some embodiments, the electronic device includes one or more elements and / or features of devices 100, 300, and 500. FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B The image on the left represents an image of the user detected by the electronic device when the user is within the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In other words, the image of the user comes from the perspective of a camera (e.g., camera module 143, optical sensor 164), which in some embodiments can be located on the electronic device (e.g., devices 100, 300, and 500), and in other embodiments can be located separately from the electronic device (e.g., an external camera or sensor that transmits data to the electronic device). In some embodiments, FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B The boundaries of the image on the left represent the boundaries of the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In some embodiments, the user's image is displayed on a display (e.g., touch screen 112, display 340, display 450, display 504) of the electronic device. In some embodiments, the user's image is transmitted to an external electronic device for display. In some embodiments, the external electronic device includes one or more elements and / or features of devices 100, 300, and 500. In some embodiments, the user's image data is collected and processed by the device, but is not immediately displayed on the electronic device or transmitted to an external device.

[0299] FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B Each of the images on the left side shows a virtual avatar (e.g., a poop avatar) in a state presented (e.g., displayed after being modified) based on a detected corresponding user image located on the left side of the figure. In some embodiments, the virtual avatar is displayed from the perspective of the user observing the virtual avatar. In some embodiments, the virtual avatar is displayed on a display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504). In some embodiments, the virtual avatar is transmitted to an external electronic device for display. In some embodiments, FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B The image on the right side indicates the position of the virtual avatar within the display area of ​​the display of the electronic device (eg, touch screen 112, display 340, display 450, display 504), and FIG. 10A to FIG. 10I , FIG. 11A to FIG. 11C , FIG. 12A to FIG. 12C , Fig.13 , FIG. 14A to FIG. 14D , FIG. 15A to FIG. 15B , FIG. 16A to FIG. 16B as well as FIG. 17A to FIG. 17B The border of the image on the right side represents the border of the display area including the virtual avatar. In some embodiments, the display area represented in the right side corresponds to the avatar display area of ​​the application user interface, such as the virtual avatar interface 643 discussed above, the message writing area 612, the message area 609 (or a portion thereof).

[0300] In some embodiments, the magnitude of the reaction of an avatar feature (e.g., a discrete element of an avatar that can be discretely moved or modified relative to other avatar features) corresponds to the magnitude of the change in a physical feature of the user (e.g., a detected or tracked feature, such as a muscle, muscle group, or anatomical feature such as an eye). For example, in some embodiments, the magnitude of the change in a physical feature is determined based on a potential range of motion of the physical feature, where the magnitude represents the relative position of the physical feature within the range of motion of the physical feature (e.g., a predicted or modeled range of motion). In such embodiments, the magnitude of the reaction of the avatar feature is similarly the relative position of the avatar feature within the range of motion of the avatar feature. In some embodiments, the magnitude of the change is determined based on a comparison or measurement (e.g., a distance) of the starting and ending positions of the change in the physical feature. In such embodiments, the change in the physical feature can be converted into a modification of the first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly, or as a scaled or adjusted value).

[0301] In some embodiments, the modification of the avatar feature has a magnitude component and a direction component, wherein the direction component of the modification of the avatar feature is based on the direction component of the change in one or more physical features to which the avatar feature reacts. In some embodiments, the direction of the reaction of the avatar feature corresponds to (e.g., directly corresponds to or inversely corresponds to) the relative direction of the change in the user's physical feature, wherein the relative direction of the change in the physical feature is determined based on the direction of movement of the physical feature from an initial position (e.g., a neutral position, a resting position of the physical feature, or in some embodiments, a position of the physical feature initially detected by the device). In some embodiments, the direction of the reaction of the avatar feature corresponds directly to the relative direction of the change in the physical feature (e.g., if the physical feature moves upward, the avatar feature also moves upward). In other embodiments, the direction of the reaction of the avatar feature corresponds inversely to the relative direction of the change in the physical feature (e.g., if the physical feature moves upward, the avatar feature moves downward).

[0302] In some embodiments, the directional component of the change of the avatar feature is mirrored relative to the directional component of the change of the physical feature. For example, when the physical feature (e.g., the user's mouth) moves to the left, the avatar feature (e.g., the avatar mouth) moves to the right. In some embodiments, for movement along the vertical axis, the directional component of the change of the avatar feature is the same as the directional component of the change of the physical feature, and for movement along the horizontal axis, the directional component of the change of the avatar feature is mirrored to the directional component of the change of the physical feature, similar to the effect seen when looking in a mirror. In some embodiments, the neutral resting position of the user's iris is determined to be a specific position relative to the periphery of the user's eyeball (e.g., a centered position).

[0303] Fig. 10A An exemplary embodiment is shown that illustrates that an electronic device modifies a poop avatar 1000 in response to detecting a change in a user's facial features. The poop avatar is shown as having four display states (1011A, 1011B, 1011C, and 1011D), wherein each of the four display states of the poop avatar corresponds to the four detection states of the user (1001A, 1001B, 1001C, and 1001D). Specifically, in Fig. 10A In each display state of the avatar, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Fig. 10A In the illustrated embodiment, the detected facial features of the user include the user's mouth 1020 (with corners 1020A and 1020B) and the user's eyebrows 1022. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 10A In an embodiment of the present invention, the features of the poop avatar include the avatar face 1000C, the mouth 1030, the avatar eyes 1032, the upper part of the avatar 1034, the lower part of the avatar 1036, and (in some states) the avatar eyebrows 1038.

[0304] As shown in 1001A, the electronic device detects a neutral facial expression of the user. For example, the electronic device detects that the mouth 1020 and eyebrows 1022 of the user are positioned in a relaxed neutral state, rather than in a position associated with a specific facial expression (e.g., smile or frown). In response to detecting the neutral facial expression of the user, the electronic device displays a poop avatar 1000 with a neutral expression (e.g., neutral state) in 1011A. Specifically, the electronic device displays a poop avatar with a relaxed neutral state rather than a avatar mouth 1030 in a position typically associated with a specific facial expression (e.g., smile or frown). In some embodiments, the neutral position of the avatar corresponds to other representations of the relevant image, such as a static poop emoticon that can be found in a messaging application. In addition, the electronic device displays a face 1000C of the poop avatar without eyebrows 1038, and displays the avatar eyes 1032 looking in a direction perpendicular to the display (or the focal plane of the camera (e.g., camera 143, optical sensor 164)). The electronic device also displays an upper portion 1034 of the poop avatar at a neutral vertical position above a lower portion 1036 .

[0305] As shown in 1001B, the electronic device detects the position of the user's mouth 1020 for forming a smiling facial expression (e.g., one or both of the corners 1020A and 1020B of the user's mouth are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression). In response to detecting the position of the user's mouth 1020, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, so that the avatar's mouth 1030 has an open smiling position, as shown in 1011B. As shown in 1011B, the eyes 1032, the upper portion 1034, and the lower portion 1036 remain unchanged relative to their original positions in 1011A.

[0306] As shown in 1001C, the electronic device detects a frowning facial expression of the user. In some embodiments, the electronic device detects a frowning facial expression by detecting that one or both of the corners 1020A and 1020B of the user's mouth are positioned in a downward posture (e.g., a downward position) and that the user's eyebrows 1022 are in a downward position (e.g., the user's eyebrows are wrinkled on the user's face or positioned at a lower position on the user's face compared to the position of the eyebrows 1022 in the relaxed neutral state in 1001A and 1001B). In response to detecting the frowning facial expression of the user, the electronic device modifies the poop avatar to have a frowning and drooping face, as shown in 1011C. For example, the electronic device modifies the poop avatar so that the corners of the avatar's mouth 1030 are rotated downward so that the mouth 1030 has a slightly open posture, and the lower portion 1036 of the poop avatar is bent downward, similar to the downward turned posture of the poop avatar's mouth 1030. Fig. 10A In an embodiment, when the electronic device detects a downward position of the user's eyebrows 1022, the anatomical counterpart of the poop avatar above the avatar's eyes 1032 is not modified.

[0307] In some embodiments, the electronic device modifies the poop avatar by displaying an animation that the mouth 1030 turns downward and the lower portion 1036 bends downward when the mouth 1030 is moving to the downward turning position, so that it has the drooping face shown in 1011C. In some embodiments, the electronic device further modifies the poop avatar so that when the poop avatar presents a drooping face, the tip 1040 of the poop avatar collapses or tilts downward. In some embodiments, the position of the tip 1040 is specifically based on the position of the user's eyebrows 1022 (physical features that are not anatomically corresponding to the tip 1040). In some embodiments, when the user no longer makes a frowning expression, the electronic device modifies the poop avatar to return it to a neutral position. In such embodiments, the electronic device modifies the poop avatar by displaying an animation that the mouth 1030 moves to a neutral position and the lower portion 1036 moves back to its neutral position, so that it returns to the neutral state in 1011A. In some embodiments, returning the poop avatar from the droopy face to the neutral state includes the electronic device displaying the tip portion 1040 of the poop avatar as straightened to its neutral position.

[0308] As shown in 1001D, the electronic device detects that the corners 1020A and 1020B of the user's mouth are slightly raised and the user's eyebrows 1022 are in a raised position (e.g., at a higher position on the user's face than the position of the eyebrows 1022 when in the relaxed neutral state shown in 1001A and 1001B). In response to detecting the position of the corners 1020A and 1020B of the user's mouth 1020, the electronic device modifies the display of the poop avatar so that the corners of the avatar's mouth 1030 are slightly raised to match the position of the corners 1020A and 1020B of the user's mouth 1020. In response to detecting the raised position of the user's eyebrow 1022, the electronic device modifies the poop avatar by introducing the eyebrow 1038 located above the poop avatar's eye 1032 into the raised position (e.g., to convey the expression that the poop avatar 1000 is raising its eyebrow 1038) and extending the upper portion 1034 of the poop avatar in an upward direction (e.g., by extending the tip 1040 of the poop avatar while maintaining the original position of the lower portion 1036). In the embodiments shown in 1001D and 1011D, when the user's eyebrow 1022 is raised, the electronic device introduces the avatar's eyebrow 1038 and extends the tip 1040 of the poop avatar. In some embodiments, when the user's eyebrow 1022 returns to its neutral position, the electronic device removes the avatar's eyebrow 1038 and relaxes the tip 1040. In some embodiments, the electronic device removes the poop avatar's eyebrows 1038 by animating the eyebrows 1038 moving downward toward the poop avatar's eyes 1032 and disappearing into the poop avatar's face 1000C above the eyes 1032 .

[0309] Fig. 10B An exemplary embodiment is shown that shows an electronic device modifying a poop avatar in response to detecting a change in a user's facial features, wherein the modification to the poop avatar includes moving the avatar features in an exaggerated manner. In some embodiments, exaggerating the features of the virtual avatar allows the user to affect the maximum changes to the avatar features without uncomfortably changing corresponding features of their face. For example, Fig. 10B As shown, the user can cause the avatar to open their mouth as wide as possible (e.g., in a surprised expression) without causing the user to open their mouth uncomfortably (e.g., without causing the user to open their mouth to the maximum range of the predicted or determined range of motion of the user's mouth).

[0310] Fig. 10B The poop avatar is shown as having three display states (1012A, 1012B, and 1012C), wherein each of the three display states of the poop avatar corresponds to the three detection states (1002A, 1002B, and 1002C) of the user. Fig. 10BIn each display state of the avatar, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Fig. 10B In the illustrated embodiment, the detected facial features of the user include the user's mouth 1020. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 10B In the embodiment of the present invention, the features of the poop avatar include an avatar mouth 1030, avatar eyes 1032, an upper portion 1034 of the avatar, and a lower portion 1036 of the avatar.

[0311] As shown in 1002A, the electronic device detects a first state of the user, in which the user's mouth 1020 is slightly open (e.g., 10% of the maximum range of the predicted or determined range of motion of the user's mouth is open). In response to detecting the slightly open user's mouth 1020, the electronic device modifies the poop avatar, as shown in 1012A, so that the poop avatar's mouth 1030 has an open position (20% of the maximum range of the simulated range of motion of the avatar's mouth, the degree of opening being greater than (e.g., within its corresponding range of motion) the degree of opening of the user's mouth 1020), while keeping other features of the avatar (such as the poop avatar's eyes 1032, upper portion 1034, and lower portion 1036) unchanged.

[0312] As shown in 1002B, the electronic device detects a change in the user's facial features in which the user's mouth 1020 is opened wider than in state 1002A (e.g., opened 25% of the maximum range of the predicted or determined range of motion of the user's mouth). In response to detecting that the user's mouth 1020 transitions from the slightly open position in 1002A to the wider open position in 1002B, the electronic device modifies the poop avatar's mouth 1030 to increase in size, as shown in 1012B, so that the mouth 1030 has a larger open position than the position shown in 1012A (e.g., 50% of the maximum range of simulated range of motion of the avatar's mouth) while still maintaining the positioning of other avatar features (including the poop avatar's eyes 1032, upper portion 1034, and lower portion 1036).

[0313] As shown in 1002C, the electronic device detects yet another change in the user's facial features, in which the user's mouth 1020 is opened even wider than in 1002B (e.g., 50% of the maximum range of the predicted or determined range of motion of the user's mouth). In response to detecting that the user's mouth 1020 transitions from the open position in 1002B to the wider open position in 1002C, the electronic device modifies the mouth 1030 of the poop avatar to further increase in size, as shown in 1012C, so that the mouth 1030 has an open position that is larger than the position shown in 1012B (e.g., 100% of the maximum range of simulated range of motion of the avatar's mouth). However, in 1012C, the opening of the mouth 1030 is greater than the height of the lower portion 1036. Therefore, the electronic device causes the lower portion 1036 of the poop avatar 1000 to expand at 1042 to maintain the structural integrity of the poop avatar 1000 in response to the user's open mouth 1020. In other words, in order to maintain a consistent positioning of the avatar and its features in response to the user's open mouth 1020, the device modifies one or more interconnected portions of the virtual avatar (e.g., lower portion 1036). For example, as shown in 1012C, the electronic device expands the lower portion 1036 of the poop avatar adjacent to the avatar's mouth 1030 at region 1042 to accommodate the increased size of the enlarged avatar's mouth 1030. If the electronic device did not modify the lower portion 1036 in this manner, the enlarged mouth 1030 would extend beyond the structure of the virtual avatar, thereby potentially hindering the context and / or tone that the user intended to convey using the virtual avatar.

[0314] In some embodiments, the increase in size of the avatar's mouth 1030 (e.g., from the position shown in 1012A to the position shown in 1012B, or from the position shown in 1012B to the position shown in 1012C) is not proportional to the increase in size of the user's mouth 1020 (e.g., from the position shown in 1002A to the position shown in 1002B, or from the position shown in 1002B to the position shown in 1002C), but is scaled to provide an exaggerated rate of change in size. For example, in some embodiments, the scaling ratio is a multiplication factor of 2, so that the relative open position of the avatar's mouth is twice the relative open position of the user's mouth. For example, if the user's mouth is open 10% of the maximum range of motion of the user's mouth, the electronic device displays the avatar's mouth opening 20% ​​of the maximum range of simulated motion of the avatar's mouth.

[0315] Fig. 10CAn exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics, wherein the modification of the poop avatar includes rotating (e.g., tilting) an upper portion of the poop avatar forward and backward while keeping a lower portion of the poop avatar still. The electronic device displays the poop avatar as having three display states (1013A, 1013B, and 1013C), wherein each of the three display states of the poop avatar corresponds to the three detection states (1003A, 1003B, and 1003C) of the user, respectively. Specifically, in Fig. 10C In each display state of the avatar, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Fig. 10C In the embodiment shown in , the user's physical features include the user's face 1024, chin 1026, and head 1028. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 10C In the embodiment of the present invention, the features of the poop avatar include a mouth 1030 of the avatar, eyes 1032 of the avatar, an upper portion 1034 of the avatar (including a tip 1040 of the poop avatar), and a lower portion 1036 of the avatar.

[0316] As shown in 1003A, the electronic device detects a user's face 1024 having a position rotated in an upward direction. For example, the user lifts his chin 1026 and tilts his head 1028 backward (e.g., away from the camera (e.g., camera 143, optical sensor 164) tilted field plane) to position his face 1024 in an upward direction. In response to detecting the upward position of the user's face 1024, the electronic device modifies the poop avatar to an upward-looking state by tilting the upper portion 1034 of the poop avatar in a direction away from the display (e.g., away from the camera (e.g., camera 143, optical sensor 164) focal plane backward), as shown in 1013A. In this backward tilted position, the electronic device displays the tip 1040 of the poop avatar positioned toward the rear side of the poop avatar 1000 to show the offset position of the tip 1040 when the poop avatar is modified to the upward-looking state. In addition, the electronic device modifies the poop avatar's eyes 1032 to an upward-looking state (e.g., by offsetting the poop avatar's pupils or irises 1032A toward the top of the avatar's eyes 1032), as shown in 1013A. When the electronic device tilts the upper portion 1034 of the poop avatar and modifies the eyes 1032 to an upward-looking state, the electronic device keeps other features of the poop avatar unchanged. For example, the electronic device maintains the position of the avatar's mouth 1030 and fixes the position of the lower portion 1036 of the poop avatar, so that the electronic device shows the upper portion 1034 rotated backward around an axis (e.g., an x-axis) that extends along the width of the lower portion 1036.

[0317] As shown in 1003B, the electronic device detects a user's face 1024 having a position rotated in a downward direction. For example, the user lowers (tucks in) his chin 1026 and tilts his head 1028 forward (e.g., toward the focal plane of the camera (e.g., camera 143, optical sensor 164)) to position his face 1024 in a downward direction. In response to detecting the downward position of the user's face 1024, the electronic device modifies the poop avatar to a downward-looking state by tilting the upper portion 1034 of the poop avatar in a direction toward the display (e.g., toward the front of the camera (e.g., camera 143, optical sensor 164) focal plane), as shown in 1013B. In this forward-tilted position, the electronic device displays the tip 1040 of the poop avatar positioned toward the front side of the poop avatar to show the offset position of the tip 1040 when the poop avatar is modified to a downward-looking state. In addition, the electronic device modifies the poop avatar's eyes 1032 to a downward-looking state (e.g., by offsetting the poop avatar's pupil or iris 1032A toward the bottom of the avatar's eyes 1032), as shown in 1013B. When the electronic device tilts the upper portion 1034 of the poop avatar and modifies the eyes 1032 to a downward-looking state, the electronic device keeps other features of the poop avatar unchanged. For example, the electronic device maintains the position of the avatar's mouth 1030 and fixes the position of the lower portion 1036 of the poop avatar, so that the electronic device shows the upper portion 1034 rotated forward about an axis (e.g., an x-axis) that extends along the width of the lower portion 1036.

[0318] As shown in 1003C, the electronic device detects that the user's position (specifically, the position of the user's face 1024 and head 1028) is in the downwardly rotated position shown in 1003B, but the user's position is also offset downward from the position shown in 1003B within the field of view of the camera (e.g., camera 143, optical sensor 164). In response, the electronic device displays the poop avatar 1000 as being in the forward tilted position shown in 1013B, but the poop avatar is also offset downward within the display area in 1013C to mirror the downward direction of the user's offset within the camera's field of view.

[0319] Fig. 10D An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a user's physical characteristics, wherein the modification of the poop avatar includes rotating an upper portion of the poop avatar while keeping a lower portion of the poop avatar stationary. The poop avatar is shown as having four display states (1014A, 1014B, 1014C, and 1014D), wherein each of the four display states of the poop avatar corresponds to four detection states of the user (1004A, 1004B, 1004C, and 1004D), respectively. Fig. 10DIn each display state of the avatar, the electronic device positions or modifies features of the poop avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Fig. 10D In the embodiment shown in , the user's physical features include the user's mouth 1020, face 1024, head 1028, and shoulders 1021. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 10D In the embodiment of the present invention, the features of the poop avatar include a mouth 1030 of the avatar, eyes 1032 of the avatar, an upper portion 1034 of the avatar (including a tip 1040 of the poop avatar), and a lower portion 1036 of the avatar.

[0320] As shown in 1004A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the user's right side while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the right (while keeping the lower portion 1036 stationary), as shown in 1014A, so that the twisting motion of the virtual avatar mirrors the rightward rotational movement of the user's head 1028 and face 1024. The electronic device also detects a smiling gesture of the user's mouth 1020 and modifies the avatar's mouth 1030 to a smiling state. As shown in 1014A, the electronic device twists the upper portion 1034 of the poop avatar around an axis 1051 (e.g., a y-axis) that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the right, the electronic device also shifts the tip 1040 of the poop avatar to the left, moves the poop avatar's eyes 1032 to the right, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, so that the poop avatar has a distorted appearance including a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnection relationship between the upper portion 1034 and the lower portion 1036. Even though parts of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the physical features of the user, these modifications to the poop avatar provide an animation effect that mimics the physical movements of the user.

[0321] 1004B and 1014B show similar effects, where the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the user's left while the user's shoulders 1021 remain positioned forward. In response, the electronic device modifies the poop avatar by twisting the upper portion 1034 of the poop avatar to the left (while keeping the lower portion 1036 stationary), as shown in 1014B, so that the twisting motion of the virtual avatar mirrors the leftward rotational movement of the user's head 1028 and face 1024. As shown in 1014B, the electronic device twists the upper portion 1034 of the poop avatar about an axis 1051 that extends vertically through the center of the poop avatar. When the electronic device twists the upper portion 1034 of the poop avatar to the left, the electronic device also shifts the tip portion 1040 of the poop avatar to the right, moves the poop avatar's eyes 1032 to the left, and increases the amount of wrinkles 1034A or layers formed in the upper portion 1034 of the poop avatar, so that the poop avatar has a distorted appearance including a slight twist 1070 of the lower portion 1036, which is modeled based on the interconnected relationship between the upper portion 1034 and the lower portion 1036.

[0322] In some embodiments, the electronic device does not track the movement (e.g., rotational movement) or positioning of the user's shoulder 1021, so that the user can affect changes in the virtual avatar without having to maintain a fixed orientation or position in front of the camera (e.g., camera 143, optical sensor 164). For example, as shown in 1004C, the user's shoulder 1021 tilts or rotates to the user's right, but the lower portion 1036 of the poop avatar remains fixed, as shown in 1014C. However, the electronic device detects that the user's head 1028 and optionally the user's face 1024 rotate to the user's right. Therefore, as shown in 1014C, the electronic device modifies the poop avatar accordingly as discussed above with respect to 1014A, without further modifying the poop avatar in response to the user rotating his shoulder 1021. A similar effect is shown in 1004D and 1014D, where the user's shoulder 1021 tilts or rotates to the user's left along with the user's head 1028, and the electronic device modifies the poop avatar as discussed above with respect to 1014B without further modifying the poop avatar (e.g., lower portion 1036) in response to the user rotating his shoulder 1021.

[0323] Fig. 10EAn exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in a physical feature of a user, wherein the modification to the poop avatar includes tilting an upper portion of the poop avatar while keeping a lower portion of the poop avatar still. The poop avatar is shown to have four display states (1015A, 1015B, 1015C, and 1015D), wherein each of the four display states of the poop avatar corresponds to four detection states of the user (1005A, 1005B, 1005C, and 1005D), respectively. In each display state, the electronic device positions or modifies features of the poop avatar in response to detecting a position or change in position of one or more physical features (such as facial features) of the user detected in a corresponding state of the user. Fig. 10E In the embodiment shown in , the user's physical features include the user's mouth 1020, face 1024, head 1028, shoulders 1021, eyes 1023, and neck 1025. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 10E In the embodiment of the present invention, the features of the poop avatar include a mouth 1030 of the avatar, eyes 1032 of the avatar, an upper portion 1034 of the avatar (including a tip 1040 and a middle portion 1031 of the poop avatar), and a lower portion 1036 of the avatar.

[0324] As shown in 1005A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are tilted to the user's right while the user's shoulders 1021 remain positioned forward. The electronic device also detects that the user's neck 1025 is tilted slightly to the user's right. In response, the electronic device modifies the poop avatar by tilting the upper portion 1034 of the poop avatar to the right (the upper portion 1034 of the poop avatar includes the tip 1040 and the middle portion 1031) while keeping the lower portion 1036 still, as shown in 1015A, so that the tilting movement of the virtual avatar mirrors the tilting to the right of the user's head 1028 (and / or face 1024) and neck 1025.

[0325] In addition to mirroring the tilt direction, the electronic device also modifies the virtual avatar to illustrate the different tilt degrees present in the various physical features of the user shown in 1005A. For example, the upper part of the user (e.g., the head 1028 of the user) is tilted to a greater extent than the lower part of the user (e.g., the neck 1025 of the user). Therefore, as shown in 1015A, the electronic device modifies the virtual avatar so that the movement or tilt amount at the top of the virtual avatar (e.g., at the tip 1040) is the largest, and the movement or tilt amount at the bottom of the virtual avatar (e.g., at the lower part 1036) is the smallest. In other words, the tilt amount gradually decreases from the top of the virtual avatar to the bottom of the virtual avatar, which is consistent with the change in the tilt degree shown in the user in 1005A. The tip 1040 with a large tilt degree, the middle part 1031 with a smaller tilt degree than the tip 1040, and the lower part 1036 that is not tilted of the poop avatar in 1015A prove this point. Even though portions of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the user's physical features, these modifications to the poop avatar provide animation effects that mimic the user's physical movements. In addition, for example, when the user tilts their head 1028 and neck 1025, the electronic device modifies the virtual avatar to have different degrees of tilt to mimic the reduced range of motion of the user's physical features.

[0326] 1005B and 1015B illustrate a similar effect, where the electronic device detects that the user's head 1028 and neck 1025 are tilted to the user's left. In response, the electronic device modifies the poop avatar by tilting the upper portion 1034 of the poop avatar to the left at a different degree of tilt (e.g., the tip portion 1040 tilts to a greater degree than the middle portion 1031), while keeping the lower portion 1036 stationary, as shown in 1015B, so that the tilting motion of the virtual avatar mirrors the leftward tilt of the user's head 1028 and neck 1025, as discussed in more detail with respect to 1005A and 1015A.

[0327] As shown in 1005C, the electronic device detects the user's eye 1023 (e.g., the iris or pupil of the user's eye) shifted to the user's right and a smiling facial expression formed by the corners 1020A and 1020B of the user's mouth 1020 positioned in an upward posture. In response to detecting the rightward shift of the user's eye 1023, the electronic device modifies the poop avatar's eye 1032 to a rightward-looking state (e.g., by shifting the poop avatar's pupil or iris 1032A toward the right side of the avatar's eye 1032), as shown in 1015C. In response to detecting one or both of the corners 1020A and 1020B of the user's mouth positioned in an upward posture, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, wherein the avatar's mouth 1030 has an open smiling position, as shown in 1015C. As shown in 1015C, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A).

[0328] 1005D and 1015D illustrate similar effects, where the electronic device detects a smiling facial expression of the user and the user's eye 1023 (e.g., the iris or pupil of the user's eye) shifted to the left of the user. In response to detecting the leftward shift of the user's eye 1023, the electronic device modifies the poop avatar's eye 1032 to a leftward-looking state (e.g., by shifting the poop avatar's pupil or iris 1032A toward the left side of the avatar's eye 1032), as shown in 1015D. In response to detecting one or both of the corners 1020A and 1020B in an upward pose, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, as discussed above with respect to 1015C. Similarly, in 1015D, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A and 1015C).

[0329] Fig.10F An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a shift in the position of a user within the field of view of a camera (e.g., camera 143, optical sensor 164). Modification of the poop avatar includes shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user within the field of view of the camera. The poop avatar is shown as having four display states (1016A, 1016B, 1016C, and 1016D) in four display areas, wherein each of the four display states of the poop avatar corresponds to four detection states (1006A, 1006B, 1006C, and 1006D) of the user, respectively. Fig.10FIn each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting a position or position change of the user detected within the camera field of view displayed in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1016A, 1016B, 1016C, and 1016D) represent the boundaries of the display area including the virtual avatar.

[0330] As shown in 1006A, the electronic device detects that the user's position is horizontally centered within the field of view of the camera (e.g., camera 143, optical sensor 164). In response to detecting the user's horizontally centered position within the camera's field of view, the electronic device displays a poop avatar with a horizontally centered position in the display area as shown in 1016A.

[0331] In 1006B, the electronic device detects that the user's position is off-center (e.g., offset or translated) in the right direction in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is offset in the left direction of the user (e.g., offset to the right relative to the field of view of the camera), but remains fully visible within the field of view of the camera. In response to detecting the offset position of the user in 1006B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) so that the poop avatar is displayed at a position offset to the left, as shown in 1016B, so as to mirror the user's directional offset to the user's left direction. As shown in 1006B, the user is offset to his left side, so that his left shoulder 1021A is close to the right edge of the field of view. Therefore, the electronic device displays a poop avatar located near the left edge of the display area of ​​1016B, which mirrors the direction of the user's offset position within the camera's field of view. In some embodiments, the offset position of the user is mirrored by the offset of the virtual avatar in direction and magnitude. In some embodiments, the offset position of the user is mirrored by the offset of the virtual avatar only in direction, and the magnitude of the offset of the virtual avatar is adjusted (e.g., attenuated) to maintain the position of the virtual avatar within the boundaries of the display area. Examples of such embodiments are discussed below with respect to 1006C, 1016C, 1006D, and 1016D.

[0332] In 1006C, the electronic device detects that the user's position is off-center (e.g., offset or translated) in the rightmost direction in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is far offset to the user's left (e.g., offset to the right relative to the field of view of the camera), but offset to such an extent that the user's left shoulder 1021A is no longer within the camera's field of view. In response to detecting the user's significantly offset position in 1006C, the electronic device offsets the horizontal position of the poop avatar so that the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) is displayed as being in a position completely offset to the left, wherein the outermost edge of the poop avatar (e.g., edge 1036A of the lower portion 1036) is positioned against the left border of the display area shown in 1016C. The offset display mirror of the poop avatar in 1016C reflects the direction of the user's offset to the left of the user, but rather than offsetting the poop avatar so that a portion of the avatar extends beyond the display area in 1016C (as the user did in 1006C), the device positions the poop avatar at the edge of the display area 1016C. By maintaining the position of the virtual avatar within the display area (e.g., 1016A, 1016B, 1016C, and 1016D) even when a portion of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect changes to the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0333] A similar effect is shown in 1006D and 1016D. In 1006D, the electronic device detects that the user is offset to the right (e.g., offset to the left in the field of view of the camera) so that the user's right shoulder is no longer within the field of view of the camera. In response to detecting the user's far offset position, the electronic device offsets the horizontal position of the poop avatar so that the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) is displayed as being in a position completely offset to the right, wherein the outermost edge of the poop avatar (e.g., edge 1036B of the lower portion 1036) is positioned against the right boundary of the display area shown in 1016D. As described above, by maintaining the position of the virtual avatar within the display area 1016D even when a portion of the user is beyond the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to affect changes in the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0334] Figure 10G An exemplary embodiment is shown in which the electronic device modifies the poop avatar in response to detecting a shift in the position of a user's physical feature within the field of view of a camera (e.g., camera 143, optical sensor 164). The modification of the poop avatar includes shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user's physical feature within the field of view of the camera. Figure 10G The embodiment shown in is similar to the above with respect to FIG. 10D to FIG. 10F In the embodiments discussed, the electronic device tracks the movement and position (e.g., rotational movement and / or translational movement) of the user's head 1028, but does not track the movement or position of the user's shoulders 1021 and optionally the user's neck 1025. In addition, Figure 10G The illustrated embodiment is similar to Fig.10F Embodiments in which modifications to the virtual avatar mirror the user's movement in direction, but not necessarily in magnitude. By implementing these techniques, the electronic device allows the user to affect changes in the virtual avatar without having to maintain a fixed orientation or position in front of a camera (e.g., camera 143, optical sensor 164).

[0335] The poop avatar is shown as having two display states (1017A and 1017B) in two display areas, wherein each of the two display states of the poop avatar corresponds to the two detection states (1007A and 1007B) of the user, respectively. Figure 10G In each of the display states, the device positions or modifies the poop avatar in response to detecting a position or change in position of a physical feature of the user detected within the camera field of view displayed in the corresponding state of the user. In each of the two display states, the boundaries of the displayed states (e.g., the boundaries of 1017A and 1017B) represent the boundaries of the display area that includes the virtual avatar.

[0336] In 1007A, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset (e.g., translated) in the left direction in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset in the user's right direction (e.g., offset to the left relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007A, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the poop avatar is displayed at a position offset to the right, as shown in 1017A, so as to mirror the offset of the user's head to the user's right direction.

[0337] A similar effect is shown in 1007B and 1017B. In 1007B, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset (e.g., translated) in the right direction in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset in the user's left direction (e.g., offset to the right relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007B, the electronic device offsets the horizontal position of the entire poop avatar (including both the upper portion 1034 and the lower portion 1036) so that the poop avatar is displayed at a position offset to the left, as shown in 1017B, so as to mirror the offset of the user's head to the user's left direction.

[0338] Fig. 10H An exemplary embodiment is shown in which an electronic device modifies a poop avatar in response to detecting a change in the position of a user's physical features within the field of view of a camera (e.g., camera 143, optical sensor 164). Modifications to the poop avatar include increasing or decreasing the size of the poop avatar, and shifting (e.g., translating) the poop avatar in a direction corresponding to the shift in the position of the user's physical features within the camera's field of view. The poop avatar is shown as having four display states (1018A, 1018B, 1018C, and 1018D) in four display areas, wherein each of the four display states of the poop avatar corresponds to four detection states (1008A, 1008B, 1008C, and 1008D), respectively, of the user. Fig. 10H In each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting a position or change in position of a physical feature of the user detected within the camera field of view displayed in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1018A, 1018B, 1018C, and 1018D) represent the boundaries of the display area including the virtual avatar.

[0339] In 1008A, the electronic device detects that the user's head 1028 is offset (e.g., translated) in an upward direction relative to its shoulder 1021 in the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is stretching its neck 1025 upward). In response to detecting the upward offset position of the user's head 1028 in 1008A, the electronic device offsets the vertical position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) so that the entire poop avatar is displayed as being in an upward offset position, wherein the uppermost edge of the poop avatar (e.g., edge 1040A of the tip 1040) is positioned close to the upper boundary of the display area shown in 1018A so as to mirror the offset of the user's head 1028 in the upward direction shown in 1008A.

[0340] In 1008B, the electronic device detects that the user's head 1028 is offset (e.g., translated) in a downward direction relative to the user's shoulder 1021 in the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is hanging his head 1028). In response to detecting the downward offset position of the user's head 1028 in 1008B, the electronic device offsets the vertical position of the entire poop avatar (including both the upper part 1034 and the lower part 1036) so that the entire poop avatar is displayed as being in a downward offset position, wherein the lowermost edge of the poop avatar (e.g., edge 1036C of the lower part 1036) is positioned close to the lower boundary of the display area shown in 1018B so as to mirror the offset of the user's head 1028 in the downward direction shown in 1008B.

[0341] In 1008C, the electronic device detects that, for example, the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) increases when the user's head 1028 is positioned closer to the camera. In response to detecting the increase in size of the user's head 1028 in 1008C, the electronic device increases the size of the entire poop avatar. In some embodiments, the electronic device changes the size of the user's head 1028 from one detected state (e.g., Fig. 10A The size of the poop avatar is increased by changing the size of the poop avatar from the neutral state in 1001A of the electronic device (e.g., the detected state 1008C). In 1018C, the electronic device increases the size of the poop avatar to fill the display area without causing a portion of the poop avatar to extend beyond the boundaries of the display area. In some embodiments, the electronic device increases the size of the virtual avatar to provide the impression that the avatar is positioned very close to the display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504).

[0342] For example, the electronic device increases the size of the poop avatar in 1018C so that the tip 1040 of the poop avatar is adjacent to the upper border of the display area at 1040A, the lower portion 1036 of the poop avatar is adjacent to the lower border of the display area at 1036C, the left edge of the lower portion 1036 is adjacent to the left border of the display area at 1036A, and the right edge of the lower portion 1036 is adjacent to the right border of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018C, the electronic device proportionally increases the size of the poop avatar so that the relative positions of various avatar features (e.g., the avatar's eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) are not deformed relative to the shape of the poop avatar. For example, when the electronic device increases the size of the poop avatar in 1018C, the sizes of the avatar's eyes 1032, mouth 1030, upper portion 1034 (including tip 1040), and lower portion 1036 also increase, but the other parts remain unchanged.

[0343] In 1008D, the electronic device detects that, for example, when the user's head 1028 is positioned away from the camera, the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) decreases. In response to detecting the decrease in size of the user's head 1028 in 1008D, the electronic device decreases the size of the entire poop avatar. In some embodiments, the electronic device changes the size of the user's head 1028 from one detected state (e.g., Fig. 10A In some embodiments, such as in 1018D, the electronic device reduces the size of the poop avatar to provide the impression that the virtual avatar is positioned away from the display of the electronic device (e.g., touch screen 112, display 340, display 450, display 504).

[0344] For example, the electronic device reduces the size of the poop avatar in 1018D so that the tip 1040 of the poop avatar is positioned away from the upper border of the display area at 1040A, the lower portion 1036 of the poop avatar is positioned away from the lower border of the display area at 1036C, the left edge of the lower portion 1036 is positioned away from the left border of the display area at 1036A, and the right edge of the lower portion 1036 is positioned away from the right border of the display area at 1036B. In some embodiments, such as the embodiment shown in 1018D, the electronic device proportionally reduces the size of the poop avatar so that the relative positions of various avatar features (e.g., the avatar's eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) are not deformed relative to the shape of the poop avatar. For example, when the electronic device reduces the size of the poop avatar in 1018D, the sizes of the avatar's eyes 1032, mouth 1030, upper portion 1034 (including tip 1040), and lower portion 1036 are also reduced, but the other parts remain unchanged.

[0345] Fig.10I An exemplary embodiment is shown that illustrates that an electronic device modifies a poop avatar 1000 in response to detecting a change in a physical feature (such as a facial feature) of a user. The poop avatar is shown as having a display state 1019 corresponding to the detection state 1009 of the user. As shown in 1009 and 1019, the electronic device detects that the user is making a pouting expression, and in response, the electronic device modifies the poop avatar 1000 by replacing the poop avatar's mouth 1030 with a set of pouting lips 1050. In some embodiments, such as the embodiment shown in 1009, the electronic device determines that the user is making a pouting facial expression by detecting that the user's jaw 1027 is in a closed position, and detecting that the corners 1020A and 1020B of the user's mouth 1020 move toward each other so that the user's lips 1029 (e.g., the user's upper lip 1029A and lower lip 1029B) extend outward from the user's mouth 1020 in a pouting posture. Although not shown in FIG. Fig.10I However, in some embodiments, the electronic device modifies the poop avatar so that it appears similar to Fig. 16B Hearts are emitted from pursed lips 1050 in the manner shown in and discussed in more detail below.

[0346] Fig.11A An exemplary embodiment is shown illustrating that an electronic device modifies a bear avatar 1100 in response to detecting a change in a user's facial features. The bear avatar is shown as having four display states (1111A, 1111B, 1111C, and 1111D), wherein each of the four display states of the bear avatar corresponds to the four detection states of the user (1101A, 1101B, 1101C, and 1101D). Fig.11AIn each display state of the avatar, the electronic device positions or modifies features of the bear avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. Fig.11A In the illustrated embodiment, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig.11A In an embodiment of the present invention, the features of the bear avatar include avatar mouth 1130, avatar eyes 1132, avatar ears 1133, avatar nose 1137, avatar head 1135, and (in some states) avatar eyebrows 1138.

[0347] As shown in 1101A, the electronic device detects a neutral facial expression of the user. For example, the device detects that the user's mouth 1120 and eyebrows 1122 are positioned in a relaxed neutral state, rather than in a position associated with a particular facial expression (such as a smile or a frown). In response to detecting the user's neutral facial expression, the electronic device displays a bear avatar 1100 with a neutral expression or state in 1111A. Specifically, the electronic device displays a bear avatar with an avatar mouth 1130 in a relaxed neutral state rather than in a position typically associated with a particular facial expression (e.g., a smile or a frown). In some embodiments, the neutral position of the avatar corresponds to other representations of the associated image, such as a static bear emoticon that can be found in a messaging application. In 1111A, the neutral state of the bear's mouth 1130 is indicated by lines 1130A and 1130B, which extend horizontally from the outside of the bear's mouth 1130 or "muzzle" area and then bend slightly upward at the base of the bear's nose 1137. The electronic device also displays a bear head with a nose 1137 and ears 1133, wherein the nose is located above a relaxed mouth 1130, and the ears are in a relaxed neutral state, positioned along the side of the bear's head 1135, and are not curled or stretched. In addition, the electronic device displays the bear head without eyebrows 1138, and displays the head's eyes 1132 looking in a direction perpendicular to the display (or the focal plane of a camera (e.g., camera 143, optical sensor 164)).

[0348] As shown in 1101B and 1111B, the electronic device detects the position of the user's mouth 1120 (e.g., detects that one or both of the corners 1120A and 1120B of the user's mouth are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression), and in response, the electronic device modifies two features of the bear avatar. For example, in response to detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture, the electronic device modifies both the bear ears 1133 and the bear mouth 1130. The electronic device modifies the bear's mouth 1130 to have a smiling expression by rotating the lines 1130A and 1130B upward, and the upward rotation of the lines 1130A and 1130B indicates the smiling expression of the bear's mouth 1130, as shown in 1111B. The electronic device modifies the bear's ears 1133 to be "uprighted" or extended in an upward direction. In some embodiments, such as the embodiment shown in 1111B, the modification of the bear ears 1133 also includes slightly narrowing the width of each ear 1133 and changing the vertical position of the ears 1133 on the side of the bear head 1135 so that when the ears are erected upward, the ears 1133 are positioned at a higher position on the bear head 1135 (compared to the vertical position of the ears 1133 on the side of the bear head 1135 in the neutral state in 1111A). As shown in 1111B, the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user smiles, the electronic device changes the bear's mouth 1130 to a smiling state and causes the ears 1133 to erect upward. When the user stops smiling, the electronic device returns the mouth 1130 and ears 1133 to their neutral positions.

[0349] As shown in 1101C, the electronic device detects that the user's eyebrows 1122 are in a raised position (e.g., at a higher position on the user's face than the position of the eyebrows 1122 when in the relaxed neutral state shown in 1101A and 1101B). In response to detecting the raised eyebrows 1122, the electronic device modifies two features of the bear avatar. For example, the electronic device modifies the bear ears 1133 to be erected upward, and modifies the bear avatar to introduce eyebrows 1138 located above the bear eyes 1132, thereby conveying the impression that the bear avatar 1100 is raising its eyebrows 1138, as shown in 1111C. As shown in 1111B, the bear's mouth 1130 returns to its neutral position, and the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's eyebrows 1122 are raised, the electronic device introduces the avatar's eyebrows 1138 and causes the ears 1133 to be erected upward. Thus, when the user's eyebrows 1122 return to their neutral position, the electronic device removes the avatar's eyebrows 1138 and relaxes the ears 1133.

[0350] In 1101D, the electronic device detects that the user's eyebrows 1122 are raised and one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture. In response, the electronic device modifies the bear head 1100 to achieve a greatly erected ear, introduces a raised eyebrow 1138, and causes it to smile, as shown in 1111D. The electronic device modifies the ears 1133 to achieve a greatly erected ear by significantly reducing the width of each ear 1133, extending the ear in an upward direction, and changing the vertical position of the ear 1133 on the side of the bear head 1135 so that the ear 1133 is positioned at a position even higher than when it is in the erected position shown in 1111B and 1111C. This combination of modifications to the bear ears 1133 causes the appearance of the ears 1133 to be greatly stretched in the upward direction, which is greater than the degree of erection of the ears shown in 1111B and 1111C. It should be understood that in this embodiment, when the user raises his eyebrows 1122 and smiles (e.g., one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture), the electronic device introduces the avatar's eyebrows 1138, modifies the bear's mouth 1130 to a smiling state, and greatly erects the ears 1133. Therefore, when the user's eyebrows 1122 and mouth 1120 return to their neutral positions, the electronic device removes the avatar's eyebrows 1138 and relaxes the ears 1133 and mouth 1130.

[0351] Fig. 11B An exemplary embodiment is shown that illustrates an electronic device modifying a bear avatar 1100 in response to detecting a change in a user's facial features. The bear avatar is shown as having four display states (1112A, 1112B, 1112C, and 1112D), wherein each of the four display states of the bear avatar corresponds to the four detection states of the user (1102A, 1102B, 1102C, and 1102D). Fig. 11B In each of the four display states, the electronic device positions or modifies features of the bear avatar in response to detecting a position or position change of one or more physical features (such as facial features) of the user detected in the corresponding state of the user. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1112A, 1112B, 1112C, and 1112D) represent the boundaries of the display area including the virtual avatar.

[0352] exist Fig. 11BIn the illustrated embodiment, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some embodiments, the tracked physical features may include other facial features, such as eyelids, irises, muscles, muscle groups, etc. Fig. 11B In an embodiment of the present invention, the features of the bear avatar include avatar mouth 1130, avatar eyes 1132, avatar ears 1133, avatar nose 1137, avatar head 1135, and (in some states) avatar eyebrows 1138.

[0353] As shown in 1102A, the electronic device detects that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture (e.g., a downward position), and in response, the electronic device modifies two features of the bear head. In some embodiments, the electronic device determines that the user is making a sad facial expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth are in a downward position, and optionally detecting that one or both of the user's eyebrows 1122 are not in a lower position (e.g., eyebrows 1122 are raised or in their neutral position). In response to detecting the downward posture or position of the corners 1120A and 1120B of the user's mouth 1120, the electronic device modifies both the bear mouth 1130 and the bear ears 1133, as shown in 1112A.

[0354] The electronic device modifies the bear's mouth 1130 to have a sad expression by rotating lines 1130A and 1130B downward, which forms the sad expression of the bear's mouth 1130, as shown in 1112A. The electronic device modifies the bear's ears 1133 to "curl" or "droop" in a downward direction by rotating the outer edges 1133A and 1133B of the bear's ears 1133 downward (e.g., folding, rotating, or tilting). In some embodiments, curling the bear's ears 1133 also includes changing the vertical position of the ears 1133 on the sides of the bear's head 1135, so that when the ears are curled, the ears 1133 are positioned at a lower position on the bear's head 1135 (compared to being in a lower position). Fig.11A 111A). As shown in 1112A, the bear eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's mouth 1120 forms a sad facial expression, the electronic device changes the bear's mouth 1130 to a sad expression and curls the bear's ears 1133. When the user no longer makes a sad facial expression, the electronic device returns the bear's mouth 1130 and ears 1133 to their neutral positions.

[0355] In 1102B and 1112B, the electronic device detects that the user has maintained a sad facial expression for a threshold amount of time, and in response, the electronic device modifies the bear avatar 1100 by introducing tears 1140 emanating from the bear's eyes 1132 to display the bear avatar as crying when the bear avatar makes a sad facial expression with curled ears 1133. In some embodiments, when the user maintains the sad expression for a first threshold amount of time (e.g., 0.3 seconds), the electronic device modifies the bear avatar to introduce only a single tear 1140 emanating from one or both of the bear's eyes 1132, and then, after the user continues to maintain the sad expression for a second threshold amount of time (e.g., 0.6 seconds), the electronic device modifies the bear avatar to introduce additional tears 1140 emanating from the eyes 1132. The electronic device displays the tears 1140 formed from the bear's eyes 1132, flowing down the bear's face, dripping from the bear's head 1135, and disappearing from view at the bottom edge of the display area in 1112B. In some embodiments, the electronic device displays tears 1140 intermittently and optionally at random intervals. In some embodiments, tears 1140 continue to appear until the user stops making a sad facial expression. As shown in 1112B, the electronic device continues to display curled bear ears 1133 and continues to display the bear mouth 1130 making a sad expression.

[0356] In 1102C and 1112C, the electronic device detects that the user maintains a frowning facial expression for a threshold amount of time, and in response, the electronic device modifies the bear avatar 1100 to have a frowning expression and a storm cloud 1142 located above the bear head 1135. The electronic device detects the user's frowning expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture and detecting that the user's eyebrows 1122 are in a downward position (e.g., the user's eyebrows are wrinkled or positioned at a lower position on the user's face compared to the position of the eyebrows 1122 in the relaxed neutral state in 1101A). In response to detecting the user's frowning facial expression, the electronic device modifies the bear avatar by rotating the bear mouth 1130 downward (e.g., by bending the lines 1130A and 1130B downward, as shown in 1112C) and introducing the bear eyebrows 1138 in a frowned state to form a frowning eyebrow posture (as shown in 1112C). In the embodiment shown in 1112B, the electronic device does not immediately introduce storm clouds 1142 in response to detecting the user's frowning facial expression, nor does the electronic device modify the bear's ears 1133 or nose 11311.

[0357] After the electronic device detects that the user maintains the frowning facial expression for a threshold amount of time, the electronic device modifies the frowning bear avatar 1100 to introduce a storm cloud 1142 located above the bear head 1135. In some embodiments, the storm cloud 1142 is animated and dynamic. For example, the electronic device can modify the storm cloud 1142 to introduce lightning 1144 and / or rain. In some embodiments, the electronic device displays rain falling from the storm cloud 1142 and landing on the bear avatar head 1135. In some embodiments, the falling rain flows down the bear face, falls from the bear head 1135, and disappears from view at the bottom edge of the dis...

Claims

1. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts to a change in a second physical feature of the face in a different manner depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; and Based on determining that the detected first change in the corresponding physical feature is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or within the second change range.

2. The method according to claim 1, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and The appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to the appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

3. The method according to claim 1, further comprising: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

4. The method according to claim 1, further comprising: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a possible variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

5. The method according to claim 1, further comprising: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

6. The method of claim 5, wherein the first manner of changing the appearance of the second avatar feature is different from the second manner of changing the appearance of the second avatar feature.

7. The method according to claim 5, further comprising: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

8. The method according to claim 5, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes avatar eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the variation of the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

9. The method according to claim 5, wherein: modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic comprises modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

10. The method according to claim 5, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; Changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature includes modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical feature; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical feature includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical feature.

11. An electronic device, comprising: one or more cameras; Display device; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature, wherein the second avatar feature reacts to a change in a second physical feature of the face in a different manner depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different from the first range of change of the second physical feature; detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras when displaying the virtual avatar; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; and Based on determining that the detected first change in the corresponding physical feature is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or within the second change range.

12. The electronic device according to claim 11, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and The appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to the appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

13. The electronic device of claim 11, wherein the one or more programs further comprise instructions for performing the following operations: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

14. The electronic device according to claim 11, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a possible variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

15. The electronic device of claim 11, wherein the one or more programs further comprise instructions for performing the following operations: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

16. The electronic device of claim 15, wherein the first manner of changing the appearance of the second avatar feature is different from the second manner of changing the appearance of the second avatar feature.

17. The electronic device according to claim 15, wherein: modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic comprises modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

18. The electronic device according to claim 15, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; Changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature includes modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical feature; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical feature includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical feature.

19. The electronic device of claim 15, wherein the one or more programs further comprise instructions for performing the following operations: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

20. The electronic device according to claim 19, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes avatar eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the variation of the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

21. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs comprising instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts to a change in a second physical feature of the face in a different manner depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature; while displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: According to determining that the virtual avatar is based on the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature; and According to determining that the virtual avatar is based on a second virtual avatar template different from the first virtual avatar template: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; and Based on determining that the detected first change in the corresponding physical feature is a change in the second physical feature, modifying the second avatar feature to reflect the first change in the second physical feature, wherein the appearance of the second avatar feature is changed in the same manner regardless of whether the change in the second physical feature is within the first change range or within the second change range.

22. The non-transitory computer readable storage medium of claim 21, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a potential variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first portion of the change within the first range of change to a second portion of the change within the second range of change; and The appearance of the second avatar feature at the end of the first portion of changes to reflect the change in the second physical characteristic is substantially similar to the appearance of the second avatar feature at the beginning of the second portion of changes to reflect the change in the second physical characteristic.

23. The non-transitory computer readable storage medium of claim 21, wherein the one or more programs further comprise instructions for: Based on determining that the detected first change is a change in the second physical characteristic and the change in the second physical characteristic is a change from the first change range to the second change range, the second avatar feature is modified based on a predetermined configuration of the second avatar feature.

24. The non-transitory computer readable storage medium of claim 21, wherein: The first variation range of the second physical characteristic and the second variation range of the second physical characteristic are adjacent ranges in a possible variation range of the second physical characteristic; The first change in the corresponding physical characteristic includes a change in the second physical characteristic, the change including a transition from a first change portion within the second range of change to a second change portion within the first range of change; and When the change of the second physical feature transitions from the first change portion within the second change range to the second change portion within the first change range, the appearance of the second avatar feature changes from the configuration of the second avatar feature based on the magnitude of the change of the second physical feature to a predetermined configuration of the second avatar feature.

25. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions for: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, the appearance of the second avatar feature is changed in a second manner to reflect the change in the second physical feature.

26. The non-transitory computer readable storage medium of claim 25, wherein the first manner of changing the appearance of the second avatar feature is different than the second manner of changing the appearance of the second avatar feature.

27. The non-transitory computer readable storage medium of claim 25, wherein: modifying the first avatar characteristic of the virtual avatar to reflect the change in the first physical characteristic comprises modifying the first avatar characteristic based on a magnitude of the change in the first physical characteristic; changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic comprises modifying the second avatar feature based on a first magnitude of the change in the second physical characteristic; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical characteristic includes modifying the second avatar feature based on a second magnitude of the change in the second physical characteristic.

28. The non-transitory computer readable storage medium of claim 25, wherein: modifying the first avatar feature of the virtual avatar to reflect the change in the first physical feature includes modifying a pose of at least a portion of the first avatar feature based on a direction of the change in pose of the first physical feature; Changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature includes modifying a pose of at least a portion of the second avatar feature based on a direction of the change in pose of the second physical feature; and Changing the appearance of the second avatar feature in the second way to reflect the change in the second physical feature includes modifying the pose of at least a portion of the second avatar feature based on the direction of the change in pose of the second physical feature.

29. The non-transitory computer readable storage medium of claim 25, wherein the one or more programs further comprise instructions for: Further in response to detecting the first change in the corresponding physical characteristic: According to determining that the virtual avatar is based on the first virtual avatar template: Based on the determination that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first change range, abandoning changing the appearance of the second avatar feature in the second manner to reflect the change in the second physical feature.

30. The non-transitory computer readable storage medium of claim 29, wherein: The second physical feature includes an eyelid of the user; The second avatar feature includes avatar eyes; Changing the appearance of the second avatar feature in the second manner within the second range of variation based on the variation of the second physical feature includes reducing the size of the avatar's eyes; and Changing the appearance of the second avatar feature in the first manner within the first range of variation based on the change in the second physical feature does not include reducing the size of the avatar's eyes.

31. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the avatar cheek, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature includes modifying the avatar cheek based on the change magnitude of the user's cheek.

32. An electronic device comprising: one or more cameras; Display device; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the avatar cheek, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature includes modifying the avatar cheek based on the change magnitude of the user's cheek.

33. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs comprising instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises a cheek of the user and the second avatar feature comprises an avatar cheek; while displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the first change range, changing the appearance of the second avatar characteristic in a first manner to reflect the change in the second physical characteristic; and Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the second change range, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical feature; wherein changing the appearance of the second avatar feature in the second manner within the second change range based on the change in the second physical feature includes introducing an exaggerated avatar effect at the avatar cheeks, and changing the appearance of the second avatar feature in the first manner within the first change range based on the change in the second physical feature includes modifying the avatar cheeks based on the change magnitude of the user's cheeks.

34. A method comprising: At an electronic device having one or more cameras and a display device: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

35. An electronic device comprising: one or more cameras; Display device; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; While displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within the first range of change, changing the appearance of the second avatar feature in a first manner to reflect the change in the second physical feature; and based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

36. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and a display device, the one or more programs comprising instructions for performing the following operations: Displaying a virtual avatar via the display device, wherein the virtual avatar comprises: a first avatar feature responsive to changes in a first physical feature of a face within a field of view of the one or more cameras; a second avatar feature that reacts differently to a change in a second physical feature of the face depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different than the first range of change of the second physical feature, wherein the second physical feature of the face comprises an eyebrow of the user and the second avatar feature comprises an upper portion of the avatar's head; while displaying the virtual avatar, detecting a first change in a corresponding physical feature of the face within the field of view of the one or more cameras; and In response to detecting the first change in the corresponding physical characteristic, modifying the virtual avatar includes: Based on determining that the detected first change in the corresponding physical characteristic is a change in the first physical characteristic, modifying the first avatar characteristic to reflect the change in the first physical characteristic; Based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the first change range, changing the appearance of the second avatar characteristic in a first manner to reflect the change in the second physical characteristic; and based on determining that the detected first change is a change in the second physical characteristic and that the change in the second physical characteristic is within the second range of change, abandoning changing the appearance of the second avatar feature in the first manner to reflect the change in the second physical characteristic and changing the appearance of the second avatar feature in a second manner to reflect the change in the second physical characteristic; Wherein changing the appearance of the second avatar feature in the second manner within the second variation range based on the change of the second physical feature includes introducing an exaggerated avatar effect at the upper part of the avatar's head, and changing the appearance of the second avatar feature in the first manner within the first variation range based on the change of the second physical feature does not include introducing the exaggerated avatar effect at the upper part of the avatar's head.

Images