Recording and sending emojis

The method of generating virtual avatars by detecting facial expression changes in real time solves the problem of complex and time-consuming generation and sending of emojis in existing technologies, and achieves more efficient generation of emojis and virtual avatars, saving device energy and improving user experience.

CN115393485BActive Publication Date: 2026-11-10APPLE INC
View PDF 22 Cites 0 Cited by

Patent Information

Application Number
CN202211050080.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-09-22
Filing Date
2018-01-23
Publication Date
2026-11-10
Estimated Expiration
2038-01-23

AI Technical Summary

Technical Problem

Existing technologies are complex and time-consuming in generating and sending emojis and virtual avatars, resulting in wasted user time and device energy, especially in battery-powered devices.

Method used

By displaying a virtual avatar generation interface on electronic devices, real-time detection of facial expression changes is achieved to generate static or animated virtual avatars. The generation time is determined by the input timing, providing a faster and more efficient interface and method.

Benefits of technology

It reduces the cognitive burden on users, improves the efficiency of generating and sending emojis, saves device power, and extends battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115393485B_ABST
    Figure CN115393485B_ABST
Patent Text Reader

Abstract

This disclosure relates throughout to generating and modifying virtual avatars. An electronic device having a camera and a display device displays a virtual avatar whose appearance changes in response to changes in a face within the camera's field of view. In response to detecting changes in one or more physical features of the face within the camera's field of view, the electronic device modifies one or more features of the virtual avatar.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application filed on January 23, 2018, with application number 201880004632.1 and titled "Recording and Sending Emojis".

[0002] Cross-references to related applications

[0003] This patent application claims priority to U.S. Provisional Application No. 62 / 507,177, filed May 16, 2017, entitled "Emoji Recording and Sending"; U.S. Provisional Application No. 62 / 556,412, filed September 9, 2017, entitled "Emoji Recording and Sending"; and U.S. Provisional Application No. 62 / 557,121, filed September 11, 2017. This patent application also claims priority to Danish Patent Application No. PA201770393, filed May 29, 2017, entitled "Emoji Recording and Sending"; Danish Patent Application No. PA201770720, filed September 22, 2017, entitled "Emoji Recording and Sending"; and Danish Patent Application No. PA201770721, filed September 22, 2017, entitled "Emoji Recording and Sending". The contents of these patent applications are hereby incorporated herein by reference in their entirety. Technical Field

[0004] This disclosure relates generally to computer user interfaces, and more specifically to techniques for generating, recording, and sending emojis and virtual avatars. Background Technology

[0005] Multimedia content (such as emojis and avatars) is sometimes sent as part of messaging communications. Emojis and avatars represent a variety of predefined people, objects, actions, and / or other things. Some messaging applications allow users to choose from a predefined library of emojis and avatars to send as part of messages that may contain other content (e.g., other multimedia and / or text content). Stickers are another type of multimedia content, sometimes sent along with messaging applications. In some ways, stickers are similar to emojis and avatars because they can represent people, objects, actions, and / or other things. Some sticker and / or messaging applications allow stickers to be associated with previously sent or received messages. Summary of the Invention

[0006] However, some technologies used to generate, send, and receive emojis and avatars on electronic devices are often cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple keystrokes or button presses. Other existing technologies use complex and time-consuming methods to manipulate and generate emojis and avatars, including requiring users to provide large amounts of input to achieve the desired emoji (e.g., desired animated or dynamic emojis). Existing technologies require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, this technology provides electronic devices with faster and more efficient methods and interfaces for sending and receiving emojis and virtual avatars. Such methods and interfaces optionally complement or replace other methods for sending and receiving emojis. These methods and interfaces reduce the cognitive burden on users and result in more efficient human-computer interfaces. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging sessions.

[0008] According to some embodiments, a method performed on an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression in the camera's field of view over a time period, based on determining that the input includes activation of a recording enable function in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera. The one or more programs include instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression in the camera's field of view over a time period, based on determining that the input includes activation of a recording enable function in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0010] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression in the camera's field of view over a time period, based on determining that the input includes activation of a recording enable representation in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; a camera; one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression in the camera's field of view over a time period, based on determining that the input includes activation of a recording enable function in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and a preview of a virtual avatar displayed in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance within the camera's field of view; means for: detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; and means for: in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression within the camera's field of view over a time period, based on determining that the input includes activation of a recording enable function in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0013] According to some embodiments, a method performed at an electronic device having a display and a camera is described. The method includes: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view; in response to receiving the request to generate the animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; and after recording the facial expressions of the face in the camera's field of view, displaying a looping version of the animated virtual avatar, the looping version including an animation sequence based on the facial expression sequence recorded in response to the request to generate the animated virtual avatar, wherein displaying the looping version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0014] According to some embodiments, a non-transitory computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view; recording a sequence of facial expressions of the face in the camera's field of view in response to receiving the request to generate the animated virtual avatar; and, after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0015] According to some embodiments, a transient computer-readable storage medium is described. A non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display and a camera, the one or more programs including instructions for performing the following operations: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view; recording a sequence of facial expressions of the face in the camera's field of view in response to receiving the request to generate an animated virtual avatar; and, after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in a camera's field of view; receiving a request to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; and after recording the facial expressions of the face in the camera's field of view, displaying a looped version of the animated virtual avatar, the looped version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looped version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a camera; a display for displaying a virtual avatar generation interface and a preview of a virtual avatar displayed in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; means for: receiving a request to generate an animated virtual avatar based on changes in facial expressions of a face in the camera's field of view; means for: recording a sequence of facial expressions of a face in the camera's field of view in response to receiving the request to generate an animated virtual avatar; and means for: after recording the facial expressions of a face in the camera's field of view, causing a looping version of the animated virtual avatar to be displayed, the looping version including an animation sequence based on the sequence of facial expressions recorded in response to the request to generate the animated virtual avatar, wherein displaying the looping version of the animated virtual avatar includes displaying the animation sequence two or more times.

[0018] In some implementations, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in facial appearance in the camera's field of view; detecting input in the virtual avatar generation interface while displaying the preview of the virtual avatar; in response to detecting input in the virtual avatar generation interface: generating a static virtual avatar representing a facial expression in the camera's field of view at a corresponding time, based on determining that input has begun on the preview of the virtual avatar, wherein the corresponding time is determined based on the timing of the input; and generating an animated virtual avatar representing a sequence of changes in facial expression in the camera's field of view over a time period, based on determining that the input includes activation of a recording enable function in the virtual avatar generation interface, wherein the time period is determined based on the timing of the input.

[0019] In some implementations, the method includes: at an electronic device having a camera and a display: displaying a virtual avatar generation interface; displaying a preview of a virtual avatar in the virtual avatar generation interface, wherein the preview of the virtual avatar responds to changes in the appearance of a face in the camera's field of view; receiving a request to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view; in response to receiving the request to generate an animated virtual avatar, recording a sequence of facial expressions of the face in the camera's field of view; and after recording the facial expressions of the face in the camera's field of view, displaying a looping version of the animated virtual avatar, the looping version including an animated sequence based on the sequence of facial expressions recorded in response to the request to generate an animated virtual avatar, wherein displaying the looping version of the animated virtual avatar includes displaying the animated sequence two or more times.

[0020] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, the virtual avatar changing its appearance in response to changes in a face within the field of view of the one or more cameras, wherein the virtual avatar includes: a first portion and a second portion different from the first portion; while displaying the virtual avatar via the display device, detecting changes in facial pose within the field of view of the one or more cameras; and changing the appearance of the virtual avatar in response to detecting the changes in facial pose, including: based on determining that the changes in facial pose include a first type of facial pose change, changing the appearance of the virtual avatar includes moving the first portion of the virtual avatar relative to the second portion of the virtual avatar according to an amount of the first type of facial pose change; and based on determining that the changes in facial pose include a second type of facial pose change, changing the appearance of the virtual avatar includes moving the first portion and the second portion of the virtual avatar based on an amount of the second type of facial pose change.

[0021] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face in the field of view of one or more cameras and a second physical feature of a face within the field of view of one or more cameras; and a second avatar feature; while displaying the virtual avatar via the display device, detecting changes in one or more physical features of a face within the field of view of one or more cameras; modifying the first avatar feature of the virtual avatar based on changes in the first physical feature, and abandoning modification of the second avatar feature based on changes in the first physical feature, based on changes in the first physical feature; and modifying the first avatar feature based on changes in the second physical feature, and abandoning modification of the second avatar feature based on changes in the second physical feature, based on changes in the second physical feature;

[0022] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of the one or more cameras; a second avatar feature that responds to changes in the first physical feature; and a third avatar feature that does not primarily respond to changes in the first physical feature; detecting changes in the first physical feature while displaying the virtual avatar; and in response to detecting changes in the first physical feature: modifying the first avatar feature based on the detected changes in the first physical feature; modifying the second avatar feature based on the detected changes in the first physical feature; and abandoning modification of the third avatar feature based on the detected changes in the first physical feature.

[0023] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of one or more cameras; a second avatar feature that responds differently to changes in a second physical feature of a face, the manner depending on whether the change in the second physical feature of the face occurs within a first range of change of the second physical feature or within a second range of change of the second physical feature, the second range of change of the second physical feature being different from the first range of change of the second physical feature; and simultaneously detecting the one or more cameras while displaying the virtual avatar. A first change in a corresponding physical feature of a face within the field of view; and, in response to detecting the first change in the corresponding physical feature, modifying a virtual avatar, including: modifying a first avatar feature to reflect the change in the first physical feature based on determining that the detected first change in the corresponding physical feature is a change in the first physical feature; and changing the appearance of a second avatar feature in a first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within a first change range; and abandoning the change in the appearance of the second avatar feature in the first manner to reflect the change in the second physical feature based on determining that the detected first change is a change in the second physical feature and that the change in the second physical feature is within a second change range.

[0024] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes a plurality of avatar features that respond to changes in one or more physical features of a face within the field of view of the one or more cameras; simultaneously displaying the virtual avatar, detecting changes in the plurality of physical features of the face, the plurality of physical features of the face including a first physical feature corresponding to one or more of the plurality of avatar features and a second physical feature not corresponding to any of the plurality of avatar features; and in response to detecting changes in the plurality of physical features of the face: changing the appearance of a corresponding avatar feature of the plurality of avatar features, wherein the magnitude and / or direction of the change in the corresponding avatar feature is based on the magnitude or direction of the change in the first physical feature; and prior to detecting changes in the one or more physical features of the face, deforming a portion of the virtual avatar that does not include avatar features, wherein the magnitude and / or direction of the deformation of the portion of the avatar feature is based on the magnitude and / or direction of the change in the second physical feature.

[0025] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar includes: a first avatar feature that responds to changes in a first physical feature of a face within the field of view of one or more cameras; detecting a change in a first physical feature having a magnitude of the change in the first physical feature while displaying the virtual avatar; in response to detecting the change in the first physical feature: changing the first avatar feature by a magnitude of the change in the first avatar feature based on a first physical feature value determined to be within a first physical feature value range; and changing the first avatar feature by a magnitude of the change in the first avatar feature based on a second physical feature value range different from and based on the magnitude of the change in the first physical feature.

[0026] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar has a corresponding spatial position in a reference frame, wherein the corresponding spatial position is based on the position of a face within the field of view of one or more cameras; while displaying the virtual avatar, detecting a corresponding amount of change in the position of the face within the field of view of one or more cameras; in response to detecting the change in the position of the face within the field of view of one or more cameras: modifying the spatial position of the virtual avatar in the reference frame based on the magnitude of the first change component and the first modification factor, according to determining that the change in the position of the face includes a first change component in a first direction; and modifying the spatial position of the virtual avatar in the reference frame based on the magnitude of the second change component and the second modification factor, the second modification factor being different from the first modification factor, according to determining that the change in position includes a second change component in a second direction different from the first direction.

[0027] In some embodiments, the method includes: at an electronic device having one or more cameras and a display device: displaying a virtual avatar via the display device, wherein the virtual avatar responds to changes in one or more physical features of a face within the field of view of the one or more cameras; while displaying the virtual avatar, detecting a first configuration of one or more physical features of the face; while detecting the first configuration of one or more physical features of the face: modifying the virtual avatar to include a first animation effect based on determining that the first configuration of one or more physical features meets an animation criterion, the animation criterion including requiring the first configuration to be maintained for at least a first threshold amount of time so that the animation criterion is met; and abandoning the modification of the virtual avatar to include the first animation effect based on the first configuration of one or more physical features not meeting the animation criterion.

[0028] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0029] Therefore, faster and more efficient methods and interfaces are provided for devices to generate, send, and receive emojis, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. These methods and interfaces can complement or replace other methods used for sending and receiving emojis. Attached Figure Description

[0030] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0031] Figure 1A This is a block diagram illustrating a portable multi-functional device with a touch-sensitive display according to some embodiments.

[0032] Figure 1B This is a block diagram illustrating exemplary components for event handling according to some implementation schemes.

[0033] Figure 2 A portable multi-functional device with a touchscreen is shown according to some embodiments.

[0034] Figure 3 This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to some implementation schemes.

[0035] Figure 4A An exemplary user interface for a menu on an application on a portable multifunction device, according to some implementation schemes, is shown.

[0036] Figure 4B An exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0037] Figure 5A A personal electronic device according to some implementation schemes is shown.

[0038] Figure 5B This is a block diagram illustrating a personal electronic device according to some implementation schemes.

[0039] Figures 6A to 6MM An exemplary user interface for generating and sending emojis, stickers, avatars, and / or other multimedia content is shown.

[0040] Figures 7A to 7J An exemplary user interface for receiving emojis, stickers, avatars, and / or other multimedia content is shown.

[0041] Figures 8A to 8B This is a flowchart illustrating methods for generating and sending emojis, stickers, avatars, and / or other multimedia content.

[0042] Figures 9A to 9B This is a flowchart illustrating methods for generating and sending emojis, stickers, avatars, and / or other multimedia content.

[0043] Figures 10A to 10I An exemplary user interface for generating and modifying poop avatars is shown.

[0044] Figures 11A to 11C An exemplary user interface for generating and modifying bear avatars is shown.

[0045] Figures 12A to 12C An exemplary user interface for generating and modifying virtual alien avatars is shown.

[0046] Figure 13 An exemplary user interface for generating and modifying a rabbit avatar is shown.

[0047] Figures 14A to 14D An exemplary user interface for generating and modifying robot virtual avatars is shown.

[0048] Figures 15A to 15B An exemplary user interface for generating and modifying unicorn avatars is shown.

[0049] Figures 16A to 16B An exemplary user interface for generating and modifying virtual chicken avatars is shown.

[0050] Figures 17A to 17B An exemplary user interface for generating and modifying virtual pig avatars is shown.

[0051] Figures 18A to 18B This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0052] Figure 19 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0053] Figure 20 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0054] Figure 21This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0055] Figure 22 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0056] Figure 23 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0057] Figure 24 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras.

[0058] Figure 25 This is a flowchart illustrating a method for generating and modifying virtual avatars based on faces detected by one or more cameras. Detailed Implementation

[0059] The following description illustrates exemplary methods, parameters, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0060] Sending messages with both multimedia and text content, or multimedia content alone, can potentially better convey the sender's message. For example, multimedia content such as virtual avatars (e.g., animated or static emoticons or stickers) can provide context and / or tone that is difficult or impossible to convey using text alone (e.g., this can be referred to as "nonverbal communication"). In some cases, predefined virtual avatars can be used to provide some of that context and / or tone. However, predefined content cannot cover all situations or provide finely tuned context or tone. Therefore, there is a need for electronic devices that provide efficient methods and interfaces for generating, sending, and receiving virtual avatars as part of a message. Such technologies can reduce the cognitive burden on users sending and receiving messages, thereby increasing productivity. Furthermore, such technologies can reduce processor and battery power wasted in other ways on redundant user input.

[0061] under, Figures 1A to 1B , Figure 2 , Figure 3 , Figures 4A to 4B as well as Figures 5A to 5B Description of an exemplary device is provided for performing techniques for generating, sending, and receiving virtual avatars. Figures 6A to 6MM and Figures 7A to 7J An exemplary user interface for receiving, generating, modifying, and sending virtual avatars is shown. Figures 8A to 8B and Figures 9A to 9BThis is a flowchart illustrating an exemplary method for receiving, generating, modifying, and sending virtual avatars. Figures 6A to 6MM as well as Figures 7A to 7J The user interface in the document is used to illustrate the processes described below, including... Figures 8A to 8B and Figures 9A to 9B The process in. Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B Exemplary user interfaces for generating and modifying virtual avatars, according to some embodiments, are shown. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 18A to 18B , Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 , Figure 24 and Figure 25 The process in. Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B User interface and Figures 18A to 18B , Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 , Figure 24 and Figure 25 The process can be used to generate in Figures 6A to 6MM as well as Figures 7A to 7J interface and Figures 8A to 8B as well as Figures 9A to 9B The virtual avatar used during the process.

[0062] Although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch may be named a second touch and similarly, a second touch may be named a first touch, without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0063] The terminology used in the description of the various embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “includes”, “including”, “comprises”, and / or “comprising”, when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0064] Depending on the context, the term "if" may optionally be interpreted as meaning "when," "at," or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" may optionally be interpreted as meaning "in response to determination..." or "in response to detection [the stated condition or event]."

[0065] This document describes implementations of electronic devices, user interfaces for such devices, and related processes for using such devices. In some implementations, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functionality. Exemplary implementations of portable multi-functional devices include, but are not limited to, those from Apple Inc. (Cupertino, California). Devices, iPod Equipment, and Device. Optionally, other portable electronic devices may be used, such as laptops or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0066] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device may optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0067] The device typically supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, game applications, phone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camcorder applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0068] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally utilizes a user interface that is intuitive and clear to the user to support various applications.

[0069] Now let’s turn our attention to implementation schemes for portable devices with touch-sensitive displays. Figure 1A This is a block diagram illustrating a portable multi-functional device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a “touchscreen” for convenience, and is sometimes referred to as or called a “touch-sensitive display system.” Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (e.g., generating haptic output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0070] As used in this specification and claims, the term "intensity" of contact on a tactile surface refers to the force or pressure (force per unit area) of a contact (e.g., finger contact) on a tactile surface, or to a substitute (alternative) for the force or pressure of a contact on a tactile surface. The intensity of contact has a range of values ​​that includes at least four different values ​​and more typically hundreds of different values ​​(e.g., at least 256). The intensity of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the tactile surface are optionally used to measure the force at different points on the tactile surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated contact force. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the tactile surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensitive surface, the capacitance and / or variation of the touch-sensitive surface near the contact, and / or the resistance and / or variation of the touch-sensitive surface near the contact may optionally be used as substitutes for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement) has been exceeded. In some embodiments, the substitute measurement of the contact force or pressure is converted into an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure) has been exceeded. Using the intensity of the contact as an attribute of user input allows the user to access additional device functions that would otherwise be inaccessible to the user on smaller devices with limited physical space, such smaller devices being used (e.g., on a touch-sensitive display) to display power indications and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls, such as knobs or buttons).

[0071] As used in this specification and claims, the term "haptic output" refers to a physical displacement of the device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device, which is detected by the user using the user's tactile sense. For example, when the device or a component of the device comes into contact with a touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or touchpad) may optionally be interpreted by the user as a "press-click" or "release-click" on a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press-click" or "release-click," even when a physically actuated button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user's movement does not move. For example, even when the smoothness of the tactile surface remains unchanged, the movement of the tactile surface can optionally be interpreted or sensed by the user as the "roughness" of the tactile surface. While such interpretations of touch by users will be limited by the individualized sensory perceptions of the user, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user.

[0072] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits.

[0073] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102.

[0074] Peripheral interface 118 can be used to couple the device's input and output peripherals to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0075] RF (Radio Frequency) circuit 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuit 108 converts electrical signals into electromagnetic signals and vice versa, and communicates with communication networks and other communication devices via these electromagnetic signals. RF circuit 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, Subscriber Identity Module (SIM) cards, memory, etc. RF circuit 108 optionally communicates with networks and other devices wirelessly, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). RF circuit 108 optionally includes well-known circuitry for detecting near-field communication (NFC) fields, such as through near-field communication radio components. Wireless communication may optionally employ any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed ​​Downlink Packet Access (HSDPA), High-Speed ​​Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Unit HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), and Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Messaging Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence with Extended Utility (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol that has not been developed as of the date of this document submission.

[0076] Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and device 100. Audio circuitry 110 receives audio data from peripheral interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves that are audible to humans. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuitry 110 converts the electrical signals into audio data and transmits the audio data to peripheral interface 118 for processing. Audio data is optionally retrieved by peripheral interface 118 from and / or transmitted to memory 102 and / or RF circuitry 108. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., ...). Figure 2 (212 in the text). The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headphone or a headset with both output (e.g., a mono or binaural headphone) and input (e.g., a microphone).

[0077] I / O subsystem 106 couples input / output peripherals on device 100, such as touchscreen 112 and other input control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, depth controller 169, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push-buttons, rocker buttons, etc.), dial pads, slide switches, joysticks, click dials, etc. In some alternative embodiments, input controller 160 is optionally coupled to (or not coupled to) any of the following: keyboard, infrared port, USB port, and pointing device such as mouse. One or more buttons (e.g., Figure 2 Optionally, 208) includes an increase / decrease button for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include a push-button (e.g., Figure 2 (206 in the middle).

[0078] A quick press of the push button optionally disengages the touchscreen 112 from its lock or optionally initiates a process of unlocking the device using gestures on the touchscreen, as described in U.S. Patent Application 11 / 322,549 (i.e., U.S. Patent No. 7,657,849), filed December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image," the entire contents of which are incorporated herein by reference. A long press of the push button (e.g., 206) optionally powers the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touchscreen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0079] The touch-sensitive display 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or sends electrical signals to the touchscreen 112. The touchscreen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0080] Touchscreen 112 has a touch-sensitive surface, sensor, or sensor array that accepts input from a user based on tactile and / or haptic contact. Touchscreen 112 and display controller 156 (along with any associated modules and / or instruction set in memory 102) detect contact on touchscreen 112 (and any movement or interruption of that contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touchscreen 112. In an exemplary embodiment, the contact point between touchscreen 112 and the user corresponds to the user's finger.

[0081] Touchscreen 112 optionally employs LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, but other display technologies are used in other embodiments. Touchscreen 112 and display controller 156 optionally employ any of a variety of touch sensing technologies now known or to be developed hereafter, along with other proximity sensor arrays or other elements for determining one or more points of contact with touchscreen 112, to detect contact and any movement or interruption thereof. These various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that from Apple Inc. (Cupertino, California). and iPod The technology used.

[0082] In some embodiments of the touchscreen 112, the touch-sensitive display optionally resembles a multi-touch-sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is incorporated herein by reference in its entirety. However, the touchscreen 112 displays visual output from the device 100, while the touch-sensitive touchpad does not provide visual output.

[0083] In some embodiments, the touchscreen 112 has a touch-sensitive display as described in the following patent applications: (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, entitled “Multipoint Touch Surface Controller”; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, entitled “Multipoint Touchscreen”; (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, entitled “Gestures For Touch Sensitive Input Devices”; (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, entitled “Gestures For Touch Sensitive Input Devices”; and (5) U.S. Patent Application No. 18, 2005, entitled “Mode-Based Graphical User Interfaces For Touch Sensitive”. U.S. Patent Application No. 11 / 038,590 entitled “Input Devices”; (6) U.S. Patent Application No. 11 / 228,758 entitled “Virtual Input Device Placement On A Touch Screen User Interface”, filed September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700 entitled “Operation Of A Computer With A Touch Screen Interface”, filed September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737 entitled “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard”, filed September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749 entitled “Multi-Functional Hand-Held Device”, filed March 3, 2006. The full text of all these applications is incorporated herein by reference.

[0084] Touchscreen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. Users optionally interact with touchscreen 112 using any suitable object or accessory such as a stylus, finger, etc. In some embodiments, the user interface is designed to operate primarily through finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform the user-desired actions.

[0085] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. Optionally, the touchpad is a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0086] The device 100 also includes a power system 162 for supplying power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in the portable device.

[0087] The device 100 may optionally also include one or more optical sensors 164. Figure 1AAn optical sensor 164 is shown coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite to a touchscreen display 112 on the front of the device, allowing the touchscreen display to be used as a viewfinder for still image and / or video image acquisition. In some embodiments, the optical sensor is located on the front of the device, allowing images of the user to be optionally acquired for video conferencing while the user views other video conferencing participants on the touchscreen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lenses and sensor within the device housing), allowing a single optical sensor 164 to be used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0088] The device 100 may optionally also include one or more contact strength sensors 165. Figure 1A A contact strength sensor 165 is shown coupled to a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric sensors, optical force sensors, capacitive touch-sensitive surfaces, or other strength sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). The contact strength sensor 165 receives contact strength information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact strength sensor is arranged juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact strength sensor is located on the rear of device 100, opposite to the touchscreen display 112 located on the front of device 100.

[0089] The device 100 may optionally also include one or more proximity sensors 166. Figure 1AA proximity sensor 166 coupled to a peripheral device interface 118 is shown. Alternatively, the proximity sensor 166 may optionally be coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 may optionally perform as described in the following U.S. patent applications: No. 11 / 241,839, entitled "Proximity Detector In Handheld Device"; No. 11 / 240,788, entitled "Proximity Detector In Handheld Device"; No. 11 / 620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, entitled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals", the entire contents of which are incorporated herein by reference. In some implementations, the proximity sensor is turned off and the touchscreen 112 is disabled when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0090] The device 100 may optionally also include one or more tactile output generators 167. Figure 1AA haptic output generator coupled to a haptic feedback controller 161 in I / O subsystem 106 is shown. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components; and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert electrical signals into haptic outputs on the device). A contact intensity sensor 165 receives haptic feedback generation instructions from a haptic feedback module 133 and generates a haptic output on device 100 that can be felt by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a haptic surface (e.g., a haptic display system 112) and optionally generates the haptic output by moving the haptic surface vertically (e.g., in / outward from the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite to the touch screen display 112 located on the front of the device 100.

[0091] The device 100 may optionally also include one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to a peripheral device interface 118 is shown. Alternatively, the accelerometer 168 may be coupled to an input controller 160 in an I / O subsystem 106. The accelerometer 168 may optionally perform as described in the following U.S. patent publications: U.S. Patent Publication 20050190059, entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication 20060017692, entitled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer," both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 may optionally include, in addition to the accelerometer 168, a magnetometer and a GPS (or GLONASS or other global navigation system) receiver for acquiring information about the location and orientation (e.g., portrait or landscape) of device 100.

[0092] The device 100 optionally also includes one or more depth camera sensors 175. Figure 1AA depth camera sensor is shown coupled to a depth camera controller 169 in I / O subsystem 106. Depth camera sensor 175 receives data projected through the environment. In conjunction with imaging module 143 (also referred to as camera module), depth camera sensor 175 is optionally used to determine depth maps of different portions of the image captured by imaging module 143. In some embodiments, the depth camera sensor is located at the front of device 100, such that user images with depth information are optionally acquired for video conferencing while the user views other video conferencing participants on a touchscreen display, and selfies with depth map data are captured. In some embodiments, the position of depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor in the device housing), such that depth camera sensor 175 is used in conjunction with the touchscreen display for both video conferencing and still image and / or video image acquisition.

[0093] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Furthermore, in some embodiments, memory 102 ( Figure 1A ) or 370 ( Figure 3 Storage device / global internal state 157, such as Figure 1A and Figure 3 As shown in the diagram. Device / global internal state 157 includes one or more of the following: active application state, which indicates which applications (if any) are currently active; display state, which indicates what applications, views or other information occupy various areas of the touchscreen display 112; sensor state, including information obtained from the device's various sensors and input control devices 116; and positional information about the device's position and / or orientation.

[0094] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or embedded operating systems such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0095] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to… (Trademark of Apple Inc.) The same or similar and / or compatible multi-pin (e.g., 30-pin) connectors used in Apple Inc. devices.

[0096] The contact / motion module 130 optionally detects contact with the touchscreen 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., touchpads or physical click-based rotary dials). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the contact intensity (e.g., the force or pressure of the contact, or an alternative to force or pressure), determining whether there is movement of the contact and tracking movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple-finger contact). In some implementations, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0097] In some implementations, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., determining whether the user has “clicked” an icon). In some implementations, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a specific physical actuator and can be adjusted without changing the physical hardware of device 100). For example, the mouse “click” threshold of a touchpad or touchscreen can be set to any threshold in a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some specific implementations, the user of the device is provided with software settings for adjusting one or more intensity thresholds in a set (e.g., by adjusting the individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using system-level clicks on the “intensity” parameter).

[0098] The touch / motion module 130 optionally detects the user's gesture input. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timings, and / or intensities of the detected contact). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger press event, and then detecting a finger lift-off (lift-away) event at the same (or substantially the same) location as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift-off (lift-away) event.

[0099] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphics" includes any object that can be displayed to a user, and non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0100] In some implementations, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like to specify the graphic to be displayed, and, if necessary, also receives coordinate data and other graphic attribute data, and then generates screen image data for output to the display controller 156.

[0101] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 167 to produce haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0102] Optionally, a text input module 134, a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., Contacts 137, Email 140, TM 141, Browser 147, and any other application that requires text input).

[0103] GPS module 135 determines the location of the device and provides that information for use in various applications (e.g., to phone 138 for use in location-based dialing; to camera 143 as image / video metadata; and to applications that provide location-based services, such as weather desktop apps, local yellow pages desktop apps, and map / navigation desktop apps).

[0104] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0105] • Contacts module 137 (sometimes called address book or contact list);

[0106] • Telephone module 138;

[0107] • Video conferencing module 139;

[0108] • Email client module 140;

[0109] • Instant Messaging (IM) module 141;

[0110] Fitness support module 142;

[0111] • Camera module 143 for still images and / or video images;

[0112] • Image management module 144;

[0113] • Video player module;

[0114] Music player module;

[0115] • Browser module 147;

[0116] • Calendar module 148;

[0117] • Desktop mini-program module 149, which optionally includes one or more of the following: weather desktop mini-program 149-1, stock market desktop mini-program 149-2, calculator desktop mini-program 149-3, alarm clock desktop mini-program 149-4, dictionary desktop mini-program 149-5, and other desktop mini-programs obtained by the user, and desktop mini-programs created by the user 149-6;

[0118] • Desktop app creator module 150 for creating user-created desktop apps 149-6;

[0119] • Search module 151;

[0120] • Video and music player module 152, which combines a video player module and a music player module;

[0121] • Notepad module 153;

[0122] • Map module 154; and / or

[0123] • Online video module 155.

[0124] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0125] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the contact module 137 is optionally used to manage an address book or contact list (e.g., in the application internal state 192 of the contact module 137 stored in memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0126] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input character sequences corresponding to telephone numbers, access one or more telephone numbers in contact module 137, modify entered telephone numbers, dial corresponding telephone numbers, initiate conversations, and disconnect or hang up when a conversation is completed. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0127] Combining RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between the user and one or more other participants based on user instructions.

[0128] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. Combined with image management module 144, email client module 140 makes it very easy to create and send emails containing still images or video images captured by camera module 143.

[0129] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting a corresponding instant message (e.g., using Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephone-based instant messaging or using XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving an instant message, and viewing a received instant message. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Services (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0130] Incorporating RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes executable instructions for creating fitness activities (e.g., with time, distance, and / or calorie burning goals); communicating with fitness sensors (executive devices); receiving fitness sensor data; calibrating sensors used to monitor fitness; selecting and playing music for fitness activities; and displaying, storing, and transmitting fitness data.

[0131] In conjunction with the touchscreen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for: capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0132] Incorporating the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0133] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as links to attachments and other files on web pages.

[0134] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and associated data (e.g., calendar entries, to-dos, etc.) according to user instructions.

[0135] In conjunction with RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop applet module 149 is optionally a micro-application downloaded and used by a user (e.g., weather desktop applet 149-1, stock market desktop applet 149-2, calculator desktop applet 149-3, alarm clock desktop applet 149-4, and dictionary desktop applet 149-5) or a user-created micro-application (e.g., user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop applet includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop applet).

[0136] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and browser module 147, the desktop applet creator module 150 can optionally be used by the user to create desktop applets (e.g., converting user-specified portions of a webpage into desktop applets).

[0137] In conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0138] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions allowing users to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touchscreen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0139] Combining the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notes, to-do items, etc., according to user instructions.

[0140] Combining RF circuitry 108, touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data related to shops and other points of interest at or near a specific location, and other location-based data) according to user instructions.

[0141] Incorporating touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, the online video module 155 includes instructions for performing the following actions: allowing users to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touchscreen or on an external display connected via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos. Further descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed June 20, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed December 31, 2007, entitled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of which are incorporated herein by reference in their entirety.

[0142] Each of the above modules and applications corresponds to an executable instruction set for performing one or more of the functions described above and the methods described in this patent application (e.g., computer-implemented methods and other information processing methods as described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, processes, or modules; therefore, various subsets of these modules may optionally be combined or otherwise rearranged in various embodiments. For example, a video player module may optionally be combined with a music player module into a single module (e.g., Figure 1A (e.g., video and music player module 152). In some embodiments, memory 102 optionally stores subgroups of the aforementioned modules and data structures. Additionally, memory 102 optionally stores other modules and data structures not described above.

[0143] In some implementations, device 100 is a device on which the operation of a predefined set of functions is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push-buttons, dials, etc.) on device 100 can be optionally reduced.

[0144] A predefined set of functions, uniquely performed via a touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push-button or other physical input control device, rather than a touchpad.

[0145] Figure 1B This is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 This includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 137 to 151, 155, 380 to 390).

[0146] Event classifier 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event classifier 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information should be delivered.

[0147] In some implementations, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or ready to be displayed by the application 136-1, a state queue for enabling the user to return to the previous state or view of the application 136-1, and a repeat / undo queue for the user's previous actions.

[0148] Event monitor 171 receives event information from peripheral device interface 118. The event information includes information about sub-events (e.g., user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripheral device interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (via audio circuitry 110). The information received by peripheral device interface 118 from I / O subsystem 106 includes information from touch-sensitive display 112 or touch-sensitive surfaces.

[0149] In some implementations, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other implementations, peripheral device interface 118 transmits event information only when a significant event occurs (e.g., receiving input above a predetermined noise threshold and / or receiving input for a predetermined duration).

[0150] In some implementations, the event classifier 170 also includes a hit view determination module 172 and / or an activity event recognizer determination module 173.

[0151] When the touch-sensitive display 112 displays more than one view, the hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. A view consists of controls and other elements that the user can see on the display.

[0152] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a procedural level within the application's procedural or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined at least in part based on the hit view of the initial touch that initiates a touch-based gesture.

[0153] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchical structure from which the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source to which it was identified as the hit view.

[0154] The activity event recognizer determination module 173 determines which views(s) within the view hierarchy should receive a specific sub-event sequence. In some embodiments, the activity event recognizer determination module 173 determines that only the hit view should receive the specific sub-event sequence. In other embodiments, the activity event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the specific sub-event sequence. In other embodiments, even if the touch sub-event is entirely confined to the area associated with a particular view, higher views in the hierarchy will still remain actively participating views.

[0155] Event assigner module 174 assigns event information to event identifiers (e.g., event identifier 180). In embodiments that include active event identifier determination module 173, event assigner module 174 delivers event information to the event identifier determined by active event identifier determination module 173. In some embodiments, event assigner module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0156] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module (such as contact / motion module 130) stored in memory 102.

[0157] In some implementations, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other implementations, one or more of the event recognizers 180 are part of a separate module, such as a user interface toolkit or a higher-level object from which application 136-1 inherits methods and other properties. In some implementations, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handlers 190 optionally utilize or invoke the data updater 176, the object updater 177, or the GUI updater 178 to update the application's internal state 192. Alternatively, one or more application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0158] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least one subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0159] Event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events, such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a longitudinal orientation to a lateral orientation, or vice versa), and the event information includes corresponding information about the device's current orientation (also referred to as device pose).

[0160] Event comparator 184 compares event information with predefined event or sub-event definitions and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, event 1 (187-1) is defined as a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift-off of a predetermined duration (touch end), a second touch (touch start) of a predetermined duration on the displayed object, and a second lift-off of a predetermined duration (touch end). In another example, event 2 (187-2) is defined as a drag on a displayed object. For example, dragging includes a touch (or contact) on the displayed object for a predetermined duration, movement of the touch on the touch-sensitive display 112, and lifting off the touch (end of touch). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0161] In some implementations, event definition 187 includes definitions of events for corresponding user interface objects. In some implementations, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0162] In some implementations, the definition of the corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined whether the sub-event sequence actually corresponds to or does not correspond to the event type of the event recognizer.

[0163] When the corresponding event recognizer 180 determines that the sub-event sequence does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events based on touch gestures are ignored. In this case, other event recognizers (if any) that remain active in the hit view continue to track and process the ongoing sub-events based on touch gestures.

[0164] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists instructing how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing how or how event recognizers can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists instructing whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0165] In some implementations, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some implementations, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and delaying) the sub-event to the corresponding hit view. In some implementations, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with that tag retrieves the tag and executes a predefined procedure.

[0166] In some implementations, event delivery instruction 188 includes a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or the actively participating view receives the event information and performs a predetermined procedure.

[0167] In some implementations, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module. In some implementations, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on touch-sensitive display.

[0168] In some implementations, event handler 190 includes, or has access to, a data updater 176, an object updater 177, and a GUI updater 178. In some implementations, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other implementations, they are included in two or more software modules.

[0169] It should be understood that the above discussion regarding event handling for user touch on a touch-sensitive display also applies to other forms of user input used to operate the multifunction device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; touch movements on the touchpad, such as taps, drags, scrolls, etc.; stylus input; device movement; verbal commands; detected eye movements; biometric input; and / or any combination thereof may optionally be used as input corresponding to sub-events that define the event to be identified.

[0170] Figure 2A portable multifunction device 100 with a touchscreen 112 is shown according to some embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment and other embodiments described below, a user can select one or more graphics by gesturing over the graphics, for example, using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, selection of one or more graphics occurs when the user breaks contact with one or more graphics. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or scrolling (from right to left, from left to right, up and / or down) of a finger already in contact with the device 100. In some specific embodiments or in some cases, unintentional contact with a graphic does not select the graphic. For example, a swipe gesture over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0171] Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204. As previously described, menu button 204 is optionally used to navigate to any application 136 of a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touchscreen 112.

[0172] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push-button 206 for powering on / off and locking the device, one or more volume control buttons 208, a SIM card slot 210, a headset jack 212, and a docking / charging external port 124. The push-button 206 is optionally used to power on / off the device by pressing the button and holding it in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing it before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In another embodiment, device 100 also accepts voice input via microphone 113 for activating or deactivating certain functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more haptic output generators 167 for generating haptic outputs for a user of device 100.

[0173] Figure 3This is a block diagram of an exemplary multifunctional device with a display and a touch-sensitive surface according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., a home controller or industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects system components and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, and a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 and sensors 359 (e.g., optical sensors, accelerometers, proximity sensors, touch sensors, and / or contact intensity sensors similar to those mentioned above) are described. Figure 1A The one or more contact strength sensors 165 mentioned above. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores data with portable multifunction device 100 (…). Figure 1A The memory 370 stores programs, modules, and data structures similar to those in the memory 102 of the portable multifunction device 100, or subsets thereof. Additionally, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunction device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a rendering module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while the portable multifunction device 100 (… Figure 1A The memory 102 may optionally not store these modules.

[0174] Figure 3Each of the elements described above is optionally stored in one or more memory devices of the previously mentioned memory devices. Each of the modules described above corresponds to an instruction set for performing the functions described above. The modules or programs (e.g., instruction sets) described above need not be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0175] Now let’s turn our attention to the implementation of the user interface, which is optionally implemented on, for example, a portable multifunction device 100.

[0176] Figure 4A An exemplary user interface for an application menu on a portable multifunction device 100 according to some embodiments is shown. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements or a subset or superset thereof:

[0177] • One or more signal strength indicators 402 for wireless communications such as cellular signals and Wi-Fi signals;

[0178] • Time 404;

[0179] Bluetooth indicator 405;

[0180] • Battery status indicator 406;

[0181] • Tray tray 408 with icons for commonly used applications, such as:

[0182] The telephone module 138 has an icon 416 labeled "telephone", which optionally includes an indicator 414 indicating the number of missed calls or voicemails.

[0183] The email client module 140 has an icon 418 labeled "Mail", which optionally includes an indicator 410 for the number of unread emails;

[0184] The browser module 147 has an icon 420 labeled "Browser"; and

[0185] The video and music player module 152 (also known as the iPod module 152, a trademark of Apple Inc.) is labeled with an icon 422 labeled "iPod"; and

[0186] • Icons of other applications, such as:

[0187] Icon 424 of the οIM module 141, labeled "Message";

[0188] The icon 426 of the calendar module 148 is labeled "Calendar";

[0189] The icon 428 of the image management module 144 is labeled "photo";

[0190] The icon 430 of the camera module 143 is labeled "camera";

[0191] The icon 432 of the online video module 155 is labeled "Online Video";

[0192] The icon 434 in the Stock Market Desktop Mini Program 149-2 is labeled "Stock Market";

[0193] The icon 436 of the map module 154 is labeled "map";

[0194] The icon 438 in the Weather Desktop Mini Program 149-1 is marked as "Weather".

[0195] The icon labeled "Clock" in the alarm clock desktop mini-program 149-4 is 440;

[0196] The icon 442 of the fitness support module 142 is labeled "fitness support";

[0197] The notepad module 153 has an icon 444 labeled "Notepad"; and

[0198] The icon 446, labeled "Settings," is used to set the settings of the device 100 and its various applications 136.

[0199] It should be pointed out that, Figure 4A The icon labels shown are merely exemplary. For example, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels may be optionally used for various application icons. In some embodiments, the label of a particular application icon includes the name of the application corresponding to that particular application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to that particular application icon.

[0200] Figure 4B A touch-sensitive surface 451 (e.g., separate from the display 450 (e.g., touchscreen display 112)) is shown. Figure 3 Devices (e.g., tablets or touchpads 355) Figure 3An exemplary user interface on the device 300. The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) for detecting the intensity of contact on the tactile surface 451 and / or one or more tactile output generators 357 for generating tactile outputs for the user of the device 300.

[0201] While some examples of input on a reference touchscreen display 112 (which combines a touch-sensitive surface and a display) are given below, in some embodiments the device detects input on a touch-sensitive surface separate from the display, such as... Figure 4B As shown in the diagram. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a spindle (e.g., on the display (e.g., 450)). Figure 4B The spindle corresponding to 453 in the middle (e.g., Figure 4B (452 in the example). According to these embodiments, the device detects the position corresponding to the corresponding position on the display (e.g., in the example). Figure 4B In the middle, 460 corresponds to 468 and 462 corresponds to 470) is in contact with the touch-sensitive surface 451 (e.g., Figure 4B (460 and 462 in the text). Thus, on touch-sensitive surfaces (e.g., Figure 4B 451) and the display of the multi-functional device ( Figure 4B When 450 is separated from the touch-sensitive surface, user input detected by the device on the touch-sensitive surface (e.g., touches on 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods may be optionally used for other user interfaces described herein.

[0202] Additionally, while the examples below are primarily given with reference to finger input (e.g., finger touch, single-finger tap, finger swipe), it should be understood that in some implementations, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of a touch), followed by movement of the cursor along the swipe path (e.g., instead of movement of the touch). Similarly, a tap gesture may optionally be replaced by a mouse click while the cursor is over the location of the tap gesture (e.g., instead of detection of touch, followed by cessation of touch detection). Likewise, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0203] Figure 5AAn exemplary personal electronic device 500 is illustrated. Device 500 includes a body 502. In some embodiments, device 500 may include components relative to devices 100 and 300 (e.g., Figures 1A to 4B Some or all of the features described herein. In some embodiments, device 500 has a touch-sensitive display 504, referred to below as touchscreen 504. As an alternative to or complement to touchscreen 504, device 500 has a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., touch). The one or more intensity sensors of touchscreen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, meaning that touches of different intensities can invoke different user interface operations on device 500.

[0204] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” published under WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” published under WIPO Patent Publication No. WO / 2014 / 105276, each of which is incorporated herein by reference in its entirety.

[0205] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) may be physical. Examples of physical input mechanisms include push-buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bangles, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow a user to wear device 500.

[0206] Figure 5B An exemplary personal electronic device 500 is illustrated. In some embodiments, device 500 may include a reference. Figure 1A , Figure 1B and Figure 3 Some or all of the components described herein. Device 500 has a bus 512 that operatively couples I / O portion 514 to one or more computer processors 516 and memory 518. I / O portion 514 may be connected to display 504, which may have touch-sensitive components 522 and optionally have an intensity sensor 524 (e.g., a contact intensity sensor). Furthermore, I / O portion 514 may be connected to communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular and / or other wireless communication technologies. Device 500 may include input mechanisms 506 and / or 508. For example, input mechanism 506 may optionally be a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 may optionally be a button.

[0207] In some examples, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which are operatively connected to the I / O section 514.

[0208] The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, may cause the computer processors to perform techniques including processes 800, 900, 1800, 1900, 2000, 2100, 2200, 2300, 2400, and 2500 (corresponding to respectively) Figures 8A to 8B , Figures 9A to 9B , Figures 18A to 18B , Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 , Figure 24 and Figure 25A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state storage such as flash memory, solid-state drives, etc. Personal electronic devices 500 are not limited to Figure 5B It can be the components and configurations, or it can include other components or additional components in a variety of configurations.

[0209] As used herein, the term "power indication" refers optionally to the power indication in devices 100, 300, and / or 500 ( Figure 1A , Figure 3 and Figures 5A to 5B A user-interactive graphical user interface object displayed on a screen. For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) optionally each constitute a functional representation.

[0210] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some specific implementations that include a cursor or other positional marker, the cursor acts as a "focus selector," such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the cursor is positioned on a touch-sensitive surface (e.g., a...). Figure 3 The touchpad 355 or Figure 4B When an input (e.g., a press input) is detected on the touch-sensitive surface 451 of the display, the specific user interface element is adjusted according to the detected input. This applies to touchscreen displays (e.g., those capable of enabling direct interaction with user interface elements on the touchscreen display) Figure 1A The touch-sensitive display system 112 or Figure 4AIn some embodiments of the touchscreen 112, a touch detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, that particular user interface element is adjusted according to the detected input. In some embodiments, focus moves from one area of ​​the user interface to another without corresponding movement of the cursor or movement of a touch on the touchscreen display (e.g., moving focus from one button to another using tab keys or arrow keys); in these embodiments, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user-controlled user interface element (or a touch on the touchscreen display) that delivers the user-expected interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user expects to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, touch, or selection box) above the corresponding button will indicate to the user that they expect to activate the corresponding button (rather than other user interface elements shown on the device's display).

[0211] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected over a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted away, before or after contact begins to move, before contact ends, before or after contact intensity is detected to increase and / or before or after contact intensity decreases). The characteristic intensity of the contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the contact intensity over time). In some implementations, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds may optionally include a first intensity threshold and a second intensity threshold. In this example, contact with a feature intensity not exceeding the first threshold results in a first action, contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a feature intensity exceeding the second threshold results in a third action. In some implementations, a comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more actions (e.g., whether to perform the corresponding action or abandon performing the corresponding action) rather than to determine whether to perform the first action or the second action.

[0212] In some implementations, a portion of the gesture is identified to determine the characteristic intensity. For example, a touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts, rather than the entire swipe contact (e.g., only the portion of the swipe contact at the ending position). In some implementations, a smoothing algorithm is optionally applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic intensity.

[0213] Optionally, the contact intensity on a touch-sensitive surface is characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device performs an operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device performs an operation different from the operation typically associated with clicking a button on a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity lower than the light press intensity threshold (e.g., and higher than the nominal contact detection intensity threshold, where contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different groups of user interface figures.

[0214] An increase in contact intensity from below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact intensity from below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact intensity from below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact intensity from above a contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact being lifted off the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0215] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the intensity of the contact (or multiple contacts) increases to above a press input intensity threshold. In some embodiments, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases to above a press input intensity threshold (e.g., a "downward stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing to above a press input intensity threshold and the intensity of the contact subsequently decreasing to below the press input intensity threshold, and the corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the press input threshold (e.g., an "upward stroke" of the corresponding press input).

[0216] In some implementations, the device employs intensity hysteresis to avoid unintended inputs sometimes referred to as "jitter," wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship with a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Therefore, in some implementations, a press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of that contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., an "upstroke" of the corresponding press input). Similarly, in some implementations, a press input is detected only when the device detects that the contact intensity increases from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and optionally the contact intensity subsequently decreases to an intensity equal to or below the hysteresis intensity threshold, and a corresponding operation is performed in response to detecting a press input (e.g., an increase or decrease in contact intensity depending on the environment).

[0217] For ease of explanation, optionally, the description of an operation triggered in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is provided in response to detecting any of the following conditions: the contact strength increases to above the press input strength threshold, the contact strength increases from below a hysteresis strength threshold to above the press input strength threshold, the contact strength decreases to below the press input strength threshold, and / or the contact strength decreases to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in the example where the operation is described as being performed in response to detecting a decrease in contact strength below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in contact strength below a hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0218] Now let’s turn our attention to the implementation of user interfaces (“UIs”) and associated processes on electronic devices (such as portable multifunction devices 100, 300 or 500).

[0219] Figures 6A to 6MM Exemplary user interfaces for generating and sending virtual avatars are shown according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 8A to 8B as well as Figures 9A to 9B The process in. Figures 7A to 7J An exemplary user interface for receiving and playing virtual avatars is shown according to some implementation schemes. Figures 6A to 6MM as well as Figures 7A to 7J Use virtual avatars as a specific example of virtual avatars.

[0220] Figure 6A Device 600 is depicted having a display 601 (in some cases, the display is a touch-sensitive display) and a camera 602 (the camera includes at least an image sensor capable of capturing data representing a portion of the spectrum, such as visible light, infrared light, or ultraviolet light). In some embodiments, camera 602 includes multiple image sensors and / or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, camera 602 is capable of capturing other types of data such as depth data. For example, in some embodiments, camera 602 also uses techniques based on speckle, time-of-flight, parallax, or focus to capture depth data. The image data captured by device 600 using camera 602 includes data corresponding to a portion of the spectrum of the scene within the camera's field of view. Additionally, in some embodiments, the captured image data also includes depth data of light data. In some other embodiments, the captured image data contains data sufficient to determine or generate depth data for that portion of the spectrum. In some embodiments, device 600 includes one or more features of device 100, 300, or 500.

[0221] In some examples, the electronic device 600 includes a depth camera, such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (e.g., a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate an object during image capture by a visible light camera and a depth camera (e.g., an IR camera), and information from the depth camera and the visible light camera is used to determine depth maps of different portions of the object captured by the visible light camera. In some embodiments, parallax information from both cameras (e.g., two visible light cameras) is used to display the lighting effects described herein for a backward image, and depth information from the depth camera is combined with image data from the visible light camera for a forward image (e.g., a self-portrait). In some embodiments, the same user interface is used when determining depth information using both visible light cameras and when determining depth information using the depth camera, thus providing a consistent user experience even when using drastically different techniques to determine the information used to produce the lighting effects. In some implementations, when displaying a camera user interface with one of the lighting effects applied, the device detects the selection of a camera switching display and switches from a front camera (e.g., a depth camera and a visible light camera) to a rear camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), while maintaining the display of user interface controls to apply the lighting effect and replacing the field of view display of the front camera with the field of view display of the rear camera (or vice versa).

[0222] exist Figure 6AIn the present invention, device 600 is displaying a home screen interface with multiple icons for various applications, including icon 603 for a messaging application. In response to a gesture on icon 603 (e.g., a tap gesture 604), device 600 displays... Figure 6B The corresponding user interface of the messaging application associated with icon 603.

[0223] exist Figure 6B In this context, device 600 is displaying a messaging interface 608. Elements 605-1 to 605-6 correspond to previous messaging communications. Each element 605-1 to 605-6 represents a communication with one or more remote users, each associated with their own electronic device. In response to a gesture on a particular element (e.g., a tap gesture 606), device 600 updates the messaging interface 608 to display a portion of a previous messaging communication with one or more remote users as part of a previous communication, such as... Figure 6C As depicted in the text.

[0224] exist Figure 6C In this embodiment, device 600 is displaying a messaging interface 608 for messaging communication with a remote user named "John" (and containing the initials or letter combinations of "JA"). Messaging interface 608 includes a message area 609 containing four previously exchanged messages 610-1 to 610-3 (message 610-3 was sent from the user of device 600 to "John," while the other two messages were received by device 600 from "John"). Messaging interface 608 also includes a message composition area 612 and message option icons (including icon 614, for example, an icon for accessing an interface for selecting stickers and / or other multimedia elements for messages) to the left of the message composition area 612. In some embodiments, the message option icons allow sending different types of messages, including photos, emoticons, stickers, and other forms of non-text messages, such as those described below.

[0225] In response to device 600 detecting (e.g., via) Figure 6C The tap gesture 616) selects the message writing area 612, such as Figure 6D The message sending interface 608 is updated as depicted in the diagram. For example, in... Figure 6D In the middle, the message option icon is hidden (but can be shown again by selecting button 618), the suggested message response 620 is displayed, and the virtual keyboard 622 is displayed. In some cases, the virtual keyboard 622 is used to enter new messages to be sent to remote users.

[0226] exist Figure 6EIn the message composition area 612, the text "Will be there later" is included, for example, by being entered via a virtual keyboard 622 or other methods such as voice input. In response to the selection of the send button 621, the device 600 sends the text as part of a message to one or more participants associated with the communication in message 609. Figure 6E In this case, device 600 will send a message to a user named "John". Figure 6F In this device 600, there is an updated message area 609 to reflect the sending of messages by updating message area 612 to include messages 610-4.

[0227] In some cases, the message option icon is accessed to add or compose a new message (e.g., by adding non-text content to the message). For example, in response to device 600 detecting a selection on display 618 (e.g., via...). Figure 6F The tap gesture 624 in the middle will again display the message option icon including icon 614, such as Figure 6G As depicted in the image. In response to the selection of icon 614 (e.g., via a gesture, such as...) Figure 6H The device 600 updates the messaging interface 608 (e.g., the virtual keyboard 622) by replacing the virtual keyboard 622 with the multimedia items interface 628 that currently displays the recent items menu 629 (sometimes called the "recent items tray"). Figure 6I (As depicted in the text), the multimedia project interface includes previously sent multimedia projects (e.g., Figure 6I The interface includes stickers 630-1 to 630-4, but can also include other types of multimedia items, such as sound, animation, or video. Using this interface, users can select previously sent multimedia items to send again. For example, a user can resend a previously sent multimedia item via a tap gesture on the selected sticker. Figure 6IA sticker is selected from the recent items menu 629. In response to such a selection, device 600 places the sticker in message composing area 612 or sends the selected sticker to one or more remote users involved in a communication represented in message area 609. In some embodiments, a tap and drag gesture is used to place the selected sticker (or other multimedia item) in message composing area 612 or message area 609 (and in some cases, on a specific message). For example, a specific sticker is selected via a tap gesture. The sticker is dragged to message composing area 612 or message area 609 via a drag gesture without breaking contact with touch-sensitive display 601. Once the sticker reaches the desired position, contact with touch-sensitive display 601 is stopped, and the sticker is placed at the last contact position. If the last contact position is in message area 609, the sticker is sent to one or more remote users associated with the communication represented in message area 609. Optionally, the sticker and data that associates the sticker with a specific message are sent to the remote user (e.g., the sticker is sent along with data instructing the sticker to be "pasted" at a specific position in a specific message). These techniques are not specific to selecting and sending stickers. They can also be applied to other types of multimedia projects that can be selected from the Recent Items menu 629 or other locations.

[0228] exist Figure 6I The multimedia project interface 628 also includes a menu selection button 632 (which allows selection of menus or interfaces other than the recent items menu 629 by displaying buttons or other optional items corresponding to the available menus) and a full-screen button 634 (which allows the multimedia project interface 628 to expand to occupy more space on the display (or occupy the entire display)). The full-screen button 634 is further described below.

[0229] In addition to using the menu selection button 632 to switch between menus or interfaces, gestures can also be optionally used to switch between menus. For example, in response to a swipe gesture (e.g., as...). Figure 6J and Figure 6K As depicted, by touching 636 (represented by a swipe across the multimedia item interface 628), the device updates the multimedia item interface 628 to replace the display of the recent items menu 629 with a virtual avatar menu 638. Although the recent items menu 629 is being replaced with the virtual avatar menu 638, the scroll indicator 639 provides feedback on how many other menus are available in the multimedia item interface 628.

[0230] exist Figure 6L In this context, the virtual avatar menu 638 has completely replaced the display of the recent items menu 629. This is in response to device 600 detecting (e.g., via gestures such as...) Figure 6M The tap gesture 642) indicates the option to continue displaying the ability 640, such as Figure 6N The virtual avatar interface 643 is depicted. This interface allows users to generate new virtual avatars that reflect the user's facial movements and expressions, as further described below. In some embodiments, the virtual avatar menu 638 is not displayed at all. Instead, the virtual avatar interface 643 is displayed, without first displaying the virtual avatar menu 638.

[0231] Figure 6N The virtual avatar interface 643 includes avatar template representations 644-1 to 644-7 (e.g., avatar characters with different appearances and behaviors) corresponding to different avatar frames. Each avatar template representation maps detected facial movements and expressions onto an avatar frame. An indicator 645 corresponds to the currently selected avatar template. The virtual avatar preview 646 is a "real-time" preview of the virtual avatar, as it is updated to reflect the user's current facial movements and expressions. For example, in some embodiments, utilizing camera 602, device 600 continuously captures image data from camera 602. The captured image data includes visible light data and depth data. Device 600 analyzes the captured image data to identify facial movements (e.g., muscle movements, head orientation, gaze direction, etc.) and / or facial expressions (e.g., smiling, frowning, angry expressions, sad expressions, confused expressions, etc.). Device 600 then updates the avatar preview 646 according to the parameters of the avatar frame currently associated with the virtual avatar preview 646 to reflect the detected user characteristics. In some implementations, device 600 automatically begins continuously updating the virtual avatar preview 646 in response to the virtual avatar interface 643 that is initially being executed or displayed. Detecting the selection of a representation of a different avatar template will cause device 600 to update the virtual avatar preview 646 based on the newly selected avatar template.

[0232] Figure 6O Several examples depict user faces in captured image data 650-1 to 650-5 and corresponding updates 651-1 to 651-5 to the virtual avatar preview. These are examples of device 600 updating emoji preview 646 to reflect the user's facial movements, expressions, and posture. In captured image data 650-1, device 600 detects (e.g., based on facial features, facial muscles, facial movements, and / or facial expressions) that the user is looking forward, smiling, and / or happy. In response, in addition to updating the eyes of the virtual avatar preview to look straight ahead, device 600 also updates the virtual avatar preview to reflect the user's smiling and / or happy expression, as depicted in update 651-1. Although the physical features of the user detected in the captured image data are sometimes updated so that the virtual avatar reflects the same physical features in the user's virtual avatar, in other cases, changes in the detected physical features of the user result in updating different types of physical features of the virtual avatar. For example, in Figure 6OIn some implementations, since monkeys lack eyebrows, changes in the user's eyebrows are mapped onto the monkey's ears (or other features), as shown in 650-2 and 651-2. In this example, the user's mouth and eyes are mapped onto the monkey's mouth and eyes. In the examples of image data 650-3 and update 651-3, the user's unpleasant expression and / or frown is reflected in the corresponding features of the virtual avatar preview. In some implementations, if the user maintains a facial expression or facial pose, as depicted in images 650-3 and 650-4, the virtual avatar preview is updated with additional features (such as tears in the case of update 651-4). This type of predefined update can also occur in response to insufficient detected movement. In some implementations, the update is also based on user movement detected in the image data. For example, device 600, which detects rotation of the user's head, generates an update that causes the virtual avatar preview to rotate similarly. In some implementations, the update is also based on a physical model of the features used for the virtual avatar. For example, in image data 650-5, device 600 detects that the user is shaking their head. In response, device 600 generates update 651-5 to reflect the head shaking. Additionally, in update 651-5, the puppy's ears protrude due to the physical model applied to the puppy's ears.

[0233] exist Figure 6P In this process, device 600 detects selection of the record button 652 via gesture (e.g., a tap gesture represented by touch 653). In response, the virtual avatar interface 643 is updated to indicate that an animated virtual avatar is being generated, such as... Figure 6Q As depicted in the description. For example, the record button 652 is replaced with a stop button 654, the avatar template representations 644-1 to 644-7 are no longer displayed, and a recording progress indicator 656 is displayed, indicating how long the animated emoticon has been recorded and the relative amount of time remaining for recording the virtual avatar. Recording can be stopped by any method, such as by the expiration of a predetermined amount of time (e.g., 15 seconds) or by selecting the stop button 654. In some embodiments, during recording, the device 600 is detecting and / or storing a series of data points for creating the animated virtual avatar. For example, in some embodiments, the device 600 records a time series of facial movements and / or facial expressions (e.g., as values ​​in a range of possible values, where each value in the range corresponds to a predetermined movement or expression), and then maps the facial movements and / or facial expressions onto the avatar template to create the animated virtual avatar. Alternatively, when device 600 updates the virtual avatar preview to reflect the user's facial movements and / or expressions, device 600 records an animated virtual avatar by creating a video recording of the virtual avatar preview. In some embodiments, device 600 also records sound captured using device 600's microphone, such that the recorded animated virtual avatar includes sound that can be played back along with the recorded animation of the virtual avatar.

[0234] Figure 6R This depicts a later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0235] Figure 6R-1 A device 600 is described that detects a change in the user's position relative to the device during avatar recording. Specifically, in conjunction with... Figure 6R-1 At the corresponding point in time, the user's face is no longer in the camera's field of view. In response, device 600 displays a virtual avatar at the edge of avatar interface 643 (e.g., the edge corresponding to the last detected position of the user's face), displays frame corners 653 around the virtual avatar, and displays message 655A ("Please put your face in view") to prompt the user to adjust their alignment relative to the device. In some embodiments, recording of the virtual avatar continues even after the user's face is no longer detected in the camera's field of view, but the virtual avatar remains still (or assumes a predetermined pose (e.g., a neutral pose)) when the user's face is not detected.

[0236] Figure 6R-2 Device 600 describes the time after a user has remained outside the camera's field of view for more than a predetermined threshold time. In response to detecting that the user has remained outside the camera's field of view for more than the predetermined time, device 600 pauses recording the virtual avatar. Figure 6R-2 As shown, in response to the recording pause, device 600 has replaced stop button 654 with record button 648. Device 600 also displays message 655B (“Tap to resume”), which indicates to the user that recording has been paused. In some implementations, the user can resume recording by tapping anywhere in the avatar interface 643 (including tapping record button 648). Pausing virtual avatar recording when the user has been outside the camera's field of view for more than a predetermined threshold amount of time and requesting another input to resume recording reduces power consumption and the use of the depth camera, thereby extending the battery life of battery-powered devices and the lifespan of the depth camera.

[0237] Figure 6S This depicts another later point in time during the recording of the animated virtual avatar. The virtual avatar preview 646 has been updated to further reflect newly detected facial movements and / or expressions from the user. The indicator 656 has also been updated to reflect further progress in recording the animated virtual avatar.

[0238] exist Figure 6SIn the process, the device 600 receives a gesture requesting the cessation of recording the animated virtual avatar (e.g., a tap gesture represented by contact 658). In response, the device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, such as... Figure 6T As described in the text. In other cases, in response to the expiration of a predetermined time period (e.g., 15 seconds), device 600 stops recording the animated virtual avatar and updates the virtual avatar interface, as... Figure 6T As depicted in the text.

[0239] Figures 6S-1 to 6S-3 Another embodiment of a virtual avatar interface 643 for recording (e.g., generating) virtual avatars is depicted. For example... Figure 6S-1 As shown, device 600 displays timer 659 (e.g., displaying 10 seconds remaining), which indicates the remaining time of the current avatar recording session (e.g., a session initiated by activating the recording button 648). Figure 6S-2 The same recording session lasted 4 seconds, and timer 659 currently displays the avatar recording session with 6 seconds remaining. Figure 6S-3 The recording session has ended (i.e., Figure 6S-3 It is more than Figure 6S-1 (10 seconds later). In response to the end of the recording session, device 600 replaces timer 659 with trash can indicator 660-1, which can be activated (e.g., by a tap gesture) to discard the completed recording session. In some embodiments, trash can indicator 660 functions similarly to... Figure 6T as well as Figures 6U to 6U-1 The discard indication is 660.

[0240] exist Figure 6T In this context, the virtual avatar interface 643 is currently playing a recorded animated virtual avatar 659, instead of displaying a virtual avatar preview, as described in the three snapshots of the playback of the animated virtual avatar 659. In some implementations, the recorded animated virtual avatar is played in a loop (e.g., playing the recorded animated virtual avatar at least twice without user input, such as...). Figure 6T(As indicated by the arrow in the image). The virtual avatar interface 643 also includes a discard button 660, a mute button 662, and a confirmation button 664 (displayed in place of the record button 652). The discard button 660 discards the displayed recorded animated virtual avatar without saving or sending it to a remote user. The mute button 662 allows the user to mute the playback of the recorded animated virtual avatar. The confirmation button 664 allows the recorded animated virtual avatar to be sent to a remote user (e.g., in response to the activation of the confirmation button 664 or before the user moves to the message composing area 612 to send it directly to one or more users associated with the communication displayed in the message area 609). After the device 600 detects the selection of the confirmation button 664, it updates the virtual avatar interface 643 to return to the state relative to the record button 652. Figure 6N The described status. In some implementations, the confirmation button 664 includes a send glyph or icon displayed in the send button of the message sent in the message composing area (e.g., Figure 6V (670) Similar or identical glyphs or icons, which are used to indicate that the recorded animated virtual avatar can be sent to a remote user by selecting the confirmation button 664.

[0241] While the animated virtual avatar 659 is playing, in response to a tap gesture 665 on the representation of an avatar template different from the currently selected template, the animated virtual avatar is updated to reflect the new avatar template without having to re-record the animated virtual avatar. This in Figure 6U It is described in the text. Figure 6U This depicts animated virtual avatar 659, which has been replaced by animated virtual avatar 666. The recording used for generation... Figure 6T The facial muscles, facial movements, facial features, and facial expressions of the animated virtual avatar 659 are reapplied. Figure 6U The new avatar templates available.

[0242] exist Figure 6U-1 In the process of playing the animated avatar 666, device 600 detects a tap gesture 661 corresponding to a selection on the discard enable representation 660. In response, device 600 discards the captured animated avatar data (e.g., abandons adding the animated avatar to the message composing area 612) and transitions to... Figure 6 U-2 The interface. Figure 6 U-2 In the middle, device 600 displays a pre-recorded avatar interface 643, which is similar to... Figure 6P The interface seen in [the document / website] (e.g., including the record button). [And / or] Figure 6P In contrast, the avatar template remains the avatar template for the robot (e.g., virtual avatar 666), instead of reverting to the monkey virtual avatar 659. That is, it preserves... Figure 6T and Figure 6UThe results of the detection of changes from avatar 659 to avatar 666 during the playback period described in the figure.

[0243] Re-reference Figure 6T In response to a gesture (e.g., a tap gesture represented by contact 667), device 600 adds a recorded animated virtual avatar 668 to message composing area 612 (see [link]). Figure 6V ) and return the virtual avatar interface 643 to Figure 6N The state described in (see) Figure 6V Then, the user can send a message on device 600 (e.g., in response to a tap gesture represented by contact 672 on the sending indication display 670, such as...). Figure 6W (as depicted in the text) before the message (see...) Figure 6V Add more message content (e.g., text or other multimedia items). Alternatively, when device 600 detects a selection of confirmation button 664, device 600 sends a recorded animated virtual avatar to one or more remote users associated with the communication displayed in message area 609, and then updates the message area to reflect that the animated virtual avatar 668 has been sent to one or more users associated with the communication included in message area 609, such as... Figure 6X As depicted in the text.

[0244] Figures 6Y to 6BB The virtual avatar interface 643 is described in response to user input while scrolling through a list of avatar templates. For example, it responds to a swipe gesture (e.g., a touch gesture) or a vertical movement across an avatar template, as shown in section 676. Figures 6Y to 6AA (As depicted in the image), device 600 scrolls through avatar templates and changes the currently selected avatar template. The avatar templates present in the avatar template indicator 645 are updated based on a swipe gesture. In response to detecting that a new avatar template is being selected, device 600 updates the virtual avatar preview. For example, in... Figure 6Z In the process, when device 600 detects a selection of avatar template representation 644-5, it displays a virtual avatar preview 678 (this virtual avatar preview is based on the avatar template corresponding to representation 644-5), and... Figure 6AA In the middle, when the avatar template representation 644-8 is selected, a virtual avatar preview 680 is displayed (this virtual avatar preview is based on the avatar template corresponding to representation 644-8).

[0245] In addition to generating recordings of animated puppet emojis, the emoji interface 643 also allows the generation of static virtual avatars (e.g., stickers with expressions / appearances determined based on the state of the virtual avatar). For example, in Figure 6CCIn response to user input on the virtual avatar preview 680 (e.g., a tap and hold gesture represented by touch 682), device 600 generates a sticker corresponding to the state of the virtual avatar preview 680 at a time associated with the user input (e.g., when input is received, when user input ends, or some other time associated with the user input). In an embodiment, device 600 displays a sticker 683 peeled off from the virtual avatar preview 680. Figure 6DD and Figure 6EE This indicates that a sticker has been generated and / or that the user can place a sticker.

[0246] After the device 600 generates the sticker, the user can optionally select from several operations for the sticker. For example, the user can instruct the device 600 to place the sticker in a recent menu or other similar interface that allows later use. The user can also instruct the device 600 to place the sticker in the message composition area 612 before the device 600 sends a message containing the sticker, and the user can place the sticker in the message area 609 (and optionally) on a specific message, so that the device 600 sends the sticker to one or more users participating in the communication in the message area 609.

[0247] For example, in Figure 6FF In the process, device 600 has detected the removal of contact 682, while the contact remains above the virtual avatar preview 680. In response, device 600 has saved the generated sticker to device 600, such as in a database or library stored in device 600's memory, accessible via the recent items menu 629. Figure 6I Access to the database or library allows stickers to be optionally selected via the recent items menu 629 or via other interfaces on the device 600. The device 600 optionally displays stickers that are locally saved via animations with different graphic versions 684 and 686, such as... Figures 6FF to 6GG As depicted, it moves toward the menu selection button 632.

[0248] For example, Figures 6HH to 6KK This illustrates an example of device 600 sending a generated sticker to one or more users participating in the communication represented in message area 609. Figure 6II In the process, device 600 detects user input on the virtual avatar preview 680 (e.g., by...). Figures 6HH to 6JJ The touch 688 in the virtual avatar preview 680 represents a tap and drag gesture. When device 600 detects that the user drags the sticker representation into message area 609, the sticker representation 690 follows the touch 688. Once device 600 detects that the touch 688 in message area 609 is lifted, device 600 sends the sticker 691 to one or more remote users who are participants in the communication represented in message area 609, such as... Figure 6KK As depicted in the text.

[0249] exist Figure 6LL In this process, device 600 responds to the selection of the full-screen button 634 (e.g., via...). Figure 6LL The touch gesture 692 in the middle (representing a tap) updates the virtual avatar interface 643 to display more screen (or in full-screen mode). Figure 6MM The virtual avatar interface 643 is depicted after being enlarged to occupy more space on the display 601. When button 692 is selected, the device 600 returns the virtual avatar interface 643 to its previous configuration.

[0250] Figures 7A to 7J A messaging interface 608 is depicted after receiving an animated emoticon from a remote user. Although Figures 7A to 7J use Figures 6A to 6MM Device 600 is used as an example, but Figures 7A to 7J The user interface and functionality described herein are also applicable to other devices (e.g., devices 100, 300, or 500), including those that did not previously send stickers or animated virtual avatars.

[0251] Figure 7A A message interface 608 is depicted after an animated virtual avatar 700 has been received from a remote user named "John" (and containing the initials or letter combinations of "JA") just before the animated virtual avatar began playing. In some embodiments, the device 600 automatically plays the animated virtual avatar 700 after it is received. When the mute button 702 is selected, it silences any sound associated with the device 600 and the animated virtual avatar 700. In some embodiments, any sound also silences if the animated virtual avatar scrolls off the display. In some embodiments, such as relative to... Figures 6A to 6MM The virtual avatar interface 643 is displayed as described (e.g., the virtual avatar interface 643 includes a preview of the virtual avatar based on detected facial movements / expressions and the selected avatar template).

[0252] exist Figure 7B In response to playing the animated virtual avatar 700 once (e.g., once from start to finish), a still frame 703 of the animated virtual avatar 700 is displayed instead of the animated virtual avatar 700. A playback button 704 is also displayed in the message area 609 and allows playback via, for example, by... Figure 7C The touch feature 706 indicates that the virtual avatar 700 will replay the animation. Figure 7D Device 600 depicts the replay of the animated virtual avatar 700 (relative to) Figure 7A (Describes playing animated emojis).

[0253] In some implementations, when the animated virtual avatar 700 is playing, if the device 600 receives user input on the mute button 702 (e.g., by...), Figure 7E If a tap gesture (represented by touch 708) is received, device 600 stops playing any sound associated with the animated avatar 700, while continuing to play the animated avatar 700 (e.g., the animated avatar 700 continues to move without sound). In some embodiments, in response to the selection of the mute button 702 (or if sound on device 600 is turned off or device 600 has enabled accessibility features), text button 714 is displayed, such as... Figure 7H As depicted in the text. Responding to the selection of the text button 714 (e.g., via...). Figure 7G The touch gesture 716 indicates a tap, and the text 718 displays the sound used for the animated virtual avatar 700, such as... Figure 7H As depicted herein. The content of text 718 is generated locally on device 600 or remotely (e.g., using computing resources of a remote server).

[0254] In response to user input on the animated virtual avatar 700 (e.g., Figure 7I The touch 720 (representing tap and hold gestures) on device 600 displays an options menu related to the animated virtual avatar 700, such as... Figure 7J As depicted in the diagram. For example, menu 722 includes several response buttons 723-1 to 723-6, which device 600 can send to one or more remote users participating in the communication represented in message area 609. Additionally, menu 724 is displayed, which has a copy button 726, a save button 728, and a more button 730. The copy button 726 copies the animated avatar 700 to the clipboard of device 600. The save button 728 saves the animated avatar 700 to device 600 (e.g., to a database or library that can be accessed later by an application installed on device 600). The more button 730 displays additional operations that can be performed relative to the animated avatar 700.

[0255] Figures 8A to 8B This is a flowchart illustrating a method 800 using an electronic device according to some embodiments. Method 800 is performed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in method 800 are optionally combined, some operations are optionally changed in order, and some operations are optionally omitted.

[0256] As described below, Method 800 provides an intuitive way to generate and send emojis, such as virtual avatars. This method reduces the cognitive burden on users when generating and sending emojis, thus creating a more efficient human-computer interface. For battery-powered computing devices, it enables users to generate and send emojis faster and more efficiently, saving power and increasing the time interval between battery charges.

[0257] An electronic device (e.g., 600) having a camera (e.g., 602) (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (802) a virtual avatar generation interface (e.g., Figure 6N (e.g., an interface for selecting emojis (animated or static), generating static stickers, and / or recording animated virtual avatars). The electronic device displays (804) a preview of the virtual avatar in the virtual avatar generation interface (e.g., Figure 6N (e.g., 646) (e.g., 2D or 3D computer-generated graphical objects, in some cases designed to convey nonverbal messages such as emotions or reactions) (e.g., animated virtual avatars selected from multiple different available virtual avatar templates). The virtual avatar preview reacts to changes in the appearance of the face within the camera's field of view (e.g., Figure 6O (For example, an animated virtual avatar would reflect the user's head movements, facial expressions, and orientation detected in image data from one or more image sensors in the camera.) While displaying a preview of the virtual avatar, the electronic device detects (806) input in the virtual avatar generation interface (e.g., touch 652, 682, or 690). In response to (808) detecting input in the virtual avatar generation interface and based on determining that the input begins on the preview of the virtual avatar (e.g., 682 or 690) (e.g., a touch and hold input on the animated virtual avatar or touchpad input controlling the cursor), the electronic device generates (810) a static virtual avatar sticker (e.g., 683 or 691) (e.g., a still image of an animated emoji "adhesive" at a specific location in the message area), which represents the facial expression in the camera's field of view at the corresponding time. In some embodiments, the corresponding time is determined based on the timing of the input (e.g., when the input is first received, when the input ends, when the gesture corresponding to the input begins to move across the touch-sensitive surface, or at any other time related to the input). Based on the determined input, including activation of a recording enablement (e.g., a tap on the recording enablement) in the virtual avatar generation interface (e.g., 648), the electronic device generates (812) an animated virtual avatar (e.g., 668) representing a sequence of facial expressions changing within the camera's field of view over a time period (e.g., such as...). Figures 6Q to 6S(As shown). In some embodiments, the time period is determined based on the timing of the input (e.g., a time period when the start of the input is detected, when the end of the input is detected, when some type of movement of the input is detected (such as when the input is a gesture on a touch-sensitive surface), or some other time period based on the input). In some embodiments, the virtual avatar is three-dimensional. In some embodiments, a preview of the virtual avatar is displayed in 3D (e.g., 646) or an animated virtual avatar (e.g., 659). Eliminating ambiguity between user input for two possible styles of communication (e.g., animated virtual avatars and static virtual avatars) avoids the need for separate interfaces to generate each type of message content. Reducing the amount of input required to deliver the desired message enhances the operability of the device and makes the user device interface more efficient (e.g., by allowing multiple types of multimedia communication from a single interface to help the user achieve the desired communication), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0258] In some implementations, the electronic device displays (814) a messaging interface (e.g., interface 608) including a message area (e.g., 609) (e.g., a messaging application such as Apple's Messages). The message area includes information from communications (e.g., Figure 6N The communication in message area 609 (e.g., message sending thread) involves two or more participants (e.g., in...) Figure 6C In the middle, messages (e.g., 610-1 to 610-4) between remote user "John" and user of device 600 (e.g., messages sent from the user of the electronic device and messages received from the remote user of different electronic devices). The virtual avatar generation interface and the message sending interface are displayed simultaneously (e.g., Figure 6N (For example, the virtual avatar generation interface is displayed in the lower half of the messaging interface.) In some implementations, a virtual avatar preview (e.g., 646) is automatically displayed as part of the initial display of the virtual avatar generation interface.

[0259] In some implementations, the messaging interface includes a message composition area (e.g., 612) (e.g., a message input area for entering text, emojis, and other content before sending a message to a recipient) and input is a tap on a virtual avatar preview (e.g., 646). In response to detecting input in the virtual avatar generation interface, the electronic device displays a static virtual avatar (e.g., 683 or 691) in the message composition area. In some implementations, displaying the virtual avatar generation interface includes replacing the display of the virtual keyboard (e.g., 622) of the messaging interface with the display of the virtual avatar generation interface (e.g., from...). Figures 6H to 6N(Conversion without interfering with data). Displaying multimedia content for a message before sending it reduces the likelihood of error messages and allows users to add more content (e.g., via text or other content) before sending the message. Reducing the number of messages required to deliver the desired message enhances device operability and makes the user device interface more efficient (e.g., by helping users achieve the intended communication while reducing the number of messages required for communication), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0260] In some implementations, the avatar generation interface includes a static virtual avatar area (e.g., 629) (e.g., a tray of previously generated stickers displayed at the bottom of the avatar generation interface), which includes a set of one or more previously generated virtual avatars (e.g., 630-1 to 630-4). In response to user input (e.g., Figure 6CC (682) The electronic device adds the generated virtual avatar to (612) one or more previously generated virtual avatars (e.g., sending an avatar, saving a virtual avatar, or otherwise tagging a virtual avatar to include it in the set of virtual avatars). In some embodiments, the set of virtual avatars (e.g., stickers) (e.g., including miniature versions of newly generated stickers in the tray) is displayed in response to user input (e.g., 626) (e.g., selection of a virtual avatar set enablement in the avatar generation user interface or messaging user interface). In some embodiments, the tray of previously generated stickers is hidden until the user receives input requesting the tray to be displayed (e.g., input 626) or until some other event detected on the electronic device indicates that the tray may be related to the current state of the messaging interface or avatar generation interface. In some embodiments, after adding a virtual avatar to the set of virtual avatars, the electronic device receives a request from the user to share the set of virtual avatars with a second user, and in response, the electronic device sends the set of virtual avatars to the second user. Retaining previously sent message multimedia content allows users to add and reuse previous content when applicable to new messages. The elimination of the need to recreate content enhances device operability and makes the user device interface more efficient (e.g., by avoiding the repetitive generation of content), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0261] In some implementations, the input begins (816) on the preview of the virtual avatar (e.g., 680) and ends at a location within the message area (e.g., see [link]). Figures 6HH to 6KK(For example, a gesture that begins with a finger touching a preview of the virtual avatar (e.g., 680), continues by dragging the finger to a message area, and ends by lifting the finger away from the message area (in some cases, the gesture may end on a specific message in the message area, and a sticker may be associated with that specific message and optionally move with that message as it moves through the session)). The electronic device sends (818) a static virtual avatar (e.g., 691) to a participant associated with the communication (e.g., one or more remote users) (e.g., Figure 6KK In some implementations, an animation is displayed in response to a gesture indicating that a static virtual avatar is being peeled off from a virtual avatar preview. Figures 6CC to 6FF ).

[0262] In some implementations, the static virtual avatar (e.g., 691) has an appearance determined based on the facial expression in the camera's field of view when input (e.g., 688) is detected on the virtual avatar preview (e.g., 680). In some implementations, in response to the start of input detection on the virtual avatar preview, the electronic device causes the virtual avatar preview to stop (820) in response to changes in the appearance of the face in the camera's field of view. This indicates to the user that a sticker has been generated and previews the appearance of the sticker sent to the user when dragging the sticker to the communication shown in message 609 by sending the sticker to the user. By displaying a preview of the sticker to be generated without requiring the user to perform additional interaction or complete a full sticker generation gesture before seeing the resulting sticker, the operability of the device is enhanced, making the human-computer interface more efficient and effective. This reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and effectively.

[0263] In some implementations, the virtual avatar preview (e.g., 680) resumes reacting to changes after the input moves away from the virtual avatar (e.g., animation resumes when dragging a static avatar toward the messaging session). In some implementations, the virtual avatar preview (e.g., 680) stops reacting to changes in facial appearance until the input of dragging the static avatar ends. Resuming updates to the virtual avatar preview allows the user to compare the appearance of the generated sticker with other possible appearances of the virtual avatar, which may potentially be the basis for different / attached stickers. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback on other content that may be generated before the user sends the generated content to help the user achieve the desired result), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0264] In some implementations, in response to the expiration of the time period used to generate the animated avatar (e.g., a timer for 5, 10, or 15 seconds expires, or user input for that time period stops), the electronic device displays a send or acknowledge enable indicator (e.g., 664) instead of a record enable indicator (e.g., instead of displaying a virtual record button, a virtual send button is displayed instead of a virtual record button). In response to receiving input selecting a send or acknowledge enable indicator (e.g., a tap gesture on the send enable indicator on a touch-sensitive display), the electronic device sends (824) the generated animated avatar to a remote user (e.g., see [link]). Figure 6U and Figure 6X Instead of first sending the animated virtual avatar to message composition area 612, such as Figure 6V and Figure 6W (As shown) (for example, sending an animated avatar to a remote user associated with a messaging thread or session, instead of initially placing the animated avatar in another area of ​​the messaging interface, such as the message composing area). Displaying a send or confirm button instead of a record button after recording the animated avatar completes allows for the display of more information in the interface by reusing areas occupied by buttons that are not applicable to the current state of the interface and by providing the user with more context-sensitive functionality. This enhances the operability of the device and makes the user device interface more efficient (e.g., by displaying more information / options on the display without cluttering it with unused elements), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0265] In some implementations, in response to the expiration of the time period used to generate the animated virtual avatar, the electronic device displays (822) a confirmation power indicator (e.g., 664) instead of recording a power indicator. In response to receiving input selecting to send a power indicator (e.g., via contact 667) (e.g., a tap gesture on the power indicator on a touch-sensitive display), the electronic device displays a message on the messaging interface (e.g., Figure 6V The animated virtual avatar representation (e.g., a static graphic element or an animated virtual avatar) is displayed in the message composition area (e.g., 612) (e.g., the area of ​​the messaging interface that will display text typed on the keyboard). In some embodiments, this time period is based on a predetermined amount of time (e.g., an amount of time represented by a progress indicator 656). After generating the animated virtual avatar, the electronic device stops displaying a preview of the virtual avatar and displays a looping version of the animated virtual avatar (e.g., ...). Figure 6TThe display of the looping version of the animated avatar includes displaying the animation sequence two or more times (e.g., as described below with respect to method 900). Displaying a send or confirm button instead of a record button after recording the animated avatar completes allows for the display of more information in the interface by reusing areas occupied by buttons that are not suitable (or less suitable) for the current state of the interface. This enhances the operability of the device and makes the user device interface more efficient (e.g., by displaying more information / options on the display without cluttering it with unused elements), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0266] In some implementations, a first virtual avatar template is used (e.g., the template represented by element 644-4, or by...). Figure 6N The indicator 645 indicates another element) to display a looping version of the animated virtual avatar. The electronic device displays representations of multiple other virtual avatar templates (e.g., elements 644-1 to 644-7) (e.g., miniature universal versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), these representations including representations of a second virtual avatar template (e.g., Figure 6Z Element 644-8), wherein the second virtual avatar template differs from the first virtual avatar template. In some embodiments, one or more of the multiple virtual avatar templates are based on emojis that can be sent via a messaging application. An animated virtual avatar in a looping version is then displayed (e.g., Figure 6T Following this, and in response to receiving user input (e.g., 665) indicating the selection of a first virtual avatar template, the electronic device updates the display of the cyclically updated animated virtual avatar to reflect the second virtual avatar template (e.g., Figure 6U (For example, while the animated avatar is still based on a sequence of facial expression changes, its appearance can be altered to reflect a new avatar template.) For instance, an animated avatar could transform from a monkey into a robot, but it would still reflect a sequence of facial expressions changing over time within the same camera field of view (e.g., see From...). Figure 6T to Figure 6U (The conversion). Updating animated avatars based on newly selected avatar templates allows users to fine-tune multimedia content for expected messages by selecting the style of the animated avatar after recording its movements and actions. Eliminating the need to re-record animated avatars to try new avatar templates enhances device operability and makes the user device interface more efficient (e.g., by avoiding repetitive content generation), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and effectively.

[0267] In some implementations, a virtual avatar preview is automatically displayed in response to the launch of the virtual avatar generation interface (e.g., 646). In some implementations, the virtual avatar preview is displayed once the virtual avatar generation interface is displayed, without requiring user input.

[0268] In some embodiments, in response to detecting characteristics (e.g., position, orientation, movement) of a first physical feature of the face (e.g., a smiling mouth, protruding tongue, wiggling ears, raised eyebrows, or any other movement of any other physical feature) in the camera's field of view, the electronic device updates the first physical feature of the displayed virtual avatar preview based on the detected characteristics, wherein the type of the first physical feature of the face (e.g., eyes, eyebrows, mouth, tongue, ears) is the same as the type of the first physical feature in the displayed preview. In some embodiments, if the user's mouth opens, the virtual avatar's mouth will open in response (e.g., ...). Figure 6O (At locations 650-1 and 651-1). Similar results can be based on facial expressions. For example, if movement of one or more physical features or characteristics of a face is detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can be updated to reflect the predefined movement by updating the corresponding physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to similar physical features of the virtual avatar allows the user to provide movements, expressions, and gestures that provide input to the system, which are intuitively mapped onto the virtual avatar without requiring cumbersome or time-consuming touch or key input. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping users achieve expected results through predictable mapping of user features to the virtual avatar, and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0269] In some implementations, in response to detecting a second physical feature of a face in the camera's field of view, the electronic device updates the second physical feature of the displayed virtual avatar preview based on the detected feature, wherein the type of the second physical feature of the face is different from the type of the second physical feature of the displayed preview (e.g., in...). Figure 6OIn the example, eyebrow movement 650-2 and ear movement 651-2. In some implementations, if the user is smiling, and this indicates that the user is happy, different features of the virtual avatar (such as a unicorn's horn or lights on a robot) can be changed to reflect the smile. Similar results can be based on facial expressions. For example, if movement of one or more physical features or characteristics of the face is detected, the electronic device can determine that a predefined emotion is being displayed. In response, the displayed virtual avatar preview can optionally be updated to reflect the predefined movement by updating different sets of physical features or characteristics to reflect the detected facial expression. Mapping the user's physical features to different physical features of the virtual avatar allows the user to provide movements, expressions, and gestures that provide input to the system, which are mapped to avatar features that the user would not otherwise be able to easily control. This enhances the operability of the device and makes the user device interface more efficient (e.g., by mapping the user's features to the virtual avatar, additional features controlling the virtual avatar help the user achieve the desired results and reduce user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0270] In some implementations, in response to detecting movement of a face within the camera's field of view, the electronic device updates a third physical feature of the displayed virtual avatar preview based on a physical model used for the virtual avatar and the detected movement (e.g., Figure 6O (See 650-5 and 651-5 in the original text). In some implementations, for example, if the virtual avatar is based on a virtual avatar template for a puppy, the virtual avatar's face will shake when the user's face is detected to be shaking, and the virtual avatar's ears may protrude to reflect the physics of the shaking motion, even if the user's ears do not protrude in response to the shaking. In some implementations, the same physical features of the displayed virtual avatar preview are updated based on the movement and physical model of the corresponding facial features in the camera's field of view (e.g., the ears move based on the movement of the user's ears, but also based on the physical model for a puppy's large ears). Updating the virtual avatar based on the physical model for the virtual avatar allows users to create realistic and interactive virtual avatars that can convey a wider range of nonverbal information. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping users deliver pre-defined messages using more realistic virtual avatar movements), thereby further reducing power consumption and extending the device's battery life by enabling users to use the device more quickly and efficiently.

[0271] In some implementations, the preview of the virtual avatar is based on a predefined virtual avatar template (e.g., with...). Figure 6NThe image represents the avatar template associated with 644-4. The electronic device updates the preview of the virtual avatar based on one or more predefined behaviors associated with the predefined virtual avatar template. In some implementations, if no movement or change in facial expression is detected on the face in the camera's field of view (e.g., 650-3 and 650-4), the preview of the virtual avatar displays a predefined response (e.g., 651-4), such as blinking, head rotation, making facial expressions, or other actions.

[0272] In some implementations, in response to determining that a face is no longer detected in the camera's field of view (e.g., face tracking failed because the face has moved out of the camera's field of view; the face is occluded and not in the camera's field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device gradually fades out the display of the virtual avatar preview (e.g., virtual avatar preview 646 will fade out). In some implementations, the device makes other modifications to the virtual avatar preview to indicate that the user's face can no longer be tracked, such as degrading the virtual avatar preview based on the last information detected by the device, including changing the size, rotation, motion, etc. of the virtual avatar preview. In some implementations, in response to determining that a face is no longer detected in the camera's field of view (e.g., face tracking failed because the face has moved out of the camera's field of view; the face is occluded and not in the camera's field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device displays a message indicating that the camera can no longer accurately detect the face (in...). Figure 6N The virtual avatar above 643 or replacing Figure 6NThe virtual avatar preview 646 displays a prompt. In some embodiments, in response to determining that a face is no longer detected in the camera's field of view (e.g., face tracking fails because the face has moved out of the camera's field of view; the face is occluded and does not appear in the camera's field of view; or the face has been repositioned so that the device can no longer accurately track the movement of features on the face), the electronic device updates the display of the virtual avatar preview based on facial appearance changes that occurred during a period of time before the face was no longer detected in the field of view (e.g., immediately before the face was no longer detected in the field of view or shortly before the face was no longer detected in the field of view). (e.g., device 600 repeatedly displays a transition from smiling to frowning or eye movement.) In some embodiments, updating the display of the virtual avatar preview based on facial appearance changes that occurred during a period of time before the face was no longer detected in the camera's field of view includes: gradually slowing down the updates of the virtual avatar preview over time, such that the updates of the virtual avatar gradually stop (e.g., a rotating avatar slowly stops rotating, eyes that are opening or closing slowly stop opening or closing, and a mouth that is opening or closing slowly stop opening or closing). Displaying feedback on whether the virtual avatar preview is tracking the user's face allows the user to determine if the device is being held correctly and if the conditions used to detect the user's face are appropriate. Providing improved feedback on the device's status enhances its operability and makes the user interface more efficient by offering better continuity to the user interface through indications that the device is still attempting to track the user's face. This provides a better and more intuitive human-computer interface and allows the user to continue interacting with the device even when it is unable to track the user's face.

[0273] It should be noted that the above description is relative to the process described in method 800 (e.g., Figures 8A to 8B The details of the above methods also apply in a similar manner to the methods described below. For example, method 900 optionally includes one or more features of the various methods described above with reference to method 800. For example, the generation of stickers described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. Similarly, the muting of sound before sending an animated virtual avatar (e.g., an animated virtual avatar) as described above with respect to method 800 is optionally combined with the user interface described below with respect to method 900. For the sake of brevity, these details will not be repeated below.

[0274] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as opposed to...). Figure 1A , Figure 3 and Figure 5A (as described above) or application-specific chips. Furthermore, the above references... Figure 8A and Figure 8BThe operation is optionally performed by Figures 1A to 1B The components depicted are used to implement this. For example, detecting input (806) in the virtual avatar generation interface is optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and an event dispatcher module 174 delivers event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand how it can be based on Figures 1A to 1B The components depicted in the diagram are used to perform other processes.

[0275] Figures 9A to 9B This is a flowchart illustrating a method 900 using an electronic device according to some embodiments. Method 900 is performed at a device (e.g., 100, 300, 500, 600) having a display and a camera. Some operations in method 900 are optionally combined, some operations are optionally changed in order, and some operations are optionally omitted.

[0276] As described below, Method 900 provides an intuitive way to generate and send emojis, such as virtual avatars. This method reduces the cognitive burden on users when generating and sending emojis, thus creating a more efficient human-computer interface. For battery-powered computing devices, it enables users to generate and send emojis faster and more efficiently, saving power and increasing the time interval between battery charges.

[0277] An electronic device (e.g., 600) having a camera (e.g., configured to have one or more sensors for capturing data representing visible light data, IR light data, depth data, etc.) and a display (e.g., 601) displays (902) a virtual avatar generation interface (e.g., Figure 6N (e.g., an interface for selecting emojis, generating static emojis, and recording animated emojis). The electronic device displays (904) a preview of the virtual avatar in the virtual avatar generation interface (e.g., ...). Figure 6N(e.g., 2D or 3D computer-generated graphical objects, in some cases designed to convey nonverbal messages such as emotions or reactions) (e.g., moving emojis selected from multiple different available emoji styles or templates). A virtual avatar preview reacts to changes in the appearance of the face within the camera's field of view (e.g., ...). Figure 6O (For example, animated emoticons would reflect the user's head movements, facial expressions, and orientations detected in image data from one or more image sensors). The electronic device receives (906) a request (e.g., contact 652) to generate an animated virtual avatar based on changes in facial expressions of the face in the camera's field of view. In response to receiving the request to generate an animated virtual avatar, the electronic device records (908) (e.g., Figures 6Q to 6S The sequence of facial expressions of the face in the camera's field of view (e.g., including a sequence of data points used to provide a mapping of points that can be applied to a virtual avatar template to generate an animated virtual avatar). After recording the facial expressions of the face in the camera's field of view, the electronic device displays (910) a looped version including the animated sequence (e.g., Figure 6T and 6U An animated virtual avatar (e.g., 659) is generated based on a sequence of facial expressions recorded in response to a request to generate the animated virtual avatar (e.g., by sequentially mapping a series of data points representing the recorded facial expressions to a predefined animated virtual avatar template). The electronic device displays a looping version of the animated virtual avatar two or more times. In some embodiments, recording the facial expression sequence includes recording a time series of values ​​for discrete mapping points used in the predefined virtual avatar template. Displaying a looping version of the animated virtual avatar allows the user to subsequently review the animated virtual avatar content to understand whether an appropriate message is being conveyed. Providing improved visual feedback to the user enhances device operability, reduces error instances, and makes the user device interface more efficient (e.g., by providing feedback indicating that the device will produce the expected output to help the user achieve the desired result, and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0278] In some implementations, the electronic device stops recording facial expression sequences in response to a timer expiration (e.g., indicated by a progress indicator 656) (e.g., a 5-second, 10-second, or 15-second timer). In some implementations, the electronic device stops recording facial expression sequences in response to receiving user input (e.g., touch 658) (e.g., a user tapping a virtual button displayed on a monitor). Limiting the recording time of animated avatars allows users to create animated avatar recordings while limiting the impact on the device's computing resources (e.g., storage). This enhances the operability of the device by conserving its computing resources.

[0279] In some implementations, the electronic device replaces the (912) preview display with a looping version of the animated virtual avatar (e.g., see From...). Figure 6S to Figure 6T (e.g., in response to the completion of recording a facial expression sequence, playing back the generated animated avatar to the user in a loop). Playing back the looped version of the recorded animated avatar automatically allows the user to preview the animated avatar before deciding whether to send, delete, or save it. This enhances device operability and makes the user device interface more efficient (e.g., by helping users achieve expected results by providing feedback indicating the outcome before submission, and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and effectively.

[0280] In some implementations, the request to generate an animated avatar includes selecting a recording enablement (e.g., 648) to be displayed in the avatar generation interface. After recording facial expressions of the face in the camera's field of view, the electronic device replaces the display of the recording enablement with a send or acknowledge enablement (e.g., 664). In some implementations, sending the enablement operates as explained above with respect to method 800. Recording an animated avatar in response to the selection of the recording enablement allows the user to use the avatar preview to verify that the device is tracking the user and that the currently selected avatar template is consistent with the message the user wishes to convey. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user achieve the expected result by providing a preview of the expected result before the user generates the expected result, and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0281] In some implementations, a first virtual avatar template (e.g., the avatar template corresponding to element 644-4) is used to display (918) a looping version of the animated virtual avatar. The electronic device displays representations of multiple other virtual avatar templates (e.g., elements 644-1 to 644-7) (e.g., miniature universal versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), including representations of a second virtual avatar template (e.g., Figure 6Z (Elements 644-8). The second virtual avatar template differs from the first virtual avatar template. In some embodiments, one or more of the multiple virtual avatar templates are based on emojis that can be sent via a messaging application. After the display of a cyclic version of the animated virtual avatar begins and in response to receiving user input that selects the representation of the first virtual avatar template, the electronic device updates (922) the display of the cyclic version of the animated virtual avatar to reflect the second virtual avatar template (e.g., changing the appearance of the animated virtual avatar to reflect the new virtual avatar template while the animated virtual avatar is still based on a sequence of facial expression changes). For example, the animated virtual avatar can change from a monkey to a robot, but it will still reflect the same sequence of facial expression changes over time in the same camera field of view (e.g., see From Figure 6T to Figure 6U (conversion). In some implementations, the electronic device displays (914) multiple representations of a virtual avatar template (e.g., miniature universal versions of different virtual avatar templates, such as smiley faces, animals, robots, or other objects), including a representation of a first virtual avatar template. In response to receiving user input corresponding to a selection of a representation of the first virtual avatar template, the electronic device updates (916) the display of a cyclical version of an animated virtual avatar to correspond to the first virtual avatar template (e.g., changing the animation to a robot animation based on a robot virtual avatar template instead of a puppy animation based on a puppy virtual avatar template without requiring the user to re-record any facial expressions). Updating the animated virtual avatar based on the newly selected avatar template allows the user to fine-tune the multimedia content for the intended message by allowing the user to select the style of the animated virtual avatar after recording its movement and actions. The elimination of the need to re-record the animated virtual avatar to preview the new avatar template enhances the operability of the device and makes the user device interface more efficient (e.g., by avoiding the repetitive generation of content), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0282] In some implementations, the virtual avatar preview is based on a second avatar template. This is in response to detecting a face in the camera's field of view (e.g., Figure 6OThe electronic device updates (920) the first physical feature of the displayed virtual avatar preview based on the detected first physical feature (e.g., movement of the user's eyebrows). The first physical feature of the displayed preview has a first feature type (e.g., eyebrows). After receiving user input corresponding to the selection of a first graphic element (e.g., switching the avatar template from a puppy to a monkey) and in response to detecting a second physical feature of the first physical feature of the face in the camera's field of view (e.g., movement of the user's eyebrows), the electronic device updates (922) the second physical feature of the displayed virtual avatar preview based on the detected second physical feature (e.g., movement of the user's eyebrows). Figure 6O (e.g., moving a monkey's ears), wherein the second physical feature of the displayed preview has a different second feature type (e.g., ears) than the first feature type (e.g., eyebrows). Mapping the same physical features of a user to different physical features of different avatar templates allows the user to have a wider range of options for conveying messages by generating multiple virtual avatars using the same input. This enhances the operability of the device and makes the user device interface more efficient (e.g., helping the user achieve the intended message by providing more options for conveying messages), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0283] In some implementations, in response to receiving and scrolling (e.g., via...) Figures 6Y to 6AA In response to a user input corresponding to a request for multiple representations of a virtual avatar template (contact 676), the electronic device scrolls the display of multiple representations of the virtual avatar template to display a second graphical element that is not part of the multiple representations of the virtual avatar template. In some implementations, scrolling is based on the speed of the user input corresponding to the request. Scrolling through the virtual avatar template allows the user to quickly view different options for the virtual avatar. Additionally, scrolling the display of multiple representations of the virtual avatar template allows the user to view the previous and next virtual avatar templates. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing feedback indicating that the device will produce the expected output to help the user achieve the expected result, and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0284] In some implementations, the scrolling speed gradually decreases over time after the end of user input is detected (e.g., scrolling gradually stops, much like the inertia of multiple representations of a virtual avatar gradually slowing down through friction). Gradually decreasing the scrolling speed over time allows the user to continue viewing different virtual avatar template options without providing additional input. This enhances device operability and makes the user interface more efficient (e.g., by helping the user achieve expected results by providing feedback on possible outcomes without requiring additional interaction, and by reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and effectively.

[0285] In some implementations, in response to receiving user input corresponding to a request, the electronic device generates auditory and / or tactile outputs corresponding to the currently selected virtual avatar template when changing from one virtual avatar template to a different one. For example, auditory and / or tactile outputs are generated as each of a plurality of representations of the virtual avatar template scrolls past a position indicating the currently selected virtual avatar template. Generating auditory or tactile feedback allows the user to determine when a new selection has occurred. This enhances the operability of the device and makes the user interface more efficient (e.g., by providing feedback indicating when a new selection was made to help the user achieve the intended result and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and effectively.

[0286] In some implementations, in response to receiving user input on the animated avatar, corresponding to a request to save the animated avatar, the electronic device stores the data used for the animated avatar into a database on the electronic device (e.g., Figures 6CC to 6GG For example, an electronic device stores data representing animated virtual avatars in a directory or library on a non-volatile storage device within the electronic device.

[0287] In some implementations, the electronic device receives a request from a remote user to send an animated virtual avatar to a remote device (e.g., contact 688). Figures 6HH to 6KKBased on the determination that the remote device meets the first set of criteria (e.g., the remote device has the required application for playing the first version of the animated avatar), the electronic device sends the first version of the animated avatar to the user of the remote device (e.g., sending non-graphical data representing a recorded sequence of facial expressions and an instruction for the avatar template, enabling the remote device to reproduce the animated avatar). Based on the determination that the remote device does not meet the first set of criteria (e.g., the remote device does not have a suitable messaging application or a suitable version of a messaging application for playing the first version of the animated avatar), the electronic device sends a second version of the animated avatar, different from the first version (e.g., sending a video file representing the animated avatar), to the user of the remote device. Determining which version of the animated avatar to send to the remote user conserves device resources by sending only the minimum amount of compatible information to the remote user. Furthermore, this reduces the need for the user to resend information in a more compatible format (e.g., in response to the remote user indicating that the initial format is not visible). Efficient and efficient data transmission enhances device operability, which in turn reduces power consumption and extends device battery life by enabling users to use the device more quickly and efficiently.

[0288] In some implementations, when displaying a looping version of the animated virtual avatar, the electronic device plays (924) audio data based on sounds recorded during the recording of the facial expression sequence and based on an audio filter associated with a predefined avatar template. In some implementations, when displaying a looping version of the animated virtual avatar, the electronic device plays audio data based on sounds recorded during the recording of the facial expression sequence. In response to receiving a mute indication (e.g., Figure 6T and Figure 6U Upon selecting the corresponding user input (662), the electronic device stops playing audio data. Filtered audio for the animated virtual avatar is played based on a filter specific to the avatar template used for the virtual avatar. This allows users to send messages more effectively by providing more options on how to deliver messages and more engaging animated virtual avatars. This enhances the device's operability and makes the user interface more efficient (e.g., by providing users with more options on how to deliver messages to help them achieve their intended message), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and effectively.

[0289] In some implementations, in response to receiving a request to send an animated avatar to a remote user (926) and based on determining that a request to send the animated avatar to the remote user is received while muting the audio data associated with the display of a looping version of the animated avatar (e.g., contact 667), the electronic device sends (928) data representing the animated avatar to the remote user without sending the sound data for the animated avatar. Based on determining that a request to send the animated avatar to the remote user is received while not muting the audio data associated with the display of a looping version of the animated avatar, the electronic device sends (930) data representing the animated avatar along with the sound data for the animated avatar to the remote user. Sending an animated avatar without sound when the user mutes the playback of the animated avatar allows the user to effectively choose whether to include sound in the message sent to the remote user. This enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user achieve the expected message with minimal interaction), which in turn reduces power consumption and extends the device's battery life by enabling the user to use the device more quickly and efficiently.

[0290] In some implementations, in response to receiving a request to generate an animated virtual avatar (e.g., contact 652), the electronic device records a first facial movement of a face within the camera's field of view, wherein displaying a looping version of the animated virtual avatar includes animaten the virtual avatar based on a physical model for the animated virtual avatar and the first facial movement (e.g., see image data 650-5, and...). Figure 6O Update 651-5). Updating animated virtual avatars based on the physical model used for them allows users to create realistic and interactive virtual avatars that can convey a wider range of nonverbal communications. This enhances device operability and makes the user device interface more efficient (e.g., by helping users move around using more realistic virtual avatars to deliver scheduled messages), thereby further reducing power consumption and extending device battery life by enabling users to use the device more quickly and efficiently.

[0291] In some implementations, in response to detecting that a specific feature of a face within the camera's field of view remains in a specific pose for more than a threshold amount of time during the recording of a facial expression sequence (e.g., see...), the mechanism is described in the following ways. Figure 6O (650-4 and 650-5), electronic devices add predefined animated expressions to animated virtual avatars that correspond to specific facial poses (e.g., see 650-4 and 650-5). Figure 6O(651-5). For example, if the face has a neutral expression for a predetermined time period, predefined movements (such as head turning or blinking) are added to the animated avatar. Similarly, if the face has an angry expression for a predetermined time period, one or more additional features indicating anger (such as ears changing color or steam emanating from the ears) are added to the animated avatar. Updating the animated avatar based on the threshold amount of time that the device detects features of the user's face to remain in a specific posture allows the user to add more actions to the animated avatar than could be achieved using only facial expressions, features, and movements. This enhances the operability of the device and makes the user device interface more efficient (e.g., by providing additional mechanisms for conveying actions that do not otherwise correspond to readily available facial expressions, movements, or features), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0292] In some implementations, when recording a sequence of facial expressions, in response to receiving user input (e.g., touch on a touch-sensitive surface, movement of an electronic device detected by a motion sensor, activation of a button, or other input) via an input mechanism separate from the camera, the electronic device adds a first facial expression to the sequence of facial expressions (e.g., recording a happy facial expression, a tongue-out facial expression, or any other facial expression not actually recorded as a facial expression made by a face in the camera's field of view when recording facial expressions to be included in the animated avatar). The first facial expression is based on the user input received via the input mechanism. In some implementations, while the animated avatar is looping, the user can use touchscreen controls to add additional expressions to the animated avatar, such that as the animated avatar loops, the user can gradually add expressions to the animated avatar, such that even when the animated avatar is initially created, those changes in expression differ from changes in expression recorded based on facial expressions in the camera's field of view (e.g., the user's face), and the animated avatar also includes changes in expressions selected by the user. Updating the animated avatar based on user input rather than with camera capture allows the user to add more actions to the animated avatar than could be achieved using only facial expressions, features, and movements. This enhances the operability of the device and makes the user device interface more efficient (for example, by providing additional mechanisms for transmitting actions that do not otherwise correspond to easily available facial expressions, movements, or features to help users achieve the desired results), which in turn reduces power consumption and extends the device's battery life by enabling users to use the device more quickly and efficiently.

[0293] It should be noted that the above description is relative to the process described in method 900 (e.g., Figures 9A to 9BDetails of the above methods can also be applied in a similar manner. For example, method 900 may optionally include one or more features of the various methods described above with reference to method 800. For example, displaying an animated virtual avatar preview based on a sequence of recorded facial features, movements, and / or expressions and based on a frame associated with the avatar template described with respect to method 900 may be applied to the sticker and animated virtual avatar interface described above with respect to method 800.

[0294] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (e.g., as opposed to...). Figure 1A , Figure 3 and Figure 5A (as described above) or application-specific chips. Furthermore, the above references... Figure 9A and Figure 9B The operation is optionally performed by Figures 1A to 1B The components depicted are used to implement this. For example, receiving a request to generate an animated virtual avatar (906) is optionally implemented by an event classifier 170, an event recognizer 180, and an event handler 190. An event monitor 171 in the event classifier 170 detects contact on the touch-sensitive surface 604, and an event dispatcher module 174 delivers the event information to the application 136-1. The corresponding event recognizer 180 of the application 136-1 compares the event information with a corresponding event definition 186 and determines whether a first contact at a first location on the touch-sensitive surface corresponds to a predefined event or sub-event, such as a selection of an object on the user interface. When a corresponding predefined event or sub-event is detected, the event recognizer 180 activates the event handler 190 associated with the detection of that event or sub-event. The event handler 190 optionally utilizes or invokes a data updater 176 or an object updater 177 to update the application's internal state 192. In some embodiments, the event handler 190 accesses a corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly understand how it can be based on Figures 1A to 1B The components depicted in the diagram are used to perform other processes.

[0295] Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B Exemplary user interfaces for generating and modifying virtual avatars, according to some embodiments, are shown. The user interfaces in these figures are used to illustrate the processes described below, including... Figures 18A to 18B , Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 , Figure 24 and Figure 25 The process in.

[0296] In some embodiments, the virtual avatar is a graphically depictable user representation. In some embodiments, the virtual avatar is non-realistic (e.g., cartoonish). In some embodiments, the avatar is anthropomorphic constructs such as stylized animals (e.g., avatars 1100, 1300, 1500, 1600, and 1700), stylized robots (e.g., avatar 1400), or stylized generally inanimate objects (e.g., avatar 1000). In some embodiments, the virtual avatar includes an avatar face having one or more avatar features (e.g., avatar facial features). In some embodiments, the avatar features correspond to (e.g., map) one or more physical features of the user's face, such that detected movement of the user's physical features affects the avatar features (e.g., affects the graphical representation of the features).

[0297] In some examples, a user can manipulate the characteristics or features of the virtual avatar using a camera sensor (e.g., camera module 143, optical sensor 164). When the user's physical features (such as facial features) and position (such as head position or head tilt) change, the electronic device detects the change and modifies the displayed image of the virtual avatar to reflect the change in the user's physical features and position. In some embodiments, changes in the user's physical features and position indicate various facial expressions, emotions, context, tone of voice, or other nonverbal communication. In some embodiments, the electronic device modifies the displayed image of the virtual avatar to represent these facial expressions, emotions, context, tone of voice, or other nonverbal communication.

[0298] Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B Corresponding variations of exemplary user input and exemplary virtual avatars (e.g., poop avatars, bear avatars, alien avatars, rabbit avatars, robot avatars, unicorn avatars, chicken avatars, and pig avatars) displayed on an electronic device are shown. In some embodiments, the electronic device includes one or more elements and / or features of devices 100, 300, and 500. Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B The image on the upper left represents an image of the user detected by the electronic device when the user is within the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensors). In other words, the user's image is derived from the viewpoint of the camera (e.g., camera module 143, optical sensor 164), which in some embodiments may be positioned on the electronic device (e.g., devices 100, 300, and 500), and in other embodiments may be positioned separately from the electronic device (e.g., transmitting data to an external camera or sensor of the electronic device). In some embodiments, Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B The upper left edge of the image represents the boundary of the field of view of one or more cameras (e.g., camera module 143, optical sensor 164) and / or other sensors (e.g., infrared sensor). In some embodiments, the user's image is displayed on a display of an electronic device (e.g., touchscreen 112, display 340, display 450, display 504). In some embodiments, the user's image is transmitted to an external electronic device for display. In some embodiments, the external electronic device includes one or more elements and / or features of devices 100, 300, and 500. In some embodiments, the user's image data is collected and processed by the device but is not immediately displayed on the electronic device or transmitted to an external device.

[0299] Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B Each of the images on the left side shows a virtual avatar (e.g., a poop avatar) in a state presented (e.g., after modification) based on the detected corresponding user image located on the left side of the image. In some embodiments, the virtual avatar is displayed from the user's perspective. In some embodiments, the virtual avatar is displayed on the display of an electronic device (e.g., touchscreen 112, display 340, display 450, display 504). In some embodiments, the virtual avatar is transmitted to an external electronic device for display. In some embodiments, Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B The image on the upper right shows the position of the virtual avatar within the display area of ​​the electronic device's display (e.g., touchscreen 112, display 340, display 450, display 504), and Figures 10A to 10I , Figures 11A to 11C , Figures 12A to 12C , Figure 13 , Figures 14A to 14D , Figures 15A to 15B , Figures 16A to 16B as well as Figures 17A to 17B The boundary of the image on the upper right represents the boundary of the display area including the virtual avatar. In some implementations, the display area represented on the right corresponds to the avatar display area of ​​the application user interface, such as the virtual avatar interface 643, message composition area 612, message area 609 (or a portion thereof) discussed above.

[0300] In some implementations, the magnitude of the response of an avatar feature (e.g., a discrete element of an avatar that can be discretely moved or modified relative to other avatar features) corresponds to the magnitude of a change in the user's physical features (e.g., detected or tracked features such as user muscles, muscle groups, or anatomical features such as eyes). For example, in some implementations, the magnitude of a change in a physical feature is determined based on the potential range of motion of the physical feature, wherein the magnitude represents the relative position of the physical feature within the range of motion of that physical feature (e.g., a predicted or modeled range of motion). In such implementations, the magnitude of the response of an avatar feature is similarly the relative position of the avatar feature within the range of motion of the avatar feature. In some implementations, the magnitude of a change is determined based on a comparison or measurement (e.g., distance) of the start and end positions of a change in a physical feature. In such implementations, a change in a physical feature can be translated into a modification of a first avatar feature by applying the measured change in the physical feature to the avatar feature (e.g., directly, or as a scaled or adjusted value).

[0301] In some embodiments, modifications to avatar features have a magnitude component and a direction component, wherein the direction component of the modification to the avatar feature is based on the direction component of changes in one or more physical features in response to the avatar feature. In some embodiments, the direction of the avatar feature's response corresponds (e.g., directly or inversely) to the relative direction of a change in the user's physical features, wherein the relative direction of the change in the physical features is determined based on the direction of movement of the physical features from an initial position (e.g., a neutral position, a stationary position, or, in some embodiments, the position of the physical features initially detected by the device). In some embodiments, the direction of the avatar feature's response directly corresponds to the relative direction of the change in the physical features (e.g., if the physical features move upward, the avatar feature also moves upward). In other embodiments, the direction of the avatar feature's response corresponds inversely to the relative direction of the change in the physical features (e.g., if the physical features move upward, the avatar feature moves downward).

[0302] In some implementations, the directional component of the change in the head feature is mirrored relative to the directional component of the change in the physical feature. For example, when the physical feature (e.g., the user's mouth) moves to the left, the head feature (e.g., the head's mouth) moves to the right. In some implementations, for movement along the vertical axis, the directional component of the change in the head feature is the same as the directional component of the change in the physical feature, and for movement along the horizontal axis, the directional component of the change in the head feature is a mirror image of the directional component of the change in the physical feature, similar to the effect seen when looking in a mirror. In some implementations, the neutral resting position of the user's iris is determined as a specific position (e.g., a centered position) relative to the periphery of the user's eyeball.

[0303] Figure 10A An exemplary embodiment illustrating how an electronic device modifies a poop avatar 1000 in response to detecting changes in a user's facial features is shown. The poop avatar is shown having four display states (1011A, 1011B, 1011C, and 1011D), each of the four display states corresponding to four detection states of the user (1001A, 1001B, 1001C, and 1001D). Specifically, in Figure 10A In each display state, the electronic device positions or modifies the features of the poop avatar in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the user's corresponding state. Figure 10A In the illustrated implementation, the detected facial features of the user include the user's mouth 1020 (with corners 1020A and 1020B) and the user's eyebrows 1022. In some implementations, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, etc. Figure 10A In the implementation scheme, the features of the poop avatar include the avatar face 1000C, mouth 1030, avatar eyes 1032, avatar upper part 1034, avatar lower part 1036, and (in some states) avatar eyebrows 1038.

[0304] As shown in 1001A, the electronic device detects a neutral facial expression of the user. For example, the electronic device detects that the user's mouth 1020 and eyebrows 1022 are positioned in a relaxed, neutral state, rather than in a position associated with a specific facial expression (e.g., smiling or frowning). In response to detecting a neutral facial expression of the user, the electronic device displays a poop avatar 1000 with a neutral expression (e.g., neutral state) in 1011A. Specifically, the electronic device displays a poop avatar with an avatar mouth 1030 in a relaxed, neutral state, rather than in a position normally associated with a specific facial expression (e.g., smiling or frowning). In some embodiments, the neutral position of the avatar corresponds to other representations of the associated image, such as a static poop emoji that can be found in messaging applications. Additionally, the electronic device displays the face 1000C of the poop avatar without eyebrows 1038 and displays avatar eyes 1032 looking in a direction perpendicular to the focal plane of the display (or the camera (e.g., camera 143, optical sensor 164)). The electronic device also displays a poop image in a neutral vertical position above the lower part 1036 at the upper part 1034.

[0305] As shown in 1001B, the electronic device detects the position of the user's mouth 1020 for forming a smiling facial expression (e.g., one or both of the corners of the user's mouth 1020A and 1020B are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression). In response to detecting the position of the user's mouth 1020, the electronic device modifies the display of the avatar mouth 1030 to have a smiling expression, such that the avatar mouth 1030 has an open smiling position, as shown in 1011B. As shown in 1011B, the eyes 1032, upper part 1034, and lower part 1036 remain unchanged relative to their original positions in 1011A.

[0306] As shown in 1001C, the electronic device detects a user's frowning facial expression. In some embodiments, the electronic device detects the frowning facial expression by detecting that one or both of the corners of the user's mouth 1020A and 1020B are positioned in a downward posture (e.g., a downward position) and that the user's eyebrows 1022 are in a downward position (e.g., the user's eyebrows are furrowed or positioned lower on the user's face compared to the position of the eyebrows 1022 in the relaxed neutral state in 1001A and 1001B). In response to detecting the user's frowning facial expression, the electronic device modifies the poop avatar to have a frowning and drooping face, as shown in 1011C. For example, the electronic device modifies the poop avatar such that the corners of the avatar's mouth 1030 are turned downward to give the mouth 1030 a slightly open posture, and the lower part 1036 of the poop avatar curves downward, similar to the downward-turned posture of the poop avatar's mouth 1030. Figure 10A In the implementation scheme, when the electronic device detects the downward position of the user's eyebrows 1022, it does not modify the anatomically corresponding part of the poop avatar above the eyes 1032.

[0307] In some embodiments, the electronic device modifies the poop avatar to have the drooping face shown in 1011C by displaying an animation of the mouth 1030 turning downwards and the lower portion 1036 bending downwards as it moves to a downward-turning position. In some embodiments, the electronic device further modifies the poop avatar such that the tip 1040 of the poop avatar collapses or tilts downwards when the poop avatar presents a drooping face. In some embodiments, the position of the tip 1040 is specifically based on the position of the user's eyebrows 1022 (a physical feature that does not anatomically correspond to the tip 1040). In some embodiments, when the user no longer makes a frowning expression, the electronic device modifies the poop avatar to return it to a neutral position. In such embodiments, the electronic device modifies the poop avatar to return it to the neutral state in 1011A by displaying an animation of the mouth 1030 moving to a neutral position and the lower portion 1036 moving back to its neutral position. In some implementations, returning the poop avatar from a droopy face to a neutral state includes an electronic device displaying the tip 1040 of the poop avatar as pulled until it reaches a neutral position.

[0308] As shown in 1001D, the electronic device detects that the corners 1020A and 1020B of the user's mouth are slightly raised and the user's eyebrows 1022 are in a raised position (e.g., at a higher position on the user's face compared to the position of the eyebrows 1022 when in the relaxed neutral state shown in 1001A and 1001B). In response to detecting the position of the corners 1020A and 1020B of the user's mouth 1020, the electronic device modifies the display of the poop avatar so that the corners of the avatar's mouth 1030 are slightly raised to match the position of the corners 1020A and 1020B of the user's mouth 1020. In response to detecting a raised position of the user's eyebrows 1022, the electronic device modifies the poop avatar by introducing the eyebrows 1038, located above the eyes 1032 of the poop avatar, into the raised position (e.g., to convey the expression that the poop avatar 1000 is raising its eyebrows 1038) and extending the upper portion 1034 of the poop avatar in an upward direction (e.g., by extending the tip 1040 of the poop avatar while maintaining the original position of the lower portion 1036). In the embodiments shown in 1001D and 1011D, when the user's eyebrows 1022 are raised, the electronic device introduces the avatar eyebrows 1038 and extends the tip 1040 of the poop avatar. In some embodiments, when the user's eyebrows 1022 return to a neutral position, the electronic device removes the avatar eyebrows 1038 and relaxes the tip 1040. In some implementations, the electronic device removes the eyebrows 1038 of the poop avatar by animing the process of moving the eyebrows 1038 downward toward the eyes 1032 of the poop avatar and disappearing into the face 1000C of the poop avatar above the eyes 1032.

[0309] Figure 10B An exemplary embodiment is illustrated, showing an electronic device modifying a poop avatar in response to detecting changes in a user's facial features, wherein the modification of the poop avatar includes exaggerating the movement of avatar features. In some embodiments, exaggerating the features of the virtual avatar allows the user to influence the maximum changes made to the avatar features without uncomfortably altering the corresponding features of their face. For example, as... Figure 10B As shown, users can make their avatars open their mouths as wide as possible (e.g., in a surprised expression) without making users uncomfortable by opening their mouths (e.g., without making users open their mouths to the maximum range of predicted or determined user mouth movements).

[0310] Figure 10B The poop avatar is displayed with three display states (1012A, 1012B, and 1012C), each corresponding to one of the user's three detection states (1002A, 1002B, and 1002C). Specifically, in Figure 10BIn each display state, the electronic device positions or modifies the features of the poop avatar in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the user's corresponding state. Figure 10B In the illustrated implementation, the detected user facial features include the user's mouth 1020. In some implementations, the tracked physical features may include other facial features such as eyelids, iris, muscles, muscle groups, etc. Figure 10B In the implementation plan, the features of the poop avatar include a mouth 1030, eyes 1032, an upper part 1034, and a lower part 1036.

[0311] As shown in 1002A, the electronic device detects a first state in which the user's mouth 1020 is slightly open (e.g., 10% of the maximum range of predicted or determined movement of the user's mouth). In response to detecting the slightly open user's mouth 1020, the electronic device modifies the poop avatar, as shown in 1012A, such that the poop avatar's mouth 1030 has an open position (20% of the maximum range of simulated movement of the avatar's mouth, which is greater than (e.g., within its corresponding range of movement) the degree of opening of the user's mouth 1020), while keeping other features of the avatar (such as the eyes 1032, upper part 1034, and lower part 1036 of the poop avatar) unchanged.

[0312] As shown in 1002B, the electronic device detects a change in the user's facial features, in which the user's mouth 1020 is opened wider than in state 1002A (e.g., 25% of the maximum range of predicted or determined movement of the user's mouth). In response to detecting a shift in the user's mouth 1020 from a slightly open position in 1002A to a wider open position in 1002B, the electronic device modifies the mouth 1030 of the poop-shaped face to increase its size, as shown in 1012B, such that the mouth 1030 has a larger open position than shown in 1012A (e.g., 50% of the maximum range of simulated movement of the face's mouth), while still maintaining the positioning of other face features (including the eyes 1032, upper part 1034, and lower part 1036 of the poop-shaped face).

[0313] As shown in 1002C, the electronic device detects another change in the user's facial features, in which the user's mouth 1020 is opened even wider than in 1002B (e.g., 50% of the maximum range of predicted or determined movement of the user's mouth). In response to detecting a change in the user's mouth 1020 from the open position in 1002B to the wider open position in 1002C, the electronic device modifies the mouth 1030 of the poop-shaped head to further increase its size, as shown in 1012C, such that the mouth 1030 has an open position larger than that shown in 1012B (e.g., 100% of the maximum range of simulated movement of the head's mouth). However, in 1012C, the opening of the mouth 1030 is greater than the height of the lower portion 1036. Therefore, the electronic device expands the lower portion 1036 of the poop-shaped head 1000 at 1042 to maintain the structural integrity of the poop-shaped head 1000 in response to the user's open mouth 1020. In other words, to maintain consistent positioning of the avatar and its features in response to the user's open mouth 1020, the device modifies one or more interconnected portions of the virtual avatar (e.g., the lower portion 1036). For example, as shown in 1012C, the electronic device extends the lower portion 1036 of the poop avatar adjacent to the avatar's mouth 1030 at region 1042 to accommodate the enlarged size of the mouth 1030. If the electronic device does not modify the lower portion 1036 in this way, the enlarged mouth 1030 will extend beyond the structure of the virtual avatar, potentially hindering the context and / or tone that the user intends to convey using the virtual avatar.

[0314] In some embodiments, the increase in the size of the portrait mouth 1030 (e.g., from the position shown in 1012A to the position shown in 1012B, or from the position shown in 1012B to the position shown in 1012C) is not proportional to the increase in the size of the user mouth 1020 (e.g., from the position shown in 1002A to the position shown in 1002B, or from the position shown in 1002B to the position shown in 1002C), but is scaled to provide an exaggerated rate of size change. For example, in some embodiments, the scaling factor is a multiplier of 2, such that the relative opening position of the portrait mouth is twice the relative opening position of the user mouth. For example, if the user mouth opens to 10% of the maximum range of motion of the user mouth, the electronic device displays the portrait mouth opening to 20% of the maximum range of motion of the portrait mouth.

[0315] Figure 10CAn exemplary embodiment of an electronic device modifying a poop avatar in response to a detected change in a user's physical characteristics is shown, wherein the modification of the poop avatar includes rotating (e.g., tilting) the upper part of the poop avatar forward and backward while keeping the lower part of the poop avatar stationary. The electronic device displays the poop avatar with three display states (1013A, 1013B, and 1013C), each of the three display states of the poop avatar corresponding to three detection states of the user (1003A, 1003B, and 1003C). Specifically, in Figure 10C In each display state, the electronic device positions or modifies the features of the poop avatar in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the user's corresponding state. Figure 10C In the illustrated implementation, the user's physical features include the user's face 1024, chin 1026, and head 1028. In some implementations, the tracked physical features may include other facial features such as eyelids, iris, muscles, muscle groups, etc. Figure 10C In the implementation scheme, the features of the poop avatar include a mouth 1030, eyes 1032, an upper part 1034 (including the tip 1040 of the poop avatar) and a lower part 1036.

[0316] As shown in 1003A, the electronic device detects a user's face 1024 in a position rotated in the upward direction. For example, the user raises their chin 1026 and tilts their head 1028 backward (e.g., away from the field of view of the camera (e.g., camera 143, optical sensor 164)) to position their face 1024 in the upward direction. In response to detecting the upward position of the user's face 1024, the electronic device modifies the poop avatar to an upward-looking state by tilting the upper portion 1034 of the poop avatar in a direction away from the display (e.g., away from the focal plane of the camera (e.g., camera 143, optical sensor 164) backward), as shown in 1013A. In this backward-tilted position, the electronic device displays the tip 1040 of the poop avatar positioned towards the rear of the poop avatar 1000 to show the offset position of the tip 1040 when the poop avatar is modified to an upward-looking state. Additionally, the electronic device modifies the eyes 1032 of the poop-shaped image to look upwards (e.g., by shifting the pupils or irises 1032A of the poop-shaped image toward the top of the eyes 1032), as shown in 1013A. While the electronic device tilts the upper portion 1034 of the poop-shaped image and modifies the eyes 1032 to look upwards, it keeps other features of the poop-shaped image unchanged. For example, the electronic device maintains the position of the mouth 1030 and fixes the position of the lower portion 1036 of the poop-shaped image, such that the electronic device displays the upper portion 1034 rotated backwards about an axis (e.g., the x-axis) that extends along the width of the lower portion 1036.

[0317] As shown in 1003B, the electronic device detects a user's face 1024 in a position rotated in the downward direction. For example, the user lowers (pulls back) their chin 1026 and tilts their head 1028 forward (e.g., toward the focal plane of the camera (e.g., camera 143, optical sensor 164)) to position their face 1024 in the downward direction. In response to detecting the downward position of the user's face 1024, the electronic device modifies the poop avatar to a downward-looking state by tilting the upper portion 1034 of the poop avatar in a direction toward the display (e.g., toward the focal plane of the camera (e.g., camera 143, optical sensor 164)), as shown in 1013B. In this forward-tilted position, the electronic device displays the tip 1040 of the poop avatar positioned toward the front side of the poop avatar to show the offset position of the tip 1040 when the poop avatar is modified to a downward-looking state. Additionally, the electronic device modifies the eyes 1032 of the poop-shaped image to a downward-looking state (e.g., by shifting the pupils or irises 1032A of the poop-shaped image toward the bottom of the eyes 1032), as shown in 1013B. While the electronic device tilts the upper portion 1034 of the poop-shaped image and modifies the eyes 1032 to a downward-looking state, it keeps other features of the poop-shaped image unchanged. For example, the electronic device maintains the position of the mouth 1030 and fixes the position of the lower portion 1036 of the poop-shaped image, such that the electronic device displays the upper portion 1034 rotated forward about an axis (e.g., the x-axis) that extends along the width of the lower portion 1036.

[0318] As shown in 1003C, the electronic device detects that the user's position (specifically, the position of the user's face 1024 and head 1028) is in the downward-rotated position shown in 1003B, but this user position is also shifted downward from the position shown in 1003B within the field of view of the camera (e.g., camera 143, optical sensor 164). In response, the electronic device displays the poop-shaped head image 1000 in a forward-tilted position shown in 1013B, but this poop-shaped head image is also shifted downward within the display area in 1013C to mirror the downward direction of the user's shift within the camera's field of view.

[0319] Figure 10D An exemplary embodiment of an electronic device modifying a poop avatar in response to a detected change in a user's physical characteristics is shown, wherein the modification of the poop avatar includes rotating the upper part of the poop avatar while keeping the lower part of the poop avatar stationary. The poop avatar is shown having four display states (1014A, 1014B, 1014C, and 1014D), wherein each of the four display states of the poop avatar corresponds to one of the user's four detection states (1004A, 1004B, 1004C, and 1004D). Figure 10DIn each display state, the electronic device positions or modifies the features of the poop avatar in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the user's corresponding state. Figure 10D In the illustrated implementation, the user's physical features include the user's mouth 1020, face 1024, head 1028, and shoulders 1021. In some implementations, the tracked physical features may include other facial features such as eyelids, iris, muscles, muscle groups, etc. Figure 10D In the implementation scheme, the features of the poop avatar include a mouth 1030, eyes 1032, an upper part 1034 (including the tip 1040 of the poop avatar) and a lower part 1036.

[0320] As shown in 1004A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the user's right side, while the user's shoulders 1021 remain forward-positioned. In response, the electronic device modifies the poop avatar by twisting the upper part 1034 of the poop avatar to the right (while keeping the lower part 1036 stationary), as shown in 1014A, such that the twisting motion of the virtual avatar mirrors the rightward rotation of the user's head 1028 and face 1024. The electronic device also detects a smiling posture of the user's mouth 1020 and modifies the avatar's mouth 1030 to a smiling state. As shown in 1014A, the electronic device twists the upper part 1034 of the poop avatar about an axis 1051 (e.g., the y-axis) that extends vertically through the center of the poop avatar. When the electronic device twists the upper part 1034 of the poop avatar to the right, it also shifts the tip 1040 of the poop avatar to the left, moves the eyes 1032 of the poop avatar to the right, and increases the amount of wrinkles 1034A or layers formed in the upper part 1034 of the poop avatar, thereby giving the poop avatar a distorted appearance including a slight twist 1070 of the lower part 1036, which is modeled based on the interconnection between the upper part 1034 and the lower part 1036. Even though parts of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the user's physical features, these modifications to the poop avatar provide an animation effect that mimics the user's physical movement.

[0321] 1004B and 1014B demonstrate a similar effect, wherein the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are rotated to the user's left side, while the user's shoulders 1021 remain forward-positioned. In response, the electronic device modifies the poop avatar by twisting the upper part 1034 of the poop avatar to the left (while keeping the lower part 1036 stationary), as shown in 1014B, such that the torsional motion of the virtual avatar mirrors the leftward rotational movement of the user's head 1028 and face 1024. As shown in 1014B, the electronic device twists the upper part 1034 of the poop avatar about an axis 1051 that extends vertically through the center of the poop avatar. When the electronic device twists the upper part 1034 of the poop head to the left, it also shifts the tip 1040 of the poop head to the right, moves the eyes 1032 of the poop head to the left, and increases the amount of wrinkles 1034A or layers formed in the upper part 1034 of the poop head, thereby giving the poop head a distorted appearance including a slight twist 1070 of the lower part 1036, which is modeled based on the interconnection between the upper part 1034 and the lower part 1036.

[0322] In some implementations, the electronic device does not track the movement (e.g., rotational movement) or positioning of the user's shoulder 1021, allowing the user to influence changes to the virtual avatar without having to maintain a fixed orientation or position in front of the camera (e.g., camera 143, optical sensor 164). For example, as shown in 1004C, the user's shoulder 1021 tilts or rotates to the user's right, but the lower portion 1036 of the poop avatar remains fixed, as shown in 1014C. However, the electronic device detects that the user's head 1028 and optionally the user's face 1024 rotate to the user's right. Therefore, as shown in 1014C, the electronic device modifies the poop avatar accordingly, as discussed above with respect to 1014A, without further modifying the poop avatar in response to the user rotating their shoulder 1021. Similar effects are shown in 1004D and 1014D, in which the user's shoulder 1021 tilts or turns to the user's left along with the user's head 1028, and the electronic device modifies the poop avatar as discussed above with respect to 1014B, without further modifying the poop avatar (e.g., the lower part 1036) in response to the user turning their shoulder 1021.

[0323] Figure 10EAn exemplary embodiment of an electronic device modifying a poop avatar in response to the detection of a change in a user's physical characteristics is shown, wherein the modification of the poop avatar includes tilting the upper part of the poop avatar while keeping the lower part of the poop avatar stationary. The poop avatar is shown having four display states (1015A, 1015B, 1015C, and 1015D), each of the four display states of the poop avatar corresponding to four detection states of the user (1005A, 1005B, 1005C, and 1005D). In each display state, the electronic device positions or modifies features of the poop avatar in response to the detection of the position or positional change of one or more physical characteristics of the user (such as facial features) detected in the corresponding state of the user. Figure 10E In the illustrated implementation, the user's physical features include the user's mouth 1020, face 1024, head 1028, shoulders 1021, eyes 1023, and neck 1025. In some implementations, the tracked physical features may include other facial features such as eyelids, iris, muscles, muscle groups, etc. Figure 10E In the implementation scheme, the features of the poop avatar include a mouth 1030, eyes 1032, an upper part 1034 (including a tip 1040 and a middle part 1031) and a lower part 1036.

[0324] As shown in 1005A, the electronic device detects that the user's head 1028 and optionally the user's face 1024 (or various physical features including the face 1024) are tilted to the right of the user, while the user's shoulders 1021 remain forward-positioned. The electronic device also detects that the user's neck 1025 is slightly tilted to the right of the user. In response, the electronic device modifies the poop avatar by tilting the upper part 1034 of the poop avatar to the right (the upper part 1034 of the poop avatar includes a tip 1040 and a middle part 1031) while keeping the lower part 1036 stationary, as shown in 1015A, such that the tilting motion of the virtual avatar mirrors the rightward tilt of the user's head 1028 (and / or face 1024) and neck 1025.

[0325] In addition to mirroring the tilt direction, the electronic device also modifies the virtual avatar to illustrate the different degrees of tilt present in various physical features of the user shown in 1005A. For example, the upper part of the user (e.g., the user's head 1028) tilts more than the lower part of the user (e.g., the user's neck 1025). Therefore, as shown in 1015A, the electronic device modifies the virtual avatar such that the amount of movement or tilt is greatest at the top of the virtual avatar (e.g., at the tip 1040) and least at the bottom of the virtual avatar (e.g., at the bottom 1036). In other words, the amount of tilt gradually decreases from the top to the bottom of the virtual avatar, consistent with the variation in the degree of tilt shown in the user in 1005A. This is demonstrated by the tip 1040 with a large degree of tilt, the middle portion 1031 with a smaller degree of tilt than the tip 1040, and the untilted bottom portion 1036 of the poop avatar in 1015A. Even if parts of the avatar (e.g., the tip 1040 of the poop avatar) do not necessarily correspond anatomically to the user's physical features, these modifications to the poop avatar provide an animation effect that mimics the user's physical movements. Furthermore, for example, when the user tilts their head 1028 and neck 1025, the electronic device modifies the virtual avatar to have different degrees of tilt to mimic the reduced range of motion of the user's physical features.

[0326] 1005B and 1015B demonstrate a similar effect, wherein the electronic device detects that the user's head 1028 and neck 1025 are tilted to the left. In response, the electronic device modifies the poop avatar by tilting the upper part 1034 to the left at a different degree of tilt (e.g., the tip 1040 tilts to a greater degree than the middle part 1031), while keeping the lower part 1036 stationary, as shown in 1015B, such that the tilting motion of the virtual avatar mirrors the leftward tilt of the user's head 1028 and neck 1025, as discussed in more detail with respect to 100SA and 1015A.

[0327] As shown in 1005C, the electronic device detects a user eye 1023 offset to the user's right (e.g., the iris or pupil of the user's eye) and a smiling facial expression formed by the corners 1020A and 1020B of the user's mouth 1020 positioned in an upward posture. In response to detecting the rightward offset of the user's eye 1023, the electronic device modifies the eyes 1032 of the poop avatar to a right-looking state (e.g., by shifting the pupil or iris 1032A of the poop avatar toward the right side of the avatar's eyes 1032), as shown in 1015C. In response to detecting one or both of the corners 1020A and 1020B of the user's mouth positioned in an upward posture, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, wherein the avatar's mouth 1030 has an open smiling position, as shown in 1015C. As shown in 1015C, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A).

[0328] 1005D and 1015D demonstrate a similar effect, wherein the electronic device detects a user's smiling facial expression and the user's eye 1023 shifted to the user's left (e.g., the iris or pupil of the user's eye). In response to the detection of the leftward shift of the user's eye 1023, the electronic device modifies the eyes 1032 of the poop avatar to a left-looking state (e.g., by shifting the pupil or iris 1032A of the poop avatar toward the left of the avatar's eye 1032), as shown in 1015D. In response to the detection of one or both of the corners 1020A and 1020B in an upward posture, the electronic device modifies the display of the avatar's mouth 1030 to have a smiling expression, as discussed above relative to 1015C. Similarly, in 1015D, the upper portion 1034 and the lower portion 1036 remain unchanged relative to their respective neutral positions (shown in 1011A and 1015C).

[0329] Figure 10F An exemplary embodiment of an electronic device is shown in which a poop avatar is modified in response to the detection of an offset in the user's position within the field of view of a camera (e.g., camera 143, optical sensor 164). Modification of the poop avatar includes offsetting (e.g., translating) the poop avatar in a direction corresponding to the offset in the user's position within the camera's field of view. The poop avatar is shown having four display states (1016A, 1016B, 1016C, and 1016D) in four display areas, each of the four display states of the poop avatar corresponding to one of the user's four detection states (1006A, 1006B, 1006C, and 1006D). Figure 10FIn each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting the user's position or positional change detected within the camera's field of view in the user's corresponding state. In each of the four display states, the boundary of the displayed state (e.g., the boundaries of 1016A, 1016B, 1016C, and 1016D) represents the boundary of the display area including the virtual avatar.

[0330] As shown in 1006A, the electronic device detects that the user's position is horizontally centered within the field of view of the camera (e.g., camera 143, optical sensor 164). In response to detecting the user's horizontally centered position within the camera's field of view, the electronic device displays a poop image with the horizontally centered position within a display area as shown in 1016A.

[0331] In 1006B, the electronic device detects that the user's position is off-center (e.g., shifted or translated) in the right direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user shifts to the left of the user (e.g., shifted to the right relative to the camera's field of view), but remains fully visible within the camera's field of view. In response to detecting the user's shifted position in 1006B, the electronic device shifts the horizontal position of the entire poop-shaped image (including both the upper 1034 and lower 1036) so that the poop-shaped image is displayed at the left-shifted position, as shown in 1016B, to mirror the user's directional shift to the left of the user. As shown in 1006B, the user shifts to their left, bringing their left shoulder 1021A closer to the right edge of the field of view. Therefore, the electronic device displays a poop-shaped image located near the left edge of the display area in 1016B, which mirrors the direction of the user's shifted position within the camera's field of view. In some implementations, the user's offset position is mirrored by offsetting the virtual avatar in both direction and magnitude. In some implementations, the user's offset position is mirrored by offsetting the virtual avatar only in direction, and the magnitude of the virtual avatar's offset is adjusted (e.g., attenuated) to maintain the virtual avatar's position within the boundaries of the display area. Examples of such implementations are discussed below with respect to 1006C, 1016C, 1006D, and 1016D.

[0332] In 1006C, the electronic device detects that the user's position is off-center (e.g., shifted or translated) in the rightmost direction within the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user is shifted far to the left of the user (e.g., shifted to the right relative to the camera's field of view), but shifted to the point that the user's left shoulder 1021A is no longer within the camera's field of view. In response to detecting the user's significant positional shift in 1006C, the electronic device shifts the horizontal position of the poop-shaped image so that the entire poop-shaped image (including both the upper 1034 and the lower 1036) is displayed in a position completely shifted to the left, wherein the outermost edge of the poop-shaped image (e.g., the edge 1036A of the lower 1036) is positioned against the left boundary of the display area shown in 1016C. The offset display mirror of the poop avatar in 1016C reflects the user's leftward offset direction, rather than offsetting the poop avatar so that a portion of the avatar extends beyond the display area in 1016C (as the user does in 1006C). The device positions the poop avatar at the edge of the display area 1016C. By keeping the position of the virtual avatar within the display area (e.g., 1016A, 1016B, 1016C, and 1016D) even when a portion of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to influence changes to the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0333] A similar effect is shown in 1006D and 1016D. In 1006D, the electronic device detects that the user has shifted to the right (e.g., shifted to the left in the camera's field of view), causing the user's right shoulder to no longer be within the camera's field of view. In response to detecting the user's significant shift, the electronic device shifts the horizontal position of the poop avatar so that the entire poop avatar (including both the upper 1034 and the lower 1036) is displayed in a completely right-shifted position, wherein the outermost edge of the poop avatar (e.g., edge 1036B of the lower 1036) is positioned against the right boundary of the display area shown in 1016D. As described above, by keeping the position of the virtual avatar within the display area 1016D even when part of the user is outside the field of view of the camera (e.g., camera 143, optical sensor 164), the electronic device allows the user to influence changes in the virtual avatar without having to maintain a fixed orientation or position in front of the camera.

[0334] Figure 10G An exemplary embodiment of an electronic device is shown in which a poop avatar is modified in response to the detection of an offset in the position of a user's physical features within the field of view of a camera (e.g., camera 143, optical sensor 164). The modification of the poop avatar includes offsetting (e.g., translating) the poop avatar in a direction corresponding to the offset in the position of the user's physical features within the camera's field of view. Figure 10G The embodiment shown is similar to the one described above relative to the one described above in the following respects. Figures 10D to 10F The embodiments discussed involve electronic devices that track the movement and positioning of the user's head 1028 (e.g., rotational and / or translational movement), but not the movement or positioning of the user's shoulders 1021 and optionally the user's neck 1025. Additionally, Figure 10G The implementation scheme shown is similar to the following in the following respects Figure 10F The implementation scheme in this case involves modifying the virtual avatar to mirror the user's movement in direction, but not necessarily in terms of magnitude. By implementing these techniques, electronic devices allow users to influence changes to the virtual avatar without having to maintain a fixed orientation or position in front of a camera (e.g., camera 143, optical sensor 164).

[0335] The poop avatar is displayed as having two display states (1017A and 1017B) in two display areas, each of the two display states of the poop avatar corresponding to two user detection states (1007A and 1007B). Figure 10G In each display state, the device positions or modifies the poop avatar in response to detecting the position or positional change of the user's physical features detected within the camera's field of view in the corresponding user state. In each of the two display states, the boundary of the displayed state (e.g., the boundary of 1017A and 1017B) represents the boundary of the display area including the virtual avatar.

[0336] In 1007A, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset to the left (e.g., translated) in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset to the right of the user (e.g., offset to the left relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007A, the electronic device offsets the horizontal position of the entire poop head image (including both the upper 1034 and the lower 1036) so that the poop head image is displayed at the right-off offset position, as shown in 1017A, in order to mirror the offset of the user's head to the right of the user.

[0337] Similar effects are shown in 1007B and 1017B. In 1007B, the electronic device detects that the user's head 1028 and optionally the user's neck 1025 are offset to the right (e.g., translated) in the field of view of the camera (e.g., camera 143, optical sensor 164). In other words, the user's head 1028 and neck 1025 are offset to the left of the user (e.g., offset to the right relative to the camera's field of view). In response to detecting the offset position of the user's head 1028 and optionally the user's neck 1025 in 1007B, the electronic device offsets the horizontal position of the entire poop head image (including both the upper 1034 and the lower 1036) so that the poop head image is displayed at the offset position to the left, as shown in 1017B, in order to mirror the offset of the user's head to the left of the user.

[0338] Figure 10H An exemplary embodiment of an electronic device modifying a poop avatar in response to detecting a change in the position of a user's physical features within the field of view of a camera (e.g., camera 143, optical sensor 164). Modification of the poop avatar includes increasing or decreasing its size and shifting (e.g., translating) it in a direction corresponding to the shift in the position of the user's physical features within the camera's field of view. The poop avatar is shown having four display states (1018A, 1018B, 1018C, and 1018D) in four display areas, each of the four display states corresponding to four detection states of the user (1008A, 1008B, 1008C, and 1008D). Figure 10H In each of the four display states, the electronic device positions or modifies the poop avatar in response to detecting the position or positional change of the user's physical features detected within the camera's field of view in the user's corresponding state. In each of the four display states, the boundary of the displayed state (e.g., the boundaries of 1018A, 1018B, 1018C, and 1018D) represents the boundary of the display area including the virtual avatar.

[0339] In 1008A, the electronic device detects that the user's head 1028 is offset (e.g., translated) in the upward direction relative to its shoulders 1021 within the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is stretching their neck 1025 upward). In response to detecting the upward offset position of the user's head 1028 in 1008A, the electronic device offsets the vertical position of the entire poop head image (including both the upper part 1034 and the lower part 1036) such that the entire poop head image is displayed in an upward offset position, wherein the uppermost edge of the poop head image (e.g., the edge 1040A of the tip 1040) is positioned close to the upper boundary of the display area shown in 1018A so as to mirror the upward offset of the user's head 1028 shown in 1008A.

[0340] In 1008B, the electronic device detects that the user's head 1028 is offset (e.g., translated) in the downward direction relative to the user's shoulder 1021 within the field of view of the camera (e.g., camera 143, optical sensor 164) (e.g., the user is drooping their head 1028). In response to detecting the downward offset position of the user's head 1028 in 1008B, the electronic device shifts the vertical position of the entire poop head image (including both the upper part 1034 and the lower part 1036) such that the entire poop head image is displayed in the downward offset position, wherein the lowest edge of the poop head image (e.g., the edge 1036C of the lower part 1036) is positioned close to the lower boundary of the display area shown in 1018B so as to mirror the downward offset of the user's head 1028 shown in 1008B.

[0341] In 1008C, the electronic device detects, for example, that the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) increases as the user's head 1028 moves closer to the camera. In response to detecting this increase in the size of the user's head 1028 in 1008C, the electronic device increases the size of the entire poop image. In some embodiments, the electronic device adjusts the size of the poop image based on the detected user head 1028, moving from a detected state (e.g., ...). Figure 10A The size of the poop avatar is increased by a size change from a neutral state (1001A) to another state (e.g., detected state 1008C). In 1018C, the electronic device increases the size of the poop avatar to fill the display area without causing any part of the poop avatar to extend beyond the boundaries of the display area. In some embodiments, the electronic device increases the size of the virtual avatar to provide the impression that the avatar is positioned very close to a display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504).

[0342] For example, in 1018C, the electronic device increases the size of the poop-shaped image such that the tip 1040 of the poop-shaped image is near the upper boundary of the display area at 1040A, the lower portion 1036 of the poop-shaped image is near the lower boundary of the display area at 1036C, the left edge of the lower portion 1036 is near the left boundary of the display area at 1036A, and the right edge of the lower portion 1036 is near the right boundary of the display area at 1036B. In some embodiments, such as the one shown in 1018C, the electronic device increases the size of the poop-shaped image proportionally such that the relative positions of various image features (e.g., the eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) are not distorted relative to the shape of the poop-shaped image. For example, when the size of the poop-shaped head is increased in the electronic device 1018C, the size of the eyes 1032, mouth 1030, upper part 1034 (including tip 1040) and lower part 1036 of the head are also increased, but other parts remain unchanged.

[0343] In 1008D, the electronic device detects, for example, that the size of the user's head 1028 within the field of view of the camera (e.g., camera 143, optical sensor 164) decreases as the user's head 1028 moves away from the camera positioning. In response to detecting this decrease in the size of the user's head 1028 in 1008D, the electronic device reduces the overall size of the poop-shaped image. In some embodiments, the electronic device adjusts the size of the poop-shaped image based on the detected user's head 1028 from a detected state (e.g., ...). Figure 10A The size of the poop avatar is reduced by changing its size from a neutral state (1001A) to another state (e.g., detected state 1008D). In some embodiments, such as in 1018D, the electronic device reduces the size of the poop avatar to provide the impression that the virtual avatar is positioned away from the display of the electronic device (e.g., touchscreen 112, display 340, display 450, display 504).

[0344] For example, in 1018D, the electronic device reduces the size of the poop-shaped image such that the tip 1040 of the poop-shaped image is positioned away from the upper boundary of the display area at 1040A, the lower portion 1036 of the poop-shaped image is positioned away from the lower boundary of the display area at 1036C, the left edge of the lower portion 1036 is positioned away from the left boundary of the display area at 1036A, and the right edge of the lower portion 1036 is positioned away from the right boundary of the display area at 1036B. In some embodiments, such as the one shown in 1018D, the electronic device reduces the size of the poop-shaped image proportionally such that the relative positions of various image features (e.g., the eyes 1032, mouth 1030, upper portion 1034, and lower portion 1036) remain unchanged relative to the shape of the poop-shaped image. For example, when the electronic device reduces the size of the poop avatar in 1018D, the sizes of the avatar's eyes 1032, mouth 1030, upper part 1034 (including tip 1040), and lower part 1036 also decrease, but other parts remain unchanged.

[0345] Figure 10I An exemplary embodiment illustrating how an electronic device modifies a poop avatar 1000 in response to detecting a change in a user's physical characteristics (such as facial features) is shown. The poop avatar is shown as having a display state 1019 corresponding to a user detection state 1009. As shown in 1009 and 1019, the electronic device detects that the user is making a pouting expression, and in response, the electronic device modifies the poop avatar 1000 by replacing the mouth 1030 of the poop avatar with a set of pouting lips 1050. In some embodiments, such as the one shown in 1009, the electronic device determines that the user is making a pouting facial expression by detecting that the user's jaw 1027 is in a closed position and that the corners 1020A and 1020B of the user's mouth 1020 are moving toward each other such that the user's lips 1029 (e.g., the user's upper lip 1029A and lower lip 1029B) extend outward from the user's mouth 1020 in a pouting gesture. Although not shown in Figure 10I As shown in the diagram, but in some implementations, the electronic device modifies the poop image to resemble... Figure 16B The way shown and discussed in more detail below is how the heart is emitted from the pouting lips 1050.

[0346] Figure 11A An exemplary embodiment illustrating how an electronic device modifies a bear head image 1100 in response to detecting changes in a user's facial features is shown. The bear head image is shown having four display states (1111A, 1111B, 1111C, and 1111D), each of which corresponds to one of the user's four detection states (1101A, 1101B, 1101C, and 1101D). Figure 11AIn each display state, the electronic device positions or modifies the features of the bear head in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the corresponding user state. Figure 11A In the illustrated implementation, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some implementations, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, etc. Figure 11A In the implementation scheme, the features of the bear head include a mouth 1130, eyes 1132, ears 1133, nose 1137, head 1135, and (in some states) eyebrows 1138.

[0347] As shown in 1101A, the electronic device detects a neutral facial expression of the user. For example, the device detects that the user's mouth 1120 and eyebrows 1122 are positioned in a relaxed, neutral state, rather than in a position associated with a specific facial expression (such as a smile or a frown). In response to detecting a neutral facial expression of the user, the electronic device displays a bear head 1100 with a neutral expression or state in 1111A. Specifically, the electronic device displays a bear head with a mouth 1130 in a relaxed, neutral state, rather than in a position normally associated with a specific facial expression (e.g., a smile or a frown). In some embodiments, the neutral position of the head corresponds to other representations of the associated image, such as static bear emoticons that can be found in messaging applications. In 1111A, the neutral state of the bear's mouth 1130 is indicated by lines 1130A and 1130B, which extend horizontally from the outside of the bear's mouth 1130, or "nose-mouth" area, and then curve slightly upward at the base of the bear's nose 1137. The electronic device also displays a bear head with a nose 1137 and ears 1133, wherein the nose is positioned above a relaxed mouth 1130, and the ears are in a relaxed, neutral state, positioned along the side of the bear's head 1135, and are neither curled nor extended. Additionally, the electronic device displays a bear head without eyebrows 1138 and displays eyes 1132 looking in a direction perpendicular to the focal plane of the display (or the camera (e.g., camera 143, optical sensor 164)).

[0348] As shown in 1101B and 1111B, the electronic device detects the position of the user's mouth 1120 (e.g., detects that one or both of the corners 1120A and 1120B of the user's mouth are positioned in an upward posture (e.g., an upward position) to form a smiling facial expression), and in response, the electronic device modifies two features of the bear's head. For example, in response to detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture, the electronic device modifies both the bear's ears 1133 and the bear's mouth 1130. The electronic device modifies the bear's mouth 1130 to have a smiling expression by rotating lines 1130A and 1130B upward, the upward rotation of lines 1130A and 1130B indicating the smiling expression of the bear's mouth 1130, as shown in 1111B. The electronic device modifies the bear's ears 1133 to be "erected upward" or extended in an upward direction. In some embodiments, such as the one shown in 1111B, the modification to the bear ears 1133 also includes slightly narrowing the width of each ear 1133 and changing the vertical position of the ears 1133 on the side of the bear head 1135 such that when the ears are erect, the ears 1133 are positioned at a higher position on the bear head 1135 (compared to the vertical position of the ears 1133 on the side of the bear head 1135 in the neutral state in 1111A). As shown in 1111B, the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user smiles, the electronic device changes the bear's mouth 1130 to a smiling state and makes the ears 1133 erect. When the user stops smiling, the electronic device returns the mouth 1130 and ears 1133 to their neutral positions.

[0349] As shown in 1101C, the electronic device detects that the user's eyebrows 1122 are in a raised position (e.g., at a higher position on the user's face compared to the position of the eyebrows 1122 when in the relaxed neutral state shown in 1101A and 1101B). In response to detecting the raised eyebrows 1122, the electronic device modifies two features of the bear head. For example, the electronic device modifies the bear ears 1133 to be erected and modifies the bear head to introduce eyebrows 1138 located above the bear eyes 1132, thereby conveying the impression that the bear head 1100 is raising its eyebrows 1138, as shown in 1111C. As shown in 1111B, the bear mouth 1130 returns to its neutral position, and the eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's eyebrows 1122 are raised, the electronic device introduces the eyebrows 1138 of the head and makes the ears 1133 erect. Therefore, when the user's eyebrows 1122 return to their neutral position, the electronic device removes the portrait eyebrows 1138 and relaxes the ears 1133.

[0350] In 1101D, the electronic device detects that the user's eyebrows 1122 are raised and one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture. In response, the electronic device modifies the bear head 1100 to maximize the erection of the ears, introduces raised eyebrows 1138, and makes it smile, as shown in 1111D. The electronic device modifies the ears 1133 to maximize the erection of the ears by significantly reducing the width of each ear 1133, extending the ears in the upward direction, and changing the vertical position of the ears 1133 on the side of the bear head 1135 so that the ears 1133 are positioned on the bear head 1135 at a position even higher than when they are in the erect positions shown in 1111B and 1111C. This combination of modifications to the bear ears 1133 causes the appearance of the ears 1133 to be stretched to a greater extent in the upward direction than the degree of erection shown in 1111B and 1111C. It should be understood that in this embodiment, when the user raises their eyebrows 1122 and smiles (e.g., one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in an upward posture), the electronic device introduces the eyebrows 1138 of the image, modifies the bear's mouth 1130 into a smiling state, and causes the ears 1133 to stand up to a great extent. Therefore, when the user's eyebrows 1122 and mouth 1120 return to their neutral positions, the electronic device removes the eyebrows 1138 of the image and relaxes the ears 1133 and mouth 1130.

[0351] Figure 11B An exemplary embodiment illustrating how an electronic device modifies a bear head image 1100 in response to detecting changes in a user's facial features is shown. The bear head image is shown having four display states (1112A, 1112B, 1112C, and 1112D), each of which corresponds to one of the user's four detection states (1102A, 1102B, 1102C, and 1102D). Figure 11B In each display state, the electronic device positions or modifies the features of the bear avatar in response to detecting the position or positional change of one or more physical features of the user (such as facial features) detected in the corresponding user state. In each of the four display states, the boundaries of the displayed state (e.g., the boundaries of 1112A, 1112B, 1112C, and 1112D) represent the boundaries of the display area including the virtual avatar.

[0352] exist Figure 11BIn the illustrated implementation, the detected facial features of the user include the user's mouth 1120 (with corners 1120A and 1120B) and the user's eyebrows 1122. In some implementations, the tracked physical features may include other facial features such as eyelids, irises, muscles, muscle groups, etc. Figure 11B In the implementation scheme, the features of the bear head include a mouth 1130, eyes 1132, ears 1133, nose 1137, head 1135, and (in some states) eyebrows 1138.

[0353] As shown in 1102A, the electronic device detects that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture (e.g., a downward position), and in response, the electronic device modifies two features of the bear's head. In some embodiments, the electronic device determines that the user is making a sad facial expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth are in a downward position, and optionally detects that one or both of the user's eyebrows 1122 are not in a low position (e.g., the eyebrows 1122 are raised or in a neutral position). In response to detecting the downward posture or position of the corners 1120A and 1120B of the user's mouth 1120, the electronic device modifies both the bear's mouth 1130 and the bear's ears 1133, as shown in 1112A.

[0354] The electronic device modifies the bear's mouth 1130 to have a sad expression by rotating lines 1130A and 1130B downwards. The downward rotation of lines 1130A and 1130B forms the sad expression of the bear's mouth 1130, as shown in 1112A. The electronic device modifies the bear's ears 1133 to be "curled" or "droopy" in the downward direction by rotating the outer edges 1133A and 1133B of the ears downwards (e.g., folding, rotating, or tilting). In some embodiments, curling the bear's ears 1133 also includes changing the vertical position of the ears 1133 on the side of the bear's head 1135, such that when the ears are curled, the ears 1133 are positioned at a lower position on the bear's head 1135 (compared to being in a lower position). Figure 11A The ears 1133 in the neutral state in 1111A are in a vertical position on the side of the bear's head 1135 (compared to the vertical position in 1111A). As shown in 1112A, the bear's eyes 1132 and nose 1137 remain unchanged relative to their original positions in 1111A. It should be understood that in this embodiment, when the user's mouth 1120 forms a sad facial expression, the electronic device changes the bear's mouth 1130 to a sad expression and causes the bear's ears 1133 to curl. When the user no longer makes a sad facial expression, the electronic device returns the bear's mouth 1130 and ears 1133 to their neutral positions.

[0355] In 1102B and 1112B, the electronic device detects that the user maintains a sad facial expression for a threshold amount of time, and in response, the electronic device modifies the bear head 1100 by introducing a tear 1140 emanating from the bear's eye 1132 to display the bear head as if it is crying while making a sad facial expression with curled ears 1133. In some embodiments, when the user maintains the sad expression for a first threshold amount of time (e.g., 0.3 seconds), the electronic device modifies the bear head to introduce only a single tear 1140 emanating from one or both of the bear's eyes 1132, and then, after the user continues to maintain the sad expression for a second threshold amount of time (e.g., 0.6 seconds), the electronic device modifies the bear head to introduce another tear 1140 emanating from the eye 1132. The electronic device displays the tear 1140 formed from the bear's eye 1132, which flows down the bear's face, drips from the bear's head 1135, and disappears from the view at the bottom edge of the display area in 1112B. In some embodiments, the electronic device displays tears 1140 intermittently and optionally at random intervals. In some embodiments, tears 1140 continue to appear until the user stops making a sad facial expression. As shown in 1112B, the electronic device continues to display curled bear ears 1133 and continues to display a bear mouth 1130 making a sad expression.

[0356] In 1102C and 1112C, the electronic device detects that the user maintains a frowning facial expression for a threshold amount of time, and in response, the electronic device modifies the bear head 1100 to have a frowning expression and a storm cloud 1142 located above the bear head 1135. The electronic device detects the user's frowning expression by detecting that one or both of the corners 1120A and 1120B of the user's mouth 1120 are positioned in a downward posture and by detecting that the user's eyebrows 1122 are in a downward position (e.g., the user's eyebrows are furrowed or positioned lower on the user's face compared to the position of the eyebrows 1122 in the relaxed neutral state in 1101A). In response to detecting the user's frowning facial expression, the electronic device modifies the bear head by rotating the bear mouth 1130 downward (e.g., by bending lines 1130A and 1130B downward, as shown in 1112C) and introducing the furrowed bear eyebrows 1138 to form a frowning eyebrow posture (as shown in 1112C). In the embodiment shown in 1112B, the electronic device does not immediately introduce a storm cloud 1142 in response to detecting a user's frowning facial expression, nor does the electronic device modify the bear's ears 1133 or nose 11311.

[0357] After the electronic device detects that the user has maintained a frowning facial expression for a threshold amount of time, the electronic device modifies the frowning bear head 1100 to introduce a storm cloud 1142 located above the bear head 1135. In some embodiments, the storm cloud 1142 is animated and dynamic. For example, the electronic device may modify the storm cloud 1142 to introduce lightning 1144 and / or rain. In some embodiments, the electronic device displays rain falling from the storm cloud 1142 and landing on the bear head 1135. In some embodiments, the falling rain flows down the bear's face, falls from the bear head 1135, and disappears from view at the bottom edge of the display area in 1112C. In some embodiments, the electronic device displays lightning 1144 and rain intermittently and optionally at random intervals. In some embodiments, the electronic device continues to display the storm cloud 1142 until the user stops making the frowning facial expression. When the user stops making the frowning facial expression, the electronic device modifies the bear head image 1100 so that the storm cloud 1142 disappears, the frowning eyebrows 1138 disappear, and the mouth 1130 returns to the neutral position shown in 1111A. In some embodiments, when the user holds the frowning facial expression for a first threshold amount of time (e.g., 0.3 seconds), the electronic device displays a storm cloud 1142 without lightning 1144 or rain, and then, after the user continues to hold the frowning facial expression for a second threshold amount of time (e.g., 0.6 seconds), the electronic device modifies the storm cloud 1142 to produce lightning 1144 and rain.

[0358] As shown in 1102D and 1112D, the electronic device detects that a user lowers one or both of its eyebrows 1122, and in response, modifies the bear ears 1133 to curl. In this embodiment, the electronic device curls the bear head's ears 1133 in response to detecting that a user has lowered its eyebrows 1122. When the user returns its eyebrows 1122 to their neutral position, the electronic device straightens the bear ears 1133. In the embodiments shown in 1102D and 1112D, the electronic device refrains from modifying other features of the bear head 1100. Thus, the electronic device maintains the neutral positions of the bear's eyes 1132, mouth 1130, and nose 1137, and does not introduce eyebrows 1138 or other objects (e.g....

Claims

1. A method comprising: At an electronic device having one or more cameras and display devices: The virtual avatar is displayed via the display device, wherein the virtual avatar includes: A first facial feature, which responds to changes in a first physical feature of the face within the field of view of the one or more cameras; The second avatar feature, which reacts to changes in the first physical feature; and The third avatar feature does not primarily respond to changes in the first physical feature; When the virtual avatar is displayed, the change in the first physical feature is detected; and In response to detecting the change in the first physical feature: Modifying the first avatar feature based on the detected change in the first physical feature includes: Based on the determination that the change in the first physical feature includes a change in a first value, the first avatar feature is modified by a first modification value based on the first value; and Based on the determination that the change in the first physical feature includes a change in a second value different from the first value, the first avatar feature is modified by a second modified value based on the second value, wherein the second modified value is different from the first modified value; Modifying the second avatar feature based on the detected change in the first physical feature includes: Based on the determination that the change in the first physical feature includes the change in the first value, the second avatar feature is modified by a third modification value based on the first value; and Based on the determination that the change in the first physical feature includes a change in a second value different from the first value, the second avatar feature is modified by a fourth modification value based on the second value, wherein the fourth modification value is different from the third modification value; and Abandon modifying the third avatar feature based on the detected changes in the first physical feature.

2. The method of claim 1, wherein the first head feature corresponds anatomically to the first physical feature, and wherein the second head feature does not correspond anatomically to the first physical feature.

3. The method of claim 1, wherein the third facial feature responds to changes in the second physical feature, wherein the third facial feature anatomically corresponds to the second physical feature, the method further comprising: When the virtual avatar is displayed, the change in the second physical feature is detected; and In response to the detection of the change in the second physical characteristic: The third avatar feature is modified based on the detected changes in the second physical feature.

4. The method of claim 3, wherein the second avatar feature responds to changes in the second physical feature.

5. The method of claim 3, wherein the first portrait feature responds to changes in the second physical feature.

6. The method according to claim 1, wherein: The changes in the first physical characteristic include changes of a first type; Modifying the first avatar feature based on the detected change in the first physical feature includes modifying the first avatar feature based on a second type of change; and Modifying the second avatar feature based on the detected change in the first physical feature includes modifying the second avatar feature based on a third type of change, which is different from the second type of change.

7. The method of claim 1, wherein the virtual avatar further comprises a fourth avatar feature that responds to changes in the first physical feature, and the method further comprises: In response to detecting the change in the first physical feature: The fourth avatar feature is modified based on the detected change in the first physical feature.

8. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having one or more cameras and display devices, the one or more programs including instructions for performing the method according to any one of claims 1-7.

9. An electronic device, comprising: One or more cameras; Display device; One or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1-7.

10. An electronic device, comprising: One or more cameras; Display device; and Apparatus for performing the method according to any one of claims 1 to 3 and 6 to 7.

11. A computer program product comprising one or more programs executed by one or more processors of a computer system configured to communicate with one or more cameras and display devices, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 3 and 6 to 7.

Citation Information

Patent Citations

  • Method and apparatus for integrating manual input

    US20020015024A1

  • Acceleration-based theft detection system for portable electronic devices

    US20050190059A1

  • Methods and apparatuses for operating a portable device based on an accelerometer

    US20060017692A1

  • Gestures for touch sensitive input devices

    US20060026521A1

  • Gestures for touch sensitive input devices

    US20060026536A1