User Interface for Multi-User Communication Sessions

Abstract

The present disclosure generally relates to user interfaces for multi-user communication sessions. In some examples, a device initiates a live stream in a communication session. In some examples, the device switches between streaming live audio and live video. In some examples, the device is capable of synchronizing media playback during a live stream.

Description

[0001] This divisional application is for an application with international application number PCT / US2018 / 048151, international filing date of August 27, 2018, date of entry into the Chinese national phase of February 28, 2020, national application number 201880056514.5, and invention title "User Interface for Multi-User Communication Sessions".

[0002] Cross - Reference to Related Applications

[0003] This patent application claims priority to U.S. Patent Application No. 16 / 035,422, filed on July 13, 2018, entitled "USER INTERFACE FOR MULTI-USER COMMUNICATION SESSION", which claims priority to U.S. Provisional Patent Application No. 62 / 566,181, filed on September 29, 2017, entitled "USER INTERFACE FOR MULTI-USER COMMUNICATION SESSION", the content of each of these patents being hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0004] The present disclosure generally relates to computer user interfaces and, more particularly, to techniques for managing communication sessions. BACKGROUND ART

[0005] Electronic devices enable remote users to communicate, such as via instant messaging and phone calls. While such exemplary conventional techniques enable communication, these techniques only provide users with limited ability to fully participate in a session, especially when the session includes a group of users. SUMMARY OF THE INVENTION

[0006] Some techniques for using an electronic device to manage a communication session are generally cumbersome and inefficient. For example, some prior art uses complex and time-consuming user interfaces, which may include multiple button presses or keystrokes. The prior art requires more time than necessary, which results in wasted user time and device energy. This latter consideration is particularly important in battery-powered devices such as mobile phones.

[0007] Accordingly, the present technology provides faster and more efficient methods and interfaces for managing communication sessions in an electronic device. Such methods and interfaces optionally supplement or replace other methods for managing communication sessions. Such methods and interfaces reduce the cognitive burden imposed on the user and result in a more effective human-machine interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges. Additionally, such techniques enable the user to more fully participate in a session with multiple participants simultaneously. Further, such techniques enable a remote user to experience media while communicating.

[0008] According to some embodiments, a method is performed at an electronic device having a display, one or more camera sensors, and one or more microphones. The method includes: receiving user input identifying one or more contacts to be included as one or more participants in a communication session; while in the communication session, simultaneously displaying on the display: a first enabling representation for transmitting a real-time media stream including real-time audio and not including real-time video and a second enabling representation for transmitting a real-time media stream including real-time audio and real-time video; receiving user input activating one of the first enabling representation and the second enabling representation; in response to receiving user input activating the first enabling representation, simultaneously: using the one or more microphones to detect audio; and transmitting the real-time audio in the real-time media stream to the one or more participants in the communication session, wherein the transmission of the real-time audio occurs without transmitting real-time video; and in response to receiving user input activating the second enabling representation, simultaneously: using the one or more microphones to detect audio; using the one or more camera sensors to detect a plurality of images of video; and transmitting the real-time audio and the real-time video in the real-time media stream to the one or more participants in the communication session.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, one or more camera sensors, and one or more microphones, the one or more programs including instructions for: receiving user input identifying one or more contacts to be included as one or more participants in a communication session; while in the communication session, simultaneously displaying on the display: a first enabling representation for transmitting a real-time media stream including real-time audio and not including real-time video and a second enabling representation for transmitting a real-time media stream including real-time audio and real-time video; receiving user input activating one of the first enabling representation and the second enabling representation; in response to receiving user input activating the first enabling representation, simultaneously: using the one or more microphones to detect audio; and transmitting the real-time audio in the real-time media stream to the one or more participants in the communication session, wherein the transmission of the real-time audio occurs without transmitting real-time video; and in response to receiving user input activating the second enabling representation, simultaneously: using the one or more microphones to detect audio; using the one or more camera sensors to detect a plurality of images of video; and transmitting the real-time audio and the real-time video in the real-time media stream to the one or more participants in the communication session.

[0010] According to some embodiments, a transitory computer-readable storage medium is described. The transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, one or more camera sensors, and one or more microphones, the one or more programs including instructions for: receiving user input identifying one or more contacts to be included as one or more participants in a communication session; while in the communication session, simultaneously displaying on the display: a first enabling representation for transmitting a real-time media stream including real-time audio and not including real-time video and a second enabling representation for transmitting a real-time media stream including real-time audio and real-time video; receiving user input activating one of the first enabling representation and the second enabling representation; in response to receiving user input activating the first enabling representation, simultaneously: using the one or more microphones to detect audio; and transmitting the real-time audio in the real-time media stream to the one or more participants in the communication session, wherein the transmission of the real-time audio occurs without transmitting real-time video; and in response to receiving user input activating the second enabling representation, simultaneously: using the one or more microphones to detect audio; using the one or more camera sensors to detect a plurality of images of video; and transmitting the real-time audio and the real-time video in the real-time media stream to the one or more participants in the communication session.

[0011] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more camera sensors; one or more microphones; one or more processors; and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: receiving user input identifying one or more contacts to be included as one or more participants in a communication session; while in the communication session, simultaneously displaying on the display: a first enabling representation for transmitting a real-time media stream including real-time audio and not including real-time video and a second enabling representation for transmitting a real-time media stream including real-time audio and real-time video; receiving user input activating one of the first enabling representation and the second enabling representation; in accordance with receiving user input activating the first enabling representation, simultaneously: using the one or more microphones to detect audio; and transmitting the real-time audio in the real-time media stream to the one or more participants in the communication session, wherein transmitting the real-time audio occurs without transmitting real-time video; and in accordance with receiving user input activating the second enabling representation, simultaneously: using the one or more microphones to detect audio; using the one or more camera sensors to detect a plurality of images of video; and transmitting the real-time audio and real-time video in the real-time media stream to the one or more participants in the communication session.

[0012] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more camera sensors; one or more microphones; means for receiving user input identifying one or more contacts to be included as one or more participants in a communication session; while in the communication session, means for simultaneously displaying on the display: a first enabling representation for transmitting a real-time media stream including real-time audio and not including real-time video and a second enabling representation for transmitting a real-time media stream including real-time audio and real-time video; means for receiving user input activating one of the first enabling representation and the second enabling representation; in accordance with receiving user input activating the first enabling representation, means for simultaneously performing the following operations: using the one or more microphones to detect audio; and transmitting the real-time audio in the real-time media stream to the one or more participants in the communication session, wherein transmitting the real-time audio occurs without transmitting real-time video; and in accordance with receiving user input activating the second enabling representation, means for simultaneously performing the following operations: using the one or more microphones to detect audio; using the one or more camera sensors to detect a plurality of images of video; and transmitting the real-time audio and real-time video in the real-time media stream to the one or more participants in the communication session.

[0013] According to some embodiments, a method is performed at an electronic device having a display. The method includes: when transmitting an outgoing real-time media stream of a first type to one or more participants of a communication session: outputting at least one or more incoming real-time media streams at the electronic device; receiving user input for switching from transmitting the outgoing real-time media stream of the first type to transmitting an outgoing real-time media stream of a second type, where the first type is different from the second type; and in response to receiving the user input for switching, switching from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, including: transmitting real-time video according to a first type of real-time media stream that includes real-time audio and does not include real-time video; and stopping transmitting real-time video according to a first type of real-time media stream that includes both real-time audio and real-time video.

[0014] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for: when transmitting an outgoing real-time media stream of a first type to one or more participants of a communication session: outputting at least one or more incoming real-time media streams at the electronic device; receiving user input for switching from transmitting the outgoing real-time media stream of the first type to transmitting an outgoing real-time media stream of a second type, where the first type is different from the second type; and in response to receiving the user input for switching, switching from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, including: transmitting real-time video according to a first type of real-time media stream that includes real-time audio and does not include real-time video; and stopping transmitting real-time video according to a first type of real-time media stream that includes both real-time audio and real-time video.

[0015] According to some embodiments, a transitory computer-readable storage medium is described. The transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for: when transmitting an outgoing real-time media stream of a first type to one or more participants of a communication session: outputting at least one or more incoming real-time media streams at the electronic device; receiving user input for switching from transmitting the outgoing real-time media stream of the first type to transmitting an outgoing real-time media stream of a second type, where the first type is different from the second type; and in response to receiving the user input for switching, switching from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, including: transmitting real-time video according to a first type of real-time media stream that includes real-time audio and does not include real-time video; and stopping transmitting real-time video according to a first type of real-time media stream that includes both real-time audio and real-time video.

[0016] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: when transmitting a first type of outgoing real-time media stream to one or more participants of a communication session: outputting at least one or more incoming real-time media streams at the electronic device; receiving a user input for switching from transmitting the first type of outgoing real-time media stream to transmitting a second type of outgoing real-time media stream, where the first type is different from the second type; and in response to receiving the user input for switching, switching from transmitting the first type of outgoing real-time media stream to transmitting the second type of outgoing real-time media stream, including: transmitting real-time video according to a first type of real-time media stream that includes real-time audio and does not include real-time video; and stopping transmitting real-time video according to a first type of real-time media stream that includes real-time audio and real-time video.

[0017] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for: when transmitting a first type of outgoing real-time media stream to one or more participants of a communication session: outputting at least one or more incoming real-time media streams at the electronic device; receiving a user input for switching from transmitting the first type of outgoing real-time media stream to transmitting a second type of outgoing real-time media stream, where the first type is different from the second type; and in response to receiving the user input for switching, means for switching from transmitting the first type of outgoing real-time media stream to transmitting the second type of outgoing real-time media stream, the switching including: transmitting real-time video according to a first type of real-time media stream that includes real-time audio and does not include real-time video; and stopping transmitting real-time video according to a first type of real-time media stream that includes real-time audio and real-time video.

[0018] According to some embodiments, a method is performed at an electronic device having a display. The method includes: when participating in a communication session with one or more participants: transmitting a link to media to one or more participants of the communication session; displaying a representation of the media on the display, including an affordance representation; detecting a user input; in response to detecting the user input: displaying a media playback user interface corresponding to the media; when the media playback user interface is displayed, detecting a media playback control user input; in response to receiving the media playback control user input: controlling playback of the media in the media playback user interface at the electronic device according to the received media playback control user input; and according to the user input being an activation of the affordance representation: transmitting instructions for controlling playback of the media at the devices of the one or more participants according to the received media playback control user input, such that playback of the media at the devices of the one or more participants of the communication session remains substantially synchronized.

[0019] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for the following operations: when participating in a communication session with one or more participants: transmitting a link to media to one or more participants of the communication session; displaying a representation of the media on the display, including an affordance representation; detecting a user input; in response to detecting the user input: displaying a media playback user interface corresponding to the media; when the media playback user interface is displayed, detecting a media playback control user input; in response to receiving the media playback control user input: controlling the playback of the media in the media playback user interface at the electronic device according to the received media playback control user input; and according to the activation of the user input being an affordance: transmitting instructions for controlling the playback of the media at the devices of the one or more participants according to the received media playback control user input, such that the playback of the media at the devices of the one or more participants of the communication session remains substantially synchronized.

[0020] According to some embodiments, a transitory computer-readable storage medium is described. The transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for the following operations: when participating in a communication session with one or more participants: transmitting a link to media to one or more participants of the communication session; displaying a representation of the media on the display, including an affordance representation; detecting a user input; in response to detecting the user input: displaying a media playback user interface corresponding to the media; when the media playback user interface is displayed, detecting a media playback control user input; in response to receiving the media playback control user input: controlling the playback of the media in the media playback user interface at the electronic device according to the received media playback control user input; and according to the activation of the user input being an affordance: transmitting instructions for controlling the playback of the media at the devices of the one or more participants according to the received media playback control user input, such that the playback of the media at the devices of the one or more participants of the communication session remains substantially synchronized.

[0021] According to some embodiments, an electronic device is described. The electronic device includes: a display; one or more processors; and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: when participating in a communication session with one or more participants: transmitting a link to media to one or more participants of the communication session; displaying a representation of the media on the display, including an affordance representation; detecting a user input; in response to detecting the user input: displaying a media playback user interface corresponding to the media; when the media playback user interface is displayed, detecting a media playback control user input; in response to receiving the media playback control user input: controlling playback of the media in the media playback user interface at the electronic device according to the received media playback control user input; and in accordance with activation of the user input being an affordance: transmitting, according to the received media playback control user input, instructions for controlling playback of the media at the devices of the one or more participants such that playback of the media at the devices of the one or more participants of the communication session remains substantially synchronized.

[0022] According to some embodiments, an electronic device is described. The electronic device includes: a display; means for: when participating in a communication session with one or more participants: transmitting a link to media to one or more participants of the communication session; displaying a representation of the media on the display, including an affordance representation; detecting a user input; in response to detecting the user input: displaying a media playback user interface corresponding to the media; when the media playback user interface is displayed, detecting a media playback control user input; in response to receiving the media playback control user input: controlling playback of the media in the media playback user interface at the electronic device according to the received media playback control user input; and in accordance with activation of the user input being an affordance: transmitting, according to the received media playback control user input, instructions for controlling playback of the media at the devices of the one or more participants such that playback of the media at the devices of the one or more participants of the communication session remains substantially synchronized.

[0023] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. The executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0024] Accordingly, a faster and more efficient method and interface for managing communication sessions are provided for the device, thereby enhancing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace other methods for managing communication sessions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals indicate corresponding parts in all the figures.

[0026] Figure 1A is a block diagram showing a portable multifunctional device having a touch-sensitive display in accordance with some embodiments.

[0027] Figure 1B is a block diagram showing exemplary components for event handling in accordance with some embodiments.

[0028] Figure 2 shows a portable multifunctional device having a touch screen in accordance with some embodiments.

[0029] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface in accordance with some embodiments.

[0030] Figure 4A shows an exemplary user interface for a menu of an application on a portable multifunctional device in accordance with some embodiments.

[0031] Figure 4B shows an exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display in accordance with some embodiments.

[0032] Figure 5A shows a personal electronic device in accordance with some embodiments.

[0033] Figure 5B is a block diagram showing a personal electronic device in accordance with some embodiments.

[0034] Figures 5C - 5D shows exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor in accordance with some embodiments.

[0035] Figures 5E - 5H shows exemplary components and a user interface of a personal electronic device in accordance with some embodiments.

[0036] Figures 6A to 6AF shows an exemplary device and user interface for managing a communication session.

[0037] Figures 7A to 7B is a flowchart showing a method for initiating a live stream.

[0038] Figure 8 is a flowchart showing a method for switching between streaming live audio and live video.

[0039] Figures 9A to 9B is a flowchart showing a method for synchronizing media playback during a live stream. DETAILED DESCRIPTION

[0040] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to limit the scope of the present disclosure, but rather is provided as a description of exemplary embodiments.

[0041] An electronic device needs to provide effective methods and interfaces for managing communication sessions. For example, an effective method and interface for initiating a live stream are needed. Also, an effective method and interface for switching between streaming live audio and live video are needed. Also, an effective method and interface for synchronizing media playback during a live stream are needed. Such techniques can reduce the cognitive burden on a user managing a communication session, thereby increasing productivity. Additionally, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user input.

[0042] Below Figures 1A to 1B , Figure 2 , Figure 3 , Figures 4A to 4B and Figures 5A to 5H provides a description of an exemplary device for performing techniques for managing event notifications. Figures 6A to 6AF shows an exemplary device and user interface for managing a communication session. Figures 7A to 7B is a flowchart showing a method for initiating a live stream according to some embodiments. Figure 8 is a flowchart showing a method for switching between streaming live audio and live video according to some embodiments. Figures 9A to 9B is a flowchart showing a method for synchronizing media playback during a live stream according to some embodiments. Figures 6A to 6AF The user interface in Figures 7A to 7B , Figure 8 and Figures 9A to 9B is used to show the processes described below, including the processes in

[0043] Although the following description uses terms such as "first" and "second" to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch could be named a second touch and similarly a second touch could be named a first touch without departing from the scope of the various described embodiments. Both the first touch and the second touch are touches, but they are not the same touch.

[0044] The terms used in the description of the various embodiments herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "comprises", "comprising", "includes", and / or "including" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] Depending on the context, the term "if" is optionally interpreted to mean "when", "upon", or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrases "if determined..." or "if [stated condition or event] is detected" are optionally interpreted to mean "when determining...", "in response to determining...", "when [stated condition or event] is detected", or "in response to detecting [stated condition or event]".

[0046] Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions such as PDA and / or music player functions. Exemplary embodiments of portable multifunctional devices include, but are not limited to, devices from Apple Inc. (Cupertino, California), devices, iPods devices, and devices. Other portable electronic devices, such as laptop computers or tablet computers having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad), may optionally be used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0047] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0048] The device generally supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, gaming applications, telephone applications, video conferencing applications, email applications, instant messaging applications, fitness support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0049] The various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or within the respective applications. Thus, a common physical architecture of the device (such as a touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0050] Attention is now turned to an embodiment of a portable device having a touch-sensitive display. Figure 1A FIG. is a block diagram of a portable multifunctional device 100 having a touch screen display 112 in accordance with some embodiments. The touch screen display 112 is sometimes called a "touch screen" for convenience, and is sometimes referred to as or called a "touch-sensitive display system". The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of a contact on the device 100 (e.g., a touch-sensitive surface, such as the touch screen display 112 of the device 100). The device 100 optionally includes one or more tactile output generators 167 for generating a tactile output on the device 100 (e.g., generating a tactile output on a touch-sensitive surface, such as the touch screen display 112 of the device 100 or the touchpad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0051] As used in this specification and the claims, the “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a surrogate for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of methods and a variety of sensors or combinations of sensors. For example, one or more force sensors beneath or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine the estimated contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in size of the contact area detected on the touch-sensitive surface, the capacitance and / or change in capacitance of the touch-sensitive surface near the contact, and / or the resistance and / or change in resistance of the touch-sensitive surface near the contact are optionally used as surrogates for the force or pressure of a contact on the touch-sensitive surface. In some embodiments, the surrogate measurements of contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurements). In some embodiments, the surrogate measurements of contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of a user input allows a user to access additional device functions that would otherwise be inaccessible to the user on a smaller device with a limited physical area in some cases, the smaller device being used to (e.g., on a touch-sensitive display) display affordances and / or receive user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0052] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of a device relative to a previous position of the device detected by a user using the user's sense of touch, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device. For example, in the case of contact between the device or a component of the device and a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a sense of touch that corresponds to a perceived change in the physical characteristics of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or a touchpad) is optionally interpreted by the user as a "press click" or "release click" of a physical actuation button. In some cases, the user will feel a sense of touch, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface will optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such interpretations of touch by the user will be limited by the user's individual sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or a component thereof that would generate the stated sensory perception of a typical (or average) user.

[0053] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of these components. Figure 1A The various components shown are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0054] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0055] The peripheral device interface 118 can be used to couple the input and output peripheral devices of the device to the CPU 120 and the memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, the peripheral device interface 118, the CPU 120, and the memory controller 122 are optionally implemented on a single chip such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0056] The RF (Radio Frequency) circuit 108 receives and transmits RF signals, which are also referred to as electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with the communication network and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 108 optionally communicates with the network and other devices via wireless communication, and these networks are such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular phone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The RF circuit 108 optionally includes well-known circuits for detecting a near field communication (NFC) field, such as via a short-range communication radio component. The wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), WiMAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the date of submission of this document.

[0057] The audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuitry 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves audible to humans. The audio circuitry 110 also receives the electrical signal converted from sound waves by the microphone 113. The audio circuitry 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuitry 108 by the peripheral interface 118. In some embodiments, the audio circuitry 110 also includes an earphone jack (e.g., Figure 2 212 in

[0058] ). The earphone jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or an earphone having both an output (e.g., a mono or stereo earphone) and an input (e.g., a microphone). Figure 2 208 in Figure 2 ). The one or more buttons optionally include up / down buttons for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., 206 in

[0059] ).Pressing the depress button quickly optionally disengages the lock of the touch screen display 112 or optionally starts the process of unlocking the device by using gestures on the touch screen, as described in U.S. Patent Application No. 11 / 322,549, filed on December 23, 2005, entitled "Unlocking a Device by Performing Gestures on an Unlock Image" (i.e., U.S. Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. Pressing the depress button for a long time (e.g., 206) optionally powers on or powers off the device 100. The functions of one or more buttons are optionally customizable by the user. The touch screen display 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

[0060] The touch screen display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch screen display 112 and / or sends electrical signals to the touch screen display 112. The touch screen display 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0061] The touch screen display 112 has a touch-sensitive surface, sensor, or group of sensors that accepts input from the user based on tactile and / or haptic contact. The touch screen display 112 and the display controller 156 (along with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of the contact) on the touch screen display 112 and convert the detected contact into an interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch screen display 112. In an exemplary embodiment, the point of contact between the touch screen display 112 and the user corresponds to the user's finger.

[0062] The touch screen display 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, but in other embodiments, other display technologies are used. The touch screen display 112 and the display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen display 112 to detect contact and any movement or interruption thereof, the variety of touch sensing technologies including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the technology used in and iPod as used in.

[0063] The touch-sensitive display in some embodiments of the touch screen display 112 optionally resembles the multi-touch sensitive touchpad described in the following U.S. patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman et al.) and / or U.S. Patent Publication 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, the touch screen display 112 displays the visual output from the device 100, while the touch-sensitive touchpad does not provide a visual output.

[0064] In some embodiments, the touch-sensitive display of the touchscreen display 112 is as described in the following patent applications: (1) U.S. Patent Application No. 11 / 381,313, entitled "Multipoint Touch Surface Controller," filed on May 2, 2006; (2) U.S. Patent Application No. 10 / 840,862, entitled "Multipoint Touchscreen," filed on May 6, 2004; (3) U.S. Patent Application No. 10 / 903,964, entitled "Gestures For Touch Sensitive Input Devices," filed on July 30, 2004; (4) U.S. Patent Application No. 11 / 048,264, entitled "Gestures For Touch Sensitive Input Devices," filed on January 31, 2005; (5) U.S. Patent Application No. 11 / 038,590, entitled "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," filed on January 18, 2005; (6) U.S. Patent Application No. 11 / 228,758, entitled "Virtual Input Device Placement On A Touch Screen User Interface," filed on September 16, 2005; (7) U.S. Patent Application No. 11 / 228,700, entitled "Operation Of A Computer With A Touch Screen Interface," filed on September 16, 2005; (8) U.S. Patent Application No. 11 / 228,737, entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed on September 16, 2005; and (9) U.S. Patent Application No. 11 / 367,749, entitled "Multi-Functional Hand-Held Device," filed on March 3, 2006. All of these applications are hereby incorporated by reference in their entirety.

[0065] The touchscreen display 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touchscreen display 112. In some embodiments, the user interface is designed to work primarily through finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of the finger on the touchscreen. In some embodiments, the device converts the finger-based rough input into an accurate pointer / cursor position or command for performing the action desired by the user.

[0066] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device, which, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen display 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0067] The device 100 also includes a power system 162 for powering various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0068] The device 100 optionally also includes one or more optical sensors 164. Figure 1AAn optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. The optical sensor 164 optionally includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representative of an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite the touch screen display 112 on the front of the device, such that the touch screen display can be used as a viewfinder for still image and / or video image capture. In some embodiments, the optical sensor is located on the front of the device such that an image of the user can optionally be obtained for video conferencing while the user views other video conferencing participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (e.g., by rotating the lens and sensor in the device housing) such that a single optical sensor 164 can be used with the touch screen display for both video conferencing and still image and / or video image capture.

[0069] The device 100 optionally further includes one or more contact intensity sensors 165. Figure 1A A contact intensity sensor coupled to an intensity sensor controller 159 in the I / O subsystem 106 is shown. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., the touch screen display 112). In some embodiments, at least one contact intensity sensor is located on the rear of the device 100, opposite the touch screen display 112 located on the front of the device 100.

[0070] The device 100 optionally further includes one or more proximity sensors 166. Figure 1AA proximity sensor 166 is shown coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is optionally coupled to an input controller 160 in the I / O subsystem 106. The proximity sensor 166 optionally operates as described in the following U.S. patent applications: No. 11 / 241,839, titled "Proximity Detector In Handheld Device"; No. 11 / 240,788, titled "Proximity Detector In Handheld Device"; No. 11 / 620,702, titled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11 / 586,862, titled "Automated Response To And Sensing Of User Activity In Portable Devices"; and No. 11 / 638,251, titled "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call), the proximity sensor turns off and disables the touch screen display 112.

[0071] Device 100 optionally further includes one or more haptic output generators 167. Figure 1AIllustrated is a haptic output generator coupled to the haptic feedback controller 161 in the I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting an electrical signal into a haptic output on the device). The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., the touchscreen display 112), and optionally generates a haptic output by moving the touch-sensitive surface vertically (e.g., into / out of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of the device 100, opposite the touchscreen display 112 located on the front of the device 100.

[0072] The device 100 optionally further includes one or more accelerometers 168. Figure 1A Illustrated is an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally operates as described in the following U.S. Patent Publications: U.S. Patent Publication No. 20050190059, titled "Acceleration-based Theft Detection System for Portable Electronic Devices" and U.S. Patent Publication No. 20060017692, titled "Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer", both of which are hereby incorporated by reference in their entireties. In some embodiments, information is displayed in a portrait view or a landscape view on the touchscreen display based on an analysis of data received from one or more accelerometers. The device 100 optionally further includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to one or more accelerometers 168 for obtaining information about the location and orientation (e.g., portrait or landscape) of the device 100.

[0073] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and application programs (or instruction sets) 136. Additionally, in some embodiments, the memory 102 ( Figure 1A ) or 370 ( Figure 3 ) stores a device / global internal state 157, as shown in Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state that indicates which applications (if any) are currently active; a display state that indicates what applications, views, or other information occupy the various regions of the touchscreen display 112; a sensor state that includes information obtained from the various sensors of the device and the input control device 116; and location information regarding the location and / or orientation of the device.

[0074] The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between the various hardware components and software components.

[0075] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted to be directly coupled to other devices or indirectly coupled through a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is the same as or similar to and / or compatible with the 30-pin connector used on (a trademark of Apple Inc.) devices and is a multi-pin (e.g., 30-pin) connector.

[0076] The contact / motion module 130 optionally detects contact with the touch screen display 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or a physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection, such as determining whether contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of the contact point being represented by a series of contact data. These operations are optionally applied to single-point contact (e.g., single-finger contact) or multi-point simultaneous contact (e.g., "multi-touch" / multiple finger contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0077] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., determining whether the user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds is determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of the touchpad or touch screen can be set to any one of a wide range of predefined thresholds without changing the touchpad or touch screen display hardware. Additionally, in some implementations, software settings are provided to the user of the device for adjusting one or more of the intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click on an "intensity" parameter).

[0078] The contact / motion module 130 optionally detects gesture inputs made by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and subsequently detecting a finger lift (lift-off) event.

[0079] The graphics module 132 includes various known software components for presenting and displaying graphics on the touchscreen display 112 or other displays, including components for altering the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual attributes). As used herein, the term "graphics" includes any object that can be displayed to the user, which non-limitingly includes text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0080] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application, etc., and also receives coordinate data and other graphic attribute data as necessary, and then generates screen image data for output to the display controller 156.

[0081] The haptic feedback module 133 includes various software components for generating instructions that are used by the haptic output generator 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0082] The text input module 134, which is optionally a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0083] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., provided to the phone 138 for location-based dialing; provided to the camera 143 as picture / video metadata; and provided to applications that provide location-based services, such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0084] The application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0085] · A contacts module 137 (sometimes referred to as an address book or contact list);

[0086] · A phone module 138;

[0087] · A video conferencing module 139;

[0088] · An email client module 140;

[0089] · An instant messaging (IM) module 141;

[0090] · A fitness support module 142;

[0091] · A camera module 143 for still images and / or video images;

[0092] · An image management module 144;

[0093] · A video player module;

[0094] · A music player module;

[0095] · A browser module 147;

[0096] · A calendar module 148;

[0097] · A desktop widget module 149, which optionally includes one or more of the following: a weather desktop widget 149-1, a stock market desktop widget 149-2, a calculator desktop widget 149-3, an alarm clock desktop widget 149-4, a dictionary desktop widget 149-5, and other desktop widgets obtained by the user, as well as a user-created desktop widget 149-6;

[0098] · A desktop widget creator module 150 for forming the user-created desktop widget 149-6;

[0099] · A search module 151;

[0100] · A video and music player module 152, which combines the video player module and the music player module;

[0101] · A notepad module 153;

[0102] · A map module 154; and / or

[0103] · An online video module 155.

[0104] Examples of other application programs 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, speech recognition, and speech reproduction.

[0105] In conjunction with the touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the contacts module 137 is optionally used to manage an address book or contact list (e.g., in the application internal state 192 of the contacts module 137 stored in the memory 102 or memory 370), including: adding one or more names to the address book; deleting names from the address book; associating a phone number, email address, physical address, or other information with a name; associating an image with a name; categorizing and classifying names; providing a phone number or email address to initiate and / or facilitate communication via the phone 138, video conferencing module 139, email 140, or IM 141; and so on.

[0106] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the phone module 138 is optionally used to input a character sequence corresponding to a phone number, access one or more phone numbers in the contacts module 137, modify the entered phone number, dial the corresponding phone number, conduct a session, and disconnect or hang up when the session is completed. As described above, wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies.

[0107] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen display 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contacts module 137, and phone module 138, the video conferencing module 139 includes executable instructions to initiate, execute, and terminate a video conference between the user and one or more other participants according to user instructions.

[0108] In conjunction with the RF circuit 108, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, the email client module 140 includes executable instructions to create, send, receive, and manage emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with static or video images captured by the camera module 143.

[0109] In combination with the RF circuitry 108, the touchscreen display 112, the display controller 156, the touch / motion module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for the following operations: inputting a character sequence corresponding to an instant message, modifying a previously input character, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0110] In combination with the RF circuitry 108, the touchscreen display 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the music player module, the fitness support module 142 includes executable instructions for creating a fitness (e.g., having time, distance, and / or calorie burn goals); communicating with fitness sensors (exercise equipment); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for the fitness; and displaying, storing, and transmitting fitness data.

[0111] In combination with the touchscreen display 112, the display controller 156, the optical sensor 164, the optical sensor controller 158, the touch / motion module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for the following operations: capturing still images or video (including video streams) and storing them in the memory 102, modifying the characteristics of still images or video, or deleting still images or video from the memory 102.

[0112] In combination with the touchscreen display 112, the display controller 156, the touch / motion module 130, the graphics module 132, the text input module 134, and the camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0113] In combination with RF circuit 108, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions for browsing the Internet according to user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, and linking to attachments and other files of web pages.

[0114] In combination with RF circuit 108, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and data associated with the calendars (e.g., calendar entries, to-do items, etc.) according to user instructions.

[0115] In combination with RF circuit 108, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop widget module 149 is a mini - application (e.g., weather desktop widget 149 - 1, stock market desktop widget 149 - 2, calculator desktop widget 149 - 3, alarm clock desktop widget 149 - 4, and dictionary desktop widget 149 - 5) optionally downloaded and used by the user or a mini - application created by the user (e.g., user - created desktop widget 149 - 6). In some embodiments, desktop widgets include HTML (HyperText Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, desktop widgets include XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop widgets).

[0116] In combination with RF circuit 108, touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147, desktop widget creator module 150 is optionally used by the user to create desktop widgets (e.g., converting a user - specified portion of a web page into a desktop widget).

[0117] In combination with touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions for searching in memory 102 for text, music, sound, images, video, and / or other files that match one or more search criteria (e.g., one or more user - specified search terms) according to user instructions.

[0118] In combination with touch screen display 112, display controller 156, touch / motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch screen display 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0119] In combination with touch screen display 112, display controller 156, touch / motion module 130, graphics module 132, and text input module 134, notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, and the like according to user instructions.

[0120] In combination with RF circuitry 108, touch screen display 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data related to stores and other points of interest at or near a particular location, and other location-based data) according to user instructions.

[0121] In combination with touch screen display 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, e-mail client module 140, and browser module 147, online video module 155 includes instructions for performing the following operations: allowing a user to access, browse, receive (e.g., via streaming and / or downloading), play back (e.g., on the touch screen or on an external display connected via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, the instant message module 141 is used instead of the e-mail client module 140 to send a link to a particular online video. Other descriptions of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed Jun. 20, 2007, and titled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” and U.S. Patent Application No. 11 / 968,067, filed Dec. 31, 2007, and titled “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” the contents of both of which are hereby incorporated by reference in their entireties.

[0122] Each of the above-described modules and applications corresponds to a set of executable instructions for performing one or more of the above-described functions and the methods described in this patent application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., the instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, the video player module is optionally combined with the music player module into a single module (e.g., Figure 1A the video and music player module 152 in ). In some embodiments, memory 102 optionally stores a subgroup of the above-described modules and data structures. In addition, memory 102 optionally stores additional modules and data structures not described above.

[0123] In some embodiments, device 100 is a device in which the operation of a predefined set of functions on the device is performed exclusively via a touch screen and / or a touchpad. By using the touch screen and / or the touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0124] The predefined set of functions performed exclusively via the touch screen and / or the touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 from any user interface displayed on device 100 to a main menu, a home menu, or a root menu. In such embodiments, the touchpad is used to implement a "menu button". In some other embodiments, the menu button is a physical push button or other physical input control device rather than the touchpad.

[0125] Figure 1B is a block diagram showing exemplary components for event handling according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in operating system 126) and corresponding application 136-1 (e.g., any one of the foregoing applications 137 to 151, 155, 380 to 390).

[0126] Event classifier 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of application 136-1. Event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on touch screen display 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is / are currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0127] In some embodiments, application internal state 192 includes additional information such as one or more of the following: recovery information to be used when application 136-1 resumes execution, user interface state information indicating information being displayed by or ready for display by application 136-1, a state queue for enabling the user to return to a previous state or view of application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0128] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (e.g., a user touch on the touch screen display 112, as part of a multi-touch gesture). The peripheral interface 118 transmits information that it receives from the I / O subsystem 106 or sensors such as the proximity sensor 166, one or more accelerometers 168, and / or the microphone 113 (via the audio circuitry 110). The information that the peripheral interface 118 receives from the I / O subsystem 106 includes information from the touch screen display 112 or a touch-sensitive surface.

[0129] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (e.g., receiving above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0130] In some embodiments, the event classifier 170 further includes a hit view determination module 172 and / or an active event recognizer determination module 173.

[0131] When the touch screen display 112 shows more than one view, the hit view determination module 172 provides a software process for determining where within one or more of the views the sub-event has occurred. A view is composed of controls and other elements that a user can see on the display.

[0132] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, within which information is displayed and touch-based gestures occur. The application view (of the corresponding application) within which a touch is detected optionally corresponds to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest-level view within which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct inputs is optionally determined at least in part based on the hit view of the initial touch that begins the touch-based gesture.

[0133] The hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchical structure, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchical structure that should handle the sub-events. In most cases, the hit view is the lowest-level view in which the initiating sub-event (e.g., the first sub-event in a sequence of sub-events that form an event or a potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view generally receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0134] The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views and thus determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with a particular view, higher views in the hierarchy will still remain as actively participating views.

[0135] The event dispatcher module 174 distributes event information to event recognizers (e.g., event recognizer 180). In embodiments that include the active event recognizer determination module 173, the event dispatcher module 174 delivers the event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, the event dispatcher module 174 stores the event information in an event queue, which is retrieved by the corresponding event receiver 182.

[0136] In some embodiments, the operating system 126 includes the event classifier 170. Alternatively, the application 136-1 includes the event classifier 170. In yet another embodiment, the event classifier 170 is an independent module or is part of another module (such as the contact / motion module 130) stored in the memory 102.

[0137] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of an independent module that is a higher-level object such as a user interface toolkit (not shown) or from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handlers 190 include one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. The event handlers 190 optionally utilize or call the data updater 176, object updater 177, or GUI updater 178 to update the internal state 192 of the application. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0138] The corresponding event recognizer 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event based on the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0139] The event receiver 182 receives event information from event classifier 170. The event information includes information about sub-events such as a touch or a touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device pose).

[0140] The event comparator 184 compares the event information with predefined event or sub - event definitions and, based on the comparison, determines an event or sub - event or determines or updates the status of an event or sub - event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 contains definitions of events (e.g., predefined sequences of sub - events), such as event 1 (187 - 1), event 2 (187 - 2), and others. In some embodiments, the sub - events within an event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi - touch. In one example, the definition of event 1 (187 - 1) is a double - tap on a displayed object. For example, a double - tap includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift - off (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift - off (touch end) of a predetermined duration. In another example, the definition of event 2 (187 - 2) is a drag on a displayed object. For example, a drag includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch - screen display 112, and lift - off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0141] In some embodiments, the event definition 187 includes a definition of an event for a corresponding user - interface object. In some embodiments, the event comparator 184 performs a hit test to determine which user - interface object is associated with a sub - event. For example, in an application view that displays three user - interface objects on the touch - screen display 112, when a touch is detected on the touch - screen display 112, the event comparator 184 performs a hit test to determine which of the three user - interface objects is associated with the touch (sub - event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, the event comparator 184 selects the event handler associated with the sub - event and the object that triggered the hit test.

[0142] In some embodiments, the definition of a corresponding event (187) also includes a delay action that delays the delivery of event information until it has been determined that the sub - event sequence does or does not correspond to the event type of the event recognizer.

[0143] When the corresponding event recognizer 180 determines that the sub - event sequence does not match any event in the event definition 186, the corresponding event recognizer 180 enters an event - impossible, event - failed, or event - ended state, after which subsequent sub - events of the touch - based gesture are ignored. In such a case, other event recognizers (if any) that remain active for the hit view continue to track and process sub - events of the ongoing touch - based gesture.

[0144] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists that indicate how the event delivery system should perform sub - event delivery to the event recognizers actively participating. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists that indicate how event recognizers interact with each other or can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists that indicate whether sub - events are delivered to different levels in the view or the programmatic hierarchy.

[0145] In some embodiments, when one or more specific sub - events of an event are recognized, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferring sending) sub - events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a token associated with the recognized event, and the event handler 190 associated with that token retrieves the token and executes a predefined process.

[0146] In some embodiments, the event delivery instruction 188 includes a sub - event delivery instruction that delivers event information about the sub - event without activating the event handler. Instead, the sub - event delivery instruction delivers the event information to the event handler associated with the sub - event sequence or to the actively participating view. The event handler associated with the sub - event sequence or with the actively participating view receives the event information and executes a predetermined process.

[0147] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137, or stores video files used in a video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.

[0148] In some embodiments, event handler 190 includes data updater 176, object updater 177, and GUI updater 178, or has access to the data updater, the object updater, and the GUI updater. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0149] It should be understood that the foregoing discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input for operating multifunctional device 100 using an input device, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses optionally in cooperation with single or multiple keyboard presses or holds; contact movement on a touchpad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal instructions; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events that define an event to be recognized.

[0150] Figure 2FIG. 0 shows a portable multifunctional device 100 having a touch screen display 112, in accordance with some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this and other embodiments described below, the user is able to select one or more of these graphics by making gestures on the graphics, such as by using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of one or more of the graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gestures optionally include one or more taps, one or more swipes (from left to right, right to left, up, and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, left to right, up, and / or down). In some implementations or in some cases, inadvertently contacting a graphic does not select the graphic. For example, when the gesture corresponding to selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0151] The device 100 optionally further includes one or more physical buttons, such as a “home” or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any of a set of applications optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display 112.

[0152] In some embodiments, the device 100 includes a touch screen display 112, a menu button 204, a depressible button 206 for powering on / off the device and for locking the device, one or more volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The depressible button 206 is optionally used to power on / off the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and / or to unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also receives voice input for activating or deactivating certain functions via a microphone 113. The device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch screen display 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0153] Figure 3FIG. 0 is a block diagram of an exemplary multi-functional device having a display and a touch-sensitive surface, in accordance with some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a media player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home or industrial controller). Device 300 generally includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects the system components and controls the communication between them. Device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touchpad 355, a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to haptic output generator 167 described above with reference to Figure 1A ), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors (similar to contact intensity sensor 165 described above with reference to Figure 1A ). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to or a subset of those stored in memory 102 of portable multi-functional device 100 ( Figure 1A ). Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multi-functional device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while memory 102 of portable multi-functional device 100 ( Figure 1A ) optionally does not store these modules.

[0154] Figure 3Each of the above elements in [element name] is optionally stored in one or more of the previously mentioned memory devices of the memory device. Each of the above modules corresponds to an instruction set for performing the above functions. The above modules or programs (e.g., instruction sets) do not have to be implemented as separate software programs, processes, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, the memory 370 optionally stores a subgroup of the above modules and data structures. Additionally, the memory 370 optionally stores additional modules and data structures not described above.

[0155] Attention is now turned to embodiments of a user interface optionally implemented on, for example, the portable multifunctional device 100.

[0156] Figure 4A An exemplary user interface of an application menu on the portable multifunctional device 100 according to some embodiments is shown. A similar user interface is optionally implemented on the device 300. In some embodiments, the user interface 400 includes the following elements or a subset or superset thereof:

[0157] · One or more signal strength indicators 402 of one or more wireless communications such as cellular signals and Wi-Fi signals;

[0158] · Time 404;

[0159] · Bluetooth indicator 405;

[0160] · Battery status indicator 406;

[0161] · A tray 408 with icons for commonly used applications, such as:

[0162] ○ An icon 416 marked "Phone" for the phone module 138, which icon 416 optionally includes an indicator 414 of the number of missed calls or voicemails;

[0163] ○ An icon 418 marked "Mail" for the email client module 140, which icon 418 optionally includes an indicator 410 of the number of unread emails;

[0164] ○ An icon 420 marked "Browser" for the browser module 147; and

[0165] ○ An icon 422 marked "iPod" for the video and music player module 152 (also known as the iPod (trademark of Apple Inc.) module 152); and

[0166] · Icons for other applications, such as:

[0167] ○ The icon 424 of the IM module 141 labeled "Message";

[0168] ○ The icon 426 of the calendar module 148 labeled "Calendar";

[0169] ○ The icon 428 of the image management module 144 labeled "Photo";

[0170] ○ The icon 430 of the camera module 143 labeled "Camera";

[0171] ○ The icon 432 of the online video module 155 labeled "Online Video";

[0172] ○ The icon 434 of the stock market desktop applet 149-2 labeled "Stock Market";

[0173] ○ The icon 436 of the map module 154 labeled "Map";

[0174] ○ The icon 438 of the weather desktop applet 149-1 labeled "Weather";

[0175] ○ The icon 440 of the alarm clock desktop applet 149-4 labeled "Clock";

[0176] ○ The icon 442 of the fitness support module 142 labeled "Fitness Support";

[0177] ○ The icon 444 of the notepad module 153 labeled "Notepad"; and

[0178] ○ The icon 446 of the settings application or module labeled "Settings", which provides access to the settings of the device 100 and its various applications 136.

[0179] It should be noted that Figure 4A The icon labels shown are merely exemplary. For example, the icon 422 of the video and music player module 152 is labeled "Music" or "Music Player". Other labels may be optionally used for the various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0180] Figure 4B A device (e.g., Figure 3 is shown having a touch-sensitive surface 451 (e.g., Figure 3Exemplary user interface on device 300). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contacts on the touch-sensitive surface 451 and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0181] Although some examples below will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as Figure 4B shown. In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451 in ) has a main axis corresponding to the main axis (e.g., Figure 4B 453 in ) on the display (e.g., Figure 4B 452 in ). According to these embodiments, the device detects contact with the touch-sensitive surface 451 at a position corresponding to a corresponding position on the display (e.g., in Figure 4B , 460 corresponds to 468 and 462 corresponds to 470) (e.g., Figure 4B 460 and 462 in ). Thus, when the touch-sensitive surface (e.g., Figure 4B 451 in ) is separate from the display of the multifunctional device (e.g., Figure 4B 450 in ), user input detected by the device on that touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0182] Additionally, while the examples below are mainly given with reference to finger input (e.g., finger contact, single-finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of a contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is above the position of the tap gesture (e.g., instead of detecting a contact, followed by stopping detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and a finger contact are optionally used simultaneously.

[0183] Figure 5AAn exemplary personal electronic device 500 is shown. The device 500 includes a body 502. In some embodiments, the device 500 may include some or all of the features described with respect to devices 100 and 300 (e.g., Figures 1A to 4B ). In some embodiments, the device 500 has a touch-sensitive display screen 504 hereinafter referred to as a touch screen 504. As an alternative or addition to the touch screen 504, the device 500 has a display and a touch-sensitive surface. As in the case of devices 100 and 300, in some embodiments, the touch screen 504 (or the touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of an applied contact (e.g., a touch). One or more intensity sensors of the touch screen 504 (or the touch-sensitive surface) may provide output data representative of the intensity of the touch. The user interface of the device 500 may respond to the touch based on the intensity of the touch, meaning that touches of different intensities may invoke different user interface operations on the device 500.

[0184] Exemplary techniques for detecting and processing touch intensity are found, for example, in the following related patent applications: International Patent Application Serial No. PCT / US2013 / 040061, filed May 8, 2013, entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application," published as WIPO Patent Publication No. WO / 2013 / 169849; and International Patent Application Serial No. PCT / US2013 / 069483, filed November 11, 2013, entitled "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships," published as WIPO Patent Publication No. WO / 2014 / 105276, each of which is hereby incorporated by reference in its entirety.

[0185] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508 (if included) can be in physical form. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms (if included) may allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watchbands, bracelets, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear device 500.

[0186] Figure 5B An exemplary personal electronic device 500 is shown. In some embodiments, device 500 may include some or all of the components referred to Figure 1A , Figure 1B and Figure 3 above. Device 500 has a bus 512 that operably couples the I / O section 514 to one or more computer processors 516 and a memory 518. The I / O section 514 may be connected to a display 504 that may have a touch-sensitive component 522 and optionally a force sensor 524 (e.g., a contact force sensor). Additionally, the I / O section 514 may be connected to a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, near field communication (NFC), cellular, and / or other wireless communication technologies. Device 500 may include an input mechanism 506 and / or 508. For example, input mechanism 506 is optionally a rotatable input device or a pressable input device and a rotatable input device. In some examples, input mechanism 508 is optionally a button.

[0187] In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as a GPS sensor 532, an accelerometer 534, an orientation sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or a combination thereof, all of which are operably connected to the I / O section 514.

[0188] The memory 518 of personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, may cause the computer processors to perform the techniques described below, including processes 700 through 900 ( Figures 7A to 7B , Figure 8 and Figures 9A to 9B)。A computer-readable storage medium can be any medium that tangibly contains or stores computer-executable instructions for use by or in connection with an instruction execution system, apparatus, and device. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transient computer-readable storage medium. Non-transient computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical discs based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash memory, solid-state drives, and the like. Personal electronic device 500 is not limited to Figure 5B the components and configurations thereof, but may include other components or additional components in a variety of configurations.

[0189] As used herein, the term "presentation" refers to a user-interactive graphical user interface object optionally displayed on the display screen of devices 100, 300, and / or 500 ( Figure 1A 、 Figure 3 and Figures 5A to 5B ). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute a presentation.

[0190] As used herein, the term "focus selector" refers to an input element for indicating the current portion of the user interface with which the user is interacting. In some specific implementations including a cursor or other position marker, the cursor acts as the "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., Figure 3 the touchpad 355 in Figure 4B ) above a particular user interface element (e.g., a button, window, slider, or other user interface element), that particular user interface element is adjusted in accordance with the detected input. In some implementations including a touchscreen display capable of enabling direct interaction with user interface elements on the touchscreen display (e.g., Figure 1A the touchscreen display 112 in Figure 4AIn some specific embodiments of the touchscreen display 112), the detected contact on the touchscreen serves as a "focus selector" such that when an input (e.g., a press input made by the contact) is detected at the position of a specific user interface element (e.g., a button, a window, a slider, or other user interface element) on the touchscreen display, the specific user interface element is adjusted according to the detected input. In some specific embodiments, the focus moves from one area of the user interface to another area of the user interface without a corresponding movement of the cursor or a movement of the contact on the touchscreen display (e.g., moving the focus from one button to another button by using the tab key or arrow keys); in these specific embodiments, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on the touchscreen display) that is controlled by the user to deliver the interaction with the user interface expected by the user (e.g., by indicating to the device the element of the user interface that the user desires to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or a touchscreen), the position of the focus selector (e.g., a cursor, a contact, or a selection box) above the corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0191] As used in the specification and claims, the term "feature intensity" of a contact refers to a feature of the contact based on one or more intensities of the contact. In some embodiments, the feature intensity is based on a plurality of intensity samples. The feature intensity is optionally based on a predefined number or set of intensity samples collected during a predefined period of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after detecting a contact, before detecting a contact lift-off, before or after detecting a contact start to move, before detecting a contact end, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact). The feature intensity of a contact is optionally based on one or more of the following: the maximum value of the intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the value at the top 10% of the intensity of the contact, the half maximum value of the intensity of the contact, the 90% maximum value of the intensity of the contact, etc. In some embodiments, the duration of the contact is used in determining the feature intensity (e.g., when the feature intensity is the average value of the intensity of the contact over time). In some embodiments, the feature intensity is compared with a set of one or more intensity thresholds to determine whether the user has performed an operation. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, a contact whose feature intensity does not exceed the first threshold results in a first operation, a contact whose feature intensity exceeds the first intensity threshold but does not exceed the second intensity threshold results in a second operation, and a contact whose feature intensity exceeds the second threshold results in a third operation. In some embodiments, the comparison between the feature intensity and one or more thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding operation or to forgo performing the corresponding operation) rather than for determining whether to perform a first operation or a second operation.

[0192] Figure 5C Illustrated is the detection of a plurality of contacts 552A - 552E on a touch-sensitive display screen 504 using a plurality of intensity sensors 524A - 524D. Figure 5C Also included is an intensity map that shows the current intensity measurements of the intensity sensors 524A - 524D relative to intensity units. In this embodiment, the intensity measurements of both intensity sensors 524A and 524D are 9 intensity units, and the intensity measurements of both intensity sensors 524B and 524C are 7 intensity units. In some specific implementations, the cumulative intensity is the sum of the intensity measurements of the plurality of intensity sensors 524A - 524D, which is 32 intensity units in this embodiment. In some embodiments, each contact is assigned a corresponding intensity, i.e., a portion of the cumulative intensity. Figure 5DAssigns cumulative intensity to contacts 552A - 552E based on their distance from the center of force 554. In this embodiment, each of contacts 552A, 552B, and 552E is assigned an intensity of 8 intensity units of cumulative intensity, and each of contacts 552C and 552D is assigned an intensity of 4 intensity units of cumulative intensity. More generally, in some specific implementations, each contact j is assigned a corresponding intensity Ij according to a predefined mathematical function Ij = A·(Dj / ΣDi), which is a part of the cumulative intensity A, where Dj is the distance of the corresponding contact j from the center of force, and ΣDi is the sum of the distances of all corresponding contacts (e.g., i = 1 to the last) from the center of force. An electronic device similar to or equivalent to devices 100, 300, or 500 can perform the operations referred to Figures 5C - 5D above. In some embodiments, the characteristic intensity of a contact is based on one or more intensities of the contact. In some embodiments, an intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). It should be noted that the intensity map is not part of the displayed user interface, but is included in Figures 5C - 5D to assist the reader.

[0193] In some embodiments, a part of the recognized gesture is used to determine the characteristic intensity. For example, the touch-sensitive surface optionally receives a continuous swiping contact that transitions from a starting position and reaches an ending position where the contact intensity increases. In this example, the characteristic intensity of the contact at the ending position is optionally based on only a part of the continuous swiping contact, rather than the entire swiping contact (e.g., only the part of the swiping contact at the ending position). In some embodiments, a smoothing algorithm is optionally applied to the intensity of the swiping contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: unweighted moving average smoothing algorithm, triangular smoothing algorithm, median filter smoothing algorithm, and / or exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swiping contact for the purpose of determining the characteristic intensity.

[0194] Optionally, characterize the contact intensity on the touch-sensitive surface relative to one or more intensity thresholds such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move a focus selector based on the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.

[0195] An increase in contact characteristic intensity from an intensity below the light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a "light press" input. An increase in contact characteristic intensity from an intensity below the deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a "deep press" input. An increase in contact characteristic intensity from an intensity below the contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting a contact on the touch surface. A decrease in contact characteristic intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting a contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0196] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or contacts), where the corresponding press input is detected at least in part based on the detected intensity of the contact (or contacts) increasing above a press input intensity threshold. In some embodiments, a corresponding operation is performed in response to detecting an increase in the intensity of the corresponding contact above the press input intensity threshold (e.g., the "downstroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the press input intensity threshold, and a corresponding operation is performed in response to detecting a subsequent decrease in the intensity of the corresponding contact below the press input threshold (e.g., the "upstroke" of the corresponding press input).

[0197] Figures 5E - 5HDetection of a gesture is shown, the gesture comprising contact 562 with an intensity from below Figure 5E The light press intensity threshold in L ”) increases to a strength higher than Figure 5H The deep compression intensity threshold in D ”). On the displayed user interface 570 including application icons 572A-572D displayed in the predefined area 574, when the cursor 576 is displayed above the application icon 572B corresponding to application 2, a gesture performed using contact 562 is detected on the touch-sensitive surface 560. In some embodiments, the gesture is detected on the touch-sensitive display 504. The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560. The device determines whether the intensity of the contact 562 is within the deep press intensity threshold (for example, “IT D ”) reaches a peak above. Contact 562 is maintained on touch-sensitive surface 560. In response to detecting the gesture, and according to the increase in intensity during the gesture to a deep press intensity threshold (e.g., “IT D ”) above contact 562, displays scaled representations 578A-578C (e.g., thumbnails) of documents recently opened for application 2, such as Figures 5F - 5H In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in the Figures 5E - 5H To assist the readers.

[0198] In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, such as Figure 5F As the animation progresses, representation 578A moves upward and representation 578B is displayed near application icon 572B, as shown in FIG. Figure 5G Then, representation 578A moves upward, 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. Figure 5H 578A-578C form an array above icon 572B. In some embodiments, the animation progresses according to the intensity of contact 562, such as Figures 5F - 5G , where representations 578A-578C appear and increase as the strength of contact 562 approaches a deep press strength threshold (eg, “IT D ”) increases and moves upward. In some embodiments, the intensity according to which the animation progresses is the characteristic intensity of the contact. The reference may be performed using an electronic device similar to or equivalent to device 100, 300, or 500. Figures 5E - 5H The operation described.

[0199] In some embodiments, the device employs strength hysteresis to avoid unexpected inputs sometimes referred to as “jitter,” where the device defines or selects a hysteresis strength threshold having a predefined relationship to a press input strength threshold (e.g., the hysteresis strength threshold is X strength units lower than the press input strength threshold, or the hysteresis strength threshold is 75%, 90%, or some reasonable percentage of the press input strength threshold). Thus, in some embodiments, a press input includes the strength of a corresponding contact increasing above the press input strength threshold and the strength of that contact subsequently decreasing below the hysteresis strength threshold corresponding to the press input strength threshold, and a corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases below the hysteresis strength threshold (e.g., the “upstroke” of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from a strength equal to or lower than the hysteresis strength threshold to a strength equal to or higher than the press input strength threshold and optionally the contact strength subsequently decreases to a strength equal to or lower than the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., depending on the context, the contact strength increases or the contact strength decreases).

[0200] For ease of explanation, optionally, the description of an operation performed in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is triggered in response to detecting any one of the following various conditions: the contact strength increases above the press input strength threshold, the contact strength increases from a strength lower than the hysteresis strength threshold to a strength higher than the press input strength threshold, the contact strength decreases below the press input strength threshold, and / or the contact strength decreases below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in an example where an operation is described as being performed in response to detecting that the strength of a contact decreases below the press input strength threshold, the operation is optionally performed in response to detecting that the strength of the contact decreases below the hysteresis strength threshold corresponding to and less than the press input strength threshold.

[0201] Attention is now turned to embodiments of a user interface (“UI”) implemented on an electronic device (such as the portable multifunctional device 100, device 300, or device 500) and associated processes.

[0202] Figures 6A to 6AF An exemplary user interface for managing a communication session is shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 7A to 7B 、 Figure 8 and Figures 9A to 9B the processes in.

[0203] Figure 6AAn electronic device 602 and an electronic device 652 are shown. Each electronic device corresponds to a different user. For example, device 602 corresponds to a first user ("Freddy") and device 602 is logged into a first user account corresponding to the first user. Also, device 652 corresponds to a second user ("Nick") and device 652 is logged into a second user account corresponding to the second user. Device 602 and device 652 are far from each other.

[0204] Device 602 receives user input to activate a messaging application (e.g., a tap on an enabling representation corresponding to the messaging application). As Figure 6A shown, in response to receiving the user input, device 602 displays a user interface 604 for the messaging application. The user interface 604 for the messaging application includes an enabling representation 604a for accessing multiple sessions and an enabling representation 604b for starting a new session. At Figure 6A , device 602 receives a tap input 606a on the enabling representation 604b (e.g., at a location on a touch-sensitive surface corresponding to the enabling representation). Figure 6A The display of device 652 in

[0205] At Figure 6B , in response to receiving the tap input 606a, device 602 displays an input field 604c for receiving contact information of participants in the session for which user input is received, a keyboard 604d for inputting information such as contact information, a field 604f for receiving a user input message (e.g., using the keyboard 604d), and an enabling representation 604e for adding a participant to the session. Figure 6B The display of device 652 in

[0206] At Figure 6C , device 602 receives a tap input 606b on the enabling representation 604e. Device 602 receives additional user input to select a contact from the address book and adds the selected contact to the session as a participant, as Figure 6D shown in the field 604c. Device 602 also receives user input such as voice input or keyboard input using the keyboard 604d, as Figure 6D shown, and the user input is displayed in the field 604f in the form of a draft message (e.g., without transmitting the message to the participants in the session). In this example, the draft message includes a first part and a second part, the first part does not include a URL, and the second part includes a URL. The URL is a link to a media file. Figures 6C to 6D The display of device 652 in

[0207] At Figure 6EAt location, device 602 receives a tap input 606c on an enabling representation 604g for sending a draft message. Figure 6E The display of device 652 in Figure 6E is turned off.

[0208] In response to receiving the tap input 606c on the enabling representation 604g, device 602 transmits the content of the draft message to the participants of the session and, as Figure 6F shown, displays message 608a and message 608b in session log 608. A real-time enabling representation 610 provides an indication as to whether device 602 is currently transmitting a real-time media stream to the participants of the session. In this example, the real-time enabling representation 610 is not bold, thus indicating that device 602 is not transmitting a real-time media stream to the participants of the session. In some examples, the indication as to whether device 602 is transmitting includes one or more of the following: the color, size, and animation of the real-time enabling representation 610. Device 602 simultaneously displays multiple avatars 618a, 620a, and 622a for the participants of the communication session other than the user of device 602. In this example, avatars 618a, 620a, and 622a are displayed above session log 608. Avatars 618a, 620a, and 622a are adjacent to the real-time enabling representation 610 above session log 608. Device 602 also displays the name of the corresponding participant represented by the avatar directly below each avatar. Session log 608 separates avatars 618a, 620a, and 622a from keyboard 604d. For example, avatar 618a is a 3D computer-generated image of participant Nick who is using device 652 to participate in the session.

[0209] At Figure 6F location, device 652 receives a message from device 602 while in a locked state and displays a notification 654a on the lock screen 654, the notification including at least a portion of the received message and optionally an indication of the sender (e.g., "Freddy").

[0210] At Figure 6G location, both device 602 and device 652 receive a new message from another participant ("Lauren") of the session. Device 602 displays the new message 608c in session log 608. Device 652 displays the new message 654b on the lock screen 654 while in a locked state.

[0211] At Figure 6H location, device 602 receives a tap input 606d on the real-time enabling representation 610. At Figure 6IAt [location], in response to receiving a tap input 606d on the live enablement indication 610, the device 602 displays a menu 612 that simultaneously includes an audio enablement indication 612a for transmitting a live media stream that includes live audio and does not include live video, a video enablement indication 612b for transmitting a live media stream that includes live audio and live video, and a cancel enablement indication 612c for continuing without transmitting a live media stream. When the audio enablement indication 612a, the video enablement indication 612b, and the cancel enablement indication 612c are simultaneously displayed, the device 602 receives a tap input 604h on the audio enablement indication 612a.

[0212] As Figures 6J to 6M shown, in response to receiving the tap input 604h on the audio enablement indication 612a, the device 602 provides a countdown 614 to the user to start transmitting the live media stream. In some examples, the countdown is provided by generating an audio (e.g., an oral countdown played using the speaker of the device 602). In some examples, the countdown is provided by haptic feedback. In this example, the countdown is provided by a visual display of the countdown 614, as Figures 6J to 6M shown. The countdown 614 includes a cancel enablement indication 614a that, when activated, cancels the countdown without starting to transmit the live media stream. Note that the device 602 does not transmit the live media stream throughout Figures 6A to 6I as indicated by the non-bolded live enablement indication 610.

[0213] At Figure 6M [location], in response to receiving the tap input 604h on the audio enablement indication 612a and after the countdown has been successfully completed, the device 602 begins to detect audio using the microphone of the device 602. In some examples, the device 602 begins to detect audio before the countdown is completed, but does not transmit the audio as part of the live media stream. In some examples, in response to receiving the tap input 604h on the audio enablement indication 612a, the device 602 abandons starting to detect video using the camera sensor of the device 602.

[0214] In response to receiving the tap input 604h on the audio enablement indication 612a and after the countdown has been successfully completed, the device 602 begins to transmit the audio (without video) in the live media stream to the participants of the session (other than the user of the device 602). Since the device 602 has received the activation of the audio enablement indication 612a rather than the video enablement indication 612b, the live media stream includes live audio and does not include live video. After the device 602 has received the activation of the enablement indication 612b, the live media stream will include live audio and live video.

[0215] As Figure 6MAs shown, the device 652 receives an indication that a participant in a session is streaming real-time media. When the device 652 is in a locked state, it displays a notification 654c on the lock screen 654 that Freddy is streaming a real-time media stream. The notification 654c includes an indication of the sender 654e of the real-time media stream and indications for the real-time media stream and optionally for the real-time audio stream (but not the real-time video stream) (such as indication 654d and indication 654f).

[0216] At Figure 6N this point, in response to the device 602 starting to stream a real-time media stream to a participant in the session, the device 602 displays a notification 608d in the session log 608 indicating that the device 602 has started streaming the real-time media stream. Similarly, the real-time enabling representation 610 provides an indication that the device 602 is streaming a real-time media stream to a participant in the session. In this example, the real-time enabling representation 610 is bolded to indicate that the device 602 is streaming the real-time media stream. Additionally, the real-time enabling representation 610 is updated to include an audio indicator 610a that varies (e.g., in shape, size, and / or color) according to the volume (or frequency) of the audio detected by the microphone of the device 602. The audio indicator 610a indicates the status of the real-time media stream to the user and confirms to the user that the device 602 is correctly detecting the audio for the real-time media stream.

[0217] Figure 6N Also shown is that the device 652 receives an input 656a on the notification 654c on the lock screen 654. In some examples, the device 652 includes one or more intensity sensors and determines whether the characteristic intensity of the input 656a exceeds an intensity threshold. Based on the characteristic intensity of the input 656a exceeding the intensity threshold, the device 652 provides a preview of the real-time media stream to the user without opening the corresponding messaging application (and optionally, without indicating to other participants that the user has joined the session). Based on the characteristic intensity of the input 656a not exceeding the intensity threshold, the device 652 provides the real-time media stream to the user by opening the corresponding messaging application (and optionally, indicating to other participants that the user has joined the session).

[0218] As Figure 6OAs shown, in response to the device 652 detecting the input 656a, the device 652 displays a user interface for a messaging application (including the session log 658). The session log 658 includes messages 658a - 658c received from other participants in the session and a notification 658d indicating that the participant (the user of the device 602) has started transmitting a real-time media stream. The notification 658d includes a join enabling indication 658e that, when activated, causes the device 658 to join the real-time media stream (e.g., start providing the real-time media stream to the user of the device 652). In some examples, in response to the device 652 detecting the input 656a, the device 652 provides (e.g., via a speaker) the real-time media stream being transmitted by the participants in the session, in which case the real-time audio is transmitted by the device 602. In some examples, in response to the device 652 detecting the input 656a, the device 652 does not provide (e.g., via a speaker) the real-time media stream being transmitted by the participants in the session until the activation of the join enabling indication 658e is received. The device 652 simultaneously displays multiple avatars 668a, 670a, and 672a for the participants in the communication session other than the user of the device 652. In this example, the avatars 668a, 670a, and 672a are displayed above the session log 658. The avatars 668a, 670a, and 672a are adjacent to the real-time enabling indication 660. Specifically, the avatar 668a represents the participant Freddy who is using the device 602 to participate in the session. The device 652 also displays the name of the corresponding participant represented by the avatar directly below each avatar.

[0219] At Figure 6O this point, the device 602 receives a tap input 604i on the real-time enabling indication 610. At Figure 6P this point, in response to receiving the tap input 604i on the real-time enabling indication 610, the device 602 displays a menu 616 that simultaneously includes a video enabling indication 616a for switching from transmitting real-time audio to transmitting real-time video and real-time audio, an end-audio enabling indication 616b for stopping the transmission of the real-time media stream, and a cancel enabling indication 616c for continuing to transmit the real-time media stream including real-time audio and excluding real-time video.

[0220] At Figure 6P this point, the device 652 receives a tap input 656b on the join enabling indication 658e. In response to detecting the activation of the join enabling indication 658e, as Figure 6QAs shown, in this case, in addition to providing the user with the real-time media stream being transmitted by the participants of the session (e.g., via the speaker), the real-time audio is transmitted by device 602, and device 652 starts transmitting a real-time media stream including real-time audio (and not including real-time video). Similarly, device 652 updates the real-time enablement representation 660 to be bolded to indicate to the user that device 652 is transmitting the real-time media stream to the participants of the session. At Figure 6Q this point, device 652 displays a notification 658f in the session log 658, which indicates that the participant Freddy has started transmitting a real-time media stream including real-time audio using device 602. Thus, device 602 receives the real-time audio from device 652 and outputs the received real-time audio on the speaker of device 602. Additionally, device 602 provides an indication that the participant Nick is providing a real-time media stream by displaying a dot 618c below the virtual avatar 618a, and the dot 618c is displayed next to the name of the participant ("Nick"). Device 652 also displays a notification 658g in the session log 658 indicating that device 652 is transmitting real-time audio to the participants of the session.

[0221] At Figure 6Q this point, when device 602 is transmitting real-time audio to the participants of the session, device 602 receives a tap input 604j on the video enablement representation 616a for converting from transmitting real-time audio to transmitting real-time video and real-time audio. In response to device 602 receiving the tap input 604j on the video enablement representation 616a, device 602 changes from not transmitting real-time video to transmitting real-time video and real-time audio.

[0222] At Figure 6RAt, in response to the device 602 receiving a tap input 604j on the video enabling indication 616a, the device 602 replaces the display of the session log 608 with the avatars 618a, 620a, and 622a of one or more participants in the session. In some examples, the device 602 displays an animation that transforms the avatars 618a, 620b, and 620c displayed above the session log into a grid that replaces the session log by magnifying the avatars, moving the avatars (e.g., moving down on the display), and changing the shape of the avatars (e.g., from circular to rectangular). The device 602 also displays the avatar 624a of the user of the device 602. The avatar 618a of participant Nick is a 3D computer-generated image. The indication 618b indicates that participant Nick is transmitting real-time audio. The avatar 620a of participant Lauren is a static image accessed from the address book of the device 602. Participant Lauren is not transmitting a real-time media stream (no real-time audio and no real-time video). The avatar 622a of participant Giancarlo is a set of the participant's initials. The thought bubble 622b indicates that participant Giancarlo is typing a draft message in the session. The avatar 624a of the user (Freddy) of the device 602 is a real-time video captured using a camera of the device 602 (such as a front camera), which is the same as the real-time video transmitted by the device 602 in the real-time media stream. The full-screen enabling indication 624b, when activated, causes the device 602 to magnify the avatar 624a and optionally replace the display of the avatars 618a, 620a, and 622a. The camera enabling indication 624c, when activated, causes the device 602 to change the source of the real-time video stream of the user from a first camera sensor (e.g., a front camera) to a second camera sensor (e.g., a rear camera). The enabling indication 624d, when activated, causes the device 602 to switch between transmitting real-time audio but not real-time video and transmitting real-time audio and real-time video.

[0223] At Figure 6R this point, the device 652 begins receiving the real-time video corresponding to participant Freddy. In response to receiving the real-time video corresponding to participant Freddy, the device 652 updates the avatar 668a of participant Freddy to include the real-time video corresponding to participant Freddy. As Figure 6RAs shown, device 652 displays a ring 668b around the virtual avatar 668a. In some examples, displaying the ring 668b indicates that participant Freddy is transmitting a real-time media stream. In some examples, the characteristics of the ring 668b (e.g., color, thickness, animation) indicate whether participant Freddy is currently transmitting a real-time media stream to the participants of the communication session. In some examples, the size (and / or color, and / or shape) of the ring 668b varies according to the volume (or frequency) of the audio received from participant Freddy.

[0224] At Figure 6S , participant Lauren has started transmitting real-time video, and thus device 602 updates the virtual avatar 620a of participant Lauren to include the real-time video being received from participant Lauren. Additionally, device 620a displays a caption-enabled representation 620b with the virtual avatar 620a based on the real-time audio received from participant Lauren. Device 602 has received a text message from participant Giancarlo, and thus device 602 displays the message in a speech bubble 622c with the virtual avatar 622a of participant Giancarlo. As Figure 6S shown, device 602a receives a tap input 604k on the caption-enabled representation 620b. In response to receiving the tap input 604k on the caption-enabled representation 620b, device 602 starts displaying the caption 620c of the audio transmitted by participant Lauren, such as at Figure 6U . The caption is optionally transcribed on device 602a or a remote server.

[0225] At Figure 6S , participant Lauren has started transmitting real-time video, and thus device 652 updates the virtual avatar 670a of participant Lauren to include the real-time video being received from participant Lauren.

[0226] As Figures 6T to 6X shown, device 652 receives a downward swipe user input 656c starting from a position adjacent to the top of the display (e.g., adjacent to the virtual avatars 668a, 670a, and 672a on the handle). As the swipe user input 656c travels towards the bottom of the display, device 652 continues to receive the swipe user input without detecting the lift-off of the swipe user input 656c. As the swipe user input 656c travels (with the handle 662), device 652 changes the shape of the displayed virtual avatars (e.g., from Figure 6T a circle in Figure 6Wthe rectangles therein), and transform the display of the participant's virtual avatars 668a, 670a, and 672a by changing the position of the displayed virtual avatars 668a, 670a, and 672a (e.g., from a 1×3 grid at the top of the display to a 2×2 grid).

[0227] At Figure 6V , the device 602 receives a tap input 604l on the camera enable indication 624c. As Figure 6V and Figure 6W shown, in response to receiving the tap input 604l on the camera enable indication 624c, the device 602 switches the live video of the virtual avatar 624a from the first camera (e.g., the front-facing camera) of the device 602 to the second camera (e.g., the rear-facing camera).

[0228] At Figure 6X , the device 602 receives a tap input 604m on the full-screen enable indication 624b. As Figure 6Y shown, in response to receiving the tap input 604m on the full-screen enable indication 624b, the device 602 enlarges the virtual avatar 624a on the display of the device 602, such as by using a smooth animation. At Figure 6Y , the virtual avatar 674a represents the participant Nick of the device 652. The device 652 receives a tap input 656d on the enable indication 674b. In response to receiving the tap input 656d, the device 652 switches from transmitting real-time audio but not real-time video to transmitting both real-time audio and real-time video (e.g., by starting to capture live video at the device 652 using the front-facing camera sensor), as Figure 6Z shown by the virtual avatar 674a that includes live video.

[0229] At Figure 6AA , the device 602 has switched back to displaying the message log 608 that includes the notification 608e, which indicates that the participant Nick of the device 652 has previously started transmitting live video as part of the real-time media stream. In this example, the device 602 continues to transmit the live video in the real-time media stream, as indicated by the bold real-time enable indication 610. Similarly, the device 652 has switched back to displaying the message log 658 that includes the notification 658h, which indicates that the device 652 has previously started transmitting live video as part of the real-time media stream.

[0230] At Figure 6AB , the user of the device 602 has provided a link to a media file, such as by entering a URL into the field 604f. In some examples, the user uses as described with respect to Figures 6D to 6FThe same (or similar) technology described is used to input the URL. In this example, the media file is a video. In response, device 602 transmits a link to the media file to the other participants in the communication session (e.g., Freddy, Lauren, Giancarlo). As Figure 6AB shown, device 602 displays message 608f, which includes an enabling representation 608g for synchronized playback of the media file and an enabling representation 608h for local (unsynchronized) playback of the media file. The visual appearance of enabling representation 608h is based on the content of the media (e.g., includes frames from the corresponding video). Message 608f also includes an indication 608i of the source of the media file.

[0231] At Figure 6AB , in response to receiving the media file transmitted by device 602, device 652 displays message 658i, which includes an enabling representation 658l for synchronized playback of the media file and an enabling representation 658j for local playback of the media file. The visual appearance of enabling representation 658j is based on the content of the media (e.g., includes frames from the corresponding video). Message 658i also includes an indication 658k of the source of the media file.

[0232] Activating enabling representation 608h causes device 602 to display a media playback user interface for playing the media file without synchronizing the playback with the other participants in the session. This allows the user of device 602 to, for example, watch a video without causing the playback of the video to start at the devices of the other participants. Activating enabling representation 608g causes device 602 to display a media playback user interface for playing the media file while synchronizing the playback with the other participants in the session. This allows the user of device 602 to, for example, watch a video with playback synchronized among one or more of the other participants' devices.

[0233] At Figure AC, device 602 detects a tap input 604n on enabling representation 608g. In response to detecting the tap input 604n on enabling representation 608g, device 602 displays media playback user interface 626 and transmits an instruction to the participants in the session to display a corresponding media playback user interface (e.g., 676), as Figure 6ADAs shown. The media playback user interface 626 includes a video 626a, a timer 626d showing the playback time of the media file, a play enabling indication 626b for starting the playback of the media file, other media controls (fast forward, rewind), and a scroll bar 626c for scrolling to different times in the media file. The media playback user interface 676 includes a video 676a, a timer 676d showing the playback time of the media file, a play enabling indication 676b for starting the playback of the media file, other media controls, and a scroll bar 676c for scrolling to different times in the media file. Either device 602 and / or device 652 can display either the media playback user interface 626 and the media playback user interface 676 and the corresponding layout of the virtual avatar.

[0234] In some examples, in response to a tap input 604n, the synchronized playback of the media file automatically starts (e.g., on device 602 and device 652). In this example, in response to device 602 detecting the activation of the play enabling indication 626b or device 652 detecting the activation of the play enabling indication 676b, the synchronized playback starts. At Figure 6AD device 602 receives a tap input 604o on the play enabling indication 626b. In response to detecting the tap input 604o on the play enabling indication 626b, device 602 starts the local playback of the media file and transmits an instruction to control the remote playback of the media file at the device of other participants (e.g., on device 652), so that the local playback and the remote playback are substantially synchronized, as Figure 6AE shown.

[0235] At Figure 6AE device 602 detects a swipe input 604p on the scroll bar 626c. In Figure 6AF , in response to detecting the swipe input 604p, device 602 updates the display of the video 626a to the time in the video corresponding to the updated scroll bar position, and transmits an instruction to the participants in the session so that the corresponding media player user interfaces of the participants are also updated to the time in the video, so that the playback of the video is substantially synchronized between the devices. Similarly, an input received at device 652 to control the playback of the video 676a (e.g., using the scroll bar 676c) causes device 652 to update the local display of the video at device 652 and transmit an instruction to control the playback of the video at the device of the session participants (e.g., at device 602), so that the playback of the video is substantially synchronized between the devices.

[0236] As Figure 6AEFurther shown, device 602 continues to receive real-time media streams from the participants of the session during the synchronous playback of video 626a, such as real-time video and real-time audio. Thus, the user of device 602 can view video 626a and simultaneously see and hear the reactions of other participants, as these participants also view the same part of the video.

[0237] In some examples, the participants of the session can share (e.g., by transmitting) a real-time view of the user interface of their respective devices instead of real-time video captured by a camera. In some examples, the above techniques can be used to synchronize game play on the respective devices instead of video playback, or game play can be synchronized in addition to synchronizing video playback. For example, two remote users can share the same view and control the same character in a first-person game.

[0238] In some examples, when device 602 is transmitting a real-time media stream, device 602 receives user input and, in response, returns a user interface 604 for a messaging application that includes an enabling representation 604a for accessing multiple sessions while continuing to transmit the real-time media stream. In some examples, returning user interface 604 causes device 602 to stop transmitting real-time video (but continue to transmit real-time audio) to the participants of the session.

[0239] In some examples, device 602 modifies the audio from the respective participants based on the display position of the respective virtual avatars of the participants on the display (e.g., using a head-related transfer function, the audio corresponding to the virtual avatar at the top of the display sounds like it is coming from above, and the audio corresponding to the virtual avatar on the right side of the display sounds like it is coming from the right). In some examples, a real-time media stream including real-time video can be redirected to a remote device for display on a television.

[0240] In some examples, the session remains even when the initiator of the session (e.g., the user of device 602) leaves the session. In some examples, the session ends when the last participant leaves the session. In some examples, a device that transmits a real-time media stream can save the real-time media stream to local memory, such as to a repository accessible by an image or video viewing application.

[0241] Figures 7A to 7B is a flowchart showing a method for using an electronic device to transmit real-time media (e.g., real-time audio and / or real-time video) according to some embodiments. Method 700 is performed at a device (e.g., 100, 300, 500) having a display, one or more camera sensors, and one or more microphones. Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0242] As described below, method 700 provides an intuitive way to transmit real-time media (e.g., real-time audio and / or real-time video). The method reduces the cognitive burden on the user for transmitting real-time media, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling the user to transmit real-time media faster and more efficiently saves power and increases the time interval between two battery charges.

[0243] In method 700, an electronic device receives (702) user input (e.g., 606b) that identifies one or more contacts to be included as one or more participants (e.g., in addition to the user of the electronic device) in a communication session. In some examples, the electronic device receives user input that selects one or more contacts from a plurality of contacts (such as contacts stored in an electronic address application).

[0244] According to some embodiments, the electronic device receives (704) (e.g., after receiving user input 606b that identifies one or more contacts) user input (e.g., 606d) for initiating a real-time streaming session (e.g., activation of the "real-time" affordance 610).

[0245] While in a communication session, the electronic device simultaneously displays (706) on a display: a first (708) affordance (e.g., 612a) for transmitting a real-time media stream that includes real-time audio and does not include real-time video (or any video) and a second (710) affordance (e.g., 612b) for transmitting a real-time media stream that includes both real-time audio and real-time video (e.g., a single stream that includes both audio and video).

[0246] According to some embodiments, in response to receiving user input (e.g., 606d) for initiating a real-time streaming session, the electronic device simultaneously displays the first affordance (e.g., 612a) and the second affordance (e.g., 612b).

[0247] The electronic device receives (712) user input (e.g., 604h) that activates one of the first affordance and the second affordance.

[0248] In accordance with (714) receiving user input (e.g., 604h) that activates the first affordance (e.g., 604a), the electronic device uses one or more microphones to detect (716) audio. In some examples, the electronic device does not use one or more camera sensors to detect multiple images of video.

[0249] Further in response to receiving a user input (e.g., 604h) that activates the first enabling indication (e.g., 604a), the electronic device (e.g., simultaneously with detecting audio) transmits (718) the real-time audio (without video, based on the detected audio) in the real-time media stream to one or more participants (other than the user of the electronic device) of the communication session, wherein the transmission of the real-time audio occurs without transmitting real-time video. Optionally, the electronic device does not use one or more camera sensors to detect images. Optionally, the electronic device does not transmit the streamed video received by the participants of the communication session (e.g., captured based on the images detected by one or more camera sensors).

[0250] In response to receiving a user input that activates the second enabling indication (e.g., 604b) according to (720), the electronic device uses one or more microphones to detect (722) audio. Further in response to receiving a user input that activates the second enabling indication (e.g., 604b) according to (720), the electronic device (e.g., simultaneously with detecting audio) uses one or more camera sensors to detect (724) multiple images of video. Further in response to receiving a user input that activates the second enabling indication (e.g., 604b) according to (720), the electronic device transmits (726) the real-time audio (e.g., based on the detected audio) and real-time video (e.g., based on the multiple images of video) in the real-time media stream to one or more participants (e.g., other than the user of the electronic device) of the communication session. In some examples, instead of (or in addition to) transmitting audio or audio / video, the electronic device transmits an invitation to the participants of the communication session to start receiving the corresponding audio or audio / video.

[0251] According to some embodiments, in response to receiving a user input (e.g., 604h) that activates the first enabling indication (e.g., 612a), the electronic device displays an indication (e.g., 608d) that the real-time audio stream has started in the recording area (e.g., 608) of the communication session. According to some embodiments, in response to receiving a user input that activates the second enabling indication (e.g., 612b), the electronic device displays an indication that the real-time video stream has started in the recording area of the communication session. Providing an indication of the type of stream that has started provides the user with feedback regarding the type of information that is being collected and transmitted to other participants. Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0252] According to some embodiments, in response to receiving user input (e.g., 606b) identifying one or more contacts, the electronic device automatically transmits a request to the one or more contacts to join a communication session. Automatically transmitting the request to the one or more contacts enables the user to share real-time audio and / or real-time video with the contacts without excessive user input. Reducing the amount of input required to perform an operation enhances the device's operability and makes the user-device interface more efficient, which in turn reduces power usage and extends the device's battery life by enabling the user to use the device more quickly and effectively.

[0253] According to some embodiments, the electronic device receives a request to transmit media (e.g., 604h, 604j) to a participant of a communication session. In response to receiving the request to transmit media to a participant of the communication session, based on determining that the communication session currently does not include transmitting a real-time media stream, the electronic device transmits a first type of notification to one or more of the first participants without transmitting a second type of notification to one or more of the first participants. In response to receiving the request to transmit media to a participant of the communication session, based on determining that the communication session currently does not include transmitting a real-time media stream, the electronic device transmits a second type of notification to one or more of the first participants without transmitting a first type of notification to one or more of the first participants. In some examples, the device transmits different types of data based on whether the current communication session includes or does not include a real-time stream of audio and / or video.

[0254] According to some embodiments, the electronic device displays one or more corresponding virtual avatars (e.g., 618a, 620a, 622a, 668a, 670a, 672a) of one or more (e.g., all, some but not all) of the participants of the communication session. In some examples, based on receiving a real-time media stream including real-time video from a participant, the corresponding virtual avatar of the participant includes at least a portion of the received real-time video. In some examples, the electronic device arranges the one or more corresponding virtual avatars in a grid style. Displaying the virtual avatars of the corresponding participants provides the user with visual feedback on who the participants in the communication session are, regardless of whether the participant is currently speaking. Providing the user with improved visual feedback enhances the device's operability and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the device's battery life by enabling the user to use the device more quickly and effectively.

[0255] According to some embodiments, an electronic device displays one or more respective status indicators (such as, 610, 660, 668b) of one or more (e.g., all, some but not all) of the participants in a communication session. The respective status indicators include respective indications of whether the respective participants are currently transmitting a real-time media stream to the participants in the communication session. Displaying the status indicators of the participants provides the user with visual feedback regarding the status of each participant in the communication session (e.g., streaming media, not streaming media, streaming only audio, streaming audio / video). Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0256] According to some embodiments, an electronic device displays corresponding avatars (e.g., 668a) of participants in a communication session. In some examples, based on a real-time media stream received from a participant that includes real-time video, the corresponding avatar of the participant includes at least a portion of the received real-time video. A corresponding status indicator (e.g., 668b) for a participant in a communication session includes a visual indicator (e.g., 668b, ring, circular avatar) around the corresponding avatar (e.g., 668a) of the participant in the communication session. Characteristics of the visual indicator (e.g., color, thickness, animation) indicate whether the corresponding participant is currently transmitting a real-time media stream to the participants in the communication session. In some examples, a ring (e.g., 668b) around the avatar (e.g., 668a) of a participant currently streaming real-time media (e.g., audio and / or video) is a first color (e.g., green). In some examples, a ring (e.g., 668b) around the avatar (e.g., 668a) of a participant not currently streaming real-time media (e.g., audio and / or video) is a second color (e.g., not green but red). In some examples, the avatar of a participant not currently streaming real-time media (e.g., audio and / or video) does not include a ring. In some examples, a green ring indicates that the participant is currently streaming real-time audio. In some examples, a red ring indicates that the participant is currently streaming real-time audio and video. Displaying different types of indicators (e.g., shape, color) based on the status of the participants provides visual feedback (e.g., streaming media, not streaming media, only streaming audio, streaming audio / video) to the user regarding the status of each participant in the communication session. Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0257] According to some embodiments, a visual indicator (e.g., 668b) is characterized by the size of the visual indicator. The visual indicator (e.g., 668b) of a corresponding participant (e.g., 668a) varies according to the volume (or frequency) of the audio received from the corresponding participant. In some embodiments, a ring (e.g., 668b) is characterized by the shape of the ring (e.g., 668b), the diameter of the ring (e.g., 668b), the width of the ring (e.g., 668b), and / or the surface area of the ring (e.g., 668b), and these characteristics vary according to the volume and / or frequency of the audio received from the corresponding participant. Displaying the variation in the visual indicator based on the volume (or frequency) allows a user to visually see whether a participant is streaming real-time audio and to see the relative volume of the real-time audio without requiring the user to turn on speakers or wear headphones. Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0258] According to some embodiments, when in a communication session and when a keyboard is not displayed on a display, an electronic device detects a first gesture (e.g., an upward swipe gesture on a touch-sensitive surface of the device). In response to detecting the first gesture, the electronic device displays a keyboard (e.g., for entering a text message in the communication session) (e.g., by sliding into view from the bottom portion of the display).

[0259] According to some embodiments, further in response to detecting the first gesture, the electronic device transforms the display of one or more virtual avatars of one or more participants in a communication session by simultaneously doing the following: reducing the size of one or more virtual avatars of the display of one or more participants, changing the shape of the virtual avatars of the display of one or more participants (e.g., from rectangular to circular), and changing the position of the virtual avatars of the display of one or more participants (e.g., from an n×n grid (where n is not 1) to a 1×n grid (where n is not 1)). In some examples, when one or more of the effects of the size reduction, shape change, and position change do not occur, in response to receiving the first gesture, one or more of these effects occur. In some examples, according to a real-time media stream received from a participant that includes real-time video, the corresponding virtual avatar of the participant includes at least a portion of the received real-time video.

[0260] According to some embodiments, after transforming the display of one or more virtual avatars of one or more participants in response to a first gesture, the electronic device detects a second gesture (e.g., 656c, a downward swipe gesture on the touch-sensitive surface of the device, different from the first gesture). In response to detecting the second gesture (e.g., 656c), the electronic device transforms the display of one or more virtual avatars (e.g., 668a, 670a, 672a) of one or more participants by simultaneously performing the following: enlarging the size of one or more virtual avatars of the display of one or more participants, changing the shape of the virtual avatars of the display of one or more participants (e.g., from circular to rectangular), and changing the position of the virtual avatars of the display of one or more participants (e.g., from a 1×n grid (where n is not 1) to an n×n grid (where n is not 1)). In some examples, when one or more of the effects of the enlargement of the size, the change of the shape, and the change of the position do not occur, in response to receiving the second gesture, one or more of these effects occur. In some examples, according to a real-time media stream including real-time video received from a participant, the corresponding virtual avatar of the participant includes at least a portion of the received real-time video. When space is available (e.g., when the keyboard is not displayed), automatically enlarging and rearranging one or more virtual avatars provides the user with a better view of the real-time video streams being received from other participants. Reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0261] According to some embodiments, the electronic device detects a third gesture (e.g., 604m, a tap gesture) at the location of a virtual avatar corresponding to a participant in the display of a communication session. In response to detecting the third gesture (e.g., 604m), the electronic device enlarges the corresponding virtual avatar of the participant. In some examples, according to a real-time media stream including real-time video received from a participant, the corresponding virtual avatar of the participant includes at least a portion of the received real-time video. In some examples, the corresponding virtual avatar of the participant is displayed full screen. In some examples, in response to detecting the third gesture, the virtual avatars of other participants stop being displayed.

[0262] According to some embodiments, an electronic device detects user input for enabling captions (e.g., 604k). In response to detecting user input for enabling captions (e.g., 604k), the electronic device displays captions (e.g., 620c) for an audio feed of one or more participants in a communication session. In some examples, the device detects user input for disabling captions and in response stops displaying captions for the participants in the communication session. In some examples, the audio feed of the participants is played simultaneously with the display of the captions. In some examples, the audio feed is transcribed by the electronic device to generate captions. In some examples, the captions are received by the electronic device for display. In some examples, other participants in the communication session can cause the captions to be displayed (or stopped being displayed) on the electronic device during the communication session. For example, the captions (e.g., 620c) help enable a user of the electronic device to participate in the communication session when the user is in a quiet environment such as a library. In some examples, the user input for enabling captions corresponds to a specific participant and the device enables captions for that participant without changing the caption state of other participants.

[0263] According to some embodiments, in response to receiving user input (e.g., 604h) that activates one of a first enabling representation (e.g., 612a) and a second enabling representation (e.g., 612b), the electronic device initiates the display of a countdown (e.g., 614) for initiating a live stream session. The transmission of audio or video in a live media stream to one or more participants (other than the user of the electronic device) in the communication session occurs after (and in response to) the completion of the displayed countdown (e.g., the countdown starts at 10 and transmission begins when the countdown reaches 0). Displaying the countdown before starting the stream provides the user with feedback on when the live media stream will start and allows the user to prepare for the live media streaming. Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0264] Note that the processes described above with respect to method 700 (e.g., Figures 7A to 7B) The details of [[ID=]] can also be applied to the methods described below in a similar manner. For example, methods 800 and 900 optionally include one or more features of the various methods described above with reference to method 700. For example, methods 700, 800, and 900 describe similar communication sessions. For another example, methods 800, 800, and 900 describe similar real-time media streams. For yet another example, the techniques of method 700 can be used to start a real-time media session before performing the techniques of methods 800 and 900. For yet another example, methods 800 and 900 can be performed during a real-time media session started using the techniques of method 700. For yet another example, the techniques of method 800 can be used to change the type of media stream started using the techniques of method 700. For the sake of brevity, these details are not repeated below.

[0265] Figure 8 is a flowchart showing method 800 for using an electronic device to switch between streaming real-time audio and real-time video according to some embodiments. Method 800 is performed at a device having a display (e.g., 100, 300, 500). Some operations in method 800 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0266] As described below, method 800 provides an intuitive way to switch between streaming real-time audio and real-time video. This method reduces the cognitive burden on the user to switch between streaming real-time audio and real-time video, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling the user to switch between streaming real-time audio and real-time video faster and more efficiently saves power and increases the time interval between two battery charges.

[0267] When transmitting (802) a first type of outgoing real-time media stream (e.g., a real-time audio stream, a real-time audio + video stream) to one or more participants (other than the user of the electronic device) of a communication session, the electronic device outputs (804) (e.g., displays on a display, plays audio on a speaker) at least one or more incoming real-time media streams (e.g., real-time video streams, real-time audio streams of one or more participants) at the electronic device.

[0268] When transmitting (802) a first type of outgoing real-time media stream to one or more participants of a communication session, the electronic device receives (806) a user input (e.g., 604j) for switching from transmitting the first type of outgoing real-time media stream to transmitting a second type of outgoing real-time media stream (e.g., a real-time audio stream, a real-time audio + video stream), where the first type is different from the second type.

[0269] According to some embodiments, receiving (806) user input for conversion includes detecting (808) a tap user input (e.g., 656d) on an enabling representation (e.g., 674b), where the enabling representation (e.g., 674b) is displayed as part of a virtual avatar (e.g., 674a) of a user of the electronic device.

[0270] In some examples, in accordance with currently using a camera of the electronic device to transmit a real-time media stream including a real-time video of the user, the corresponding virtual avatar of the user includes at least a portion of the real-time video. In some examples, the virtual avatar of the user includes video captured by the camera of the electronic device.

[0271] In response to receiving user input for conversion (e.g., 604j), the electronic device transitions (810) from transmitting a first type of outgoing real-time media stream to transmitting a second type of outgoing real-time media stream, including: in accordance with the first type of real-time media stream that includes real-time audio and does not include real-time video, (initiating, starting) transmitting (812) real-time video (and optionally maintaining or restarting transmission of real-time audio), and in accordance with the first type of real-time media stream that includes real-time audio and real-time video, stopping (814) transmission of real-time video (and optionally maintaining or restarting transmission of real-time audio).

[0272] According to some embodiments, after transitioning from transmitting a first type of outgoing real-time media stream to transmitting a second type of outgoing real-time media stream, the electronic device receives user input for transitioning from transmitting the second type of outgoing real-time media stream to transmitting the first type of outgoing real-time media stream. In response to receiving the user input for conversion, the electronic device transitions from transmitting the second type of outgoing real-time media stream to transmitting the first type of outgoing real-time media stream, including: in accordance with the second type of real-time media stream that includes real-time audio and does not include real-time video, (initiating, starting) transmitting real-time video (and optionally maintaining or restarting transmission of real-time audio), and in accordance with the second type of real-time media stream that includes real-time audio and real-time video, stopping transmission of real-time video (and optionally maintaining or restarting transmission of real-time audio).

[0273] According to some embodiments, the electronic device further includes one or more camera sensors and one or more microphones. According to some embodiments, transmitting the second type of outgoing real-time media stream (e.g., audio + video) includes: transmitting an image detected using one or more camera sensors, and transmitting audio detected using one or more microphones. Thus, the real-time media stream includes capturing real-time audio / video from the environment of the electronic device and transmitting the real-time audio / video to other participants.

[0274] According to some embodiments, when transmitting real-time video, the electronic device displays the real-time video on a display (e.g.,Figure 6R In 624a). Accordingly, the electronic device simultaneously displays a live video (which is being transmitted) on a display of the device. Displaying the live video being transmitted provides the user with feedback regarding the content of the video being transmitted. This enables the user to more easily adjust the camera that is being used to capture the live video. Providing improved visual feedback to the user enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0275] According to some embodiments, one or more cameras include a plurality of cameras, the plurality of cameras including a first camera and a second camera. When a first image is detected using the first camera and not using the second camera and when the first image is transmitted to one or more participants of a communication session (and when the first image is displayed on the display), the electronic device displays an enabling representation for switching cameras (e.g., 624c). Additionally, when the first image is detected and displayed, the electronic device detects an activation (e.g., 604l) of the enabling representation for switching cameras (e.g., 624c). In response to detecting an activation (e.g., 604l) of the enabling representation for switching cameras (e.g., 624c), the electronic device uses the second camera and not the first camera to detect a second image. Further in response to detecting an activation (e.g., 604l) of the enabling representation for switching cameras (e.g., 624c), the electronic device transmits the second image to one or more participants of the communication session (and optionally displays the second image on the display without transmitting the first image or displays the first image on the display). Accordingly, the electronic device displays an enabling representation for switching cameras. The electronic device changes the source of the video transmitted to the participants of the communication session and changes the source of the video displayed on the display when activated. A single input that changes both the source of the transmitted video and the source of the displayed video reduces the number of inputs required to provide the operation. Reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0276] According to some embodiments, in response to receiving user input for transformation and further based on a first type of real-time media stream including real-time audio and real-time video, an electronic device replaces the display of the real-time video captured by one or more cameras of the device with a static virtual avatar image on a display. In some examples, the static virtual avatar image replaces the real-time video. The transformation of the display of the user avatar from real-time video to a static image indicates the status of the device to the user (not transmitting the real-time video to the participants).

[0277] According to some embodiments, in response to receiving user input for transformation and further based on a first type of real-time media stream including real-time audio and real-time video, an electronic device replaces the display of the real-time video captured by one or more cameras of the device with a text display area on a display. Replacing the display of the real-time video with a text display area provides the user with improved visual feedback that enables the user to determine the source of the text being received and displayed (e.g., based on the location on the display, based on the replacement of the previously displayed video). Providing the user with improved visual feedback enhances the operability of the device and makes the user device interface more effective (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0278] According to some embodiments, in response to receiving user input for transformation and further based on a first type of real-time media stream including real-time audio and real-time video, an electronic device transforms (e.g., by animation) the display of one or more virtual avatars of one or more participants in a communication session by simultaneously: reducing the size of one or more virtual avatars of the display of one or more participants, changing the shape of the virtual avatars of the display of one or more participants (e.g., from rectangular to circular), and changing the position of the virtual avatars of the display of one or more participants (e.g., from an n×n grid (where n is not 1) to a 1×n grid (where n is not 1)). In some examples, when one or more of the effects of reducing the size, changing the shape, and changing the position do not occur, in response to receiving a first gesture, one or more of these effects occur. In some examples, according to the real-time media stream including real-time video received from a participant, the corresponding virtual avatar of the participant includes at least a portion of the received real-time video. In some examples, in response to receiving user input for transformation and further based on a first type of real-time media stream including real-time audio and real-time video, the electronic device also simultaneously displays a record of the communication session and a keyboard.

[0279] According to some examples, the electronic device detects a user input requesting to end the transmission of an outgoing real-time media stream. In response to detecting the user input requesting to end the transmission of the outgoing real-time media stream, the electronic device stops transmitting the outgoing real-time media stream (e.g., if only audio, stops transmitting the audio to the participants; if audio + video, stops transmitting the audio + video to the participants). Further in response to detecting the user input requesting to end the transmission of the outgoing real-time media stream, the electronic device stops receiving (and stops displaying) the real-time media stream from the participants of the communication session.

[0280] Note that the details of the processes described above with respect to method 800 (e.g., Figure 8 ) can also be applied in a similar manner to the methods described below and above. For example, method 700 and 900 optionally include one or more features of the various methods described above with reference to method 800. For example, methods 700, 800, and 900 describe similar communication sessions. Also, methods 800, 800, and 900 describe similar real-time media streams. Also, the techniques of method 700 can be used to start a real-time media session before performing the techniques of methods 800 and 900. Also, methods 800 and 900 can be performed during a real-time media session started using the techniques of method 700. Also, the techniques of method 800 can be used to change the type of media stream started using the techniques of method 700. For the sake of brevity, these details are not repeated below.

[0281] Figures 9A to 9B is a flowchart showing method 900 for using an electronic device to synchronize media playback during a live stream. Method 900 is performed at a device having a display (e.g., 100, 300, 500). Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0282] As described below, method 900 provides an intuitive way to synchronize media playback during a live stream. The method reduces the cognitive burden on the user to switch between streaming real-time audio and real-time video, thereby creating a more efficient human-machine interface. For battery-powered computing devices, enabling the user to synchronize media playback faster and more efficiently during a live stream saves power and increases the time interval between two battery charges.

[0283] When participating (902) in a communication session with one or more participants (one or more remote participants other than the user of the electronic device), the electronic device transmits (904) a link to the media (e.g., a link to audio, video, audio + video, time-based media such as a URL) to one or more participants of the communication session.

[0284] When participating in a (902) communication session, an electronic device displays (906) on a display a representation of media (e.g., 608f) that includes (908) an enabling indication (e.g., 608g, a “live” enabling indication).

[0285] According to some embodiments, the representation of media (e.g., 608f) further includes (910) an image (e.g., 608h) based on the media content. In some examples, the media is a video and the image is a frame of the video. In some examples, the media is a song of an album and the image is the album's song cover. In some examples, the image based on the media content is separate from the enabling indication (e.g., the live enabling indication). In some examples, the image based on the media content is retrieved from a remote server.

[0286] When participating in a (902) communication session, the electronic device detects (912) a user input (e.g., 608n).

[0287] In response to detecting the user input (e.g., 608n) and when participating in a (902) communication session, the electronic device displays (916) a media playback user interface (e.g., 626, a video playback UI, an audio playback UI) corresponding to the media (and optionally, stops displaying the enabling indication and / or the image based on the media content).

[0288] According to some embodiments, further in response to detecting the user input and when participating in a (902) communication session, based on the user input being an activation of an enabling indication (e.g., 608g, a “live” enabling indication of the media), the electronic device synchronizes (918) the playback of the media on the devices of one or more participants in the communication session with the playback of the media on the electronic device. User input that automatically synchronizes playback among multiple devices enables participants to share a media experience without repeatedly manually coordinating the individual controls (play, pause, rewind, etc.) of the media. Automating the operation enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively. Automatic synchronization also reduces the need for the user to provide multiple inputs to synchronize playback with other devices. Reducing the number of inputs required to perform a task also enhances the operability of the device and makes the user-device interface more efficient. In some examples, participants performing media playback control user inputs at their respective devices causes corresponding control of the media playback on the electronic device. Further in response to detecting the user input and based on the user input not being an activation of an enabling indication, the electronic device forgoes synchronizing the playback of the media on the devices of one or more participants in the communication session with the playback of the media on the electronic device.

[0289] When displaying a media playback user interface and participating in (902) a communication session, the electronic device detects (920) (e.g., by the user at the electronic device via a touch-sensitive surface) a media playback control user input (e.g., 604o, 604p, an input to scroll to a specific time in the media, an input to play or pause the media).

[0290] In response to (924) receiving the media playback control user input, the electronic device controls (926) the playback of the media in the media playback user interface at the electronic device according to the received media playback control user input.

[0291] Further in response to (924) receiving the media playback control user input, and based on (928) the activation of the user input being an enabling indication (e.g., a "live" enabling indication of the media), the electronic device transmits (930) an instruction according to the received media playback control user input to control the playback of the media at the devices of one or more participants, such that the playback of the media at the devices of one or more participants in the communication session remains substantially synchronized.

[0292] According to some embodiments, further in response to (924) receiving the media playback control user input, and based on (932) the activation of the user input being based on an image of the media content (e.g., 608h), the electronic device abandons (934) transmitting an instruction for controlling the playback of the media at the devices of one or more participants. According to some embodiments, based on the activation of the user input being based on an image of the media content (e.g., 608h), the playback of the media on the devices of one or more participants in the communication session is not synchronized with the playback of the media on the electronic device.

[0293] According to some embodiments, each participant in the communication session has equal control over the media file and can start / stop / rewind / fast forward the video. The electronic device receives control instructions from the participants and modifies the playback accordingly. According to some embodiments, when displaying the media playback user interface, the electronic device receives one or more media playback control instructions from the remote devices of the participants in the communication session. In response to receiving the one or more media playback control instructions from the remote device, the electronic device modifies the playback of the media at the electronic device according to the received one or more media playback control instructions.

[0294] According to some embodiments, based on the activation of the user input being an enabling indication (e.g., 608g, a "live" enabling indication of the media), the electronic device outputs (e.g., displays on a display, plays audio using a speaker) one or more real-time media streams received from one or more participants in the communication session simultaneously with the display of the media playback user interface corresponding to the media (e.g., 626, video playback UI, audio playback UI).Figure 6AD 618a, 620a, 622a, 624a therein, real-time audio stream, real-time video stream). One or more real-time media streams are captured at the remote devices of one or more participants during media playback at the remote devices (e.g., using one or more cameras, using one or more microphones). In some examples, in response to activation of an indication represented by a user input, the electronic device simultaneously plays the audio of the media playback with one or more real-time audio feeds received from one or more participants of the communication session, wherein one or more real-time audio feeds are captured at the remote devices of one or more participants during media playback at the remote devices.

[0295] Note that the processes described above with respect to method 900 (e.g., Figures 9A to 9B ) can also be applied to the methods described above in a similar manner. For example, methods 700 and 800 optionally include one or more features of the various methods described above with reference to method 900. For example, methods 700, 800, and 900 describe similar communication sessions. For another example, methods 800, 800, and 900 describe similar real-time media streams. For yet another example, the techniques of method 700 can be used to start a real-time media session before performing the techniques of methods 800 and 900. For yet another example, methods 800 and 900 can be performed during a real-time media session started using the techniques of method 700. For yet another example, the techniques of method 800 can be used to change the type of media stream started using the techniques of method 700. For the sake of brevity, these details are not repeated below.

[0296] For purposes of explanation, the foregoing description has been presented by reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the techniques and their practical applications. Other technicians in the art can thus best utilize the techniques and various embodiments with various modifications suitable for the particular purposes contemplated.

[0297] Although the present disclosure and examples have been described in full with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the present disclosure and examples as defined by the claims.

[0298] As described above, one aspect of the technology of the present invention lies in collecting and using data obtained from various sources to improve the delivery of inspiring content or any other content that a user may be interested in. The present disclosure anticipates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0299] The present disclosure recognizes that the use of such personal information data in the technology of the present invention can be used to benefit the user. For example, the personal information data can be used to deliver targeted content that the user is more interested in. Thus, the use of such personal information data enables planned control over the delivered content. In addition, the present disclosure also anticipates other uses of personal information data that are beneficial to the user. For example, health and fitness data can be used to provide insights into a user's overall health condition, or can be used as positive feedback for an individual who uses technology to pursue health goals.

[0300] The present disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other uses of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Users should have easy access to such policies and they should be updated as the collection and / or use of data changes. A user's personal information should be collected for the legitimate and reasonable use of the entity and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should occur after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to safeguard and secure access to such personal information data and to ensure that other persons authorized to access the personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to account for the specific types of personal information data being collected and / or accessed and to apply the applicable laws and standards that include specific considerations of the jurisdiction. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Thus, different privacy practices should be maintained for different types of personal data in each country.

[0301] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of an advertising delivery service, the techniques of the present invention may be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service. As another example, a user may choose not to provide location information for a targeted content delivery service. As yet another example, a user may choose not to provide precise location information, but permit the transmission of location area information.

[0302] In addition, it is an object of the present disclosure to manage and process personal information data to minimize the risk of inadvertent or unauthorized access or use. Once data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated, when appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of the data stored (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data among users), and / or other methods.

[0303] Accordingly, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also anticipates that various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the techniques of the present invention will not fail to operate properly due to the lack of all or a portion of such personal information data. For example, preferences may be inferred by relying on non-personal information data or an absolute minimum amount of personal information such as the content requested by a device associated with the user, other non-personal information available to the content delivery service, or publicly available information, and content may be selected and delivered to the user accordingly.

Claims

1. An electronic device, the electronic device comprising: A display; One or more processors; And A memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: When transmitting a first type of outgoing real-time media stream to one or more participants of a communication session: Output at least one or more incoming real-time media streams at the electronic device; Display, via the display, a virtual avatar of a user associated with the outgoing real-time media stream; When displaying the virtual avatar of the user associated with the outgoing real-time media stream, receive user input for switching from transmitting the first type of the outgoing real-time media stream to transmitting a second type of outgoing real-time media stream, wherein the first type is different from the second type; And In response to receiving the user input for switching, switch from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream, including: According to the first type of real-time media stream that includes real-time audio and does not include real-time video, transmit real-time video and update the virtual avatar to include the real-time video; and According to the first type of real-time media stream that includes real-time audio and real-time video: Stop transmitting real-time video; and While continuing to output real-time audio, update the virtual avatar on the display by replacing the real-time video captured by one or more cameras of the electronic device with a predefined static image.

2. The electronic device according to claim 1, wherein receiving the user input for switching includes: Detecting a tap user input on an affordance representation, wherein the affordance representation is displayed as part of a virtual avatar of a user associated with the electronic device.

3. The electronic device according to claim 1, wherein the one or more programs further include instructions for performing the following operations: After switching from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream: Receive user input for converting from transmitting the second-type outgoing real-time media stream to transmitting the first-type outgoing real-time media stream; And In response to receiving the user input for switching, switch from transmitting the second type of the outgoing real-time media stream to transmitting the first type of the outgoing real-time media stream, including: According to the second type of real-time media stream that includes real-time audio and does not include real-time video, transmit real-time video; And According to the second type of real-time media stream that includes real-time audio and real-time video, stop transmitting real-time video.

4. The electronic device according to claim 1, wherein: The electronic device includes one or more camera sensors and one or more microphones; And Transmitting the second type of the outgoing real-time media stream includes: Transmitting an image detected by the one or more camera sensors; And Transmitting audio detected by the one or more microphones.

5. The electronic device according to claim 1, wherein the one or more programs further include instructions for performing the following operations: When transmitting real-time video, display the real-time video on the display.

6. The electronic device according to claim 1, wherein: the electronic device includes a first camera and a second camera; and the one or more programs further include instructions for performing the following operations: when detecting a first image using the first camera and not using the second camera and when transmitting the first image to one or more participants of the communication session: display an enabling representation for switching the camera; detect activation of the enabling representation for switching the camera; and in response to detecting activation of the enabling representation for switching the camera: detect a second image using the second camera and not using the first camera; and transmit the second image to one or more participants of the communication session.

7. The electronic device according to claim 1, wherein the predefined static image is a static avatar image.

8. The electronic device according to claim 1, wherein the predefined static image includes a text display area.

9. The electronic device according to claim 1, wherein the one or more programs further include instructions for performing the following operations: in response to receiving the user input for conversion and further based on the first type of real-time media stream including real-time audio and real-time video: transform the display of one or more avatars of one or more participants of the communication session by simultaneously performing the following operations: reduce the size of the one or more avatars of the one or more participants being displayed, change the shape of the one or more avatars of the one or more participants being displayed, and change the position of the one or more avatars of the one or more participants being displayed.

10. The electronic device according to claim 1, wherein the one or more programs further include instructions for performing the following operations: detect a user input requesting to end the transmission of the outgoing real-time media stream; and in response to detecting the user input requesting to end the transmission of the outgoing real-time media stream: stop transmitting the outgoing real-time media stream; and stop receiving real-time media streams from one or more participants of the communication session.

11. The electronic device according to claim 1, wherein the one or more programs further include instructions for performing the following operations: before receiving a user input for switching from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream, simultaneously display: A representation of the first type of outgoing real-time media stream being transmitted to one or more participants of the communication session at a first location on the display; and a plurality of representations of incoming real-time media streams from one or more participants of the communication session at one or more corresponding positions on the display, wherein: in response to receiving the user input for conversion and based on the first type of real-time media stream including real-time audio and real-time video: replacing the display of the real-time video captured by one or more cameras of the electronic device with the predefined static image on the display while continuing to output real-time audio includes: Stop displaying a first representation of the outgoing real-time media stream at the first location and display a second representation of the outgoing real-time media stream at the first location, where the first representation of the outgoing real-time media stream is a representation of real-time video and the second representation of the outgoing real-time media stream is the predefined static image; And Continue to display the multiple representations of the one or more incoming real-time media streams from the one or more participants in the communication session at the one or more respective locations on the display, while displaying the second representation of the outgoing real-time media stream at the first location on the display.

12. The electronic device according to claim 1, wherein the predefined static image identifies the user of the electronic device.

13. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device having a display, the one or more programs including instructions for performing the following operations: When transmitting a first type of outgoing real-time media stream to one or more participants in a communication session: Output at least one or more incoming real-time media streams at the electronic device; Display a virtual avatar of the user associated with the outgoing real-time media stream via the display; When displaying the user's virtual avatar associated with the outgoing real-time media stream, receive user input for transitioning from transmitting the outgoing real-time media stream of the first type to transmitting an outgoing real-time media stream of a second type, where the first type is different from the second type; And In response to receiving the user input for conversion, convert from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream, including: Transmit real-time video according to the first type of real-time media stream including real-time audio and not including real-time video, and update the virtual avatar to include the real-time video; and According to the first type of real-time media stream including real-time audio and real-time video: Stop transmitting real-time video; and While continuing to output real-time audio, update the virtual avatar on the display by replacing the real-time video captured by one or more cameras of the electronic device with the predefined static image.

14. The non-transitory computer-readable storage medium according to claim 13, wherein receiving the user input for conversion includes: Detecting a tap user input on an enabling representation, where the enabling representation is displayed as part of a virtual avatar of the user associated with the electronic device.

15. The non-transitory computer-readable storage medium according to claim 13, wherein the one or more programs further include instructions for performing the following operations: After converting from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream: Receive user input for converting from transmitting the second-type outgoing real-time media stream to transmitting the first-type outgoing real-time media stream; And In response to receiving the user input for conversion, convert from transmitting the second type of the outgoing real-time media stream to transmitting the first type of the outgoing real-time media stream, including: Transmit real-time video according to the second type of real-time media stream including real-time audio and not including real-time video; And According to the second type of real-time media stream including real-time audio and real-time video, stop transmitting real-time video.

16. The non-transitory computer-readable storage medium according to claim 13, wherein: the electronic device includes one or more camera sensors and one or more microphones; and transmitting the second type of the outgoing real-time media stream includes: transmitting images detected using the one or more camera sensors; and transmitting audio detected using the one or more microphones.

17. The non-transitory computer-readable storage medium according to claim 13, wherein the one or more programs further include instructions for performing the following operations: when transmitting real-time video, display the real-time video on the display.

18. The non-transitory computer-readable storage medium according to claim 13, wherein: the electronic device includes a first camera and a second camera; and the one or more programs further include instructions for performing the following operations: when detecting a first image using the first camera and not using the second camera and when transmitting the first image to one or more participants of the communication session: display an enabling representation for switching the camera; detect activation of the enabling representation for switching the camera; and in response to detecting activation of the enabling representation for switching the camera: detect a second image using the second camera and not using the first camera; and transmit the second image to one or more participants of the communication session.

19. The non-transitory computer-readable storage medium according to claim 13, wherein the predefined static image is a static avatar image.

20. The non-transitory computer-readable storage medium according to claim 13, wherein the predefined static image includes a text display area.

21. The non-transitory computer-readable storage medium according to claim 13, wherein the one or more programs further include instructions for performing the following operations: in response to receiving the user input for conversion and further based on the first type of real-time media stream including real-time audio and real-time video: convert the display of one or more avatars of one or more participants of the communication session by simultaneously performing the following operations: reduce the size of the one or more avatars of the one or more participants being displayed, change the shape of the one or more avatars of the one or more participants being displayed, and change the position of the one or more avatars of the one or more participants being displayed.

22. The non-transitory computer-readable storage medium according to claim 13, wherein the one or more programs further include instructions for performing the following operations: detect a user input requesting to end transmission of the outgoing real-time media stream; and in response to detecting the user input requesting to end transmission of the outgoing real-time media stream: stop transmitting the outgoing real-time media stream; and stop receiving real-time media streams from one or more participants of the communication session.

23. The non-transitory computer-readable storage medium according to claim 13, wherein the one or more programs further include instructions for performing the following operations: Prior to receiving user input to transition from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, simultaneously display: A representation of the first type of outgoing real-time media stream being transmitted to one or more participants of the communication session at a first location on the display; and one or more representations of one or more incoming real-time media streams from one or more participants of the communication session at one or more corresponding locations on the display, wherein: In response to receiving the user input for the transition and based on the real-time media stream of the first type including real-time audio and real-time video: Replacing the display of the real-time video captured by one or more cameras of the electronic device with the predefined static image on the display while continuing to output real-time audio includes: Stopping displaying the first representation of the outgoing real-time media stream at the first location and displaying the second representation of the outgoing real-time media stream at the first location, wherein the first representation of the outgoing real-time media stream is a representation of real-time video and the second representation of the outgoing real-time media stream is the predefined static image; and Continuing to display one or more representations of one or more incoming real-time media streams from one or more participants of the communication session at the one or more corresponding locations on the display while displaying the second representation of the outgoing real-time media stream at the first location on the display.

24. The non-transitory computer-readable storage medium according to claim 13, wherein the predefined static image identifies the user of the electronic device.

25. A method, comprising: At an electronic device having a display: When transmitting an outgoing real-time media stream of a first type to one or more participants of a communication session: Outputting at least one or more incoming real-time media streams at the electronic device; Displaying, via the display, an avatar of a user associated with the outgoing real-time media stream; When displaying the avatar of the user associated with the outgoing real-time media stream, receiving user input to transition from transmitting the outgoing real-time media stream of the first type to transmitting an outgoing real-time media stream of a second type, wherein the first type is different from the second type; and In response to receiving the user input for the transition, transitioning from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, including: Transmitting real-time video and updating the avatar to include the real-time video based on the real-time media stream of the first type including real-time audio and not including real-time video; and Based on the real-time media stream of the first type including real-time audio and real-time video: Stopping transmitting real-time video; and Updating the avatar by replacing the real-time video captured by one or more cameras of the electronic device with a predefined static image on the display while continuing to output real-time audio.

26. The method according to claim 25, wherein receiving the user input for the transition includes: Detect a tap user input on an enabling representation, where the enabling representation is displayed as part of a virtual avatar of a user associated with the electronic device.

27. The method according to claim 25, further comprising: After transitioning from transmitting the first type of the outgoing real-time media stream to transmitting the second type of the outgoing real-time media stream: Receiving a user input for transitioning from transmitting the second type of the outgoing real-time media stream to transmitting the first type of the outgoing real-time media stream; And In response to receiving the user input for the transition, transitioning from transmitting the second type of the outgoing real-time media stream to transmitting the first type of the outgoing real-time media stream, including: Transmitting real-time video according to the second type of real-time media stream that includes real-time audio and does not include real-time video; And Stopping transmitting real-time video according to the second type of real-time media stream that includes real-time audio and real-time video.

28. The method according to claim 25, further comprising: The electronic device includes one or more camera sensors and one or more microphones; And Transmitting the second type of the outgoing real-time media stream includes: Transmitting an image detected using the one or more camera sensors; And Transmitting audio detected using the one or more microphones.

29. The method according to claim 25, further comprising: When transmitting real-time video, displaying the real-time video on the display.

30. The method according to claim 25, wherein: The electronic device includes a first camera and a second camera; and The method further comprises: When detecting a first image using the first camera and not using the second camera and when transmitting the first image to one or more participants of the communication session: Displaying an enabling representation for switching the camera; Detecting an activation of the enabling representation for switching the camera; and In response to detecting the activation of the enabling representation for switching the camera: Detecting a second image using the second camera and not using the first camera; and Transmitting the second image to one or more participants of the communication session.

31. The method according to claim 25, wherein the predefined static image is a static virtual avatar image.

32. The method according to claim 25, wherein the predefined static image includes a text display area.

33. The method according to claim 25, further comprising: In response to receiving the user input for the transition and further according to the first type of real-time media stream that includes real-time audio and real-time video: Transforming the display of one or more virtual avatars of one or more participants of the communication session by simultaneously: Reducing the size of the one or more virtual avatars of the one or more participants being displayed, Changing the shape of the one or more virtual avatars of the one or more participants being displayed, and Changing the position of the one or more virtual avatars of the one or more participants being displayed.

34. The method according to claim 25, further comprising: Detect user input to end transmission of an outgoing real-time media stream; and in response to detecting user input to end transmission of the outgoing real-time media stream: stop transmitting the outgoing real-time media stream; and stop receiving real-time media streams from one or more participants in the communication session.

35. The method according to claim 25, further comprising: prior to receiving user input to transition from transmitting the outgoing real-time media stream of the first type to transmitting the outgoing real-time media stream of the second type, simultaneously display: a representation of the outgoing real-time media stream of the first type being transmitted to one or more participants in the communication session at a first location on the display; and one or more representations of one or more incoming real-time media streams from the one or more participants in the communication session at one or more corresponding locations on the display, wherein: in response to receiving the user input for the transition and based on the real-time media stream of the first type including real-time audio and real-time video: replacing, on the display, the display of real-time video captured by one or more cameras of the electronic device with the predefined static image while continuing to output real-time audio includes: stopping displaying the first representation of the outgoing real-time media stream at the first location and displaying a second representation of the outgoing real-time media stream at the first location, wherein the first representation of the outgoing real-time media stream is a representation of real-time video and the second representation of the outgoing real-time media stream is the predefined static image; and continuing to display the one or more representations of the one or more incoming real-time media streams from the one or more participants in the communication session at the one or more corresponding locations on the display while displaying the second representation of the outgoing real-time media stream at the first location on the display.

36. The method according to claim 25, wherein the predefined static image identifies the user of the electronic device.

Images