User Interface Camera Effect

Abstract

To provide a technique that provides electronic devices with faster, more efficient methods and interfaces for managing camera effects.SOLUTION: The methods and interfaces optionally complement or replace other methods for managing camera effects, reduce the cognitive burden on a user and produce a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges. Such a technique provides simulated visual effects in camera viewfinders and captured images without requiring additional hardware components. An electronic device provides for transitioning between simulated lighting effects, applies a simulated lighting effect to an image, provides user interfaces for applying a filter to an image, provides a reduced filter interface, and provides a visual aid displayed in the viewfinder.SELECTED DRAWING: Figure 5D

Description

Technical Field

[0001] This application claims priority to U.S. Patent Application No. 15 / 728,147, filed Oct. 9, 2017, entitled "USER INTERFACE CAMERA EFFECTS", which claims priority to U.S. Provisional Patent Application No. 62 / 556,414, filed Sep. 9, 2017, entitled "USER INTERFACE CAMERA EFFECTS", and U.S. Provisional Patent Application No. 62 / 514,947, filed Jun. 4, 2017, entitled "USER INTERFACE CAMERA EFFECTS". This application also claims priority to Danish Patent Application No. PA201770563, filed Jul. 10, 2017, entitled "USER INTERFACE CAMERA EFFECTS", and Danish Patent Application No. PA201770719, filed Sep. 22, 2017, entitled "USER INTERFACE CAMERA EFFECTS". The contents of these applications are hereby incorporated by reference in their entirety for all purposes.

[0002] This disclosure generally relates to electronic devices, and more particularly, to computer user interfaces for devices having built-in cameras.

Background Art

[0003] The use of electronic devices for recording videos and taking pictures has increased significantly in recent years. Exemplary electronic devices for recording videos and taking pictures include smartphones and handheld cameras. Such devices often include a viewfinder that can be used by a user to preview before taking a picture or recording a video.

Summary of the Invention

[0004] However, some techniques for manipulating camera effects using an electronic device are generally cumbersome and inefficient. For example, changing the visual effects in the viewfinder so that the captured images and recorded videos exhibit visual effects often requires a great deal of user input and is inaccurate. Existing techniques take more time than necessary and waste the user's time and the device's energy. The latter problem is particularly significant in battery-operated devices.

[0005] Accordingly, the present technology provides an electronic device with a faster and more efficient method and interface for managing camera effects. Such a method and interface optionally complement or replace other methods for managing camera effects. Such a method and interface reduces the cognitive burden on the user and creates a more efficient human-machine interface. In the case of battery-operated computing devices, such a method and interface conserves power and lengthens the battery charging interval. In some embodiments, the technology provides pseudo-visual effects in the camera viewfinder and captured images without requiring additional hardware components. In some embodiments, the technology provides the ability to quickly move between user interfaces with limited user input. In some embodiments, the technology efficiently provides improved image editing capabilities that result in visually pleasing results for the displayed digital viewfinder and captured videos. In some embodiments, the technology efficiently provides a user interface for moving between different sets of options with minimal input. In some embodiments, the technology efficiently provides a user interface for providing additional functionality without requiring any direct input. Such technology reduces the number of user inputs required and conserves battery power.

[0006] In an electronic device comprising one or more cameras, one or more input devices, and a display, the exemplary method includes simultaneously displaying on the display a camera application user interface including a digital viewfinder that includes a live preview of the field of view of the one or more cameras, and a representation of a filter picker user interface that overlays the digital viewfinder; detecting, via the one or more input devices, a first input starting at a position corresponding to each portion of the live preview while simultaneously displaying the digital viewfinder and the representation of the filter picker user interface; in response to detecting the first input, applying a preview of a first filter to the live preview of the field of view of the camera that has not been applied before the first input is detected, according to a determination that a first criterion including a requirement that the filter picker user interface overlays each portion of the live preview is satisfied when the first input is detected; and performing respective operations in the camera application without applying the preview of the first filter to the live preview, according to a determination that the filter picker user interface does not overlay each portion of the live preview when the first input is detected.

[0007] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising one or more cameras, one or more input devices, and a display. The one or more programs simultaneously display, on the display, a camera application user interface including a digital viewfinder including a live preview of a field of view of the one or more cameras, and a representation of a filter picker user interface overlapping the digital viewfinder, and while simultaneously displaying the digital viewfinder and the representation of the filter picker user interface, detect, via the one or more input devices, a first input starting at a position corresponding to each respective portion of the live preview, and in response to detecting the first input, according to a determination that a first criterion including a requirement that the filter picker user interface overlaps each respective portion of the live preview is satisfied when the first input is detected, add a preview of a first filter to the live preview of the field of view of the camera that has not been added before the first input is detected, and according to a determination that the filter picker user interface does not overlap each respective portion of the live preview when the first input is detected, execute respective operations in the camera application without applying the preview of the first filter to the live preview, including instructions.

[0008] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising one or more cameras, one or more input devices, and a display. The one or more programs include instructions to simultaneously display, on the display, a camera application user interface including a digital viewfinder including a live preview of the field of view of the one or more cameras, and a representation of a filter picker user interface overlapping the digital viewfinder; detect, via the one or more input devices, a first input starting at a position corresponding to each respective portion of the live preview while simultaneously displaying the digital viewfinder and the representation of the filter picker user interface; and, in response to detecting the first input, perform respective operations in the camera application without applying a preview of a first filter to the live preview of the field of view of the camera if, according to a determination that a first criterion including a requirement that the filter picker user interface overlaps each respective portion of the live preview when the first input is detected is satisfied, the preview of the first filter was not added to the live preview of the field of view of the camera before the first input was detected, and apply the preview of the first filter to the live preview according to a determination that the filter picker user interface does not overlap each respective portion of the live preview when the first input is detected.

[0009] Exemplary electronic devices include one or more cameras, one or more input devices, a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions to simultaneously display on the display a camera application user interface including a digital viewfinder including a live preview of the field of view of the one or more cameras and a representation of a filter picker user interface overlapping the digital viewfinder, to detect, via the one or more input devices, a first input starting at a position corresponding to each portion of the live preview while simultaneously displaying the digital viewfinder and the representation of the filter picker user interface, and in response to detecting the first input, to add a preview of a first filter to the live preview of the field of view of the camera that was not added prior to the detection of the first input according to a determination that a first criterion including a requirement that the filter picker user interface overlaps each portion of the live preview is satisfied when the first input is detected, and to perform respective operations in the camera application without applying the preview of the first filter to the live preview according to a determination that the filter picker user interface does not overlap each portion of the live preview when the first input is detected.

[0010] An exemplary electronic device includes one or more cameras, one or more input devices, a display, a camera application user interface including a digital viewfinder that includes a live preview of the field of view of one or more cameras on the display, and means for simultaneously displaying an expression of a filter picker user interface that overlaps the digital viewfinder; means for detecting a first input via one or more input devices at a position corresponding to each part of the live preview while simultaneously displaying the digital viewfinder and the expression of the filter picker user interface; and means for executing respective operations in the camera application without applying a preview of a first filter to the live preview of the field of view of the camera that has not been applied before the first input is detected, according to a determination that a first criterion including a requirement that the filter picker user interface overlaps each part of the live preview is satisfied in response to detecting the first input, and for applying the preview of the first filter to the live preview of the field of view of the camera according to a determination that the filter picker user interface does not overlap each part of the live preview when the first input is detected.

[0011] An exemplary method includes, in an electronic device including one or more input devices and a display, displaying, on the display, a representation of image data associated with depth map information; detecting a first input via one or more input devices while displaying the representation of the image data on the display; applying, in accordance with detecting the first input, a first lighting effect based on the depth map information to the representation of the image data; detecting a second input via one or more input devices; and applying, in accordance with detecting the second input, a second lighting effect based on the depth map information, different from the first lighting effect, to the representation of the image data.

[0012] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising one or more input devices and a display. The one or more programs display, on the display, a representation of image data associated with depth map information, detect a first input via the one or more input devices while displaying the representation of the image data on the display, apply, in accordance with detecting the first input, a first lighting effect based on the depth map information to the representation of the image data, detect a second input via the one or more input devices, and apply, in accordance with detecting the second input, a second lighting effect based on the depth map information that is different from the first lighting effect to the representation of the image data.

[0013] An exemplary transitory computer-readable storage medium stores one or more programs configured to be executable by one or more processors of an electronic device comprising one or more input devices and a display. The one or more programs display, on the display, a representation of image data associated with depth map information, detect a first input via the one or more input devices while displaying the representation of the image data on the display, apply, in accordance with detecting the first input, a first lighting effect based on the depth map information to the representation of the image data, detect a second input via the one or more input devices, and apply, in accordance with detecting the second input, a second lighting effect based on the depth map information that is different from the first lighting effect to the representation of the image data.

[0014] An exemplary electronic device includes one or more input devices, a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs are configured to display, on the display, a representation of image data associated with depth map information, detect a first input via the one or more input devices while displaying the representation of the image data on the display, apply, in accordance with detecting the first input, a first lighting effect based on the depth map information to the representation of the image data, detect a second input via the one or more input devices, and apply, in accordance with detecting the second input, a second lighting effect based on the depth map information, different from the first lighting effect, to the representation of the image data.

[0015] An exemplary electronic device includes one or more cameras, one or more input devices, a display, means for displaying, on the display, a representation of image data associated with depth map information, means for detecting a first input via the one or more input devices while displaying the representation of the image data on the display, applying, in accordance with detecting the first input, a first lighting effect based on the depth map information to the representation of the image data, detecting a second input via the one or more input devices, and applying, in accordance with detecting the second input, a second lighting effect based on the depth map information, different from the first lighting effect, to the representation of the image data.

[0016] An exemplary method, in an electronic device comprising one or more input devices and a display, detects a first input corresponding to a selection of a first image filter of a representation of image data having a first appearance via the one or more input devices, and, in accordance with a determination that the image data is associated with depth information that enables a foreground region of the representation of the image data to be distinguishable from a background region of the representation of the image data in response to detecting the first input, applies the first image filter to a foreground region of the representation of the image data having a first level of adjustment indicating a first degree to which the first image filter changes the appearance of the representation of the image data, to change the appearance of the foreground region of the representation of the image data, applies the first image filter to a background region of the representation of the image data having a second level of adjustment indicating a second degree to which the first image filter changes the appearance of the representation of the image data, different from the first level of adjustment, to change the appearance of the background region of the representation of the image data, and, after applying the first image filter to the representation of the image data, displays on the display a representation of each image having the first filter applied to the representation of the image data, including applying the first image filter to the representation of the image data.

[0017] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device including one or more input devices and a display. The one or more programs detect a first input corresponding to a selection of a first image filter for a representation of image data having a first appearance via the one or more input devices, and in response to detecting the first input, in accordance with a determination that the image data is associated with depth information that distinguishes a foreground region of the representation of the image data from a background region of the representation of the image data, apply the first image filter to the foreground region of the representation of the image data having a first level of adjustment indicating a first degree to which the first image filter changes the appearance of the representation of the image data to change the appearance of the foreground region of the representation of the image data, apply the first image filter to the background region of the representation of the image data having a second level of adjustment indicating a second degree to which the first image filter changes the appearance of the representation of the image data and different from the first level of adjustment to change the appearance of the background region of the representation of the image data, and after applying the first image filter to the representation of the image data, display on the display a representation of each image having the first filter applied to the representation of the image data. The one or more programs include instructions for applying a first image filter to a representation of image data.

[0018] An exemplary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices and a display. The one or more programs detect a first input corresponding to the selection of a first image filter for a representation of image data having a first appearance via the one or more input devices, and in response to detecting the first input, according to a determination that the image data is associated with depth information that distinguishes a foreground region of the representation of the image data from a background region of the representation of the image data, apply the first image filter to a foreground region of the representation of the image data having a first level of adjustment indicating a first degree to which the first image filter changes the appearance of the representation of the image data, to change the appearance of the foreground region of the representation of the image data, apply the first image filter to a background region of the representation of the image data having a second level of adjustment indicating a second degree to which the first image filter changes the appearance of the representation of the image data and different from the first level of adjustment, to change the appearance of the background region of the representation of the image data, and after applying the first image filter to the representation of the image data, cause the display to display a representation of each image having the first filter applied to the representation of the image data. The one or more programs include instructions for applying a first image filter to a representation of image data.

[0019] An exemplary electronic device includes one or more input devices, a display, one or more processors, and a memory storing one or more programs executable by the one or more processors. The one or more programs detect a first input corresponding to the selection of a first image filter for a representation of image data having a first appearance via the one or more input devices. In response to detecting the first input, according to a determination that the image data is associated with depth information that enables a foreground region of the representation of the image data to be distinguishable from a background region of the representation of the image data, the first image filter is applied to the foreground region of the representation of the image data with a first level of adjustment indicating a first degree to which the first image filter changes the appearance of the representation of the image data, and to change the appearance of the foreground region of the representation of the image data. The first image filter is applied to the background region of the representation of the image data with a second level of adjustment indicating a second degree to which the first image filter changes the appearance of the representation of the image data, different from the first level of adjustment, to change the appearance of the background region of the representation of the image data. After applying the first image filter to the representation of the image data, the display is caused to display a representation of each image having the first filter applied to the representation of the image data. The instructions include applying the first image filter to the representation of the image data.

[0020] An exemplary electronic device includes one or more input devices, a display, means for detecting a first input corresponding to a selection of a first image filter for a representation of image data having a first appearance via the one or more input devices, and, in response to detecting the first input, according to a determination that the image data is associated with depth information that enables a foreground region of the representation of the image data to be distinguishable from a background region of the representation of the image data, applying the first image filter to the foreground region of the representation of the image data with a first level of adjustment indicating a first degree to which the first image filter changes the appearance of the representation of the image data, changing the appearance of the foreground region of the representation of the image data, applying the first image filter to the background region of the representation of the image data with a second level of adjustment indicating a second degree to which the first image filter changes the appearance of the representation of the image data and different from the first level of adjustment, to change the appearance of the background region of the representation of the image data, and after applying the first image filter to the representation of the image data, means for displaying on the display a representation of each image having the first filter applied to the representation of the image data, the means for applying the first image filter to the representation of the image data.

[0021] An exemplary method in an electronic device comprising one or more input devices and a display includes displaying on the display a filter selection interface including representations of a plurality of filters from a series of filters, detecting a first input at a position corresponding to the filter selection interface while the representation of the image data and the filter selection interface are simultaneously displayed on the display via the one or more input devices while a first filter of the series of filters meets a selection criterion, in response to detecting the first input, stopping a first subset of representations of a plurality of filters of the series of filters including one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter of the filter selection user interface, and maintaining a display of a second subset of representations of a plurality of filters of the series of filters including at least the representation of the first filter.

[0022] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device including one or more input devices and a display. The one or more programs cause the display to present a filter selection interface including representations of a plurality of filters from a set of filters, and while presenting on the display a representation of image data and the filter selection interface simultaneously, detect, via one or more input devices, a first input at a position corresponding to the filter selection interface while a first filter of the set of filters meets a selection criterion, and in response to detecting the first input, pause a first subset of representations of a plurality of filters of the set of filters, including one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter, and maintain a display of a second subset of representations of a plurality of filters of the set of filters, including at least the representation of the first filter.

[0023] An exemplary transitory computer-readable storage medium stores one or more programs configured to be executable by one or more processors of an electronic device including one or more input devices and a display. The one or more programs cause the display to present a filter selection interface including representations of a plurality of filters from a set of filters, and while presenting on the display a representation of image data and the filter selection interface simultaneously, detect, via one or more input devices, a first input at a position corresponding to the filter selection interface while a first filter of the set of filters meets a selection criterion, and in response to detecting the first input, pause a first subset of representations of a plurality of filters of the set of filters, including one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter, and maintain a display of a second subset of representations of a plurality of filters of the set of filters, including at least the representation of the first filter.

[0024] An exemplary electronic device includes one or more input devices, a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs are configured to display, on the display, a filter selection interface including representations of a plurality of filters from a series of filters, and while simultaneously displaying, on the display, a representation of image data and the filter selection interface, detect, via the one or more input devices, a first input at a position corresponding to the filter selection interface while a first filter of the series of filters meets a selection criterion, and in response to detecting the first input, stop a first subset of representations of a plurality of filters of the series of filters, including one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter, of the filter selection user interface, and maintain a display of a second subset of representations of a plurality of filters of the series of filters, including at least the representation of the first filter.

[0025] An exemplary electronic device includes one or more input devices, a display, means for displaying, on the display, a filter selection interface including representations of a plurality of filters from a series of filters, means for detecting, via the one or more input devices, a first input at a position corresponding to the filter selection interface while a first filter of the series of filters meets a selection criterion while simultaneously representing, on the display, a representation of image data and the filter selection interface, and means for stopping, in response to detecting the first input, a first subset of representations of a plurality of filters of the series of filters, including one or more filters in a first direction from the representation of the first filter and one or more filters in a second direction from the representation of the first filter, of the filter selection user interface, and maintaining a display of a second subset of representations of a plurality of filters of the series of filters, including at least the representation of the first filter.

[0026] An exemplary method, in an electronic device comprising a camera, a sensor, one or more input devices, and a display, includes displaying, on the display, a camera viewfinder for capturing media; while displaying the camera viewfinder, based on data from the sensor, making a determination that the device meets a position adjustment guide display criterion that includes a requirement that a relative difference between an orientation of a plane of focus of the camera and a predetermined orientation is within respective position adjustment thresholds for a position adjustment guide display criterion to be met, and in accordance with the determination, displaying, on the display, a position adjustment guide within the camera viewfinder, the appearance of which changes as an orientation of the plane of focus of the camera changes relative to the predetermined orientation; and based on data from the sensor, in accordance with a determination that the position adjustment guide display criterion is not met, ceasing to display the position adjustment guide within the camera viewfinder.

[0027] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising a camera, a sensor, one or more input devices, and a display. The one or more programs include instructions to display, on the display, a camera viewfinder for capturing media; while displaying the camera viewfinder, based on data from the sensor, make a determination that the device meets a position adjustment guide display criterion that includes a requirement that a relative difference between an orientation of a plane of focus of the camera and a predetermined orientation is within respective position adjustment thresholds for a position adjustment guide display criterion to be met, and in accordance with the determination, display, on the display, a position adjustment guide within the camera viewfinder, the appearance of which changes as an orientation of the plane of focus of the camera changes relative to the predetermined orientation; and based on data from the sensor, in accordance with a determination that the position adjustment guide display criterion is not met, cease to display the position adjustment guide within the camera viewfinder.

[0028] An exemplary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a camera, a sensor, one or more input devices, and a display. The one or more programs display, on the display, a camera viewfinder for capturing media, and while displaying the camera viewfinder, based on data from the sensor, the device determines that the relative difference between the orientation of the plane of focus of the camera and a predetermined orientation is within respective position adjustment threshold values for a position adjustment guide display criterion to be satisfied, and in accordance with a determination that the position adjustment guide display criterion is satisfied, displays, on the display, a position adjustment guide within the camera viewfinder whose appearance changes when the orientation of the plane of focus of the camera changes relative to the predetermined orientation, and based on data from the sensor, in accordance with a determination that the position adjustment guide display criterion is not satisfied, stops displaying the position adjustment guide within the camera viewfinder, including instructions.

[0029] An exemplary electronic device includes a camera, a sensor, one or more input devices, a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs display, on the display, a camera viewfinder for capturing media, and while displaying the camera viewfinder, based on data from the sensor, the device determines that the relative difference between the orientation of the plane of focus of the camera and a predetermined orientation is within respective position adjustment threshold values for a position adjustment guide display criterion to be satisfied, and in accordance with a determination that the position adjustment guide display criterion is satisfied, displays, on the display, a position adjustment guide within the camera viewfinder whose appearance changes when the orientation of the plane of focus of the camera changes relative to the predetermined orientation, and based on data from the sensor, in accordance with a determination that the position adjustment guide display criterion is not satisfied, stops displaying the position adjustment guide within the camera viewfinder, including instructions.

[0030] An exemplary electronic device includes a camera, a sensor, one or more input devices, a display, means for displaying a camera viewfinder on the display for capturing media, and while displaying the camera viewfinder, based on data from the sensor, the device determines that the relative difference between the orientation of the focal plane of the camera and a predetermined orientation is within respective position adjustment threshold values for a position adjustment guide display criterion to be satisfied. According to the determination that the position adjustment guide display criterion is satisfied, a position adjustment guide within the camera viewfinder is displayed on the display, where the appearance changes when the orientation of the focal plane of the camera changes relative to the predetermined orientation. Based on the determination that the position adjustment guide display criterion is not satisfied based on data from the sensor, means for ceasing to display the position adjustment guide within the camera viewfinder are provided.

[0031] An exemplary method in an electronic device comprising one or more input devices, one or more cameras, and a display includes displaying, on the display, a representation of image data associated with depth map information; detecting, while the representation of the image data is displayed on the display, a first input via the one or more input devices for selecting each of a plurality of lighting effects based on the depth map information; after detecting the first input, detecting a second input corresponding to a request to capture image data corresponding to the field of view of the one or more cameras; in response to detecting the second input, capturing the image data corresponding to the field of view of the one or more cameras and associating the first lighting effect with the representation of the image data according to a determination that each of the lighting effects selected based on the first input is a first lighting effect based on the depth map information; and capturing the image data corresponding to the field of view of the one or more cameras and associating the second lighting effect with the representation of the image data according to a determination that each of the lighting effects selected based on the first input is a second lighting effect based on the depth map information that is different from the first lighting effect, including capturing the image data corresponding to the field of view of the one or more cameras.

[0032] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device comprising one or more input devices, one or more cameras, and a display. The one or more programs display, on the display, a representation of image data associated with depth map information, and while displaying the representation of the image data on the display, detect a first input via one or more input devices to select a respective filter of a plurality of lighting effects based on the depth map information, after detecting the first input, detect a second input corresponding to a request to capture image data corresponding to the field of view of one or more cameras, and in response to detecting the second input, capture image data corresponding to the field of view of one or more cameras and associate a first lighting effect with the representation of the image data according to a determination that each lighting effect selected based on the first input is a first lighting effect based on the depth map information, and capture image data corresponding to the field of view of one or more cameras and associate a second lighting effect with the representation of the image data according to a determination that each lighting effect selected based on the first input is a second lighting effect based on the depth map information different from the first lighting effect, including instructions to capture image data corresponding to the field of view of one or more cameras.

[0033] An exemplary non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more input devices, one or more cameras, and a display. The one or more programs cause the display to display a representation of image data associated with depth map information, detect a first input for selecting, via one or more input devices, a respective filter of a plurality of lighting effects based on the depth map information while the representation of the image data is being displayed on the display, detect a second input corresponding to a request to capture image data corresponding to a field of view of the one or more cameras after detecting the first input, and in response to detecting the second input, capture image data corresponding to the field of view of the one or more cameras and associate a first lighting effect with the representation of the image data according to a determination that each lighting effect selected based on the first input is a first lighting effect based on the depth map information, and capture image data corresponding to the field of view of the one or more cameras and associate a second lighting effect with the representation of the image data according to a determination that each lighting effect selected based on the first input is a second lighting effect based on the depth map information that is different from the first lighting effect, including instructions to capture image data corresponding to the field of view of the one or more cameras.

[0034] An exemplary electronic device includes one or more input devices, one or more cameras, a display, one or more processors, and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs display, on the display, a representation of image data associated with depth map information, and while displaying the representation of the image data on the display, detect a first input via the one or more input devices to select a respective filter of a plurality of lighting effects based on the depth map information, and after detecting the first input, detect a second input corresponding to a request to capture image data corresponding to the field of view of the one or more cameras, and in response to detecting the second input, capture the image data corresponding to the field of view of the one or more cameras according to a determination that each lighting effect selected based on the first input is a first lighting effect based on the depth map information, and associate the first lighting effect with the representation of the image data, and capture the image data corresponding to the field of view of the one or more cameras according to a determination that each lighting effect selected based on the first input is a second lighting effect based on the depth map information that is different from the first lighting effect, and associate the second lighting effect with the representation of the image data, including instructions to capture the image data corresponding to the field of view of the one or more cameras.

[0035] An exemplary electronic device includes one or more input devices, one or more cameras, a display, means for displaying a representation of image data associated with depth map information on the display, and while displaying a representation of the image data on the display, displaying a representation of the image data associated with depth map information on the display, and while displaying a representation of the image data on the display, detecting a first input for selecting, via one or more input devices, a respective filter of a plurality of lighting effects based on the depth map information, and after detecting the first input, detecting a second input corresponding to a request to capture image data corresponding to the field of view of one or more cameras, and in response to detecting the second input, capturing, according to a determination that each lighting effect selected based on the first input is a first lighting effect based on the depth map information, image data corresponding to the field of view of one or more cameras and associating the first lighting effect with the representation of the image data, and capturing, according to a determination that each lighting effect selected based on the first input is a second lighting effect based on the depth map information different from the first lighting effect, image data corresponding to the field of view of one or more cameras and associating the second lighting effect with the representation of the image data, and means for capturing image data corresponding to the field of view of one or more cameras.

[0036] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. The executable instructions for performing these functions are optionally included in a temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0037] Accordingly, a faster and more efficient method and interface for managing camera effects are provided to the device, thereby increasing the effectiveness, efficiency, and user satisfaction of such a device. Such a method and interface can complement or replace other methods for managing camera effects.

[0038] To better understand the various embodiments described, the following "Modes for Carrying Out the Invention" should be referred to in conjunction with the following drawings, and like reference numerals refer to corresponding parts throughout the following figures.

Brief Description of the Drawings

[0039]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

Figure 6I

Figure 6J

Figure 6K

Figure 6L

Figure 6M

Figure 6N

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 8G

Figure 8H

Figure 8I

Figure 8J

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 10G

Figure 10H

Figure 10I

Figure 10J

Figure 10K

Figure 10L

Figure 10M

Figure 10N

Figure 11A

Figure 11B

Figure 11C

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 12F

Figure 12G

Figure 12H

Figure 12I

Figure 12J

Figure 12K

Figure 12L

Figure 12M

Figure 12N

Figure 12O

Figure 12P

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 13F

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 14G

Figure 14H

Figure 14I

Figure 14J

Figure 14K

Figure 15A

Figure 15B

Figure 15C

Figure 15D

Figure 15E

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 16E

Figure 16F

Figure 16G

Figure 16H

Figure 16I

Figure 16J

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 17E

Figure 17F

Figure 17G

DETAILED DESCRIPTION OF THE INVENTION

[0040] The following description describes exemplary methods, parameters, etc. However, it should be recognized that such description is not intended as a limitation on the scope of the present disclosure, but rather is provided as an explanation of exemplary embodiments.

[0041] Electronic devices are designed and manufactured using more advanced camera features and sensors. However, due to the nature of their design, some electronic devices cannot capture the richness of point-light photography without additional hardware. Many point-light photography techniques require a large number of expensive light sources placed around the subject and a separate backdrop. However, many electronic devices have only a single flash that emits light in one direction. As a result, many point-light photography techniques simply cannot be achieved via conventional electronic devices.

[0042] The embodiments described herein include electronic devices that utilize depth map information to provide an improved camera function. In some embodiments, the depth map information is used when applying a filter to an image. In some embodiments, visual aids are provided to assist the user in capturing a complete image. The described embodiments also include a complementary user interface that enables these improved camera functions.

[0043] Also, the described embodiments enable efficient packaging and manufacturing of thin and lightweight devices while improving the performance of the device's camera optical function. Using a fixed focal length camera is advantageous because it is thin and small.

[0044] Below, FIGS. 1A - 1B, 2, 3, 4A - 4B, and 5A - 5H provide an illustration of exemplary devices for performing techniques for managing event notifications. FIGS. 6A - 6N show exemplary user interfaces for managing a user interface. FIG. 7 is a flowchart showing a method for managing a user interface according to some embodiments. The user interfaces of FIGS. 6A - 6G are used to illustrate the processes described below, including the process of FIG. 7.

[0045] FIGS. 8A - 8J illustrate exemplary user interfaces for applying a pseudo - lighting effect. FIG. 9 is a flowchart showing a method for simulating a lighting effect according to some embodiments. The user interfaces of FIGS. 8A - 8D are used to illustrate the processes described below, including the process of FIG. 9.

[0046] FIGS. 10A - 10N show exemplary user interfaces for applying a filter to an image. FIG. 11 is a flowchart showing a method for applying a filter to an image according to some embodiments. The user interfaces of FIGS. 10A - 10D are used to illustrate the processes described below, including the process of FIG. 11.

[0047] Figures 12A - 12P show exemplary user interfaces for displaying a degraded filter user interface. FIG. 13 is a flowchart showing a method for displaying a degraded filter user interface according to some embodiments. The user interfaces of FIGS. 12A - 12D are used to illustrate the processes described below, including the process of FIG. 13.

[0048] Figures 14A - 14K show exemplary user interfaces for providing visual materials. FIG. 15 is a flowchart showing a method for providing visual materials according to some embodiments. The user interfaces of FIGS. 14A - 14D are used to illustrate the processes described below, including the process of FIG. 15.

[0049] Figures 16A - 16J show exemplary user interfaces for providing visual materials when applying a pseudo - optical effect. FIG. 17 is a flowchart showing a method for providing visual materials when applying a pseudo - optical effect according to some embodiments. The user interfaces of FIGS. 16A - 16J are used to illustrate the processes described below, including the process of FIG. 17.

[0050] In the following description, terms such as "first", "second", etc. are used to describe various elements, but these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch can be called the second touch, and similarly, the second touch can be called the first touch. The first touch and the second touch are both touches, but they are not the same touch.

[0051] The terms used in the description of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. In the description of the various embodiments described and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It should be understood that the terms "includes", "including", "comprises", and / or "comprising", when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0052] The term "if" optionally, depending on context, is interpreted to mean "when" or "upon", or "in response to determining" or "in response to detecting". Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" optionally, depending on context, are interpreted to mean "upon determining" or "in response to determining", or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0053] Embodiments of an electronic device, a user interface for such a device, and related processes for using such a device are described. In some embodiments, the device is a portable communication device such as a cellular phone that also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunctional devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Optionally, other portable electronic devices such as a laptop or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad) are also used. Also, in some embodiments, it should be understood that the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0054] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, a mouse, and / or a joystick.

[0055] The device typically corresponds to various applications such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0056] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or changed for each application and / or within each respective application. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and transparent to the user.

[0057] Attention is now directed to an embodiment of a portable device having a touch-sensitive display. FIG. 1A is a block diagram showing a portable multifunctional device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display 112 may be referred to as a "touch screen" for convenience, and may be known or referred to as a "touch-sensitive display system." Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more light sensors 164. Device 100 optionally includes one or more contact intensity sensors 165 (such as a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100) for detecting the intensity of contact on device 100. Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (such as generating haptic output on a touch-sensitive surface, such as the touch-sensitive display system 112 of device 100 or the touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0058] In this specification and the claims, the term "intensity" of a contact on a touch-sensing surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensing surface, or a proxy for the force or pressure of the contact on the touch-sensing surface. The intensity of the contact has a range of values that includes at least four distinct values, and more typically, hundreds (e.g., at least 256) of distinct values. The intensity of the contact is optionally specified (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average) to determine the estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to specify the pressure of the stylus on the touch-sensing surface. Alternatively, the size and / or change in size of the contact area detected on the touch-sensing surface, the capacitance and / or change in capacitance of the touch-sensing surface in proximity to the contact, and / or the resistance and / or change in resistance of the touch-sensing surface in proximity to the contact are optionally used as surrogates for the force or pressure of the contact on the touch-sensing surface. In some implementations, the alternative measurements for the force or pressure of the contact are used directly to determine whether they exceed an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurements). In some implementations, the proxy measurement for the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of the contact as an attribute of a user input, the user can access additional device functions that would normally be inaccessible to the user on a device of reduced size with a limited area for displaying affordances (e.g., on a touch-sensing display) and / or receiving user input (e.g., via a touch-sensing display, a touch-sensing surface, or a physical / mechanical control such as a knob or button).

[0059] As used in this specification and the claims, the term "haptic output" refers to the physical displacement of the device relative to its previous position, the physical displacement of a component of the device (e.g., a touch-sensing surface) relative to another component of the device (e.g., the housing), or the displacement of a component relative to the center of mass of the device, which will be detected by the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a surface of the user that is sensitive to touch (e.g., the finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, the movement of a touch-sensing surface (e.g., a touch-sensing display or a trackpad) may optionally be interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, the user may feel a tactile sensation such as a "down click" or "up click" even when there is no movement of the physical actuator button associated with the touch-sensing surface that has been physically pressed (e.g., displaced) by the user's action. As another example, the movement of a touch-sensing surface may optionally be interpreted or sensed by the user as the "roughness" of the touch-sensing surface even when there is no change in the smoothness of the touch-sensing surface. Such interpretation of touch by the user depends on the user's individual sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Therefore, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise stated, the generated haptic output corresponds to the physical displacement of the device or a component of the device that generates the described sensory perception of a typical (or average) user.

[0060] Device 100 is merely an example of a portable multifunctional device, and it should be understood that Device 100 may optionally have more or fewer components than those shown, may optionally combine two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in FIG. 1A are implemented in a combination of hardware, software, or both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0061] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0062] Peripheral interface 118 can be used to couple the input and output peripheral devices of the device to CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data. In some embodiments, peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0063] The RF (radio frequency) circuit 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals or vice versa and communicates with a communication network and other communication devices via the electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, such as, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also called the World Wide Web (WWW), an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN), and with other devices. The RF circuit 108 optionally includes well-known circuits for detecting a near field communication (NFC) field, such as by a short-range communication radio. Wireless communication optionally includes, but is not limited to only, Global System for Mobile Communications (GSM) for mobile communication, Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long Termevolution, LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth (registered trademark), Bluetooth Low Energy (BTLE (registered trademark)), Wireless Fidelity (Wi-Fi (registered trademark)) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX (registered trademark), protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the filing date of this specification.

[0064] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. Also, the audio circuit 110 receives the electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 118 for processing. The audio data is optionally obtained from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 of FIG. 2). The headset jack provides an interface between the audio circuit 110 and a detachable audio input / output peripheral device such as an output-only headset or a headset having both an output (e.g., mono or stereo headphones) and an input (e.g., a microphone).

[0065] The I / O subsystem 106 couples input / output peripheral devices on the device 100, such as the touch screen 112 and other input control devices 116, to the peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, a light sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, a depth camera controller 169, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from and transmit electrical signals to the other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller 160 is optionally coupled to (or not coupled to any of) a keyboard, an infrared port, a USB port, and pointer devices such as a mouse. One or more buttons (e.g., 208 in FIG. 2) optionally include up and down buttons for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., 206 in FIG. 2).

[0066] As described in U.S. Patent Application No. 11 / 322,549, filed December 23, 2005, "Unlocking a Device by Performing Gestures on an Unlock Image," U.S. Patent No. 7,657,849, which is hereby incorporated by reference in its entirety, a quick press of a push button optionally disengages the lock of touch screen 112 or, optionally, initiates a process of unlocking the device using gestures on the touch screen. A longer press of a push button (e.g., 206) optionally turns the power to device 100 on or off. The functionality of one or more of the buttons is optionally customizable by the user. Touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0067] Touch-sensitive display 112 provides an input interface and an output interface between the device and the user. Display controller 156 receives electrical signals from and / or transmits electrical signals to touch screen 112. Touch screen 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0068] The touch screen 112 has a touch sensing surface, sensor, or set of sensors that receives input from a user based on tactile and / or haptic contact. The touch screen 112 and the display controller 156 (along with any associated modules and / or instruction sets within the memory 102) detect contacts (and any movement or interruption of the contacts) on the touch screen 112 and convert the detected contacts into interactions with user interface objects displayed on the touch screen 112 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

[0069] The touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although in other embodiments other display technologies are also used. The touch screen 112 and the display controller 156 optionally use any of a plurality of touch sensing technologies, now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch screen 112, to detect contacts and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California, is used.

[0070] The touch-sensing displays in some embodiments of touch screen 112 are optionally similar to multi-touch sensing touch pads as described in U.S. Patent Nos. 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and / or 6,677,932 (Westerman), and / or U.S. Patent Publication No. 2002 / 0015024A1, each of which is hereby incorporated by reference in its entirety. However, while touch screen 112 displays visual output from device 100, touch sensing touch pads do not provide visual output.

[0071] Touch sensing displays in some embodiments of touch screen 112 are described in (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller"; (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen"; (3) U.S. Patent Application No. 10 / 903,964, filed Jul. 30, 2004, "Gestures For Touch Sensitive Input Devices"; (4) U.S. Patent Application No. 11 / 048,264, filed Jan. 31, 2005, "Gestures For Touch Sensitive Input Devices"; (5) U.S. Patent Application No. 11 / 038,590, filed Jan. 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices"; (6) U.S. Patent Application No. 11 / 228,758, filed Sep. 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface"; (7) U.S. Patent Application No. 11 / 228,700, filed Sep. 16, 2005, "Operation Of A Computer With A Touch Screen Interface"; (8) U.S. Patent Application No. 11 / 228,737, filed Sep. 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; and (9) U.S. Patent Application No. 11 / 367,749, filed Mar. 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are hereby incorporated by reference in their entirety.

[0072] The touch screen 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally touches the touch screen 112 using any suitable object or appendage such as a stylus, finger, etc. In some embodiments, the user interface is designed to primarily handle finger-based contact and gestures, but this may be less precise than stylus-based input because the contact area of the finger on the touch screen is larger. In some embodiments, the device converts the coarse finger-based input into an accurate pointer / cursor position or command for performing the action desired by the user.

[0073] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touch pad (not shown) for activating or deactivating certain functions. In some embodiments, the touch pad, unlike the touch screen, is a touch-sensing area of the device that does not display visual output. The touch pad is optionally a touch-sensing surface separate from the touch screen 112 or an extension of the touch-sensing surface formed by the touch screen.

[0074] The device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within a portable device.

[0075] Device 100 also optionally includes one or more optical sensors 164. FIG. 1A shows an optical sensor coupled to an optical sensor controller 158 within I / O subsystem 106. Optical sensor 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light from the environment projected through one or more lenses and converts that light into data representative of an image. Optical sensor 164, in conjunction with imaging module 143 (also referred to as a camera module), optionally captures a still image or video. In some embodiments, the optical sensor is located on the back surface of device 100, opposite touch screen display 112 on the front surface of the device, and thus the touch screen display can be effectively used as a viewfinder for acquiring still and / or video images. In some embodiments, the optical sensor is located on the front surface of the device, and thus an image of the user can optionally be obtained for a video conference, and the user can view other video conference participants on the touch screen display. In some embodiments, the position of optical sensor 164 can be varied by the user (e.g., by rotating the lens and sensor within the device housing), and thus a single optical sensor 164 can be used for both video conferencing and for acquiring still and / or video images in conjunction with the touch screen display.

[0076] Additionally, device 100 optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 within I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with, or in proximity to, a touch sensing surface (e.g., touch sensing display system 112). In some embodiments, at least one contact intensity sensor is located on the back surface of device 100, opposite a touch screen display 112 located on the front surface of device 100.

[0077] Device 100 also optionally includes one or more proximity sensors 166. FIG. 1A shows proximity sensor 166 coupled to peripheral device interface 118. Alternatively, proximity sensor 166 is optionally coupled to input controller 160 within I / O subsystem 106. Proximity sensor 166 functions as described in U.S. Patent Application Nos. 11 / 241,839, "Proximity Detector In Handheld Device", 11 / 240,788, "Proximity Detector In Handheld Device", 11 / 620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output", 11 / 586,862, "Automated Response To And Sensing Of User Activity In Portable Devices", and 11 / 638,251, "Methods And Systems For Automatic Configuration Of Peripherals", which are hereby incorporated by reference in their entirety. In some embodiments, the proximity sensor turns off and disables touch screen 112 when the multifunction device is placed near the user's ear (e.g., when the user is making a phone call).

[0078] Also, device 100 optionally includes one or more haptic output generators 167. FIG. 1A shows a haptic output generator coupled to a haptic feedback controller 161 within I / O subsystem 106. The haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output on the device), into linear movement. The contact intensity sensor 165 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is disposed on or proximate to a touch-sensing surface (e.g., touch-sensing display system 112) and optionally generates a haptic output by moving the touch-sensing surface in a vertical direction (e.g., in / out of the surface of device 100) or a horizontal direction (e.g., back and forth within the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the back surface of device 100, opposite to the touch screen display 112 located on the front surface of device 100.

[0079] Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows an accelerometer 168 coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 within the I / O subsystem 106. The accelerometer 168 functions as described, for example, in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are hereby incorporated by reference in their entirety. In some embodiments, information is displayed on the touch screen display in portrait view or landscape view based on analysis of data received from one or more accelerometers. Device 100 optionally includes, in addition to the accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the position and orientation of the device 100 (e.g., portrait or landscape orientation).

[0080] Device 100 also optionally includes one or more depth camera sensors 175. FIG. 1A shows a depth camera sensor coupled to a depth camera controller 169 within the I / O subsystem 106. The depth camera sensor 175 receives data from the environment projected through the sensor. The depth camera sensor 175 is used, optionally in conjunction with the imaging module 143 (also referred to as the camera module), to determine depth maps of different portions of an image captured by the imaging module 143. In some embodiments, while a user is viewing other video conference participants on a touch screen display, an image of the user with depth information is optionally acquired for video conferencing and a self-portrait image with depth map data is captured, and a depth camera sensor is disposed on the front surface of the device 100. In some embodiments, the position of the depth camera sensor 175 can be changed by the user (e.g., by rotating the lens and sensor within the device housing), such that the depth camera sensor 175 is used for video conferencing as well as for acquiring still and / or video images with the touch screen display.

[0081] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Further, in some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) stores a device / global internal state 157 as shown in FIGS. 1A and 3. The device / global internal state 157 includes an active application state indicating which application is active if there is a currently active application, a display state indicating which application, view, or other information occupies various regions of the touch screen display 112, a sensor state including information obtained from various sensors and input control devices 116 of the device, and one or more of location information regarding the location and / or orientation of the device.

[0082] The operating system 126 (e.g., an embedded operating system such as Darwin (registered trademark), RTXC (registered trademark), LINUX (registered trademark), UNIX (registered trademark), OS X (registered trademark), iOS (registered trademark), WINDOWS (registered trademark), or VxWorks (registered trademark)) includes various software components and / or drivers that control and manage normal system tasks (e.g., memory management, control of storage devices, power management, etc.) and facilitate communication between various hardware and software components.

[0083] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE (registered trademark), etc.) are adapted to connect to other devices either directly or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used on iPod (registered trademark) devices (a trademark of Apple Inc.).

[0084] The contact / motion module 130 optionally detects contact with the touch screen 112 (in conjunction with the display controller 156) and other touch-sensing devices (e.g., a touch pad or a physical click wheel). The contact / motion module 130 includes various software components that perform various operations related to the detection of contact, such as determining whether contact has occurred (e.g., detecting the event of a finger being lowered), determining the intensity of the contact (e.g., the force or pressure of the contact, or an alternative to the force or pressure of the contact), determining whether there is movement of the contact, tracking movement across the touch-sensing surface (e.g., detecting one or more events of a finger being dragged), and determining whether the contact has stopped (e.g., detecting the event of a finger being raised or an interruption of the contact). The contact / motion module 130 receives contact data from the touch-sensing surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., the contact of one finger) or multiple simultaneous contacts (e.g., "multi-touch" / contact of multiple fingers). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0085] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" on an icon). In some embodiments, at least one subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds can be adjusted without changing the physical hardware of the device 100, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold of a trackpad or touch screen display can be set to any of a wide range of predetermined thresholds without changing the trackpad or touch screen display hardware. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more of a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or adjusting multiple intensity thresholds at once with a system-level click "intensity" parameter).

[0086] The contact / motion module 130 optionally detects gesture inputs from a user. Different gestures on the touch-sensing surface have different contact patterns (e.g., the detected movement, timing, and / or intensity of the contact is different). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event, followed by detecting a finger up (lift off) event at the same position (or substantially the same position) as the finger down event (e.g., the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensing surface includes detecting a finger down event, followed by detecting one or more finger drag events, and then followed by detecting a finger up (lift off) event.

[0087] The graphic module 132 includes various known software components that render and display graphics on the touch screen 112 or other display, including components that vary the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term "graphic" includes, but is not limited to, any object that can be displayed to a user, including text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

[0088] In some embodiments, the graphic module 132 stores data representing the graphics that will be used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives one or more codes specifying the graphics to be displayed, along with coordinate data and other graphic characteristic data as needed, from an application or the like, and then generates screen image data that is output to the display controller 156.

[0089] The haptic feedback module 133 includes various software components for generating instructions to be used by the haptic output generator(s) 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0090] The text input module 134 is optionally a component of the graphic module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0091] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as metadata for images / videos, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0092] The application 136 optionally includes the following modules (or sets of instructions) or subsets or supersets thereof. ● Contact module 137 (sometimes called an address book or contact list), ● Phone module 138, ● Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ● Training support module 142, ● Camera module 143 for still and / or video images, ● Image management module 144, ● Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, ● Widget module 149 that optionally includes one or more of weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widget 149-6, ● Widget creator module 150 for creating user-created widget 149-6, ● Search module 151, ● Video and music player module 152 that integrates the video player module and the music player module ● Memo module 153, ● Map module 154, and / or ● Online video module 155.

[0093] Examples of other applications 136 optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption, digital rights management, voice recognition, and voice duplication.

[0094] In conjunction with the touch screen 112, the display controller 156, the contact / motion module 130, the graphic module 132, and the text input module 134, the contact module 137 is optionally used to manage the address book or contact list (e.g., stored in the internal application state 192 of the contact module 137 in the memory 102 or the memory 370) by adding name(s) to the address book, deleting name(s) from the address book, associating phone number(s), email address(es), address(es), or other information with the name, associating an image with the name, classifying and sorting the names, providing a phone number or email address to initiate and / or facilitate communication by phone 138, the video conferencing module 139, email 140, or IM 141, etc.

[0095] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the telephone module 138 is optionally used for input of a character sequence corresponding to a telephone number, access to one or more telephone numbers in the contact module 137, modification of an input telephone number, dialing of each telephone number, conducting a conversation, and call stop or hang-up when the conversation ends. As described above, the wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.

[0096] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact / motion module 130, graphic module 132, text input module 134, contact module 137, and telephone module 138, the videoconference module 139 includes executable instructions for starting, conducting, and ending a videoconference between the user and one or more other participants according to a user command.

[0097] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. In conjunction with the image management module 144, the email client module 140 greatly facilitates the creation and sending of emails with still or video images captured by the camera module 143.

[0098] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, instant messaging module 141 includes executable instructions for input of a character sequence corresponding to an instant message, modification of previously input characters, transmission of each instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone communication-based instant messages, or XMPP, SIMPLE, or IMPS for Internet-based instant messages), reception of instant messages, and viewing of received instant messages. In some embodiments, the instant messages transmitted and / or received optionally include graphics, photos, audio files, video files, and / or other attached files supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone communication-based messages (e.g., messages transmitted using SMS or MMS) and Internet-based messages (e.g., messages transmitted using XMPP, SIMPLE, or IMPS).

[0099] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and music player module, the training support module 142 includes executable instructions for creating a training (e.g., having time, distance, and / or calorie burn goals), communicating with a training sensor (sports device), receiving training sensor data, calibrating sensors used to monitor the training, selecting and playing music for the training, and displaying, storing, and transmitting training data.

[0100] In conjunction with the touch screen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion module 130, graphic module 132, and image management module 144, the camera module 143 includes executable instructions for capturing a still image or video (including a video stream) and storing it in the memory 102, modifying the characteristics of a still image or video, or deleting a still image or video from the memory 102.

[0101] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise operating on still images and / or video images, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing them.

[0102] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the browser module 147 includes executable instructions for browsing the Internet according to user commands, including searching for, linking to, receiving, and displaying a web page or a portion thereof, as well as attached files and other files linked to the web page.

[0103] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, email client module 140, and browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar entries, to-do lists, etc.) according to user commands.

[0104] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and browser module 147, the widget module 149 is mini-applications (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) that are optionally downloaded and used by the user, or mini-applications (e.g., user-created widget 149-6) created by the user. In some embodiments, the widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript (registered trademark) files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! widget).

[0105] In conjunction with the RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, and browser module 147, the widget creator module 150 is used by the user to optionally create widgets (e.g., make user-specified portions of a web page into widgets).

[0106] In conjunction with the touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, the search module 151 includes executable instructions for searching for characters, music, sound, images, video, and / or other files in the memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to a user command.

[0107] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, video and music player module 152 includes executable instructions that enable a user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing video (e.g., on touch screen 112 or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple).

[0108] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, and text input module 134, memo module 153 includes executable instructions for creating and managing memos, to-do lists, etc. according to user instructions.

[0109] In conjunction with RF circuit 108, touch screen 112, display controller 156, contact / motion module 130, graphic module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally used to receive, display, modify, and store maps and map-related data (e.g., driving directions, data regarding specific locations or stores and other target locations in the vicinity thereof, and other location-based data) according to user instructions.

[0110] In conjunction with touch screen 112, display controller 156, contact / motion module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes instructions that enable a user to access a particular online video, browse a particular online video, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen or on an external display connected via external port 124), send an email having a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141, rather than email client module 140, is used to send a link to a particular online video. Additional explanation of the online video application can be found in U.S. Provisional Patent Application No. 60 / 936,562, filed Jun. 20, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," and U.S. Patent Application No. 11 / 968,067, filed Dec. 31, 2007, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," the contents of which are hereby incorporated by reference in their entirety.

[0111] The modules and applications identified above each correspond to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules may optionally be combined or otherwise reconfigured. For example, a video player module may optionally be combined with a music player module to form a single module (e.g., the video and music player module 152 of FIG. 1A). In some embodiments, the memory 102 optionally stores a subset of the modules and data structures identified above. Further, the memory 102 optionally stores additional modules and data structures not described above.

[0112] In some embodiments, the device 100 is a device in which the operation of a set of predetermined functions in the device is performed only through a touch screen and / or a touch pad. By using the touch screen and / or the touch pad as the main input control device for the operation of the device 100, the number of physical input control devices (push buttons, dials, etc.) on the device 100 is optionally reduced.

[0113] The set of predetermined functions performed only through the touch screen and / or the touch pad optionally includes navigation between user interfaces. In some embodiments, when touched by a user, the touch pad navigates the device 100 from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, the "menu button" is implemented using the touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device rather than a touch pad.

[0114] FIG. 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) includes an event sorter 170 (e.g., within operating system 126) and respective applications 136-1 (e.g., any of the aforementioned applications 137-151, 155, 380-390).

[0115] The event sorter 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates the current application view displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, the device / global internal state 157 is used by the event sorter 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 to which the event information is to be delivered.

[0116] In some embodiments, the application internal state 192 includes additional information such as resume information used when the application 136-1 resumes execution, user interface state information indicating information that is being or is ready to be displayed by the application 136-1, a state queue for waiting for the user to return to a previous state or view of the application 136-1, and a redo / undo queue for previous actions performed by the user.

[0117] The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information regarding sub-events (e.g., a user touch on the touch-sensitive display 112 as part of a multi-touch gesture). The peripheral device interface 118 transmits information received from the I / O subsystem 106, or sensors such as the proximity sensor 166, the accelerometer(s) 168, and / or the microphone 113 (via the audio circuit 110). The information that the peripheral device interface 118 receives from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0118] In some embodiments, the event monitor 171 transmits requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 transmits event information. In other embodiments, the peripheral device interface 118 transmits event information only when there is an important event (e.g., receipt of an input that exceeds a predetermined noise threshold and / or exceeds a predetermined duration).

[0119] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition unit determination module 173.

[0120] The hit view determination module 172 provides a software procedure for determining where within one or more views a sub-event occurs when the touch-sensitive display 112 displays two or more views. A view is composed of control devices and other elements that a user can view on the display.

[0121] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as an application view or user interface window, within which information is displayed and touch-based gestures occur. The application view (for each respective application) where a touch is detected optionally corresponds to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest level view where a touch is detected is optionally referred to as the hit view, and the set of events recognized as appropriate input is optionally determined based at least in part on the hit view of the initial touch that initiates the touch-based gesture.

[0122] The hit view determination module 172 receives information related to sub-events of a touch-based gesture. When the application has a plurality of hierarchically configured views, the hit view determination module 172 identifies the hit view as the lowest level view within the hierarchy in which the sub-events are to be processed. In most situations, the hit view is the lowest level view where the start sub-event (e.g., the first sub-event within a sub-event sequence that forms an event or potential event) occurs. After the hit view is identified by the hit view determination module 172, this hit view typically receives all sub-events related to the same touch or input source that was identified as the hit view.

[0123] The active event recognition unit determination module 173 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition unit determination module 173 determines that all views including the physical location of the sub-event are views that are actively involved, and thus determines that all views that are actively involved should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area related to one particular view, the upper-level views within the hierarchy will still continue to be views that are actively involved.

[0124] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., event recognition unit 180). In embodiments including the active event recognition unit determination module 173, the event dispatcher module 174 distributes event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information obtained by each event receiver 182 in an event queue.

[0125] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, the application 136-1 includes the event sorter 170. In still other embodiments, the event sorter 170 is an independent module or part of another module stored in the memory 102, such as the touch / motion module 130.

[0126] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each including instructions to process touch events occurring within respective views of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, the corresponding event processing unit 190 includes one or more of event data 179 received from data update unit 176, object update unit 177, GUI update unit 178, and / or event sorter 170. The event processing unit 190 optionally utilizes or invokes data update unit 176, object update unit 177, or GUI update unit 178 to update the internal state of the application 192. Alternatively, one or more of the application views 191 include one or more respective event processing units 190. Also, in some embodiments, one or more of data update unit 176, object update unit 177, and GUI update unit 178 are included in the corresponding application view 191.

[0127] Each event recognition unit 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from this event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparing unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution instructions 188 (optionally including sub-event distribution instructions).

[0128] The event receiving unit 182 receives event information from the event sorter 170. The event information includes information about sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information such as the position of the sub-event. When the sub-event is related to the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (for example, from portrait to landscape or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0129] The event comparison unit 184 compares the event information with the definition of a defined event or sub-event, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (for example, a sequence of predetermined sub-events) such as event 1 (187-1) and event 2 (187-2). In some embodiments, the sub-events within an event (187) include, for example, a touch start, a touch end, a touch movement, a touch cancellation, and multiple touches. In one example, the definition of event 1 (187-1) is a double-tap on a displayed object. The double-tap includes, for example, a first touch (touch start) on the displayed object for a predetermined stage, a first lift-off (touch end) for a predetermined stage, a second touch (touch start) on the displayed object for a predetermined stage, and a second lift-off (touch end) for a predetermined stage. In another example, the definition of event 2 (187-2) is a drag on a displayed object. The drag includes, for example, a touch (or contact) of the displayed object for a predetermined stage, a movement of the touch in the touch-sensitive display 112, and a lift-off of the touch (touch end). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0130] In some embodiments, event definition 187 includes the definition of events for each user interface object. In some embodiments, event comparison unit 184 performs a hit test to determine which user interface object is associated with the sub - event. For example, in an application view where three user interface objects are displayed on touch - sensitive display 112, when a touch is detected on touch - sensitive display 112, event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with that touch (sub - event). If each displayed object is associated with a corresponding event processing unit 190, the event comparison unit determines which event processing unit 190 should be activated using the result of the hit test. For example, event comparison unit 184 selects the event processing unit associated with the sub - event and object that triggered the hit test.

[0131] In some embodiments, the definition for each event (187) also includes a delay action that delays the transmission of event information until it is determined whether the sub - event sequence corresponds to the event type of the event recognition unit.

[0132] If each event recognition unit 180 determines that a series of sub - events does not match any of the events in event definition 186, each event recognition unit 180 enters a state of event - impossible, event - failed, or event - ended, and then ignores subsequent sub - events of the touch gesture. In this situation, if there are other event recognition units that remain active for the hit view, that event recognition unit continues to track and process the sub - events of the ongoing touch gesture.

[0133] In some embodiments, the corresponding event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists indicating how the event distribution system should actively participate in the event recognition unit that should execute sub - event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating how event recognition units interact with each other or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists indicating whether sub - events are distributed at various levels in the view hierarchy or program hierarchy.

[0134] In some embodiments, each event recognition unit 180 activates the event processing unit 190 associated with the event when one or more specific sub - events of the event are recognized. In some embodiments, each event recognition unit 180 distributes event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and deferring sending) sub - events to each hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag catches the flag and executes a predefined process.

[0135] In some embodiments, the event distribution command 188 includes a sub - event distribution command that distributes event information about sub - events without activating the event processing unit. Instead, the sub - event distribution command distributes event information to an event processing unit associated with a series of sub - events or to a view that is actively involved. The event processing unit associated with a series of sub - events or an actively involved view receives the event information and executes a predetermined process.

[0136] In some embodiments, data update unit 176 creates and updates data used in application 136-1. For example, data update unit 176 updates a phone number used in contact module 137, or stores a video file used in a video player module. In some embodiments, object update unit 177 creates and updates objects used in application 136-1. For example, object update unit 177 creates a new user interface object or updates the position of a user interface object. GUI update unit 178 updates the GUI. For example, GUI update unit 178 prepares display information and sends the display information to graphic module 132 for display on a touch-sensitive display.

[0137] In some embodiments, event processing unit(s) 190 includes or has access to data update unit 176, object update unit 177, and GUI update unit 178. In some embodiments, data update unit 176, object update unit 177, and GUI update unit 178 are included in a single module of respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0138] The above considerations regarding event processing of user touches on a touch-sensitive display also apply to other forms of user input for operating multifunctional device 100 having an input device, and it will be understood that not all of those user inputs necessarily originate on the touch screen. For example, movement of a mouse and pressing of a mouse button, contact movements such as taps, drags, scrolls on a touch pad, pen stylus input, movement of the device, spoken commands, detected eye movements, biometric input, and / or any combination thereof, optionally, are utilized as input corresponding to sub-events that define events to be recognized.

[0139] Figure 2 shows a portable multifunctional device 100 having a touch screen 112 according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, as well as in other embodiments described below, the user can select one or more of those graphics by performing gestures on the graphics using, for example, one or more fingers 202 (not drawn to scale in the figure) or one or more styli 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with the one or more graphics. In some embodiments, the gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, upward, and / or downward), and / or a roll of a finger in contact with the device 100 (from right to left, from left to right, upward, and / or downward). In some implementations or situations, an accidental contact with a graphic does not select that graphic. For example, if the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon does not optionally select the corresponding application.

[0140] The device 100 also optionally includes one or more physical buttons such as a "home" button or a menu button 204. As described above, the menu button 204 is optionally used to navigate to any application 136 within a set of applications optionally running on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on the touch screen 112.

[0141] In some embodiments, device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, volume adjustment buttons 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. Push button 206 is optionally used to turn the device on / off by pressing the button and holding it pressed for a predetermined period, to lock the device by pressing the button and releasing it before a predetermined period has elapsed, and / or to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 also accepts verbal input via microphone 113 to activate or deactivate some functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on touch screen 112 and / or one or more haptic output generators 167 for generating haptic output to the user of device 100.

[0142] FIG. 3 is a block diagram of an exemplary multi-functional device having a display and a touch sensing surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home or business controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more communication buses 320 interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touch pad 355, a haptic output generator 357 that generates haptic output on device 300 (e.g., similar to haptic output generator 167 described above with reference to FIG. 1A), and a sensor 359 (e.g., an optical, acceleration, proximity, touch sensing, and / or contact intensity sensor similar to contact intensity sensor 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 102 of portable multi-functional device 100 (FIG. 1A).Furthermore, memory 370 optionally stores additional programs, modules, and data structures that do not exist in memory 102 of the portable multifunctional device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while memory 102 of the portable multifunctional device 100 (FIG. 1A) optionally does not store these modules.

[0143] Each of the elements identified above in FIG. 3 is optionally stored in one or more of the aforementioned memory devices. Each of the modules identified above corresponds to a set of instructions that perform the aforementioned functions. The modules or programs (e.g., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules, and thus in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0144] Next, attention is directed to embodiments of a user interface that are optionally implemented, for example, on the portable multifunctional device 100.

[0145] FIG. 4A shows an exemplary user interface of a menu of an application on the portable multifunctional device 100 according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicator(s) 402 for wireless communication(s) such as cellular signal and Wi-Fi signal, ● Time 404, ● Bluetooth indicator 405, ● Battery status indicator 406, ● Tray 408 having icons of frequently used applications as follows, ○ Icon 416 of phone module 138 labeled "Phone", optionally including indicator 414 of the number of missed calls or voicemail messages, ○ Icon 418 of email client module 140 labeled "Mail", optionally including indicator 410 of the number of unread emails, ○ Icon 420 of browser module 147 labeled "Browser", and ○ Icon 422 of video and music player module 152, also referred to as iPod (trademark of Apple Inc.) module 152, labeled "iPod", and ● Icons of other applications as follows, ○ Icon 424 of IM module 141 labeled "Message", ○ Icon 426 of calendar module 148 labeled "Calendar", ○ Icon 428 of image management module 144 labeled "Photos", ○ Icon 430 of camera module 143 labeled "Camera", ○ Icon 432 of online video module 155 labeled "Online Video", ○ Icon 434 of stock widget 149-2 labeled "Stock Price", ○ Icon 436 for map module 154 labeled "Maps", ○ Icon 438 of weather widget 149-1 labeled "Weather", ○ The icon 440 of the alarm clock widget 149-4 labeled as "Clock", ○ The icon 442 of the training support module 142 labeled as "Training Support", ○ The icon 444 of the memo module 153 labeled as "Memo", and ○ An icon 446 for a settings application or module labeled "Settings" that provides access to settings for the device 100 and its various applications 136.

[0146] Note that the icon labels shown in FIG. 4A are merely illustrative. For example, other labels such as "Music" or "Music Player" may be optionally used for the icon 422 for the video and music player module 152. In some embodiments, the label for each application icon includes the name of the application corresponding to that application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0147] FIG. 4B shows an exemplary user interface on a device (e.g., device 300 of FIG. 3) having a touch sensing surface 451 (e.g., the tablet or touch pad 355 of FIG. 3) separate from the display 450 (e.g., touch screen display 112). The device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of the sensors 359) that detect the intensity of contact on the touch sensing surface 451, and / or one or more haptic output generators 357 that generate haptic output for the user of the device 300.

[0148] Some of the following examples are given with reference to inputs on a touch screen display 112 (where a touch sensing surface and a display are combined), but in some embodiments, the device detects inputs on a touch sensing surface separate from the display shown in FIG. 4B. In some embodiments, the touch sensing surface (e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) that corresponds to a primary axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact (e.g., 460 and 462 in FIG. 4B) with the touch sensing surface 451 at locations (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470) that correspond to respective locations on the display. In this way, user input (e.g., contacts 460 and 462 and their movements) detected by the device on the touch sensing surface (e.g., 451 in FIG. 4B) is used by the device to operate the user interface on the display (e.g., 450 in FIG. 4B) of the multifunctional device when the touch sensing surface is separate from the display. It should be understood that a similar method is optionally used for other user interfaces described herein.

[0149] In addition, the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures), but in some embodiments, one or more of the finger inputs are understood to be replaced by inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by the movement of a cursor along a swipe path (e.g., instead of the movement of a contact) followed by a mouse click (e.g., instead of a contact). As another example, a tap gesture is optionally replaced by a mouse click while the cursor is positioned over the location of the tap gesture (e.g., instead of detecting a contact and then stopping detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger contacts are optionally used simultaneously.

[0150] Figure 5A shows an exemplary personal electronic device 500. The device 500 includes a body 502. In some embodiments, the device 500 can include some or all of the features described with respect to devices 100 and 300 (e.g., FIGS. 1A - 4B). In some embodiments, the device 500 has a touch - sensitive display screen 504, hereinafter touch screen 504. As an alternative or addition to the touch screen 504, the device 500 has a display and a touch - sensitive surface. Similar to devices 100 and 300, in some embodiments, the touch screen 504 (or touch - sensitive surface) optionally includes one or more intensity sensors that detect the intensity of an applied contact (e.g., touch). One or more intensity sensors of the touch screen 504 (or touch - sensitive surface) can provide output data representing the intensity of the touch. The user interface of the device 500 can respond to the touch based on the intensity of the touch, which means that touches of different intensities can call different user - interface operations on the device 500.

[0151] Exemplary techniques for detecting and processing touch intensity can be found in related applications such as International Patent Application No. PCT / US2013 / 040061, filed on May 8, 2013, published as International Patent No. WO / 2013 / 169849, "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application", and International Patent Application No. PCT / US2013 / 069483, filed on November 11, 2013, published as International Patent No. WO / 2014 / 105276, "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships", each of which is incorporated herein by reference in its entirety.

[0152] In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508, if included, can be physical mechanisms. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can enable device 500 to be attached to, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, pants, belts, shoes, wallets, backpacks, etc. These attachment mechanisms enable the user to wear device 500.

[0153] FIG. 5B shows an exemplary personal electronic device 500. In some embodiments, device 500 can include some or all of the components described with respect to FIGS. 1A, 1B, and 3. Device 500 has a bus 512 that operably couples an I / O section 514 to one or more computer processors 516 and a memory 518. The I / O section 514 can be connected to a display 504, which can have a touch sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). Additionally, the I / O section 514 can be connected to a communication unit 530 that receives application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular, and / or other wireless communication techniques. The device 500 can include an input mechanism 506 and / or 508. The input mechanism 506 can optionally be, for example, a rotatable input device or a depressible and rotatable input device. In some examples, the input mechanism 508 can optionally be a button.

[0154] In some examples, the input mechanism 508 can optionally be a microphone. The personal electronic device 500 can optionally include various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or combinations thereof, all of which can be operably connected to the I / O section 514.

[0155] The memory 518 of the personal electronic device 500 can include one or more non-transitory computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 516, can cause, for example, the computer processor to execute the techniques described below, including processes 700, 900, 1100, 1300, 1500, and process 1700 (FIGS. 7, 9, 11, 13, 15, and 17). A computer-readable storage media can be any media that can tangibly contain or store computer-executable instructions for use by or in connection with an instruction execution system, apparatus, or device. In some examples, the storage media is a transitory computer-readable storage media. In some examples, the storage media is a non-transitory computer-readable storage media. Non-transitory computer-readable storage media can include, but is not limited to, magnetic, optical, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memories such as flash, solid-state drives, and the like. The personal electronic device 500 is not limited to the components and configurations of FIG. 5B and can include other or additional components in a plurality of configurations.

[0156] As used herein, the term "affordance" optionally refers to a user-interaction graphical user interface object displayed on the display screen of devices 100, 300, and / or 500 (FIGS. 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0157] As used herein, the term "focus selector" refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as the "focus selector," and thus when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 of FIG. 3 or touch-sensitive surface 451 of FIG. 4B) while the cursor is positioned over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display (e.g., touch-sensitive display system 112 of FIG. 1A or touch screen 112 of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, the detected contact on the touch screen acts as the "focus selector," and thus when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one region of the user interface to another region of the user interface without movement of the corresponding cursor or movement of the contact on the touch screen display (e.g., by using the tab key or arrow keys to move the focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of the focus between various regions of the user interface. Regardless of the particular form the focus selector takes, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user (e.g., by indicating to the device the element of the user interface with which the user intends to interact) to communicate the user's intended interaction to the user interface.For example, while a press input is detected on a touch sensing surface (e.g., a touch pad or a touch screen), the position of a focus selector (e.g., a cursor, a contact, or a selection box) over the corresponding button indicates that the user intends to activate that corresponding button (as opposed to other user interface elements shown on the device's display).

[0158] As used in this specification and the claims, the term "characteristic strength" of a contact refers to the characteristics of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected during a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined number of strength samples, i.e., a predetermined event (e.g., after detecting the contact, before detecting the lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of the maximum value of the strength of the contact, the mean value of the strength of the contact, the average value of the strength of the contact, the top 10 percentile value of the strength of the contact, the median value of the strength of the contact, the top 90 percent value of the strength of the contact, etc. In some embodiments, the duration of the contact is used when specifying the characteristic strength (e.g., when the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by a user. For example, a set of one or more strength thresholds optionally includes a first strength threshold and a second strength threshold. In this example, a contact having a characteristic strength that does not exceed the first threshold results in a first action, a contact having a characteristic strength that exceeds the first strength threshold but does not exceed the second strength threshold results in a second action, and a contact having a characteristic strength that exceeds the second threshold results in a third action. In some embodiments, the comparison between the characteristic strength and one or more thresholds is not used to determine whether to perform a first action or a second action, but rather to determine whether to perform one or more actions (e.g., whether to perform each action or whether to refrain from performing each action).

[0159] FIG. 5C shows detecting a plurality of contacts 552A - 552E on a touch - sensing display screen 504 by a plurality of intensity sensors 524A - 524D. FIG. 5C additionally includes an intensity diagram showing the current intensity measurements of the intensity sensors 524A - 524D in intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the aggregated intensity is the sum of the intensity measurements of the plurality of intensity sensors 524A - 524D, which is 32 intensity units in this example. In some embodiments, to each contact, a respective intensity that is a portion of the aggregated intensity is assigned. FIG. 5D shows assigning the aggregated intensity to contacts 552A - 552E based on the distance from the center of force 554. In this example, to each of contacts 552A, 552B, and 552E, a contact intensity of 8 intensity units of the aggregated intensity is assigned, and to each of contacts 552C and 552D, a contact intensity of 4 intensity units of the aggregated intensity is assigned. More generally, in some implementation examples, to each contact j, a respective intensity Ij that is a portion of the total intensity A is assigned according to a predetermined mathematical function Ij = A·(Dj / ΣDi), where Dj is the distance from the center of force to each respective contact j, and ΣDi is the sum of the distances from the center of force to all respective contacts (e.g., from i = 1 to the last). The operations described with reference to FIGS. 5C - 5D can be performed using an electronic device similar or identical to device 100, 300, or 500. In some embodiments, the characteristic intensity of a contact is based on one or more intensities of the contact. In some embodiments, the intensity sensors are used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). Note that the intensity diagram is included in FIGS. 5C - 5D to assist the reader and is not part of the display user interface.

[0160] In some embodiments, for the purpose of identifying the characteristic intensity, a portion of the gesture is identified. For example, the touch sensing surface optionally receives a continuous swipe contact that transitions from a starting location to reach an ending location, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending location is optionally based on only a portion of the continuous swipe contact (e.g., only the portion of the swipe contact at the ending location) rather than the entire swipe contact. In some embodiments, optionally, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of a non - weighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate small increases or decreases in the intensity of the swipe contact for the purpose of identifying the characteristic intensity.

[0161] The intensity of a contact on the touch sensing surface is optionally characterized relative to one or more intensity thresholds such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold typically corresponds to the intensity at which the device performs an operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, the deep press intensity threshold typically corresponds to the intensity at which the device performs an operation different from the operation associated with clicking a button or trackpad of a physical mouse. In some embodiments, when a contact is detected having a characteristic intensity below the light press intensity threshold (e.g., above a nominal contact detection intensity threshold below which the contact is not detected), the device moves the focus selector in accordance with the movement of the contact on the touch sensing surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent among various sets of user interface figures.

[0162] An increase in the characteristic strength of contact from a strength below the light press strength threshold to a strength between the light press strength threshold and the deep press strength threshold may be referred to as an input of "light press". An increase in the characteristic strength of contact from a strength below the deep press strength threshold to a strength above the deep press strength threshold may be referred to as an input of "deep press". An increase in the characteristic strength of contact from a strength below the contact detection strength threshold to a strength between the contact detection strength threshold and the light press strength threshold may be referred to as the detection of contact on the touch surface. A decrease in the characteristic strength of contact from a strength above the contact detection strength threshold to a strength below the contact detection strength threshold may be referred to as the detection of lift-off of contact from the touch surface. In some embodiments, the contact detection strength threshold is zero. In some embodiments, the contact detection strength threshold is greater than zero.

[0163] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes each press input, or in response to detecting each press input performed by each contact (or contacts), and each press input is detected based at least in part on detecting an increase in the strength of a contact (or contacts) above a press input strength threshold. In some embodiments, each operation is performed in response to detecting an increase in the strength of each contact above the press input strength threshold (e.g., the "downstroke" of each press input). In some embodiments, the press input includes an increase in the strength of each contact above the press input strength threshold and a subsequent decrease in the strength of the contact below the press input strength threshold, and each operation is performed in response to detecting a subsequent decrease in the strength of each contact below the press input threshold (e.g., the "upstroke" of each press input).

[0164] Figures 5E - 5H show from a strength below the light press strength threshold of Figure 5E (e.g., "IT L "), to the deep press strength threshold of Figure 5H (e.g., "IT DIt shows the detection of a gesture including a press - down input corresponding to an increase in the intensity of contact 562 to an intensity exceeding 」). The gesture performed by contact 562 is detected on the touch - sensing surface 560, and on the display user interface 570 including application icons 572A - 572D displayed within a predetermined region 574, a cursor 576 is displayed over the application icon 572B corresponding to App 2. In some embodiments, the gesture is detected on the touch - sensing display 504. The intensity sensor detects the intensity of the contact on the touch - sensing surface 560. The device determines that the intensity of contact 562 has reached a peak exceeding the intensity threshold of a deep press (e.g., 「IT D 」). Contact 562 is maintained on the touch - sensing surface 560. In response to the detection of the gesture, in accordance with contact 562 having an intensity exceeding the intensity threshold of a deep press (e.g., 「IT D 」) during the gesture, as shown in FIGS. 5F - 5H, reduced - scale representations 578A - 578C (e.g., thumbnails) of the most recently opened documents for App 2 are displayed. In some embodiments, this intensity compared to one or more intensity thresholds is the characteristic intensity of the contact. Note that the intensity diagram for contact 562 is included in FIGS. 5E - 5H to assist the reader, not as part of the display user interface.

[0165] In some embodiments, the display of representations 578A - 578C includes an animation. For example, as shown in FIG. 5F, representation 578A is first displayed in proximity to the application icon 572B. As the animation progresses, as shown in FIG. 5G, representation 578A moves upward, and representation 578B is displayed in proximity to the application icon 572B. Then, as shown in FIG. 5H, representation 578A moves upward, representation 578B moves upward towards representation 578A, and representation 578C is displayed in proximity to the application icon 572B. Representations 578A - 578C form an array over the icon 572B. In some embodiments, the animation progresses in accordance with the intensity of contact 562, as shown in FIGS. 5F - 5G, and the intensity of contact 562 exceeds the intensity threshold of a deep press (e.g., 「ITD As it increases towards ") ", expressions 578A - 578C appear and move upward. In some embodiments, the intensity based on the progression of the animation is the characteristic intensity of the contact. The operations described with reference to FIGS. 5E - 5H can be performed using an electronic device similar or identical to devices 100, 300, or 500.

[0166] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs sometimes referred to as "jitter", and the device defines or selects a hysteresis intensity threshold having a predefined relationship with the press - input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press - input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press - input intensity threshold). Thus, in some embodiments, a press - down input includes an increase in the intensity of each contact above the press - input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press - input intensity threshold, and each operation is executed in response to detecting a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (e.g., the "up - stroke" of each press - down input). Similarly, in some embodiments, a press - down input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the press - input intensity threshold and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and each operation is executed in response to detecting the press - down input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact depending on the situation).

[0167] For ease of explanation, the description of an operation performed in response to a push input associated with a push input strength threshold, or a gesture including the push input, is optionally triggered in response to detecting any one of an increase in the intensity of contact exceeding the push input strength threshold, an increase in the intensity of contact from an intensity below a hysteresis strength threshold to an intensity exceeding the push input strength threshold, a decrease in the intensity of contact below the push input strength threshold, and / or a decrease in the intensity of contact below a hysteresis strength threshold corresponding to the push input strength threshold. Further, in an example where the operation is described to be performed in response to detecting a decrease in the intensity of contact below a push input strength threshold, the operation is optionally performed in response to detecting a decrease in the intensity of contact below a hysteresis strength threshold corresponding to and lower than the push input strength threshold.

[0168] Next, attention is directed to embodiments of a user interface ("UI") and related processes implemented on an electronic device such as the portable multifunctional device 100, device 300, or device 500.

[0169] Figures 6A - 6N show exemplary user interfaces for managing camera effects, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described later, including the process in Figure 7.

[0170] FIG. 6A shows an electronic device 600 having a plurality of cameras 602 and 603 (e.g., on the back of the electronic device 600). In some embodiments, device 600 includes one or more features of devices 100, 300, and / or 500. In some examples, the electronic device (e.g., 600) has a plurality of cameras that are fixed but have different focal lengths. In some examples, the electronic device (e.g., 600) has a plurality of cameras that are fixed but have different focal lengths. In some embodiments, in addition to having different fixed focal lengths, the plurality of cameras have different fixed fields of view and different fixed optical magnification characteristics. In some embodiments, a camera (e.g., 602) captures image data using multiple focal lengths. In some embodiments, one camera (e.g., 602) captures multiple focal lengths and thus produces the same results as a plurality of cameras that are fixed but have different focal lengths. In some examples, the electronic device includes a depth camera such as an infrared camera, a thermographic camera, or a combination thereof. In some examples, the device further includes a light-emitting device (e.g., a light projector) such as an IR floodlight, a structured light projector, or a combination thereof. The light-emitting device is optionally used to illuminate an object during image capture by a visible light camera and a depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine a depth map of different portions of the object captured by the visible light camera. In some embodiments, the illumination effects described herein are displayed using inconsistent information from two cameras (e.g., two visible light cameras) for a rear-facing image and depth information from a depth camera combined with image data from a visible light camera for a forward-facing image (e.g., a selfie image). In some embodiments, the same user interface is used when determining depth information using two visible light cameras and when determining depth information using a depth camera, providing the user with a consistent experience even when dramatically different techniques are used to determine the information used to generate the illumination effect.In some embodiments, while displaying a camera user interface with one of the lighting effects applied, the device detects a selection of a camera switching affordance and switches from a forward-facing camera (e.g., a depth camera and a visible light camera) to a rear-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), and replaces (or vice versa) the display from the field of view of the forward-facing camera to the field of view of the rear-facing camera while maintaining the display of the user interface control for applying the lighting effect.

[0171] As shown in FIG. 6B, the electronic device 600 includes a touch-sensitive display 604 (e.g., a touch screen), and the display displays image data received from the camera 602. In some embodiments, the display is different from the touch sensing surface.

[0172] FIG. 6B further shows an electronic device 600 that displays a camera application user interface 606 for capturing an image by the camera 602 and / or 603 on the display 604. The camera application user interface 606 further includes a digital viewfinder 608 that includes a live preview of the field of view of the camera 602. As shown in FIG. 6B, the field of view of the camera includes a foreground object (e.g., a person) and a background object (e.g., a wall).

[0173] Furthermore, in FIG. 6B, the camera application user interface includes a filter container 610, represented as a hexagon overlapping the digital viewfinder 608. In some embodiments, the filter container (e.g., 610) is represented as a circle, triangle, or any other geometric shape. In some embodiments, the filter container (e.g., 610) is an image, icon, or textual representation that provides the user with an indication regarding the current filter. In the embodiment of FIG. 6B, the filter container (e.g., 610) is a transparent representation of an object (e.g., a hexagon) that enables various filter representations to appear to move through a transparent object. In some embodiments, the filter container can be displayed above, below, to the left, or to the right of the digital viewfinder (e.g., 608) (e.g., not overlapping the viewfinder).

[0174] The user interface of FIG. 6B further shows the filter picker user interface 612 in a folded (e.g., minimized) state. The folded filter picker user interface 612 is arranged along the edge of the digital viewfinder 608. In some embodiments, the folded filter picker user interface is optionally displayed above, below, to the left, or to the right of the digital viewfinder (e.g., 608). In some embodiments, the folded filter picker user interface (e.g., 612) includes one or more icons corresponding to a plurality of filter representations arranged in one or more rows and columns, or icons arranged in a circular orientation. In some embodiments, the folded filter picker user interface (e.g., 612) is displayed at any position corresponding to the digital viewfinder. In some embodiments, the folded filter picker user interface (e.g., 612) is outlined (e.g., bounded) to distinguish the filter picker user interface from the digital viewfinder (e.g., 608). In some embodiments, the folded filter picker user interface (e.g., 612) is translucent (or partially translucent) and has no visible boundary. As a result, in some embodiments, the boundary of the folded filter picker user interface blends (e.g., is indistinguishable) with the digital viewfinder (e.g., 608).

[0175] As shown in FIG. 6B, the folded filter picker user interface 612 includes one or more filter representations (e.g., 614A, 614B, 614C, 614D, 614E) displayed on the display 604. In some embodiments, the folded filter picker user interface (e.g., 612) optionally includes filter representations that are not displayed on the display (e.g., these are off-screen). Filter representations not displayed within the folded filter picker user interface (e.g., 612) are optionally displayed on a device that detects an input (e.g., a swipe gesture), which results in scrolling of the filter representations through the filter container (e.g., 610).

[0176] As further shown in FIG. 6B, the electronic device 600 displays a filter representation (e.g., 614A) within a filter container (e.g., 610) and indicates a filter applied to the live preview displayed within the digital viewfinder 608. In some embodiments, the filter representation 614A corresponds to a “natural light” lighting (e.g., filter) option. Accordingly, an object (e.g., a person) within the digital viewfinder 608 is displayed without applying any additional lighting effects to the image. In some examples, the filter representation (e.g., 614A) corresponds to a “studio light” lighting effect filter, a “rim light” lighting effect filter, a “stage light” lighting effect filter, or a “stage light MONO” lighting effect filter. Each of the foregoing lighting effect filters affects the visual characteristics of the image displayed within the digital viewfinder 608 (e.g., by simulating the effect of shining different sets of light on the face of the object within the digital viewfinder, based on a depth map of the face of the object). In some embodiments, the “natural light” lighting option includes modifying the image based on depth information without applying additional lighting effects, e.g., by blurring the background of the image without blurring at least a portion of the foreground of the image.

[0177] As further shown in FIG. 6B, when in the folded state, filter representations not displayed in the filter container (e.g., 614B, 614C, 614D, 614E) are displayed using different visual characteristics (e.g., shading) from the filter representations displayed in the filter container (e.g., 614A). In some embodiments, different visual characteristics include color, shape, and size. As further shown in FIG. 6B, filter representations not displayed in the filter container (e.g., 614B, 614C, 614D, 614E) are displayed at different distances from each other (e.g., gradually shorter, gradually longer) based on their positions within the list of filter representations in the folded filter picker user interface (e.g., 612). Thus, in some embodiments, within the list of filter representations (e.g., 614E), the farther away from the filter container (e.g., 610), the closer on the display the filter representation (e.g., 614D) is placed relative to other filter representations.

[0178] In some embodiments, in response to receiving an input (e.g., a tap) at a position corresponding to one of the filter representations (e.g., 614B, 614C, 614D, 614E) within the folded filter picker user interface (e.g., 612), the electronic device (e.g., 600) applies the filter corresponding to the filter representation corresponding to the input position. Thus, a user's tapping on one of the filters not in the filter container results in the filter representation scrolled to the filter container by the tap, and the electronic device 600 applies the filter corresponding to the filter representation of the filter container (e.g., 610). In some embodiments, a swipe input results in a scroll input and the filter representations scroll through the filter container. Thus, when the filters within the filter container change, the electronic device applies the filters currently within the filter container (e.g., 610).

[0179] Figures 6C-6E illustrate user interfaces showing activation of an expanded filter picker user interface. As shown in FIG. 6C, the electronic device 600 receives a tap input 616 at the location of the filter container 610. In some embodiments, the tap input exists over any region along the edge of the digital viewfinder 608. In some embodiments, the tap input exists over any region corresponding to the folded filter picker user interface 612. In some embodiments, the input corresponds to a swipe gesture, a tap and hold (e.g., tap and hold for a predetermined period) gesture, or an input having a characteristic intensity that exceeds respective intensity thresholds.

[0180] As shown in FIG. 6D, in response to receiving a tap input 616, the electronic device 600 displays a transformation to an enlarged filter picker user interface 613 of the previously folded (e.g., minimized) filter picker user interface (e.g., 612) of FIG. 6B. In some embodiments, the filter picker user interface displayed in the folded state in FIG. 6B begins to transform into an arc (e.g., wheel) shaped object in FIG. 6D. In some examples, when the enlarged filter picker user interface (e.g., 613) expands on the display (e.g., overlapping beyond the digital viewfinder), the filter picker user interface transforms into a wheel or rectangular shape. In some examples, when the enlarged filter picker user interface (e.g., 613) expands from the edge of the digital viewfinder (e.g., 608) towards the center of the display 604, the filter picker user interface (e.g., 612) overlaps the digital viewfinder 608. In some examples, the enlarged filter picker user interface (e.g., 613) is translucent, semi-translucent, transparent, or semi-transparent. In some examples, the enlarged filter picker user interface (e.g., 613) is displayed as opaque or semi-opaque. In FIG. 6D, when the enlarged filter picker user interface (e.g., 613) begins to expand, the filter container 610 shifts upward to correspond to the expanding filter picker user interface (e.g., 613).

[0181] As shown in FIG. 6E, when the enlarged filter picker user interface (e.g., 613) is fully enlarged, the filter container 610 has shifted upward to the upper part of the enlarged filter picker user interface 613. In some embodiments, additional information (e.g., 618) about the filter corresponding to the filter representation (e.g., 614A) is displayed within the filter container (e.g., 610). In some examples, the additional information is displayed within the enlarged filter picker user interface. In some examples, the additional information (e.g., 618) is displayed at a position (e.g., above, below) near the filter picker user interface. In some examples, the additional information is displayed as an icon, text, or image.

[0182] As shown in FIG. 6F, the electronic device 600 receives an input (e.g., tap 620) at a position corresponding to one representation (e.g., 614B) of the filter that is not within the filter container. In response to the tap input, in some embodiments, the electronic device applies the filter corresponding to the filter representation (e.g., 614B) corresponding to the position of the tap input on the display 604. In some examples, the input is optionally a swipe, a press and hold, or an input having a characteristic strength above each respective strength threshold. In some examples, an input having a characteristic strength above each respective strength threshold in one representation (614A - 614F) of the filter optionally results in the display of additional functionality of the corresponding filter representation associated with the position of the input having a characteristic strength above each respective strength threshold.

[0183] Figures 6G-6H show the transition of the filter container 610 as a result of the electronic device 600 receiving an input (e.g., tap 620) in Figure 6F. As shown in Figure 6G, the filter container 610 is displayed such that the filter representations (e.g., 614A-614E) appear to rotate as they scroll through the filter container 610. As shown in Figure 6G, the filter container 610 appears as a three-dimensional object (e.g., a cube) during rotation. In some embodiments, when the filter container (e.g., 610) is stationary, it is represented as a two-dimensional image (e.g., a hexagon). In some examples, the three-dimensional representation of the filter container is optionally a sphere or a cone. In some examples, the filter container appears to be two-dimensional when the filter representation moves through the filter container. As further shown in Figure 6G, the electronic device 600 gradually applies the filter corresponding to the filter representation 614B to the live preview displayed in the digital viewfinder as the filter representation scrolls through the filter container.

[0184] As further shown in Figure 6H, the filter container visually changes when a new representation of the filter (e.g., 614B) is displayed within the container. In some embodiments, the visual change of the filter container 610 is based on the type of filter within the filter container (e.g., the type of visual / lighting effect added to the live preview of the digital viewfinder). In some examples, different sides (e.g., surfaces) of the filter container (e.g., 610) reflect the visual (e.g., lighting) effect of the filter corresponding to the filter representation 614B. In some examples, the volume (e.g., the interior portion) of the filter container (e.g., 610) reflects the visual effect of the filter corresponding to the filter representation 614B.

[0185] In some embodiments, the filter representations (e.g., icons 614A - 614E) correspond to filters that the electronic device 600 simulates by applying various lighting effects corresponding to light point sources. Thus, the visual characteristics of the filter container (e.g., 610) change to simulate the light point sources of the corresponding filter. In some embodiments, the filters displayed within the filter container (e.g., 610) correspond to simulated studio light visual effects (e.g., 614B) that simulate a plurality of light point sources. As a result, the filter container is visually modified to simulate a plurality of light point sources corresponding to the studio lighting filter effect. In some embodiments, the plurality of light point sources are simulated and displayed on the three - dimensional object surface during a filter selection transition. In some embodiments, when the corresponding filter is displayed within the filter container (e.g., 610), the light point sources are displayed on the surface of a two - dimensional object (e.g., a hexagon). In some embodiments, the filter representation 614E is visually distinguishable (e.g., a different representation) when within the boundaries of the filter container compared to when outside the boundaries of the filter container (e.g., 610).

[0186] As further shown in FIGS. 6F - 6H, in some embodiments, the electronic device 600 gradually applies the filter corresponding to the newly selected filter representation 614B to the transitioning digital viewfinder. As shown in FIG. 6G, the electronic device 600 first applies the filter corresponding to the filter representation 614B to the digital viewfinder at 50% intensity. In FIG. 6H, the filter corresponding to the filter representation 614B is fully applied (e.g., 100% intensity) by the electronic device. In some embodiments, the electronic device applies the filter while increasing the intensity (e.g., 10%, 25%, 50%, 75%) during the filter transition until the transition is complete.

[0187] As shown in FIGS. 6I - 6J, an input received by the electronic device (e.g., swipe 622) causes the filter representations (614A - E) to scroll through the filter container (e.g., 610). In some embodiments, in response to a swipe gesture (e.g., 622), the filter representations being displayed cross the top of the filter picker user interface and scroll left. In some examples, one swipe gesture results in an incremental scroll of the filter representation (e.g., one filter movement). In some embodiments, the number of scrolled filter representations depends on the magnitude of the swipe gesture. Thus, in some examples, a long swipe causes a longer scroll than a short swipe. FIG. 6J shows the result of the swipe gesture in FIG. 6I where the “Stage Light MONO” illumination filter is applied to the live preview of the digital viewfinder (e.g., by simulating making the set of lights corresponding to “Stage Light MONO” shine on the face of the subject within the field of view of the camera based on the depth map of the subject's face).

[0188] In some examples, the electronic device 600 detects contact with the filter picker user interface (e.g., 613) and must continue to maintain the filter picker user interface (e.g., 613) in the expanded mode. In some examples, when the electronic device detects the lifting of the contact, the expanded filter picker user interface (e.g., 613) starts to fold and does so until it reaches the folded filter picker user interface state (e.g., 612) (e.g., until it is minimized). In some examples, the expanded filter picker user interface (e.g., 613) remains displayed in the expanded mode until it reaches a predetermined time without further contact. When the electronic device 600 detects that the predetermined time has been reached, the expanded filter picker user interface is displayed in a shrunk state until it reaches (e.g., minimizes to) the folded filter picker user interface state (e.g., 612).

[0189] In FIG. 6K, the filter picker user interface returns to the folded state (e.g., 612). As shown in FIG. 6K, the electronic device receives an input (e.g., tap 624) at a position corresponding to a position where the enlarged filter picker user interface (e.g., 613) overlaps the digital viewfinder. Since the enlarged filter picker user interface is not displayed, as shown in FIG. 6L, the electronic device interprets the input as a focus command, and a bounding box (e.g., 626) is displayed at a position corresponding to the position of the tap input (e.g., 624). As shown in FIG. 6L, the processing of the focus command occurs without any change in filter application or presentation of the filter representation.

[0190] As shown in FIG. 6M, at a position corresponding to a position where the enlarged filter picker user interface (e.g., 613) overlaps the digital viewfinder, the electronic device receives a swipe gesture 628. Since the enlarged filter picker user interface is not displayed, as shown in FIG. 6N, the electronic device interprets the swipe gesture 628 as a mode change command, and the electronic device (e.g., 600) changes the camera selection user interface (e.g., 606) to a different camera mode (e.g., "square" camera mode 630, video camera mode, non-portrait camera mode, slow motion camera mode, time lapse camera mode, or panorama camera mode).

[0191] FIGS. 7A-7F are flowcharts showing a method of varying a pseudo-illumination effect into an image data representation using an electronic device according to some embodiments. Method 900 is executed on a device (e.g., 100, 300, 500, 600) having one or more input devices (e.g., a touch sensing surface, keyboard, mouse) and a display. In some embodiments, the display is a touch sensing display. In some embodiments, the display is not a touch sensing display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0192] As described below, method 700 provides an intuitive way to vary a pseudo-illumination effect into an image data representation. This method reduces the cognitive burden on the user who provides the input corresponding to the function, thereby creating a more efficient human-machine interface. In the case of a battery-operated computing device, power is conserved and the battery charging interval is increased by enabling the user to initiate various functions faster and more efficiently.

[0193] In blocks 702-706, an electronic device (e.g., 600) simultaneously displays a camera application user interface (e.g., 606) on a display (e.g., 604), and the camera application user interface (e.g., 606) includes a digital viewfinder (e.g., 608) that includes a live preview (e.g., 602 and 603) of the field of view of one or more cameras (e.g., including a live preview or a nearly live preview image), and a representation (e.g., an image, icon, text representation indicating each filter currently applied to the live preview) of a filter picker user interface (e.g., 610, 612) that overlays the digital viewfinder (e.g., 608) (at a first position). In some embodiments, the representation of the filter picker user interface (e.g., 610) is at the first position. In some embodiments, the representation of the filter picker user interface (e.g., 610) first starts at the first position and transitions to a second position. In some embodiments, the filter picker user interface (e.g., 610) is for picking between filters that apply different lighting effects to an image based on depth map information (as described in more detail below, e.g., with reference to method 900). By simultaneously displaying the digital viewfinder (e.g., 608) and the filter picker user interface (e.g., 613), the user is provided with visual feedback about objects in the field of view of the camera (e.g., 602, 603) and the filters applicable to the preview. Providing the user with improved visual feedback improves the operability of the device and makes the interface between the user and the device more effective (e.g., by providing feedback that indicates an input that generates the intended result of the device, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further reducing power usage and improving the battery life of the device by enabling the user to use the device more quickly and effectively.

[0194] In some embodiments, in block 708-710, the representation of the filter picker user interface (e.g., 612) includes a representation of a first filter (e.g., 614A), a representation of a second filter (e.g., 614B), and a representation of a third filter (e.g., 614C), and the values of the visual characteristics (e.g., size, shadow, color) of the representation of the first filter are different from the values of the visual characteristics of the representation of the second filter and the values of the visual characteristics of the representation of the third filter. In some embodiments, the filter representations are displayed within the filter picker user interface (e.g., 612), and are displayed using visual characteristics (e.g., they are smaller, have a different shadow, have a different color) that are different from those of the first filter displayed within the representation of the filter picker user interface (e.g., 612). In some embodiments, the second and third filters have visual characteristics with the same values. By making the various representations of the filters visually different, the user is provided with feedback as to which filter is currently selected, or alternatively or additionally, the user is provided with feedback as to the type of filter effect that each filter provides. Providing the user with improved visual feedback improves the operability of the device, makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that will generate the intended result of the device, thereby assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), and thereby further enables the user to use the device more quickly and effectively, reducing power usage and improving the battery life of the device.

[0195] In some embodiments, in block 712, the representation of the filter picker user interface (e.g., 610) is a representation of a first three-dimensional object (e.g., 610) having a plurality of faces.

[0196] In blocks 714 - 716, while simultaneously displaying a digital viewfinder (e.g., 608) and a representation of a filter picker user interface (e.g., 612), an electronic device (e.g., 600) detects a first input (e.g., 616) that starts at a position (e.g., on or near an icon of a filter indicator) corresponding to each part of a live preview (e.g., a swipe, a tap and hold, a button press) via one or more input devices. In some embodiments, the input can be present at any position along the entire edge of the digital viewfinder (e.g., 608). In some embodiments, in block 718, the first input (e.g., 616) is a tap gesture.

[0197] In blocks 720 - 724, in response to detecting the first input (e.g., 616), according to a determination that a first criterion is met (e.g., a filter application criterion) that includes a requirement that the filter picker user interface overlaps each part of the live preview when the first input is detected (e.g., the filter picker user interface (e.g., 613) is in an expanded mode of operation, where the filter picker user interface (e.g., 613) extends further into the live preview than in the collapsed mode of operation), the electronic device (e.g., 600) applies a preview of a first filter to a live preview of a field of view of a camera (e.g., 602, 603) that was not applied prior to the detection of the first input (e.g., to indicate that the first filter is being applied to media captured while the first filter was the currently selected filter).

[0198] In blocks 720, and 726 - 728, in response to detecting a first input (e.g., 616), when the first input is detected, according to the determination that the filter picker user interface (e.g., 613) does not overlap with respective parts of the live preview (e.g., the filter picker user interface (e.g., 613) is in the collapsed mode of operation, where, compared to the expanded state, the filter picker user interface (e.g., 613) does not extend far into the live preview), the electronic device (e.g., 600) executes respective operations in the camera application without applying the preview of the first filter to the live preview. By performing operations based on a first criterion being met (applying a filter to the preview or performing respective operations without applying a filter), the user is provided with visual feedback (in the form of an update in the viewfinder) as to whether the capture of an image and / or the recording of a video includes a filter. Providing the user with improved visual feedback improves the device's operability and makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that causes the device to produce the intended result, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the device's battery life. When less or more access to filters is required, folding or expanding the filter user interface removes obstacles from the viewfinder, thereby providing an interface for easily switching between filters while allowing the user to perform camera operations (e.g., focus, brightness, switching camera modes) on more viewfinder elements captured in the camera's field of view.By expanding the filter user interface when the user uses it, it becomes easier for the user to operate the filter. By folding the filter user interface when the user does not use it, it becomes easier for the user to perform camera operations (such as focusing, brightness, and switching camera modes). By providing additional control while limiting the obstruction of related user interface elements, the operability of the device is improved, and the interface between the user and the device becomes more efficient (for example, by assisting the user in achieving the intended result and by reducing the user's failures during device operation / interaction with the device). As a result, it becomes possible for the user to use the device more quickly and efficiently, reducing the power consumption of the device and improving the battery life.

[0199] In some embodiments, at block 730, to perform each operation, in each respective portion of the live preview where an input is detected, at an object located within the field of view of a camera (e.g., 602, 603), selecting a focus for media capture (e.g., and optionally, updating the display of a digital viewfinder (e.g., 608) to focus the representation of each object of one or more objects within the field of view of one or more cameras (e.g., 602, 603) (e.g., as represented by box 626)) is included. By receiving a tap gesture targeting an object for focusing, an object for focus is selected and the focus is changed until the desired object is placed within focus, providing an accurate targeting mechanism to the user that avoids the need for multiple or extended inputs, thereby reducing the number of inputs required to select a focus. By reducing the number of inputs required to select a focus, the operability of the device is improved and the user interface of the device is made more effective (e.g., by providing feedback indicating an input that causes a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), which further enables the user to use the device more quickly and efficiently, reducing the power usage of the device and improving battery life.

[0200] In some embodiments, at blocks 732 - 734, a filter picker user interface (e.g., 613) is displayed and while overlapping each respective portion of the live preview (e.g., while the filter picker user interface (e.g., 613) is displayed in an expanded state) and while a first filter is applied to the live preview of the field of view of a camera (e.g., 602, 603), the electronic device (e.g., 600) detects a third input (e.g., 620) (e.g., a swipe or a tap) that begins at a position corresponding to the filter picker user interface (e.g., 613).

[0201] In some embodiments, in block 736-740, in response to detecting a third input (e.g., 620), an electronic device (e.g., 600) moves the representation of a second filter (e.g., 614B) in the representation of a plurality of filters to the position of a display (e.g., 604) corresponding to the currently selected filter, and (e.g., when different filter representations are displayed within the representation of the filter picker UI) the electronic device (e.g., 600) applies a preview of the second filter (e.g., 614B) to the live preview in the field of view (e.g., to indicate that the second filter is applied to the captured media while the second filter is the currently selected filter). By moving the representation of the selected filter to the position corresponding to the currently selected filter, visual feedback about the device state, including the filter applied (or to be applied) to the live preview, is provided to the user. Providing improved visual feedback to the user improves the operability of the device, makes the user interface of the device more effective (e.g., by providing feedback that indicates the input that causes the device to produce the intended result, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device faster and more effectively, reducing power consumption, and improving the battery life of the device.

[0202] In some embodiments, in block 742, an electronic device (e.g., 600) provides a haptic output (e.g., the haptic output is provided when switching the currently selected filter from one filter to another).

[0203] In some embodiments, in blocks 744-746, while the filter picker user interface (e.g., 613) is displayed and not overlaid on respective portions of the live preview (e.g., while the filter picker is displayed in a collapsed state), the electronic device (e.g., 600) detects a second input (e.g., 616) that begins at a location corresponding to the filter picker user interface (e.g., 612). In some examples, an input corresponding to any displayed portion of the collapsed filter picker user interface expands the interface.

[0204] In block 748, in response to detecting the second input, the electronic device (e.g., 600) expands the filter picker user interface (e.g., 613) and overlays it on respective portions of the live preview. By expanding the filter picker user interface (e.g., 613), the ability is provided to more accurately target a desired filter by spreading out the individual representations of the filters, such that the user can accurately select a desired target using, for example, less precise inputs. By providing more accurate targeting control, the operability of the device is improved, making the interface between the user and the device more effective (e.g., by providing feedback that indicates an input that causes a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power usage and improving battery life.

[0205] In some embodiments, in blocks 750-752, while the representation of the filter picker user interface (e.g., 610) is associated with the sixth of the plurality of filter representations, and while the representation of the filter picker user interface (e.g., 610) presents the first of the plurality of faces, a sixth input (e.g., 620) (e.g., a swipe or a tap) that begins at a position corresponding to the filter picker user interface (e.g., 613) is detected.

[0206] In some embodiments, in blocks 754-760, in response to detecting a sixth input (e.g., 620), the electronic device (e.g., 600) rotates the container object to present a second face of a plurality of faces that were not presented before detecting the sixth input, switches from a first filter that is the currently selected filter to a second filter that is different from the first filter and is the currently selected filter, and applies a preview of the second filter to the live preview of the field of view. In some examples, the 3D object (e.g., 610) appears as a 2D object until the animation (e.g., spin) starts. In some embodiments, if the amount of filters is greater than the amount of faces available on the 3D object (e.g., 610), one face can be used to show two or more different filters. As an example, the first face of a cube (e.g., 610) displays a first filter, and after the cube rotates four times, the first face displays a fifth filter from among the plurality of filters. By rotating the container object to present the currently selected filter, visual feedback about the device state is provided to the user, including that the applied filter is changing. Providing improved visual feedback to the user improves the device's operability and makes the user-device interface more effective (e.g., by providing feedback that indicates the input that causes the intended result to occur on the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the device's battery life.

[0207] In some embodiments, the filter picker user interface (e.g., 613) includes a representation of the filter picker user interface (e.g., 610) and representations of a plurality of filters. In some embodiments, the filter picker user interface (e.g., 613) overlaps the digital viewfinder (e.g., 608). In some embodiments, the filter picker user interface (e.g., 613) is displayed as a wheel, dial, half-dial, part of a dial, or slider. In some embodiments, the filter picker user interface (e.g., 613) is rectangular.

[0208] In some embodiments, expanding the filter picker user interface (e.g., 613) to overlay each portion of the live preview occurs without applying the filter preview to the live preview of the field of view of the camera (e.g., 602, 603) that was not applied prior to detection of a second input.

[0209] In some embodiments, the representation of the filter picker user interface (e.g., 610) when in a first state (e.g., moving, expanding) is a representation of a container object (e.g., 610) having a plurality of faces that enclose a content volume containing a representation of a second three-dimensional object. In some embodiments, the representation of the filter picker user interface (e.g., 610) appears as a 2D object (e.g., a pseudo-cube arranged to appear hexagonal) when not moving. By representing the container object as having a content volume containing a three-dimensional object when in the first state, the user is provided with feedback regarding placing a source, such as a light source, within a virtual three-dimensional environment corresponding to the previewed environment. Providing the user with improved visual feedback improves the operability of the device and makes the interface between the user and the device more effective (e.g., by providing feedback that indicates an input that generates the intended result of the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further reducing power usage and improving the battery life of the device by enabling the user to use the device more quickly and effectively.

[0210] In some embodiments, the representation of the filter picker user interface (e.g., 610) is in a second state (e.g., stationary, minimized) that is different from the first state, and the representation of the container object changes such that the shadow and / or lighting effects are removed (e.g., thereby causing the representation of the container object to appear flattened to a 2D object such as a hexagon). By changing the representation of the container object and removing the shadow and / or lighting effects, visual clutter to the user is reduced, which helps to prevent the user's attention from leaving the camera viewfinder. Further, by removing such effects when they are not needed, the number of computer processing requirements is reduced. By reducing visual clutter and reducing computer processing, the operability of the device is improved, the interface between the user and the device becomes more efficient (e.g., by assisting in achieving the results intended by the user and by reducing user errors during operation of the device / interaction with the device), and furthermore, the power consumption of the device is reduced and the battery life is improved.

[0211] In some embodiments, the representation of the filter picker user interface (e.g., 610) is in a first state (e.g., moving, expanding), and one or more of the plurality of faces, or the content volume, has a visual appearance based on the currently selected filter of the plurality of filters presented. (e.g., showing lighting on the side of a cube and indicating a change in lighting) In some embodiments, different filter representations are displayed within the representation of the filter picker UI as the cube rotates. Based on the visual appearance of the plurality of faces, or the content volume of the representation of the container object, while in the first state, within a virtual 3D environment corresponding to the previewed environment in the currently selected filter, the user is provided with feedback regarding the positioning of sources such as light sources. The user is provided with feedback regarding the state of the light source (enabled, disabled, brightness level). Providing the user with improved visual feedback improves the operability of the device and makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that causes the intended result to occur in the device, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power consumption, and improving the battery life of the device.

[0212] According to some embodiments, the representation of the filter picker user interface (e.g., 610) is in a second state different from the first state (e.g., stationary, minimized), and the visual appearance of the representation of the filter picker user interface (e.g., 610) is not based on the currently selected filter of the plurality of filters displayed within the content volume. In some embodiments, when the representation for the filter picker UI is minimized or stationary, the effect of the filter is not displayed, but is displayed when the filter picker is displayed as an extension, 3D representation.

[0213] In some embodiments, while the filter picker user interface (e.g., 613) is displayed and does not overlap with each part of the live preview (e.g., while the filter picker is displayed in a folded state), the representation of the filter picker user interface (e.g., 610) and the representation of the plurality of filters are arranged along a line substantially parallel to the edge of the display (e.g., 604). By arranging the representation of the filter picker user interface (e.g., 610) and the representation of the plurality of filters along a line substantially parallel to the edge of the display (e.g., 604), while providing the rest to the user of the available lighting effects, the visibility of the subject in the live preview is increased, and it becomes possible for the user to more easily perform camera operations (e.g., focus, brightness, switching of camera modes) on the viewfinder element. By providing additional controls while maintaining the visibility of the relevant viewfinder element, the operability of the device is improved, and the interface between the user and the device becomes more efficient (e.g., by assisting the user in achieving the intended result and by reducing user failures during device operation / interaction with the device), thereby further enabling the user to use the device more quickly and efficiently, reducing the power consumption of the device and improving the battery life. In some embodiments, the additional filter options are arranged along the edge of the display (e.g., 604) extending in both directions away from the representation of the filter picker user interface (e.g., 610), and in some embodiments, the additional filter options are arranged along the edge of the display (e.g., 604) extending in one direction from the representation of the filter picker user interface (e.g., 610). In some embodiments, the representation of the filter picker user interface (e.g., 610) is displayed near the edge of the live preview.

[0214] In some embodiments, while the filter picker user interface (e.g., 613) overlaps each part of the live preview (while the filter picker user interface (e.g., 613) is displayed in an expanded state), the representation of the filter picker user interface (e.g., 610) and the representation of the plurality of filters are arranged along a curve (e.g., a line not parallel to the edge of the display (e.g., 604)). In some embodiments, the shifted location appears as an arc or a wheel. In some embodiments, the bi-directional action includes using a gesture (swipe or tap) along any location on the bottom edge of the display (e.g., 604). Representing the shifted location as an arc or a wheel and bi-directionally acting with the arc or wheel by using a gesture (swipe or tap) along the bottom edge of the display (e.g., 604) provides continuity in the user interface that reduces user confusion, such that the user provides less input and performs the desired operation. By reducing the number of inputs required to perform an operation, the operability of the device is improved, and the interface between the user and the device becomes more efficient (e.g., by assisting the user in achieving the intended result and reducing user failures during device operation / interaction with the device), which in turn enables the user to use the device more quickly and efficiently, reducing the device's power consumption and improving battery life.

[0215] In some embodiments, the first input is a tap gesture, and the position corresponding to each part of the live preview is the position corresponding to the representation of the first filter in the representation of the plurality of filters. In some embodiments, the expanded filter picker UI is displayed in an expanded state. In some embodiments, the filter picker user interface (e.g., 613) is displayed in a collapsed state.

[0216] In some embodiments, the representation of the second filter is in a first direction from the representation of the first filter, the representation of the third filter is in a first direction from the representation of the second filter, and the value of the visual characteristic changes gradually (e.g., monotonically increases or monotonically decreases) in the first direction from the representation of the first filter to the representation of the second filter and then to the representation of the third filter (e.g., the opacity of the filter representation gradually becomes smaller and / or decreases as it moves farther from the representation of the currently selected filter). The value of the visual characteristic that changes gradually in the first direction from the representation of the first filter to the representation of the second filter and then to the representation of the third filter provides a structured system that can reduce user distraction, improve the visibility of the live preview, and facilitate identification and access to the filters. By providing improved visual feedback, the operability of the device is improved, and the interface between the user and the device becomes more efficient (e.g., by assisting the user in achieving the intended result and reducing user errors during device operation / interaction with the device), thereby enabling the user to use the device more quickly and efficiently, reducing the power consumption of the device, and improving the battery life. In some embodiments, the filter representations are displayed within a filter picker user interface (e.g., 613) and use different visual characteristics based on their respective corresponding filter representations, which are from the representation of the filter picker user interface (e.g., 610).

[0217] In some embodiments, the filter picker user interface (e.g., 613) overlaps each part of the live preview (the filter picker user interface (e.g., 613) is displayed in an expanded state), and the filter picker user interface (e.g., 613) includes additional information (e.g., display of the filter name) about the first filter that is displayed in association with the representation of the first filter.

[0218] According to some embodiments, the first input is a swipe gesture (e.g., 628), and performing each operation includes changing the camera capture mode of an electronic device (e.g., 600) (e.g., the mode changes to any of video, photo, portrait, rectangle, panorama, slow motion, time lapse).

[0219] In some embodiments, applying the preview of the second filter includes gradually transitioning between applying the preview of the first filter and applying the preview of the second filter. In some embodiments, gradually transitioning includes 100% of the first at the first time, 0% of the second, 90% of the first and 10% of the second at the second time, and so on.

[0220] Note that the details of the processes described above with respect to method 700 (e.g., FIGS. 7A - 7F) are also applicable in a similar manner to the methods described later. For example, methods 900, 1100, 1300, 1500, and 1700 optionally include one or more of the various method characteristics described above with respect to method 700. For example, filter user interfaces, affordances, and control elements can be combined from among the various methods. As another example, the viewfinder in method 700 is similar to the viewfinders in methods 900, 1100, 1300, 1500, 1700. For the sake of brevity, these details are not repeated below.

[0221] In some embodiments, electronic device 800 includes some or all of the features of device 600, as shown in FIG. 6A. In some embodiments, device 800 includes a plurality of cameras 602 and 603 (e.g., on the back surface of electronic device 800). In some embodiments, device 800 includes one or more features of devices 100, 300, and / or 500. In some examples, an electronic device (e.g., 800) has a plurality of cameras that are fixed but have different focal lengths. In some examples, the plurality of cameras are on the front, back, or both sides of the electronic device (e.g., 800). In some embodiments, in addition to having different fixed focal lengths, the plurality of cameras have different fixed fields of view and different fixed optical magnification characteristics. In some embodiments, a camera (e.g., 602) captures image data using a plurality of focal lengths. In some embodiments, one camera (e.g., 602) captures a plurality of focal lengths and thus produces the same results as a plurality of cameras that are fixed but have different focal lengths. In some examples, an electronic device (e.g., 800) includes a depth camera such as an infrared camera, a self-thermometer camera, or a combination thereof. In some examples, the device further includes a light-emitting device (e.g., a light projector) such as an IR floodlight, a structured light projector, or a combination thereof. The light-emitting device is optionally used to illuminate an object during image capture by a visible light camera and a depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine a depth map of different portions of the object captured by the visible light camera. In some embodiments, the illumination effects described herein are displayed using inconsistent information from two cameras (e.g., two visible light cameras) for a rear-facing image and depth information from a depth camera combined with image data from a visible light camera for a forward-facing image (e.g., a selfie image).In some embodiments, the same user interface is used when determining depth information using two visible light cameras and when determining depth information using a depth camera, and a consistent experience is provided to the user even when dramatically different techniques are used to determine the information used to generate the lighting effect. In some embodiments, while displaying the camera user interface with one of the lighting effects applied, the device detects a selection of a camera switching affordance and switches from a forward-facing camera (e.g., a depth camera and a visible light camera) to a rear-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), and replaces (or vice versa) the display from the field of view of the forward-facing camera to the field of view of the rear-facing camera while maintaining the display of the user interface control for applying the lighting effect.

[0222] As shown in FIG. 8A, the electronic device 800 includes a touch-sensitive display 804 (e.g., a touch screen), and the display displays image data received from a camera (e.g., 602). In some embodiments, the display is different from the touch sensing surface. In some examples, a plurality of cameras (e.g., 602 and 603) are disposed on the front, back, or both sides of the electronic device (e.g., 800).

[0223] FIG. 8A further shows an electronic device 800 that displays a camera application user interface 805 for capturing an image by a camera (e.g., 602) on a display 804. The camera application user interface 805 further includes a digital viewfinder 810 that includes a live preview of the field of view of the camera (e.g., 602 or 603). In some embodiments, the camera captures depth information associated with the image data in real time. FIG. 8A further shows a camera that captures different depth levels in the field of view, including a subject (e.g., a woman) in the foreground region (e.g., 808) and a fence (e.g., 811) in the background region. In some examples, the camera (e.g., 602) captures 3, 4, 5, 10, 20, or more depth levels in the field of view. The electronic device 800 utilizes the various depth levels when applying a filter to the representation of the image data (e.g., 806) displayed within the digital viewfinder (e.g., 810), as discussed in more detail below.

[0224] Further, FIG. 8A shows an electronic device 800 that displays a filter picker user interface 813 in an expanded state. The filter picker user interface 813 is disposed along the edge of the digital viewfinder 810. In some examples, the filter picker user interface (e.g., 813) is optionally displayed above, below, to the left, or to the right of the digital viewfinder (e.g., 810). In some examples, the filter picker user interface (e.g., 813) includes one or more representations of filters (e.g., representations of visual effects) arranged in one or more rows and columns or in a circular orientation.

[0225] In some embodiments, the filter picker user interface (e.g., 813) is displayed at any position corresponding to the digital viewfinder. In some embodiments, the filter picker user interface (e.g., 813) is outlined (e.g., bounded) to distinguish the filter picker user interface from the digital viewfinder. In some embodiments, when in the folded state, the filter picker user interface (e.g., 813) is translucent (or partially translucent) and has no visible boundary. As a result, in some embodiments, the filter picker user interface (e.g., 813) appears to blend in with (e.g., be indistinguishable from) the digital viewfinder.

[0226] As shown in FIG. 8A, the electronic device 800 displays a filter picker user interface 813 that includes one or more filter representations (e.g., 814A, 814B, 814C, 814D, 814E) corresponding to visual effects. In some embodiments, the filter picker user interface (e.g., 813) optionally includes filter representations that are not displayed on the display (e.g., they are off-screen). In some embodiments, the non-displayed filter representations are displayed when the electronic device receives an input (e.g., a swipe gesture), thereby causing the filter representations to scroll through a filter container (e.g., 816).

[0227] As further shown in FIG. 8A, in some embodiments, the electronic device 800 displays a filter representation (e.g., 814A) within a filter picker user interface (e.g., 813) to indicate the currently selected visual effect. In some embodiments, the filter representation 814A corresponds to a "natural light" illumination effect filter. Accordingly, the foreground region 808 and the background region 809 are displayed using the "natural light" illumination effect filter (e.g., using natural light from the scene). Since the image representation of FIG. 8A is shown without using any synthetic light, the natural light from the scene creates various shadows on the objects (e.g., face, neck, and clothing). In some examples, possible filter representations (e.g., 814A - 814E) corresponding to illumination effects include a "studio light" illumination effect, a "rim light" illumination effect, a "stage light" illumination effect, and a "stage light MONO" illumination effect. When added to the representation of the image data (e.g., 806), each of the preceding illumination effects affects the visual characteristics of the representation of the image data displayed on the display 804.

[0228] In some embodiments, when the electronic device 800 applies a natural lighting effect, no synthetic lighting is added to the image (e.g., the original image is displayed). In contrast, a studio lighting effect includes modeling a plurality of discrete point light sources (e.g., light within a photo studio) evenly arranged around the object (e.g., creating a shiny full illumination effect). A rim lighting effect includes modeling a plurality of discrete point light sources arranged along the perimeter of the object (e.g., creating a slimming effect, creating shadows on the side of the object's face and / or on the object's jaw). A stage light illumination effect includes modeling one discrete point light source placed on the object (e.g., creating a spotlight effect). A stage light MONO illumination effect includes modeling one discrete point light source placed on the object in monochrome (e.g., creating a monochrome spotlight effect).

[0229] In some embodiments, an electronic device (e.g., 800) detects a face of an object in a representation of image data. Accordingly, when applying an illumination effect, the electronic device uses depth map information of the image data and corresponding face features. As a result, the illumination effect is applied more accurately around the face of the object, and specific face features are illuminated differently based on the selected illumination effect (e.g., an increase or decrease in shading around the object's jaw and / or cheekbones). In some embodiments, the image data includes depth map information including depth contours of the object. As a result, the electronic device uses the contour data to apply the illumination effect more accurately around the object.

[0230] As shown in FIG. 8B, the electronic device 800 receives an input (e.g., a swipe 818) at a position corresponding to a filter picker user interface (e.g., 813). As shown in FIG. 8C, the input (e.g., the swipe 818) scrolls a filter representation (814A - E) through a filter container (e.g., 816). In some embodiments, in response to the input (e.g., the swipe 818), the filter representation scrolls left across the upper boundary of the filter picker user interface (e.g., 813). In some embodiments, one swipe gesture results in an incremental scroll of the filter representation. In some embodiments, the number of scrolled filter representations depends on the magnitude of the swipe gesture. Accordingly, in some embodiments, a long swipe causes a longer scroll than a short swipe.

[0231] In response to an input (e.g., an 818 swipe), in some embodiments, the electronic device 800 applies an illumination effect corresponding to a filter representation (e.g., 814B) corresponding to a position on the display 804 of the tap input. In some examples, the input is a swipe, a press and hold, or an input having a characteristic intensity above respective intensity thresholds. In some examples, an input having a characteristic intensity above a respective intensity threshold detected in one representation of a filter (814A - 814F) optionally results in a display of additional functionality for the corresponding filter representation associated with the position of the input having a characteristic intensity above the respective intensity threshold.

[0232] In some embodiments, an additional visual effect may be applied to the entire representation of the image data (e.g., 806) before applying the illumination effect corresponding to the filter representation (e.g., 814B). For example, applying a slight gradient filter to the representation of the image data (e.g., 806) before applying a stage lighting filter enables a smoother transition from no filter to the illumination effect filter.

[0233] Figures 8C-8D show an electronic device 800 that gradually applies an illumination effect as a result of the electronic device receiving an input (e.g., a swipe 818) in Figure 8B. In some embodiments, the illumination effect corresponding to the newly selected filter representation 814B is gradually applied to the representation of the image data (e.g., 806) in the live preview. Since the selected illumination effect is "studio light", the corresponding visual effect simulates a plurality of point light sources that affect the object within the foreground region 808. As a result, during the transition phase (Figure 8C), the illumination effect corresponding to the filter representation 814B is applied to the live preview at 50% intensity. The filter corresponding to the filter representation 814B is fully applied (e.g., 100%) in Figure 8D. In some examples, the filter is gradually applied (10%, 25%, 50%, 75%) while the electronic device 800 applies the illumination effect until the transition is complete. In some examples, when the electronic device 800 applies the "studio light" light effect to the representation of the image data, the background region (e.g., 809) is completely darkened. In some embodiments, when the electronic device 800 applies the "studio light" light effect to the representation of the image data, the background region (e.g., 809) is completely darkened.

[0234] As shown in Figures 8C-8D, since the image data captured by the camera (e.g., 602) includes depth map information associated with the image data, the electronic device can use the available depth map information and simulate the effects of various point light sources in the representation of the image data 806. In some embodiments, the same illumination effect is applied differently in the background region compared to the foreground region based on the depth map information associated with the image data. As a result, foreground objects can appear more prominent, and background objects can become less prominent due to the darkening effect. Further, as shown in Figure 8D, since the illumination effect simulates point light sources, the depth map information is used to cast various shadows on the face of the object within the foreground region (e.g., 808). As shown in Figure 8D, since the "studio light" illumination effect simulates a plurality of point light sources, the electronic device 800 uses the depth map information to reduce the shadows on the face of the object compared to the "natural light" illumination effect.

[0235] As shown in FIGS. 8E-8F, the electronic device 800 receives an input (e.g., tap 820) and scrolls the representation of the filters (814A-E) through a filter container (e.g., 816). In some embodiments, in response to a tap gesture (e.g., 820) corresponding to a filter representation (e.g., 814E), the representation of the filter scrolls left across the upper periphery of the filter picker user interface (e.g., 813). FIG. 8F shows the result of the tap gesture (e.g., 820) of FIG. 8E where the electronic device 800 applies the “Stage Light MONO” filter to the digital viewfinder. The “Stage Light MONO” lighting effect simulates a single point light source and consequently resembles a spotlight effect. Using depth map information, from above an object in the foreground region (e.g., 808), the electronic device 800 applies the “Stage Light MONO” effect. In some examples, the point light source may be simulated and start from any direction. In some examples, the “Stage Light MONO” effect is simulated starting from the front, and as a result, a particular focus (e.g., a face) is emphasized while the remainder of the representation of the image data is darkened. As shown in FIG. 8F, since the simulated point light source starts from above the object, the electronic device can cast deeper shadows on the object (e.g., the face and neck) using the depth map information of the image data. In some embodiments, the background of the image is removed and replaced with a solid color such as black or white, or a color selected by the user, to further draw attention to the object in the foreground and simulate a studio setting where the user can take a photo against a unified background.

[0236] As shown in 8G, the electronic device may not be in a suitable state to capture depth information. FIG. 8G shows a user operating the electronic device 800 after taking a few steps down from the position where the user was when operating the electronic device in FIG. 8F. As a result of the user stepping back, the electronic device can no longer capture depth information (e.g., the conditions for the camera to capture depth information are no longer detected). In some embodiments, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect gradually fades out. In some examples, when the conditions for capturing the depth effect are no longer met, the previously applied lighting effect disappears (e.g., the device returns to a state without a lighting filter that was applied without a transition). In some embodiments, optionally, a temporary filter (e.g., a tilt), which is part of the lighting effect filter, is applied to the image representation when the filter disappears. The temporary filter helps to smooth (e.g., make more harmonious) the transition when the lighting effect is applied again.

[0237] In some embodiments, when the electronic device does not detect the conditions necessary to capture depth map data, the electronic device 800 displays a graphic indication (e.g., 822) to show the user what actions the electronic device (e.g., 800) should take to capture depth map information. In some examples, the electronic device detects an object, but the object is too far away (e.g., the focus is between 2.5 m and 10 m), and the electronic device instructs the user (e.g., using a graphic indication) to move closer to the camera (e.g., within 8 feet). In some examples, the electronic device determines that the light level is too low (e.g., below 400 lux) and instructs the user (e.g., using a graphic indication) to provide more light. In some examples, affordances are displayed in the viewfinder so that the user can enable or disable such instructions. In some embodiments, when the conditions for capturing the depth effect map are met, the electronic device 800 stops displaying the graphic indication that instructs the user. Thus, in some embodiments, when the user's actions are not useful for applying the lighting effect, the electronic device 800 does not instruct the user.

[0238] As shown in FIG. 8H, when the user takes a few steps forward and the electronic device detects again the conditions necessary to capture depth map information, the lighting effect disappears (e.g., without transition). In some embodiments, when the conditions for capturing depth map information are met again, the electronic device 800 gradually applies the lighting effect to the representation of the image data.

[0239] As shown in FIG. 8I, the electronic device detects an input (e.g., a tap 824) at a position corresponding to the photo viewer application (e.g., 826). In response to receiving the input (e.g., 826), as shown in FIG. 8J, the electronic device 800 switches to the image viewing mode (e.g., a mode for viewing a previously captured image rather than a live preview of camera data).

[0240] FIG. 8J shows a user interface for a photo viewer application. The photo viewer application includes a thumbnail strip of previously captured images (e.g., 828A - 828D) where 828D is the most recent captured image. In some embodiments, the previously captured images were captured using a camera corresponding to an electronic device (e.g., 800). In some embodiments, the electronic device (e.g., 800) receives the previously captured images (e.g., 828A - 828D) from a remote source (e.g., a server), and optionally, the previously captured images were captured by a different electronic device (e.g., not 800).

[0241] FIG. 8J further shows that the last captured image (e.g., 828D) is captured in combination with visual effects (e.g., a simulated shallow depth of field photographic effect (bokeh) 830 where the background is blurred, part of the foreground is not blurred, and the foreground is in the plane of focus, and an illumination effect 832). In some embodiments, the electronic device displays a "portrait" visual indicator (e.g., 834) at the top of the display as an indication to the user that previously captured image data includes depth map information. In some examples, the electronic device (e.g., 800) receives an input at a position corresponding to the visual indicator and toggles the on / off of the simulated depth effect (e.g., bokeh) effect. In some embodiments, when the simulated depth effect is toggled off, the illumination effect remains the same. In some examples, when the visual indicator is activated, it toggles both the simulated depth effect and the illumination effect. In some examples, the electronic device optionally receives an input and uses the filter picker user interface (described above) to vary the illumination effect within the photo viewer application to a different illumination effect. In some examples, when previously captured image data does not have depth map information, the electronic device (e.g., 800) does not provide the option to apply the simulated depth effect or the illumination effect. In some examples, when previously captured image data does not have depth map information associated with the image data, the electronic device does not display the visual indicator (e.g., 834).

[0242] In some examples, the electronic device (e.g., 800) stores the depth map information and the image data in one file. In some examples, the electronic device (e.g., 800) stores the depth map information separately from the image data. In some embodiments, when the electronic device stores an image as a flat image (e.g., without depth map information) along with the depth map information, the electronic device (e.g., 800) is unable to apply the illumination effect to the representation of the image data.

[0243] Figures 9A - 9D are flowcharts showing a method for applying a pseudo - illumination effect to the representation of image data using an electronic device according to some embodiments. Method 900 is executed on a device (e.g., 100, 300, 500, 800) comprising one or more input devices (e.g., a touch - sensitive surface, a keyboard, a mouse) and a display. In some embodiments, the display is a touch - sensitive display. In some embodiments, the display is not a touch - sensitive display. In some embodiments, the electronic device includes a plurality of cameras. In some embodiments, the electronic device has only one camera. Some operations of method 900 may optionally be combined, the order of some operations may optionally be changed, and some operations may optionally be omitted.

[0244] As described below, method 900 provides an intuitive method for applying a pseudo - illumination effect to the representation of image data. This method reduces the cognitive burden on the user who provides inputs corresponding to functions, thereby creating a more efficient human - machine interface. In the case of a battery - operated computing device, power is conserved and the battery - charging interval is increased by enabling the user to initiate various functions more quickly and efficiently.

[0245] In some embodiments, at block 902, before displaying a representation of the image data (e.g., 806), the electronic device (e.g., 800) receives the image data and depth map information associated with the image data at the device (e.g., from a camera, from memory, from a server). In some embodiments the image data includes RGB and depth map values. In some embodiments, the image data and depth map are received from a source external to the electronic device (e.g., 800) (e.g., the data is received from a server). In some embodiments, the image is stored in a file format that allows separation of the depth information (e.g., depth map) and RGB data within one file. In some embodiments, the image data includes depth map information. In some embodiments, the depth map information and the depth map information are separate. By receiving the image data and the depth map information corresponding to the image data before displaying a representation of the image data (e.g., 806), the device can provide visual feedback to the user about the content of the depth map information, such as whether an item is in the background (e.g., 809) or foreground (e.g., 808) of the scene, via the representation of the image data (e.g., 806). Providing improved visual feedback to the user improves the operability of the device and makes the user interface of the device more effective (e.g., by providing feedback that indicates an input that causes a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further allowing the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device.

[0246] In block 904, the electronic device (e.g., 800) displays a representation of the image data associated with the depth map information on a display (e.g., 804). In some embodiments, an image or photograph is displayed on the device's display (e.g., 804). In some embodiments, a live preview of the image data is displayed within a digital viewfinder (e.g., 810). Displaying a live preview of the image data within the digital viewfinder (e.g., 810) enables the user to quickly and efficiently frame a photograph without having to repeatedly capture the photograph, thereby reducing the number of inputs required to capture the desired photograph, reducing the memory requirements for storing the photograph, and making the user interface more efficient. Reducing the number of inputs required to capture the desired image and reducing the memory requirements improves the device's operability and makes the user interface with the device more efficient (e.g., by assisting the user in achieving the intended result and reducing user errors during device operation / interaction with the device), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power consumption, and improving battery life.

[0247] In some embodiments, in block 906, the electronic device (e.g., 800) further includes one or more cameras (e.g., 602 and / or 603), and the representation of the image data (e.g., 806) is a live preview of the image data captured within the field of view of one or more cameras (e.g., 602 and / or 603) that is displayed in the digital viewfinder (e.g., 810). In some embodiments, the device includes a plurality of cameras (e.g., 602 and / or 603) having different focal lengths.

[0248] In some embodiments, at block 908, the depth map information associated with the image data includes information corresponding to at least three different depth levels. For example, the image data includes information corresponding to at least a background depth level, a foreground depth level, and an intermediate depth level. Depth map information including three or more different depth levels provides a framework for the user to apply depth-specific filters and provides the user with more accurate feedback regarding the depth positioning of objects in the field of view of the camera(s) (e.g., 602 and / or 603). Providing the user with improved visual feedback improves the operability of the device, makes the user interface of the device more effective (e.g., by providing feedback that indicates an input that generates a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device.

[0249] In some embodiments, at block 910, the depth map information associated with the image data includes an object's information-specific depth contour in the representation of the image data (e.g., 806).

[0250] At block 912, while displaying a representation of the image data (e.g., 806) on the display (e.g., 804), the electronic device (e.g., 800) optionally performs the techniques of blocks 914-936. At block 914, the electronic device (e.g., 800) detects a first input (e.g., 818) (e.g., a swipe, tap and hold, tap, button press; the gesture can be on top of an icon representing a lighting filter, or can be another user interface system used to select a filter) via one or more input devices.

[0251] In some embodiments, at block 916, the first input (e.g., 818) is an input received while a first criterion (e.g., a series of lighting effect application criteria) is met, where the first criterion meets the requirement that an object is detected in the field of view within a predetermined distance from the electronic device (e.g., 800) (e.g., as a series of other conditions, the focal length of the first camera (e.g., 602 or 603) exceeds a minimum distance threshold, the focal length of the first camera (e.g., 602 or 603) does not exceed a maximum distance threshold, the object is detected beyond a predetermined minimum distance from the device, the amount of detected light exceeds a minimum light threshold, the amount of detected light does not exceed a maximum light threshold). In some embodiments, if the first criterion is not met, applying the first lighting effect or the second lighting effect is aborted. By applying the lighting effect when it is determined that there is an object within a predetermined distance when the first input (e.g., 818) is received, the user is provided with visual feedback that the optimal (or near-optimal) effect can be achieved by the filter with the object being properly positioned. Similarly, when the object is not within the predetermined distance, by not applying the lighting effect, the user is provided with feedback that the object is not properly positioned, indicating to the user that corrective action is required. Providing the user with improved visual feedback improves the operability of the device, makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that causes the intended result to occur on the device, assisting the user in achieving the intended result, and reducing the user's errors when operating or interacting with the device), thereby further enabling the user to use the device faster and more effectively, reducing power consumption, and improving the battery life of the device.

[0252] In some embodiments, at block 918, the first input (e.g., 818) is an input received while a first criterion (e.g., a series of lighting effect application criteria) is not met, where the first criterion includes requirements that are met when an object is detected in a field of view within a predetermined distance from an electronic device (e.g., 800) (e.g., as a series of other conditions / criteria, the focal length of the first camera (e.g., 602 or 603) exceeds a minimum distance threshold, the focal length of the first camera (e.g., 602 or 603) does not exceed a maximum distance threshold, the object is detected beyond a predetermined minimum distance from the device, the amount of detected light exceeds a minimum light threshold, the amount of detected light does not exceed a maximum light threshold). In some embodiments, if the first criterion is not met, the electronic device (e.g., 800) discontinues applying the first lighting effect or the second lighting effect.

[0253] In some embodiments, at block 920, in response to a first input (e.g., 818), the electronic device (e.g., 800) applies a placeholder filter to the live preview (e.g., blurs the background (e.g., 809) or reduces the saturation of the background) without applying a first lighting effect to the live preview. In some embodiments, in response to detecting that a first criterion is met, the electronic device (e.g., 800) applies the first lighting effect to the live preview while continuing to apply the placeholder filter to the live preview (e.g., the placeholder filter is part of the first lighting effect that does not take into account depth map information and is therefore displayed regardless of whether the first criterion is met). By applying the placeholder filter regardless of whether the first criterion is met, the user is provided with visual feedback regarding the depth map related to the viewfinder (e.g., 810) content, e.g., which portions of the image correspond to portions of the depth map in the background (e.g., 809) compared to the foreground (e.g., 808). Providing the user with improved visual feedback improves the operability of the device, makes the user interface of the device more effective (e.g., by providing feedback that indicates the input that causes the intended result of the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further allowing the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device.

[0254] In some embodiments, at block 922, after displaying the live preview without applying the first lighting effect, the electronic device (e.g., 800) detects that the first criterion is met. In response to detecting that the first criterion is met, the electronic device (e.g., 800) applies the first lighting effect to the live preview. By applying the lighting effect when the first criterion is met, the user is provided with visual feedback that the first criterion is met (e.g., the subject is properly positioned) and that an optimal (or near-optimal) effect can be achieved using the filter. Similarly, by not applying the lighting effect when the first criterion is not met, the user is provided with feedback that the first criterion is not met, indicating to the user that corrective action is needed. Providing the user with improved visual feedback improves the device's operability and makes the user-device interface more effective (e.g., by providing feedback that instructs the input that causes the intended result to occur on the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further reducing power usage and improving the device's battery life by enabling the user to use the device more quickly and effectively.

[0255] In block 924, in accordance with detecting a first input (e.g., 818), an electronic device (e.g., 800) applies a first lighting effect (e.g., natural light, studio light, contour light, stage light, stage light MONO) to the representation of the image data (e.g., 806), where the first lighting effect is based on depth map information (e.g., based on measurements of a depth sensor or based on differences in mapping of two images taken simultaneously at different positions). By displaying a lighting effect based on depth map information, visual feedback about the depth map information is provided to the user. For example, by placing (or emphasizing) shadows and removing (or thinning) them, feedback about a particular orientation of an object corresponding to the depth map information is provided to the user. Providing improved visual feedback to the user improves the operability of the device, makes the interface between the user and the device more effective (e.g., by providing feedback that indicates an input that will produce an intended result of the device, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and effectively.

[0256] In some embodiments, in block 926, applying the first lighting effect includes applying a placeholder filter to the representation of the image data (e.g., 806) displayed on the digital viewfinder (e.g., 810) (e.g., blurring the background (e.g., 809) or reducing the saturation of the background), where the placeholder filter is based on (e.g., selected by) the first lighting effect and is applied regardless of whether the first criterion is met. By applying the placeholder filter, a smoother / more comfortable transition to the lighting filter is made. By applying the placeholder filter regardless of whether the first criterion is met, the user is provided with visual feedback about the depth map related to the viewfinder (e.g., 810) content, e.g., which parts of the image correspond to portions of the depth map in the background (e.g., 809) compared to the foreground (e.g., 808). Providing the user with improved visual feedback improves the operability of the device and makes the user-interface of the device more effective (e.g., by providing feedback that indicates the input that causes the intended result of the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power consumption, and improving the battery life of the device.

[0257] In some embodiments, at block 928, applying the first lighting effect includes applying a simulation of one or more point light sources spatially to the representation of the image data (e.g., 806) displayed on the viewfinder (e.g., 810) based on depth map information associated with the image data. Lighting options include natural light, studio light, contour light, stage light, and stage light MONO. Each of the lighting effects models (e.g., simulates) the result of one or more points of light sources in the space based on the depth map of the image data. The natural lighting option does not add synthetic lighting to the image (e.g., the original image is displayed, or a portion of the original image is displayed and blurring is applied to different portions of the original effect to simulate a blurring effect). The studio lighting effect includes modeling multiple different points of light sources arranged around the subject (e.g., creating a glowing fill of the lighting effect). The contour lighting effect includes modeling multiple different points of light sources arranged at fewer points around the subject and creating shadows on the subject's face (e.g., creating a slimming effect, creating shadows on the sides of the subject's face, and / or on the subject's jaw). The stage light lighting effect includes modeling one different point light source placed on the subject (e.g., creating a spotlight effect). The stage light MONO lighting effect includes modeling one different point light source placed around on the subject in monochrome (e.g., creating a monochrome spotlight effect). In some embodiments, the lighting filter simulates a point light source. In some embodiments, as detailed above, initially (e.g., when the lighting effect application criteria are met), the lighting effect disappears. In some embodiments, when the system detects a face, facial features are considered when applying the lighting effect. As a result, based on the specific facial features and facial shape of the subject, the lighting effect changes the appearance of the representation of the image data (e.g., 806). By applying the simulation of the point light source, the user is provided with a visual representation of the content of the depth map information, enabling the device to provide the user with visual feedback about the shape and depth positioning of objects in the field of view of the camera(s) (e.g., 602 and / or 603).Providing improved visual feedback to a user improves the operability of a device and makes the user interface between the user and the device more effective (e.g., by providing feedback that indicates an input that causes a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device. Further, by applying a simulation of a point light source without the need for actual physical studio light, the electronic device is less expensive and smaller than when actual studio light and a background are required, increasing the portability of the device and reducing manufacturing costs.

[0258] In block 930, an electronic device (e.g., 800) detects a second input (e.g., a swipe, tap and hold, tap; button press, gesture that can be above an icon representing an illumination filter (e.g., 814A) or another user interface used to select a filter) via one or more input devices.

[0259] In block 932, in accordance with detecting a second input, the electronic device (e.g., 800) adds a second lighting effect (e.g., natural light, studio light, rim light, stage light, stage light MONO) different from the first lighting effect to the representation of the image data (e.g., 806), where the second lighting effect is based on depth map information (e.g., based on measurements of a depth sensor or based on differences in mapping of two images taken simultaneously at different positions). By displaying the second lighting effect based on the depth map information, additional visual feedback about the depth map information is provided to the user. For example, the second lighting effect can include one or more light sources at different positions, intensities, or types (directional, ambient, point) that provide the user with feedback about a special orientation of an object corresponding to the depth map information. Providing improved visual feedback to the user improves the operability of the device and makes the user interface of the device more effective (e.g., by providing feedback that indicates an input that causes a result intended for the device, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further reducing power usage and improving the battery life of the device by enabling the user to use the device more quickly and effectively.

[0260] In some embodiments, in block 934, applying the second lighting effect includes applying a simulation of one or more point light sources spatially to the representation of the image data (e.g., 806) displayed on the digital viewfinder (e.g., 810) based on depth map information associated with the image data. Lighting options include natural light, studio light, contour light, stage light, and stage light MONO. Each of the lighting effects models (e.g., simulates) the result of one or more points of the light sources in the space based on the depth map of the image data. The natural lighting option does not add synthetic lighting to the image (e.g., the original image is displayed, or a portion of the original image is displayed and blurring is applied to different portions of the original effect to simulate a blurring effect). The studio lighting effect includes modeling multiple different points of light sources arranged around the subject (e.g., creating a glowing fill of the light effect). The contour lighting effect includes modeling multiple different points of light sources arranged at fewer points around the subject and creating shadows on the face of the subject (e.g., creating a slimming effect, creating shadows on the side of the subject's face, and / or on the subject's jaw). The stage light lighting effect includes modeling one different point light source arranged on the subject (e.g., creating a spotlight effect). The stage light MONO lighting effect includes modeling one different point light source arranged around on the subject in monochrome (e.g., creating a monochrome spotlight effect). In some embodiments, the lighting filter simulates a point light source. In some embodiments, as detailed above, the lighting effect disappears when the first criterion is met. In some embodiments, when the system detects a face, the facial features are considered when applying the lighting effect. As a result, based on the specific facial features and the shape of the face of the subject, the lighting effect changes the appearance of the representation of the image data (e.g., 806). By applying the simulation of the point light source, the user is provided with a visual representation of the content of the depth map information, enabling the device to provide the user with visual feedback about the shape and depth positioning of the objects in the field of view of the camera(s) (e.g., 602 and / or 603).Providing improved visual feedback to a user improves the operability of the device and makes the user-device interface more effective (e.g., by providing feedback that indicates an input that causes an intended result to occur on the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device.

[0261] In some embodiments, at block 936, based on the position and curvature of the object's contour, the lighting effect changes the appearance of the representation of the image data (e.g., 806). By including the depth contour of the object in the depth map information, the device can provide more accurate visual feedback to the user about the shape and depth positioning of the object in the field of view of the camera(s) (e.g., 602 or 603). Providing improved visual feedback to a user improves the operability of the device and makes the user-device interface more effective (e.g., by providing feedback that indicates an input that causes an intended result to occur on the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device. Further, by including the depth contour of the object in the depth map information, it becomes possible to preview and apply a simulation of a light source without the need for actual physical studio lighting, making the electronic device less expensive and smaller than when actual studio lighting and a background are required, increasing the portability of the device and reducing the manufacturing cost.

[0262] In some embodiments, while the first lighting effect (or second filter) is being applied, the electronic device (e.g., 800) determines that the first criterion is not met, and in response to determining that the first criterion is not met (e.g., no object is detected in the field of view within a predetermined distance from the electronic device (e.g., 800)), ceases applying the first (or second) lighting effect to the representation of the image data (e.g., 806) (in some embodiments, the visual effect of the filter is reduced, but the filter remains applied), displays on the display (e.g., 804) the representation of the image data without the application of the first lighting effect (e.g., an unaltered image with no filter or with a partial filter applied), and displays on the display (e.g., 804) a graphical indication (e.g., 822) (e.g., text, icon, image) of the first criterion (e.g., lighting condition application criterion) that has not been met. In some embodiments, the filter is reapplied when the condition is again met.

[0263] In some embodiments, the representation of the image data is previously captured image data. (For example, not a live preview of image data captured by one or more cameras (such as 602 and / or 603), but an image retrieved from memory / storage.) In some embodiments, since the depth map information is stored for the stored image, lighting effects added to the image can be changed and / or removed after the image is captured. By storing depth map information for the stored image, the ability to modify lighting after the image is captured is provided, reducing the number of photos the user has to take to obtain the desired effect, thereby reducing the number of inputs required to capture the desired photo, reducing the memory requirements for storing the photos, and making the user interface more efficient. Reducing the number of inputs required to capture the desired image and reducing the memory requirements improves the operability of the device and makes the interface between the user and the device more efficient (e.g., by assisting the user in achieving the intended result and reducing user errors during device operation / interaction with the device), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power consumption and improving battery life.

[0264] In some embodiments, while displaying a representation of the image data (such as 806), the electronic device (such as 800) displays a visual indication (such as 834) on the display (such as 804) that the image data includes depth map information (e.g., a "portrait mode" badge is optionally displayed, indicating the availability of depth map information).

[0265] In some embodiments, while a first lighting effect is being applied, the electronic device (such as 800) maintains at least one value of a previously applied visual effect (such as blur, lighting). Thus, it is possible to achieve a lighting effect and a blur effect in one representation of the image data.

[0266] In some embodiments, the previously applied visual effect is a color filter.

[0267] In some embodiments, the second input is an input received while the first lighting effect is being applied to the representation of the image data (e.g., 806), and applying the second lighting effect includes gradually transitioning between the application of the first lighting effect and the second lighting effect (in some embodiments, gradually transitioning includes 100% first, 0% second at a first time, 90% first, 10% second at a second time, etc.). By gradually transitioning between lighting effects, the user distraction created by the on-off blinking of the filter is reduced, thereby focusing the user on taking the desired photo, thereby reducing the number of inputs required to capture the desired photo and reducing the memory requirements for storing the photo. Reducing the number of inputs required to capture the desired image and reducing the memory requirements improves the device's operability and makes the user-device interface more efficient (e.g., by assisting in achieving the result intended by the user and by reducing user errors during device operation / interaction with the device), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power consumption and improving battery life.

[0268] Note that the details of the processes described above with respect to method 900 (e.g., FIGS. 9A-9D) are also applicable in a similar manner to the methods described below and above. For example, methods 700, 1100, 1300, 1500, 1700 optionally include one or more of the various method characteristics described above and the characteristics referenced with respect to method 900. For example, filter user interfaces, affordances, and control elements can be combined from among the various methods. As another example, the viewfinder of method 900 is similar to the viewfinders of methods 900, 1100, 1300, 1500, 1700. For the sake of brevity, these details are not repeated below.

[0269] In some embodiments, the electronic device 1000 includes some or all of the components of the device 600 shown in FIG. 6A. In some embodiments, the device 1000 includes a plurality of cameras 602 and 603 (e.g., on the back surface of the electronic device 1000). In some embodiments, the device 1000 includes one or more features of the devices 100, 300, and / or 500. In some examples, the electronic device (e.g., 1000) has a plurality of cameras 602 and 603 that are fixed but have different focal lengths. In some examples, the plurality of cameras are on the front, back, or both sides of the electronic device (e.g., 1000). In some embodiments, in addition to having different fixed focal lengths, the plurality of cameras have different fixed fields of view and different fixed optical magnification characteristics. In some embodiments, a camera (e.g., 602) captures image data using a plurality of focal lengths. In some embodiments, by a camera (e.g., 602) capturing a plurality of focal lengths, the same result is produced as with a plurality of (e.g., two or more) cameras that are fixed but have different focal lengths. In some examples, the electronic device (e.g., 1000) includes a depth camera such as an infrared camera, a self-thermometer camera, or a combination thereof. In some examples, the device further includes a light-emitting device (e.g., a light projector) such as an IR floodlight, a structured light projector, or a combination thereof. The light-emitting device is optionally used to illuminate an object during image capture by a visible light camera and a depth camera (e.g., an IR camera), and the information from the depth camera and the visible light camera is used to determine a depth map of different portions of the object captured by the visible light camera. In some embodiments, the illumination effects described herein are displayed using inconsistent information from two cameras (e.g., two visible light cameras) for a rear-facing image, and depth information from a depth camera combined with image data from a visible light camera for a forward-facing image (e.g., a selfie image).In some embodiments, the same user interface is used when determining depth information using two visible light cameras and when determining depth information using a depth camera, and a consistent experience is provided to the user even when dramatically different techniques are used to determine the information used to generate the lighting effects. In some embodiments, while displaying a camera user interface with one of the filters applied, the device detects a selection of a camera switching affordance and switches from a forward-facing camera (e.g., a depth camera and a visible light camera) to a rear-facing camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), and replaces the display from the field of view of the forward-facing camera to the field of view of the rear-facing camera (or vice versa) while maintaining the display of the user interface control for applying the filter.

[0270] As shown in FIG. 10A, the electronic device 1000 includes a touch-sensitive display 1004 (e.g., a touch screen), and the display displays information received from a camera (e.g., 602). In some embodiments, the display is different from the touch sensing surface. In some examples, the camera (e.g., 602) is disposed on the front, back, or both sides of the electronic device (e.g., 1000).

[0271] As shown in FIG. 10A, the electronic device 1000 displays on the display 1004 a camera application user interface for capturing an image by a camera (e.g., 602). The camera application user interface further includes a representation of image data (e.g., 1006) that includes a live preview of the field of view of the camera (e.g., 602). In some embodiments, including the embodiment of FIG. 10A, the field of view of the camera captures depth information associated with the image data in real time. FIG. 10A further shows a camera that captures the depth level of the field of view, including a subject (e.g., a woman) in the foreground region (e.g., 1008) and a subject (e.g., a woman) surrounded by a tree in the background region (e.g., 1010). In some embodiments, the representation of the image data 1006 consists of the background region (e.g., 1008 and 1006), as well as the foreground region. As seen in FIG. 10A, since the image data includes depth map information, the electronic device 1000 applies a simulated depth effect (e.g., bokeh) to the representation of the representation of the image data (e.g., 1006) before any other filters are applied, for the pseudo-depth effect (exemplified by the representation of the subject and the tree).

[0272] Further, in FIG. 10A, the electronic device 1000 displays a filter selection interface 1012 below the representation of the image data 1006. In some embodiments, the filter selection interface overlaps the representation of the image data. In some embodiments, the filter selection interface 1012 is arranged along the edge of the representation of the image data 1006. In some examples, the filter selection interface is displayed by the electronic device 1000 above, below (as seen in FIG. 10A), to the left, or to the right of the representation of the image data (e.g., 1006). As seen in FIG. 10A, the filter selection interface (e.g., 1012) includes one or more representations (e.g., 1014A to 1014C) (e.g., visual effects) of filters arranged in any one or more of rows and columns or in a circular orientation.

[0273] In some embodiments, the filter selection interface is delineated by a contour (e.g., a boundary) that differentiates the filter selection interface from the representation of the image data. In some embodiments, the filter selection interface (e.g., 1012) is translucent, semi-translucent, transparent, or semi-transparent and is displayed without a visible boundary. As a result, in some embodiments, the filter selection interface (e.g., 1012) appears to blend (e.g., be indistinguishable) with the representation of the image data (e.g., 1006) on the display 1004.

[0274] As shown in FIG. 10A, the filter selection interface 1012 includes one or more filter representations (e.g., 1014A, 1014B, 1014C, 1014D) displayed on the display 1004 corresponding to visual effects. In some embodiments, the filter selection interface (e.g., 1012) optionally includes filter representations that are not displayed (e.g., they are off-screen). The non-displayed filter representations filter the optionally displayed representations upon input (e.g., a swipe gesture), thereby causing scrolling through the filter selection interface (e.g., 1012) in the filter representation.

[0275] As shown in FIG. 10B, the electronic device 1000 receives an input (e.g., a tap 1016) corresponding to a position of one of the filter representations within the filter selection interface 1012. In FIG. 10B, the filter corresponding to the position of the tap input is “Vivid Warm”. In response to receiving the input, as shown in FIG. 10, the electronic device 1000 applies the corresponding filter (e.g., “Vivid Warm”) to the representation of the image data (e.g., 1006).

[0276] As shown in FIG. 10C, using the depth map information, the electronic device 1000 applies the selected filter (e.g., 1014C) differently to the background region (e.g., 1010) rather than the foreground region (e.g., 1008). The foreground region 1008 (including the woman in the front) is modified using different filter values from the background region 1010 (e.g., the woman and the tree). As can be seen in FIG. 10C, the electronic device 1000 displays the background region using a warmer (more darkly drawn) shade than the foreground region.

[0277] As further shown in FIG. 10C, the selected filter (e.g., "Vivid Warm") includes a color tone protection algorithm. The electronic device 1000 uses the depth map information associated with the image to minimize the deviation from a predetermined color (or a range of predetermined color tones) when the filter is applied. For example, in the foreground region (1008) without the color tone protection algorithm, the skin color tone of the subject may potentially deviate drastically from the original color tone after the application of the Vivid Warm filter. To handle the undesirable color tone deviation, the electronic device 1000 applies the filter to the image representation, but the change is limited to a predetermined color tone (or a range of predetermined color tones). Thus, in some embodiments, the color tone corresponding to the skin color tone may stay within a specific predetermined range. In contrast, color tones not associated with the skin (e.g., blue or green) can be modified more significantly by the selected filter.

[0278] FIG. 10C further shows that the tone protection algorithm applies the algorithm differently to tones in both the background region and the foreground region. For example, the object in the background region (e.g., 1010) may appear to have the same skin tone as the object in the foreground, and potentially deviate drastically from the original skin tone (the appearance of the image becomes unrealistic) by applying the "Vivid Warm" filter without any color correction. To correct the tone deviation, the electronic device 1000 applies the filter to the image representation, but the change is limited to a predetermined tone. Thus, in some embodiments, the tone corresponding to the skin tone may stay within a specific predetermined range. However, the allowable range of tone protection in the background is different from the allowable range of color protection in the foreground (e.g., since it is not in focus, there is less or no tone protection in the tones in the background). Thus, as shown in FIG. 10C, the skin tone of the object in the background region 1010 changes to a warmer tone than the skin tone of the object in the foreground region 1008.

[0279] As shown in FIG. 10D, the electronic device detects a tap 1018 at position 1020, which corresponds to the photo viewer application. In response to receiving the tap 1018, the electronic device switches to the image viewing mode as shown in FIG. 10E.

[0280] FIG. 10E shows a user interface displayed for a photo viewer application. The photo viewer application includes a thumbnail strip of representations of previously captured images (e.g., 1020A - 1020D) where 1020D was most recently captured. A representation of the image data (e.g., 1006) corresponding to the most recently captured image (1020D) is displayed on display 1004. In some embodiments, the previously captured images were captured using a camera corresponding to the electronic device (e.g., 1000). In some embodiments, the electronic device receives the previously captured images (e.g., 1020A - 1020D) from a remote source (e.g., a server). In some embodiments, the previously captured images were captured by a different electronic device (e.g., not 1000). FIG. 10E further shows that the most recently captured image does not have any associated depth map information (e.g., there is no visual indicator representing a depth map, no visible blurring effect).

[0281] As shown in FIG. 10F, the electronic device detects a tap 1022 at position 1024, which corresponds to a photo viewer edit mode (e.g., 1024). In response to receiving the tap 1022, the electronic device switches to an image view mode as shown in FIG. 10G.

[0282] FIG. 10G shows that, as described above with respect to FIG. 10A, a filter selection interface 1012 is displayed below the representation of the image data 1006. In some embodiments, the filter selection interface (e.g., 1012) includes any one or more (e.g., 1014A - 1014C) (e.g., visual effects) of filter representations arranged in one or more rows and columns or in a circular orientation.

[0283] As shown in FIG. 10H, the device 1000 detects the position of the tap 1026 and the "VividWorm" filter representation 1014C. In response to receiving the tap 1026, the electronic device applies the VividWorm filter, which correlates the filter representation 1014C with the image representation and displays the result in FIG. 10I.

[0284] As shown in FIG. 10I, since the image data does not have associated depth information, the electronic device applies the "VividWorm filter" uniformly to the background region 1010 and the foreground region 1008. Further, even when the image data does not have depth information associated with the image, the color tone protection algorithm still applies the algorithm to the color tones in the image (e.g., applies the color tone protection algorithm non-uniformly to the entire image). For example, the electronic device 1000 applies the color protection algorithm using the same value to both the skin color tones of the subject (e.g., foreground and background). Thus, as shown in FIG. 10C, after the electronic device applies the filter to the reorientation of the image data, in some embodiments, the skin color tones of the subject (e.g., foreground and background) are displayed using the same filter value, which is different from the filter value when applied to the rest of the image.

[0285] As shown in FIG. 10J, the electronic device 1000 displays a color wheel indicator affordance 1028 that overlaps the representation of the image data 1006.

[0286] As shown in FIG. 10K, the device 1000 detects a tap 1030 at the position of the color wheel indicator affordance 1028. In response to detecting the tap 1030, as shown in FIG. 10L, the electronic device displays an expanded color wheel over the representation of the image data. The color wheel allows the user to rotate the wheel by swiping and tap on any of the colors represented within the wheel, thereby applying the selected color filter to the representation of the image data.

[0287] As shown in FIG. 10M, the device 1000 detects a tap 1034 at a position corresponding to the representation of a color within the color wheel. In response to receiving the tap 1032, the electronic device 1000 applies the selected color filter to the representation of the image data 1006, as shown in FIG. 10N. As shown in FIG. 10N, the visual indicator 1034 is displayed over the selected color filter and indicates the currently selected filter.

[0288] FIGS. 11A - 11C are flowcharts showing a method for applying a pseudo - visual effect to a representation of image data using an electronic device according to some embodiments. The method 1100 is implemented on a device (e.g., 100, 300, 500, 1000) having one or more input devices (e.g., a touch - sensitive surface, a mouse, a keyboard) and a display (e.g., 1004). (In some embodiments, the device has one camera. In some embodiments, the device has multiple cameras, each with a different focal length). Some operations of the method 1100 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0289] As described below, the method 1100 provides an intuitive way to apply a pseudo - visual effect to the representation of image data. This method reduces the cognitive burden on the user who provides an input corresponding to a function, thereby creating a more efficient human - machine interface. In the case of a battery - operated computing device, power is conserved and the battery charging interval is increased by enabling the user to initiate various functions faster and more efficiently.

[0290] In some embodiments, at block 1102, the electronic device (e.g., 1000) simultaneously displays on a display (e.g., 1104) a representation of image data (e.g., 1006) (e.g., an image or photograph displayed on the device's display (e.g., 1104)) and a filter selection interface (e.g., 1012).

[0291] In block 1104, an electronic device (e.g., 1000) detects a first input (e.g., 1016) corresponding to a selection of a swipe, tap, and hold, tap, button press, etc. at a position corresponding to (e.g., on, near) a first image filter (e.g., 1014C) (e.g., illumination filter, vivid, vivid warm, vivid cool, dramatic, dramatic warm, dramatic cool, mono, silverstone, noir) of a representation of image data (e.g., 1006) having a first appearance via one or more input devices.

[0292] In some embodiments, in block 1106, the electronic device (e.g., 1000) further includes one or more cameras (e.g., 602 and / or 603), and the representation of the image data (e.g., 1006) is a live preview of the image data captured within the field of view of the one or more cameras. In some embodiments, the device includes a plurality of cameras having various focal lengths. In some embodiments, the image data and depth information are captured by one or more cameras in the electronic device (e.g., 1000). In some embodiments, the device includes a plurality of cameras having various focal lengths.

[0293] In some embodiments, in block 1108, the first input is detected while a filter selection user interface (e.g., 1012) is being displayed.

[0294] In some embodiments, at block 1110, the filter selection user interface (e.g., 1012) includes a plurality of filter representations (e.g., 1014A - 1014D) including a representation of a first image filter (e.g., 1014C), where the first input corresponds to the selection of the representation of the first image filter (e.g., 1014C). The first filter is displayed as part of a series of one or more filter representations arranged in a row along an edge of the display (e.g., 1004). In some embodiments, the series of one or more filter representations (e.g., 1014A - 1014D) is in direct contact with the edge of the display (e.g., 1004). In some embodiments, the filter representations (e.g., 1014A - 1014D) are proximate to but not adjacent to the edge of the display (e.g., 1004). In some embodiments, the row is arranged along the short edge of the display (e.g., 1004). In some embodiments, the row is arranged along the long edge of the display (e.g., 1004). By displaying a plurality of filter representations, the user is provided with different selectable filters that are available for application to the representation of the image data (e.g., 1006), and optionally, different filters provide different techniques for visualizing depth map information corresponding to the representation of the image data (e.g., 1006), thereby providing the user with additional information about the shape and positioning of objects in the representation of the image data (e.g., 1006). Providing the user with improved visual feedback improves the operability of the device, makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that will produce the intended result of the device, thereby assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), and thereby further enables the user to use the device more quickly and effectively, reducing power usage and improving the battery life of the device.

[0295] In block 1112, in response to detecting a first input, an electronic device (e.g., 1000) applies a first image filter (e.g., 1014C) to a representation of image data (e.g., 1006) optionally including the techniques of blocks 1114 - 1126.

[0296] In block 1114, according to the determination that the image data is associated with depth information that differentiates a foreground region (e.g., 1008) of the representation of the image data (e.g., 1006) from a background region (e.g., 1010) of the representation of the image data (e.g., 1006) (e.g., a depth map), the techniques of blocks 1116 - 1120 are optionally implemented. In some embodiments, the depth information is stored as a separate track of an image file. In some embodiments, the depth information is calculated using image data received from at least two cameras having different focal lengths.

[0297] In block 1116, an electronic device (e.g., 1000) applies a first image filter (e.g., 1014C) to a foreground region of a representation of image data (e.g., 1006) at a first level of adjustment to change the appearance of the foreground region (e.g., 1008) of the representation of the image data (e.g., 1006), where the first level of adjustment indicates a first degree to which the first image filter (e.g., 1014C) changes the appearance of the representation of the image data (e.g., 1006). In some embodiments, the filter changes any one or more of color warmth, saturation reduction, color skew, light intensity, contrast, hue shift, luminance.

[0298] In some embodiments, in block 1118, applying a first image filter (e.g., 1014C) to a foreground region (e.g., 1008) includes shifting (e.g., changing, modifying, replacing) the value of a first color of a background region (e.g., 1010) using a first level of color value adjustment according to a determination that the image data corresponding to the foreground region (e.g., 1008) includes a value of a first color (e.g., a hue value, a tone value, one or more colors associated with skin tone) and the image data corresponding to the background includes the value of the first color, and further includes shifting (e.g., changing, modifying, replacing) the value of the first color of the foreground region (e.g., 1008) using a second level of color value adjustment different from the first level of color value adjustment. In some embodiments, the skin color in the background is colored differently from the skin color in the foreground. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some examples, the skin tone is not modified at all. In some embodiments, even when the background region has a color tone corresponding to the skin tone, the filter is applied uniformly to the background. Applying the filter differently includes removing the constraints of the filter from the foreground or the background (e.g., removing the requirement for skin protection in the background). By applying different levels of color adjustment to similar (or identical) color objects based on whether they are in the foreground or the background, the user is provided with visual feedback about the depth information corresponding to the image data, such as indicating that a particular object is in the foreground and a different object is in the background, and the user is provided with visual feedback for distinguishing the foreground object from similar objects in the background.Providing improved visual feedback to a user improves the operability of a device and makes the user-device interface more effective (e.g., by providing feedback that indicates an input that causes a result intended for the device, by assisting the user in achieving the intended result, and by reducing user errors when operating or interacting with the device), thereby further reducing power usage and improving the device's battery life by enabling the user to use the device more quickly and effectively.

[0299] In block 1120, an electronic device (e.g., 1000) applies a first image filter (e.g., 1014C) to a background region (e.g., 1010) of a representation of image data (e.g., 1006) at a second level of adjustment to change the appearance of the background region (e.g., 1010) of the representation of the image data (e.g., 1006). The second level of adjustment indicates a second degree to which the first image filter (e.g., 1014C) changes the appearance of the representation of the image data (e.g., 1006), and the first level of adjustment and the second level of adjustment are different.

[0300] At block 1122, after applying a first image filter (e.g., 1014C) to the representation of image data (e.g., 1006), the electronic device (e.g., 1000) displays on the display (e.g., 1004) the representation of each image to which the first filter has been applied to the representation of the image data (e.g., 1006). In some embodiments, the results are displayed on the display (e.g., 1004). In some embodiments, the electronic device (e.g., 1000) displays the results in response to receiving a request and applies the first image filter (e.g., 1014C) to each image. By applying the image filter to the foreground and background at different levels of adjustment and displaying the representation of the updated image data (e.g., 1006), visual feedback regarding depth information is provided to the user, in particular, regarding which objects are identified as the foreground and which objects are identified as the background. Providing improved visual feedback to the user improves the operability of the device and makes the user-device interface more effective (e.g., by providing feedback that indicates the input that causes the intended result to occur on the device, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further reducing power consumption and improving the battery life of the device by enabling the user to use the device more quickly and effectively. Further, by applying the image filter to the foreground and background at different levels of adjustment and displaying the representation of the updated image data (e.g., 1006), the electronic device automatically applies the filter, and without distorting the skin tone of the subject in the viewfinder where the user would otherwise need to manually correct after applying the filter, it is possible to create a more dramatic effect.Applying the filter to the relevant part of the image and automatically avoiding distorting the tone of the target skin without further user input improves the operability of the device and makes the user-device interface more efficient (e.g., by assisting in achieving the result intended by the user and by reducing the number of user inputs), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power consumption and improving battery life.

[0301] In some embodiments, in block 1124, applying a first image filter (e.g., 1014C) to the representation of the image data (e.g., 1006) further includes shifting (e.g., changing, modifying, replacing) the values of the third color according to a determination that the image data includes values of a third color (e.g., hue values, tone values, one or more colors associated with skin tone) using a third level of color value adjustment (e.g., specific colors for skin tone protection). In some examples, the skin color is colored with a color different from the rest of the image. In some examples, the skin tone color is modified so as not to dramatically change the appearance of the skin color. In some examples, the skin tone is not modified at all. In some examples, applying different filters includes removing constraints on the filter from the image (e.g., removing skin protection requirements in the background). For example, when adjusting the color of an image not only in the foreground but also the whole image when depth information is not available, the device optionally applies a skin protection algorithm. For this purpose, in some features in the background of the image close to the skin tone protected by the skin protection algorithm, the color does not shift. By using different (e.g., third level color adjustment) techniques to shift the colors of some parts of the image data, the user is provided with visual feedback as to which parts of the image do not correspond (or alternatively, correspond) to a particular color, such as by shifting the color of the skin tone different from the background, making people with a particular skin tone more distinguishable compared to other objects. Providing improved visual feedback to the user improves the operability of the device and makes the interface between the user and the device more effective (e.g., by providing feedback that indicates the input that generates the result intended for the device, assisting the user in achieving the intended result, and reducing the user's mistakes when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power consumption, and improving the battery life of the device.Furthermore, by using different (e.g., third-level color adjustment) techniques to shift the colors of a portion of the image data, while still avoiding distorting the skin tone of the subject regardless of whether depth information is available, when depth information is available, a more dramatic effect for the filter becomes possible, thereby avoiding the need for the user to manually correct the skin tone later. Applying the filter to the relevant portion of the image and automatically avoiding distorting the skin tone of the subject without further user input improves the device's operability and makes the user-device interface more efficient (e.g., by assisting in achieving the result intended by the user and by reducing the number of user inputs), thereby further enabling the user to use the device more quickly and efficiently, reducing the device's power usage and improving battery life.

[0302] In some embodiments, at block 1126, in accordance with detecting a first input and determining that the image data is not associated with depth information, in response to applying a first image filter (e.g., 1014C) to a representation of the image data (e.g., 1006), the electronic device (e.g., 1000) applies the first image filter (e.g., 1014C) uniformly to the representation of the image data (e.g., 1006) at a first level of adjustment (e.g., thereby uniformly changing the appearance of the foreground region (e.g., 1008) of the representation of the image data (e.g., 1006), and the background region (e.g., 1010) of the representation of the image data (e.g., 1006)). By uniformly applying the image filter to the representation of the image data (e.g., 1006), visual feedback is provided to the user about the state of the device and, in particular, that depth information is not available for the image data. Providing improved visual feedback to the user improves the operability of the device and makes the user interface of the device more effective (e.g., by providing feedback that indicates an input that causes an intended result to occur, assisting the user in achieving the intended result, and reducing user errors when operating or interacting with the device), thereby further enabling the user to use the device more quickly and effectively, reducing power usage, and improving the battery life of the device. Further, by uniformly applying the image filter to the representation of the image data (e.g., 1006), a more dramatic effect for the filter becomes possible when depth information is available while avoiding distorting the tone of the subject's skin, regardless of whether depth information is available, thereby avoiding the need for the user to manually correct the skin tone later.Applying the filter to the relevant part of the image and automatically avoiding distorting the tone of the target skin without further user input improves the device's operability and makes the user-device interface more efficient (e.g., by assisting in achieving the result intended by the user and by reducing the number of user inputs), thereby further enabling the user to use the device faster and more efficiently, reducing the device's power consumption and improving battery life.

[0303] In some embodiments, before detecting the first input, an electronic device (e.g., 1000) receives image data (e.g., 1006) represented by a representation of the image data in the electronic device (e.g., from a camera, from memory, from a server). In some embodiments, the image data includes RGB and depth map values. In some embodiments, the image data and depth information are received from a source outside the electronic device (e.g., the data is received from a server).

[0304] In some embodiments, when the first image filter (e.g., 1014C) is applied, the electronic device (e.g., 1000) maintains at least one value of a previously applied visual effect (e.g., blur, lighting). Thus, in one representation of the image data (e.g., 1006), it is possible to achieve a photo-curing and blur effect. By maintaining at least one value of a previously applied visual effect (e.g., blur, lighting) while the first image filter (e.g., 1014C) is being applied, the need to generate a blur effect with a bulky lens and device is eliminated.

[0305] In some embodiments, an electronic device (e.g., 1000) displays a camera application user interface on a display (e.g., 1004), where the camera application user interface includes a live preview of the field of view of one or more cameras and a representation (e.g., icon, affordance, button) of a color wheel user interface (e.g., 1028) (including a live preview or a near-live preview image) at a first position overlapping with a digital viewfinder (e.g., an image, an icon, a textual representation indicating a filter) in the digital viewfinder. In some embodiments, the representation of the color wheel interface is displa...

Claims

1. In an electronic device comprising a camera, a sensor, and a display, displaying a camera viewfinder on the display for capturing media; while displaying the camera viewfinder, based on data from the sensor, the electronic device determines that the relative difference between the orientation of the plane of focus of the camera and a predetermined orientation meets the position adjustment guide display criteria including the requirement that it is within respective position adjustment thresholds for a position adjustment guide display criteria to be satisfied, and in accordance with the determination, on the display, when the orientation of the plane of focus of the camera changes with respect to the predetermined orientation, the appearance of the position adjustment guide changes, displaying the position adjustment guide within the camera viewfinder; based on data from the sensor, in accordance with the determination that the position adjustment guide display criteria are not satisfied, ceasing to display the position adjustment guide within the camera viewfinder; A method comprising the above.

2. The position adjustment guide display criteria include the requirement that the relative difference between the orientation of the plane of focus of the camera and the predetermined orientation is maintained within the respective position adjustment thresholds for at least a threshold time for the position adjustment guide display criteria to be satisfied. The method according to claim 1.

3. While the relative difference between the orientation of the plane of focus of the camera and the predetermined orientation is maintained within the respective position adjustment thresholds for the position adjustment guide display criteria to be satisfied, the position adjustment guide display criteria include the requirement that the orientation of the electronic device does not change by more than a threshold amount for a threshold time. The method according to claim 1.

4. The predetermined orientation corresponds to a horizontal orientation. The method according to claim 1.

5. The predetermined orientation is a vertical orientation. The method according to claim 1.

6. The position adjustment guide includes at least two visual indicators. The method according to any one of claims 1 to 5.

7. When the orientation of the plane of focus of the camera changes with respect to the predetermined orientation, at least one of the at least two visual indicators remains stationary. The method according to claim 6.

8. The method according to claim 6 or 7, wherein at least one of the at least two visual indicators is displayed close to the center of the camera viewfinder.

9. The method according to any one of claims 6 to 8, wherein the distance between the two visual indicators is dynamically based on the relative difference between the orientation of the plane of focus of the camera and the predetermined orientation.

10. Detecting, based on data from the sensor, a change in the orientation of the plane of focus of the camera from a first orientation to a second orientation while the position adjustment guide is being displayed and while the orientation of the plane of focus of the camera is in the first orientation; Changing the display location of a first visual indicator of the at least two visual indicators in response to detecting the change in the orientation of the plane of focus of the camera from the first orientation to the second orientation, wherein the display location of the first visual indicator is changed based on the relative difference between the first orientation and the second orientation; The method according to any one of claims 6 to 9, further comprising:

11. Further in response to detecting the change in the orientation of the plane of focus of the camera from the first orientation to the second orientation, Displaying the first visual indicator at the changed display location according to a determination that the relative difference between the second orientation of the plane of focus of the camera and the predetermined orientation is not at a first visual indicator position adjustment threshold; Displaying the first visual indicator at a predetermined display location according to a determination that the relative difference between the second orientation of the plane of focus of the camera and the predetermined orientation is within the first visual indicator position adjustment threshold; The method according to claim 10, further comprising:

12. The method according to claim 11, wherein displaying the first visual indicator at the predetermined display location includes displaying a corresponding animation at one or more positions of the visual indicator.

13. Detecting, based on data from the sensor, a change in the orientation of the plane of focus of the camera from a third orientation having a relative difference with respect to the predetermined orientation within the first visual indicator position adjustment threshold to a fourth orientation while the first visual indicator is being displayed at the predetermined display location; In response to detecting the change in the orientation of the plane of focus of the camera from the third orientation to the fourth orientation, maintaining the display of the first visual indicator at the predetermined display location according to a determination that the relative difference between the fourth orientation of the plane of focus of the camera and the predetermined orientation is not outside a second visual indicator position adjustment threshold; displaying the first visual indicator at a second updated display location based on the relative difference between the orientation of the plane of focus of the camera and the predetermined orientation according to a determination that the relative difference between the fourth orientation of the plane of focus of the camera and the predetermined orientation is outside the second visual indicator position adjustment threshold; The method according to claim 11 or 12, further comprising.

14. While the orientation of the plane of focus of the camera is in a fifth orientation having a relative difference with respect to the predetermined orientation within the respective position adjustment thresholds, and while the second visual indicator of the at least two visual indicators is displayed having a first value of a visual characteristic, detecting a change in the orientation of the plane of focus of the camera from the fifth orientation to a sixth orientation based on data from the sensor; In response to detecting the change in the orientation of the plane of focus of the camera from the fifth orientation to the sixth orientation, displaying the second visual indicator having a second value of the visual characteristic different from the first value; The method according to any one of claims 6 to 13, further comprising, wherein the relative difference between the sixth orientation of the plane of focus of the camera and the predetermined orientation is within the respective position adjustment thresholds.

15. The fifth orientation of the plane of focus of the camera has a first relative difference between the orientation of the plane of focus of the camera and a predetermined orientation, The sixth orientation of the plane of focus of the camera has a second relative difference between the orientation of the plane of focus of the camera and a predetermined orientation, and the second relative difference is greater than the first relative difference, The method according to claim 14, wherein the visual characteristic is display intensity and the first value is greater than the second value.

16. While the orientation of the plane of focus of the camera is in a seventh orientation having a relative difference with respect to the predetermined orientation within the respective position adjustment thresholds, and while a third visual indicator among the at least two visual indicators is being displayed, detecting, based on data from the sensor, a change in the orientation of the plane of focus of the camera from the seventh orientation to an eighth orientation; In response to detecting the change in the orientation of the plane of focus of the camera from the seventh orientation to the eighth orientation; Maintaining the display of the third visual indicator according to a determination that a relative difference between the eighth orientation of the plane of focus of the camera and the predetermined orientation is within the respective position adjustment thresholds; Aborting the display of the third visual indicator according to a determination that the relative difference between the eighth orientation of the plane of focus of the camera and the predetermined orientation is not within the respective position adjustment thresholds; The method according to any one of claims 6 to 13, further comprising.

17. A computer-readable storage medium storing one or more programs configured to be executable by one or more processors of an electronic device including a camera, a sensor, and a display, the one or more programs including instructions for performing the method according to any one of claims 1 to 16.

18. An electronic device, comprising: A camera; A sensor; One or more input devices; A display; One or more processors; A memory storing one or more programs configured to be executable by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 16. The electronic device according to claim 14, further comprising.

19. An electronic device, comprising: A camera; A sensor; A display; Means for performing the method according to any one of claims 1 to 16. The electronic device according to claim 21, further comprising.

Images