Device and method for capturing an extended digital image and for bidirectional action with an extended digital image

Abstract

To a method for capturing and interacting with a moment photographed or recorded with a camera, and an electronic device with an improved interface.SOLUTION: A method includes displaying a live preview on a display by a device during a first media acquisition mode for a camera, detecting shutter button activation while displaying the live preview, and grouping a plurality of images acquired by the camera in the temporal vicinity of activation of the shutter button into an image sequence in response to the detection of activation of the shutter button. The image sequence includes a plurality of images captured by the camera before detecting shutter button activation, a representative image representative of a first image sequence and acquired by the camera after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera after acquiring the representative image.SELECTED DRAWING: Figure 9A

Description

Technical Field

[0001] This application generally relates to an electronic device having a touch sensing surface. Examples of electronic devices having a touch sensing surface include, but are not limited to, electronic devices that capture, display, and / or otherwise manipulate digital content captured by a camera, such as digital content captured by a camera and then displayed or otherwise manipulated on a touch sensing surface.

Background Art

[0002] In recent years, the use of electronic devices to capture, view, edit, and share digital content has increased significantly. Users frequently record digital content (e.g., images and / or videos) using their portable electronic devices (e.g., smartphones, tablets, and dedicated digital cameras), view and edit the digital content using image management applications (e.g., Photos by Apple Inc. (Cupertino, California)) and / or digital content management applications (e.g., iTunes (registered trademark) by Apple Inc. (Cupertino, California)), and share the digital content with others through instant messaging, email, social media applications, and other communication applications.

[0003] Portable electronic devices generally capture two types of digital content: still images and videos. Still images are generally captured by simply pressing a shutter button. Still images capture a moment without blur, but the details of the moments before and after that moment are lost. Videos are recorded for a long time and can include both interesting and less interesting moments. Generally, significant editing is required to remove the less interesting moments.

Summary of the Invention

[0004] Accordingly, there is a need for an electronic device that improves methods and interfaces for capturing moments captured or recorded by a camera and for two-way interaction with the moments. Such methods and interfaces optionally complement or replace conventional methods for capturing still images and videos and for two-way interaction with still images and videos.

[0005] The device of the present disclosure develops photography beyond still images by providing a new and improved method for capturing moments and for two-way interaction with the moments. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device such as a wristwatch). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a plurality of functions. In some embodiments, the user interacts with the GUI primarily via stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, the functions optionally include image editing, drawing, presenting, word processing, spreadsheet creation, game play, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital video shooting, web browsing, playing digital music, taking summary notes, and / or playing digital videos. The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0006] According to some embodiments, the method is executed on an electronic device comprising a display and a camera. The method includes displaying a live preview on the display while in a first media acquisition mode for the camera. The method further includes detecting activation of a shutter button at a first time while the live preview is being displayed, and grouping a plurality of images acquired by the camera in a time vicinity of the activation of the shutter button at the first time into a first image sequence in response to detecting the activation of the shutter button at the first time. The first image sequence includes a plurality of images acquired by the camera before detecting the activation of the shutter button at the first time, a representative image that represents the first image sequence and that is acquired by the camera after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera after acquiring the representative image.

[0007] According to some embodiments, the method is executed on an electronic device comprising a display and a touch sensing surface. The method includes displaying a representative image on the display. The representative image is one image in an image sequence captured by the camera. The image sequence includes one or more images acquired by the camera after acquiring the representative image. The image sequence also includes one or more images acquired by the camera before acquiring the representative image. The method includes detecting a first portion of a first input while the representative image is being displayed on the display, and replacing the display of the representative image with a display in an order of one or more images acquired by the camera after acquiring the representative image in response to detecting the first portion of the first input. The method further includes detecting a second portion of the first input after detecting the first portion of the first input, and displaying in order one or more images acquired by the camera before acquiring the representative image, the representative image, and one or more images acquired by the camera after acquiring the representative image in response to detecting the second portion of the first input.

[0008] According to some embodiments, the method is executed on an electronic device having a display and a touch sensing surface. The method includes displaying a representative image on the display. The representative image is one of the images in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is acquired. The image sequence includes one or more images captured by the camera before the representative image is acquired. The method further includes detecting a first portion of a first input while the representative image is displayed on the display. The method further includes transitioning from displaying the representative image to displaying each previous image in the image sequence in response to detecting the first portion of the first input, wherein each previous image is captured by the camera before the representative image is acquired. The method further includes, after transitioning from displaying the representative image to displaying each previous image in response to detecting the first portion of the first input, sequentially starting from each previous image, displaying at least a portion of one or more images captured by the camera before the representative image is acquired and at least a portion of one or more images captured by the camera after the representative image is acquired.

[0009] According to one embodiment, in an electronic device comprising a display, a touch sensing surface, and one or more sensors for detecting the intensity of contact with the touch sensing surface, a method is performed. The method includes displaying a representative image on the display. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is acquired. The method further includes detecting a first input, including detecting that a characteristic intensity of contact on the touch sensing surface has increased to a first intensity greater than a first intensity threshold while the representative image is being displayed on the display. The method further includes advancing, at a rate determined at least in part based on the first intensity, in a first direction between one or more images captured by the camera after the representative image is acquired, in response to detecting that the characteristic intensity of contact has increased. The method further includes detecting that the intensity of contact has decreased to a second intensity less than the first intensity after advancing between one or more images captured by the camera after the representative image is acquired at a rate determined based on the first intensity. The method further includes continuing to advance, at a second rate, in the first direction between one or more images captured by the camera after the representative image is acquired, in accordance with a determination that the second intensity exceeds the first intensity threshold, in response to detecting that the characteristic intensity of contact has decreased to the second intensity. The second rate is determined at least in part based on the second intensity, and the second rate is slower than the first rate. The method further includes moving, in a second direction opposite to the first direction, between one or more images captured by the camera after the representative image is acquired, at a rate determined at least in part based on the second intensity, in accordance with a determination that the second intensity is less than the first intensity threshold, in response to detecting that the characteristic intensity of contact has decreased to the second intensity.

[0010] According to some embodiments, the method is executed on an electronic device having a display and a touch sensing surface. The method includes storing a plurality of image sequences. Each image sequence includes each representative image captured by a camera, one or more images captured by the camera after acquiring each representative image, and one or more images captured by the camera before acquiring each representative image. The method further includes displaying a first representative image for a first image sequence in a first movable area on the display. The method further includes detecting a drag gesture on the touch sensing surface. The method further includes, in accordance with a determination that the drag gesture is in a first direction on the touch sensing surface, replacing, in the first movable area, the display of the first representative image for the first image sequence with a time series display of at least a portion of one or more images for the first image sequence captured by the camera after acquiring the first representative image for the first image sequence. The method further includes moving the first area in the first direction.

[0011] According to some embodiments, the method is executed in an electronic device having a display and a touch sensing surface. The method includes storing a plurality of image sequences. Each image sequence includes each representative image captured by the camera and one or more images captured by the camera before acquiring each representative image. The method further includes displaying a first representative image for a first image sequence in a first movable area on the display. The method further includes detecting a gesture on the touch sensing surface, the gesture including a movement by contact corresponding to a movement in a first direction on the display. In response to detecting the gesture on the touch sensing surface, the method includes moving the first area in the first direction on the display, moving a second movable area in the first direction on the display, and, according to a determination that a sequence display criterion is satisfied, while moving the second area in the first direction, displaying at least a portion of one or more images for a second image sequence, captured by the camera before acquiring a second representative image for the second image sequence, in a time series in the second area.

[0012] According to some embodiments, the method is executed in an electronic device having a display and a touch sensing surface. The method includes storing a plurality of image sequences. Each image sequence includes each representative image captured by a camera, one or more images captured by the camera after acquiring each representative image, and one or more images captured by the camera before acquiring each representative image. The method further includes storing a plurality of images different from the images in the plurality of image sequences. Each image in the plurality of images is not part of an image sequence in the plurality of image sequences. The method further includes displaying a first image on the display and detecting a first input while the first image is being displayed on the display. The method further includes performing a first operation that includes displaying at least a portion of the images in the first image sequence in addition to the first image according to a determination that the first image is an image in the first image sequence in response to detecting the first input. The method further includes performing a second operation different from the first operation that includes the first image according to a determination that the first image is an image in a plurality of images different from the images in the plurality of image sequences.

[0013] According to some embodiments, the method is executed on an electronic device having a display and a touch sensing surface. The method includes displaying a representative image on the display. The representative image is one of the images in the image sequence captured by the camera. The image sequence includes one or more images captured by the camera after obtaining the representative image. The image sequence includes one or more images captured by the camera before obtaining the representative image. The method further includes detecting an input for modifying the representative image while the representative image is displayed on the display. The method further includes, in response to detecting an input for modifying the representative image, modifying the representative image, one or more images captured by the camera after obtaining the representative image, and one or more images captured by the camera before obtaining the representative image according to a determination that the device is in a first editing mode, and modifying the representative image without modifying one or more images captured by the camera after obtaining the representative image and without modifying one or more images captured by the camera before obtaining the representative image according to a determination that the device is in a second editing mode different from the first editing mode.

[0014] According to one embodiment, in a first electronic device having a display, a method is executed. The method includes displaying a representative image on a user interface of an application configured to communicate with another electronic device on the display. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after obtaining the representative image. The image sequence includes one or more images captured by the camera before obtaining the representative image. The method further includes detecting, while the representative image is being displayed on the display, an input corresponding to a request to transmit the representative image or a request to select a representative image for transmission from the first electronic device to a second electronic device remote from the first electronic device using the application. The method includes displaying a first set of optional selections for transmitting at least a portion of the image sequence to the second electronic device according to a determination that the second electronic device is configured to act bidirectionally as a group with the image sequence, and displaying a second set of optional selections for transmitting at least a portion of the image sequence to the second electronic device according to a determination that the second electronic device is not configured to act bidirectionally as a group with the image sequence, wherein the second set of optional selections is different from the first set of optional selections, in response to detecting an input corresponding to a request to transmit the representative image or a request to select a representative image for transmission to the second electronic device.

[0015] According to one embodiment, the method is executed in an electronic device equipped with a display and a camera. The method includes displaying a live preview of a scene on the display while in a first media acquisition mode for the camera, and performing scene recognition on the scene. The method further includes detecting a single activation of the shutter button at a first time while displaying the live preview of the scene. In response to detecting the single activation of the shutter button at the first time, according to a determination that the scene meets an action capture criterion based at least in part on the scene recognition performed on the scene, retaining a plurality of images acquired by the camera in the temporal vicinity of the activation of the shutter button at the first time, and grouping the plurality of images into a first image sequence. The first image sequence includes a plurality of images acquired by the camera before detecting the activation of the shutter button at the first time, a representative image that represents the first image sequence and is acquired by the camera after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera after acquiring the representative image. In response to detecting the single activation of the shutter button at the first time, according to a determination that the scene does not meet the action capture criterion, the method further includes retaining a single image in the temporal vicinity of the activation of the shutter button at the first time.

[0016] According to some embodiments, the method is executed on an electronic device having a display and a touch sensing surface. The method includes displaying an image on the display. The image is one of the images in an image sequence captured by a camera. The image sequence includes a representative image. The image sequence includes one or more images captured by the camera after obtaining the representative image. The image sequence includes one or more images captured by the camera before obtaining the representative image. The method further includes detecting a first input while the images in the image sequence are being displayed on the display. The method further includes displaying a user interface for trimming the image sequence to a subset less than all of the image sequence in response to detecting the first input. The user interface includes an area including a representation of the images in the image sequence, a user-adjustable trimming start icon for defining a start image in a subset of the image sequence via a position of the trimming start icon in the area including the representation of the images in the image sequence, and a user-adjustable trimming end icon for defining an end image in a subset of the image sequence via a position of the trimming end icon in the area including the representation of the images in the image sequence. The trimming start icon is located at a first position automatically selected by the device in the area including the representation of the images in the image sequence. The trimming end icon is located at a second position automatically selected by the device in the area including the representation of the images in the image sequence. The method further includes detecting a second input while the user interface for trimming the image sequence is being displayed. The method further includes trimming the image sequence to a subset of the image sequence according to the current position of the trimming start icon and the current position of the trimming end icon in response to detecting the second input.

[0017] According to some embodiments, an electronic device includes a display unit configured to display a live preview, a camera unit configured to acquire an image, and a processing unit connected to the display unit and the camera unit. The processing unit is configured to display the live preview on the display unit while in a first media acquisition mode for the camera unit. While displaying the live preview, the processing unit is further configured to detect activation of a shutter button at a first time, and in response to detecting the activation of the shutter button at the first time, group a plurality of images acquired by the camera unit in a time vicinity of the activation of the shutter button at the first time into a first image sequence. The first image sequence includes a plurality of images acquired by the camera unit before detecting the activation of the shutter button at the first time, a representative image that represents the first image sequence and is acquired by the camera unit after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera unit after acquiring the representative image.

[0018] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, and a processing unit connected to the display unit and the touch sensing surface unit. The processing unit is configured to display a representative image on the display unit. The representative image is one of the images in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is obtained. The image sequence includes one or more images captured by the camera before the representative image is obtained. The processing unit is further configured to detect a first portion of a first input while the representative image is being displayed on the display unit. In response to detecting the first portion of the first input, the processing unit is further configured to replace the display of the representative image on the display unit with a display in the order of one or more images captured by the camera after the representative image is obtained. After detecting the first portion of the first input, the processing unit is further configured to detect a second portion of the first input. In response to detecting the second portion of the first input, the processing unit is further configured to display, on the display unit, in order, one or more images captured by the camera before the representative image is obtained, the representative image, and one or more images captured by the camera after the representative image is obtained.

[0019] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, and a processing unit connected to the display unit and the touch sensing surface unit. The processing unit is configured to be able to display a representative image on the display unit. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is obtained. The image sequence includes one or more images captured by the camera before the representative image is obtained. While being able to display the representative image on the display unit, the processing unit is further configured to detect a first portion of a first input. In response to detecting the first portion of the first input, the processing unit is to transition from displaying the representative image to displaying each previous image in the image sequence, where each previous image is captured by the camera before the representative image is obtained, and after transitioning from displaying the representative image to displaying each previous image, to be able to display, in order starting from each previous image, at least a portion of one or more images captured by the camera before the representative image is obtained and at least a portion of one or more images captured by the camera after the representative image is obtained.

[0020] According to one embodiment, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, one or more sensor units configured to detect the intensity of contact with the touch sensing surface unit, and a processing unit connected to the display unit, the touch sensing surface unit, and the one or more sensor units. The processing unit is configured to be able to display a representative image on the display unit. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is acquired. While being able to display the representative image on the display unit, the processing unit is further configured to detect a first input, including detecting that the characteristic intensity of the contact on the touch sensing surface unit has increased to a first intensity greater than a first intensity threshold. In response to detecting that the characteristic intensity of the contact has increased, the processing unit is further configured to advance between one or more images captured by the camera after the representative image is acquired at a rate determined at least in part based on the first intensity in a first direction. After advancing between one or more images captured by the camera after the representative image is acquired at a rate determined based on the first intensity, the processing unit is further configured to detect that the intensity of the contact has decreased to a second intensity less than the first intensity. In response to detecting that the characteristic intensity of the contact has decreased to the second intensity, the processing unit is further configured to continue to advance between one or more images captured by the camera after the representative image is acquired in the first direction at a second rate according to a determination that the second intensity exceeds the first intensity threshold. The second rate is determined at least in part based on the second intensity, and the second rate is slower than the first rate. In response to detecting that the characteristic intensity of the contact has decreased to the second intensity, the processing unit is further configured to move between one or more images captured by the camera after the representative image is acquired in a second direction opposite to the first direction at a rate determined at least in part based on the second intensity according to a determination that the second intensity is below the first intensity threshold.

[0021] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, a memory unit configured to store an image, and a processing unit connected to the display unit, the memory unit, and the touch sensing surface unit. The processing unit is configured to store a plurality of image sequences in the memory unit. Each image sequence includes each representative image captured by a camera, one or more images captured by the camera after obtaining each representative image, and one or more images captured by the camera before obtaining each representative image. The processing unit is further configured to display, on the display unit, a first representative image for a first image sequence in a movable first area on the display unit. The processing unit is further configured to detect a drag gesture on the touch sensing surface unit. The processing unit is further configured to, in accordance with a determination that the drag gesture is in a first direction on the touch sensing surface unit, in the movable first area, replace the display of the first representative image for the first image sequence with a time-series display of at least a part of one or more images for the first image sequence captured by the camera after obtaining the first representative image for the first image sequence on the display unit, and move the first area in the first direction on the display unit.

[0022] According to one embodiment, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, and a processing unit connected to the display unit and the touch sensing surface unit. The processing unit is configured to store a plurality of image sequences. Each image sequence includes each representative image captured by a camera and one or more images captured by the camera before obtaining each representative image. The processing unit is further configured to be able to display a first representative image for a first image sequence in a first movable area on the display unit. The processing unit is configured to detect a gesture on the touch sensing surface unit, the gesture including a movement by contact corresponding to a movement in a first direction on the display unit. In response to detecting the gesture on the touch sensing surface unit, the processing unit moves the first area in the first direction on the display unit, moves a second movable area in the first direction on the display unit, and according to a determination that a sequence display criterion is satisfied, at least a part of one or more images for a second image sequence captured by the camera before obtaining a second representative image for the second image sequence is made to be displayable in time series in the second area while moving the second area in the first direction.

[0023] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, a memory unit configured to store images, and a processing unit connected to the display unit, the memory unit, and the touch sensing surface unit. The processing unit is configured to store a plurality of image sequences in the memory unit. Each image sequence includes each representative image captured by a camera, one or more images captured by the camera after each representative image is obtained, and one or more images captured by the camera before each representative image is obtained. The processing unit is further configured to store in the memory unit a plurality of images different from the images in the plurality of image sequences. Each image in the plurality of images is not part of an image sequence in the plurality of image sequences. The processing unit is further configured to display a first image on the display unit. The processing unit is further configured to detect a first input while the first image is being displayed on the display unit. In response to detecting the first input, the processing unit is further configured to perform a first operation including displaying on the display unit at least a portion of the images in the first image sequence in addition to the first image according to a determination that the first image is an image in the first image sequence. The processing unit is further configured to perform a second operation different from the first operation including the first image according to a determination that the first image is an image in a plurality of images different from the images in the plurality of image sequences.

[0024] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, and a processing unit connected to the display unit and the touch sensing surface unit. The processing unit is configured to be able to display a representative image on the display unit. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after acquiring the representative image. The image sequence includes one or more images captured by the camera before acquiring the representative image. The processing unit is further configured to detect an input for modifying the representative image while being able to display the representative image on the display unit. In response to detecting an input for modifying the representative image, the processing unit modifies the representative image, one or more images captured by the camera after acquiring the representative image, and one or more images captured by the camera before acquiring the representative image according to a determination that the device is in a first editing mode, and modifies the representative image without modifying one or more images captured by the camera after acquiring the representative image and without modifying one or more images captured by the camera before acquiring the representative image according to a determination that the device is in a second editing mode different from the first editing mode.

[0025] According to one embodiment, an electronic device includes a display unit configured to display an image, and a processing unit connected to the display unit. The processing unit is configured to be able to display a representative image on a user interface of an application configured to communicate with another electronic device on the display unit. The representative image is one image in an image sequence captured by a camera. The image sequence includes one or more images captured by the camera after the representative image is acquired. The image sequence includes one or more images captured by the camera before the representative image is acquired. While the representative image can be displayed on the display unit, the processing unit is further configured to detect, using the application, an input corresponding to a request to transmit the representative image or a request to select a representative image for transmission from the electronic device to a second electronic device remote from the electronic device. In response to detecting an input corresponding to a request to transmit the representative image or a request to select a representative image for transmission to the second electronic device, the processing unit is further configured to be able to display a first set of optional selections for transmitting at least a portion of the image sequence to the second electronic device according to a determination that the second electronic device is configured to act bidirectionally with the image sequence as a group, and to be able to display a second set of optional selections for transmitting at least a portion of the image sequence to the second electronic device according to a determination that the second electronic device is not configured to act bidirectionally with the image sequence as a group, wherein the second set of optional selections is different from the first set of optional selections.

[0026] According to some embodiments, an electronic device includes a display unit configured to display an image, a camera unit configured to acquire an image, and a processing unit connected to the display unit and the camera unit. The processing unit is configured to display a live preview of a scene on the display unit while in a first media acquisition mode for the camera unit, and to perform scene recognition on the scene. The processing unit is further configured to detect a single activation of a shutter button at a first time while being able to display the live preview of the scene. In response to detecting the single activation of the shutter button at the first time, the processing unit holds a plurality of images acquired by the camera unit in the temporal vicinity of the activation of the shutter button at the first time and groups the plurality of images into a first image sequence according to a determination that the scene meets an action capture criterion based at least in part on the scene recognition performed on the scene. The first image sequence includes a plurality of images acquired by the camera unit before detecting the single activation of the shutter button at the first time, a representative image that represents the first image sequence and is acquired by the camera unit after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera unit after acquiring the representative image. In response to detecting the single activation of the shutter button at the first time and according to a determination that the scene does not meet the action capture criterion, the processing unit is further configured to hold a single image in the temporal vicinity of the activation of the shutter button at the first time.

[0027] According to some embodiments, an electronic device includes a display unit configured to display an image, a touch sensing surface unit configured to detect user input, and a processing unit connected to the display unit and a camera unit. The processing unit is configured to enable an image to be displayed on the display unit. The image is one image in an image sequence captured by a camera. The image sequence includes a representative image. The image sequence includes one or more images acquired by the camera after acquiring the representative image. The image sequence includes one or more images acquired by the camera before acquiring the representative image. The processing unit is further configured to detect a first input while enabling an image in the image sequence to be displayed on the display unit. In response to detecting the first input, the processing unit is further configured to enable a user interface for trimming the image sequence to a subset less than all of the image sequence to be displayed. The user interface includes an area including a representation of an image in the image sequence, a user-adjustable trimming start icon that defines a start image in a subset of the image sequence via a position of the trimming start icon in the area including the representation of the image in the image sequence, and a user-adjustable trimming end icon that defines an end image in a subset of the image sequence via a position of the trimming end icon in the area including the representation of the image in the image sequence. The trimming start icon is located at a first position automatically selected by the device in the area including the representation of the image in the image sequence. The trimming end icon is located at a second position automatically selected by the device in the area including the representation of the image in the image sequence. The processing unit is further configured to detect a second input while enabling the user interface for trimming the image sequence to be displayed. In response to detecting the second input, the processing unit is further configured to trim the image sequence to a subset of the image sequence according to the current position of the trimming start icon and the current position of the trimming end icon.

[0028] According to some embodiments, an electronic device includes a display, a touch sensing surface, one or more optional sensors for detecting the intensity of contact with the touch sensing surface, one or more processors, a memory, and one or more programs, where the one or more programs are configured to be stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing any of the operations of the methods described herein or instructions for causing the execution of operations for performing any of the methods described herein. According to some embodiments, a computer-readable storage medium stores instructions therein, and when the instructions are executed by an electronic device including a display, a touch sensing surface, and one or more optional sensors for detecting the intensity of contact with the touch sensing surface, the device is caused to perform any of the operations of the methods described herein or to execute operations. According to some embodiments, a graphical user interface on an electronic device including a display, a touch sensing surface, one or more optional sensors for detecting the intensity of contact with the touch sensing surface, a memory, and one or more processors for executing the one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, and the elements are updated in response to input as described in any of the methods described herein. According to some embodiments, an electronic device includes a display, a touch sensing surface, one or more optional sensors for detecting the intensity of contact with the touch sensing surface, and means for performing or causing the execution of any of the operations of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device including a display, a touch sensing surface, and one or more optional sensors for detecting the intensity of contact with the touch sensing surface includes means for performing or causing the execution of any of the operations of the methods described herein.

[0029] In this way, an electronic device is provided that improves a method and an interface for capturing a moment captured or recorded by a camera and for two-way interaction with the moment. Such a method and interface can complement or replace conventional methods for capturing still images and videos and for two-way interaction with still images and videos.

Brief Description of the Drawings

[0030] For a better understanding of the various embodiments described, reference should be made to the following "Modes for Carrying Out the Invention" in conjunction with the following drawings. In the drawings, like reference numerals refer to corresponding parts throughout the drawings.

[0031]

Figure 1A

[0032]

Figure 1B

[0033]

Figure 2

[0034]

Figure 3

[0035]

Figure 4A

[0036]

Figure 4B

[0037]

Figure 4C

Figure 4D

Figure 4E

[0038]

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 5I

Figure 5J

Figure 5K

[0039]

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

Figure 6I

Figure 6J

Figure 6K

Figure 6L

Figure 6M

Figure 6N

Figure 6O

Figure 6P

Figure 6Q

Figure 6R

Figure 6S

Figure 6T

Figure 6U

Figure 6V

Figure 6W

Figure 6X

Figure 6Y

Figure 6Z

Figure 6AA

Figure 6BB

Figure 6CC

Figure 6DD

Figure 6EE

Figure 6FF

[0040]

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

Figure 7J

Figure 7K

Figure 7L

Figure 7M

Figure 7N

Figure 7O

Figure 7P

Figure 7Q

Figure 7R

Figure 7S

Figure 7T

Figure 7U

Figure 7V

Figure 7W

Figure 7X

Figure 7Y

Figure 7Z

Figure 7AA

Figure 7BB

Figure 7CC

[0041]

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 8E

Figure 8F

Figure 8G

Figure 8H

Figure 8I

Figure 8J

Figure 8K

Figure 8L

[0042]

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 9E

Figure 9F

Figure 9G

[0043]

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

[0044]

Figure 10F

Figure 10G

Figure 10H

Figure 10I

[0045]

Figure 10J

Figure 10K

Figure 10L

Figure 10M

[0046]

Figure 11A

Figure 11B

Figure 11C

Figure 11D

Figure 11E

[0047]

Figure 11F

Figure 11G

Figure 11H

Figure 11I

[0048]

Figure 12A

Figure 12B

[0049]

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

[0050]

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 20E

Figure 20F

Figure 20G

Figure 20H

Figure 20I

Figure 20J

Figure 20K

Figure 20L

[0051]

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 21E

Figure 21F

Figure 21G

Figure 21H

Figure 21I

Figure 21J

[0052]

Figure 22A

Figure 22B

Figure 22C

Figure 22D

[0053]

Figure 23A

Figure 23B

Figure 23C

Figure 23D

Figure 23E

[0054]

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 24E

[0055]

Figure 25A

Figure 25B

Figure 25C

[0056]

Figure 26A

Figure 26B

Figure 26C

Figure 26D

[0057]

Figure 27A

Figure 27B

Figure 27C

Figure 27D

Figure 27E

[0058]

Figure 28

Figure 29

Figure 30

Figure 31

DETAILED DESCRIPTION OF THE INVENTION

[0059] As described above, portable electronic devices generally capture two types of digital content: still images and videos. Still images are generally captured simply by pressing a shutter button. Still images capture a moment without blur, but lose the details of the moments before and after that moment. Videos record for a long time but can include both interesting and less interesting moments. Generally, significant editing is required to remove the less interesting moments.

[0060] Here, new and improved devices and methods for instant capture and two-way interaction with instants will be described.

[0061] In some embodiments, in response to pressing the shutter button, the device groups together an image sequence that includes a representative image (similar to the moment captured in a conventional still image) and the images acquired before and after the representative image. In this way, pressing the shutter button captures not only the moment but also the moments around that moment. In some embodiments, additional information about the moment, such as audio and metadata, is also captured. From the user's perspective, this process makes capturing a moment (the grouped image sequence including the representative image) as easy as capturing a moment (a single still image). The user only needs to press the shutter button. To distinguish it from a single still image, the term "extended photo" may be used for brevity to refer to the grouped image sequence.

[0062] In some embodiments, while viewing the representative image, the extended photo may "come to life" in response to user input (e.g., a long-press gesture or a deep-press gesture on the extended photo) and play back that moment.

[0063] In some embodiments, while navigating between extended photos, for each extended photo, as the extended photo comes into view on the display, the image taken just before the representative image of the extended photo is shown and / or as the extended photo goes out of the display, the image taken just after the representative image is shown, thereby extending and displaying the moment.

[0064] In some embodiments, while navigating between an extended photo and a conventional still image, the extended photo is "played back" as it comes into view and / or goes out of the display, while for a conventional still image, when the still image is being displayed, additional information (e.g., location data) and / or an animation within the still image is shown.

[0065] In some embodiments, the user can modify the representative image in the enhanced photo and apply the modification to only the representative image in the enhanced photo or to all of the images in the enhanced photo (e.g., the user can switch between an all - apply mode and a still - image editing mode).

[0066] In some embodiments, when each user sends an enhanced photo to another user, each user's device presents various options for sending the enhanced photo depending on whether the other user's device is compatible with the enhanced photo (e.g., if the other user's device is compatible with the enhanced photo, present the option to send the enhanced photo as an enhanced photo, and if the other user's device is not compatible with the enhanced photo, present the option to send only the representative image).

[0067] In some embodiments, the device performs scene recognition (e.g., while in the image - capture mode). If the scene is useful for holding an enhanced photo (e.g., the scene includes movement or faces), in response to pressing the shutter button, the device holds an enhanced photo. If the scene is not useful for holding an enhanced photo (e.g., the scene is a photo of a receipt), in response to pressing the shutter button, the device holds a single still image.

[0068] In some embodiments, the user can trim an image sequence into a subset of the image sequence. The device provides a handle for trimming the sequence at a position automatically selected in the sequence (e.g., based on scene recognition). The handle can also be used to manually trim the sequence.

[0069] The following FIGS. 1A - 1B, 2, and 3 provide an illustration of an exemplary device. FIGS. 4A - 4B, 5A - 5K, 6A - 6FF, 7A - 7CC, 8A - 8L, 20A - 20L, 21A - 21J, 22A - 22D, and 23A - 23E show exemplary user interfaces for capturing related image sequences, navigating related image sequences, and performing operations on or related to related image sequences. FIGS. 9A - 9G are flowcharts showing methods of capturing related image sequences according to some embodiments. FIGS. 10A - 10M are flowcharts showing methods of displaying (or playing back) related image sequences according to some embodiments. FIGS. 11A - 11I are flowcharts showing methods of navigating between photos including related image sequences according to some embodiments. FIGS. 12A - 12B are flowcharts showing methods of performing operations different from still photos on photos including related image sequences according to some embodiments. FIGS. 24A - 24E are flowcharts showing methods of modifying images in an image sequence according to some embodiments. FIGS. 25A - 25C are flowcharts showing methods of transmitting images from an image sequence to a second electronic device according to some embodiments. FIGS. 26A - 26D are flowcharts showing methods of acquiring photos (e.g., extended photos or still photos) using scene recognition according to some embodiments. FIGS. 27A - 27E are flowcharts showing methods of trimming an image sequence (e.g., an extended photo) according to some embodiments. The user interfaces in FIGS. 5A - 5K, 6A - 6FF, 7A - 7CC, 8A - 8L, 20A - 20L, 21A - 21J, 22A - 22D, and 23A - 23E are used to illustrate the processes in FIGS. 9A - 9G, 10A - 10M, 11A - 11I, 12A - 12B, 24A - 24E, 25A - 25C, 26A - 26D, and 27A - 27E.

[0070] Exemplary device Here, refer in detail to the embodiments shown in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0071] In this specification, terms such as first, second, etc. are used in some embodiments to describe various elements, and it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Although both the first contact and the second contact are contacts, they are not the same contact unless the context clearly indicates otherwise.

[0072] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. When used in the description of the various embodiments described and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural form as well, unless the context clearly dictates otherwise. As used herein, the term "and / or" is to be understood to mean any and all combinations of one or more of the associated listed items, and to include the same. It is further to be understood that the terms "includes", "including", "comprises" and / or "comprising", when used herein, specify the presence of the stated feature, integer, step, operation, element and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0073] As used herein, the term "if (when)" is optionally construed to mean "when", "upon", "in response to determining", or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" is optionally construed to mean "upon determining", or "in response to determining", or "upon detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)", depending on the context.

[0074] Embodiments of an electronic device, a user interface for such a device, and related processes for using such a device are described. In some embodiments, the device is a portable communication device such as a mobile phone that includes other functions such as PDA functionality and / or music playback functionality. Exemplary embodiments of portable multifunctional devices include, but are not limited to, the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) devices of Apple Inc. (Cupertino, California). Other portable electronic devices such as laptop or tablet computers having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad) may optionally be used. Also, in some embodiments, it should be understood that the device is not a portable communication device but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0075] In the following description, an electronic device having a display and a touch-sensitive surface is described. However, it should be understood that this electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, a mouse, and / or a joystick.

[0076] This device generally supports various applications such as one or more of a summary note application, a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music playback application, and / or a digital video playback application.

[0077] Various applications executed on this device optionally use at least one common physical user interface device such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or changed for each application and / or within the corresponding application. In this way, the common (such as touch-sensitive surface) physical architecture of the device optionally supports various applications having an intuitive and transparent user interface for the user.

[0078] Next, attention is directed to an embodiment of a portable device with a touch-sensing display. FIG. 1A is a block diagram showing a portable multifunctional device 100 with a touch-sensing display system 112 according to some embodiments. The touch-sensing display system 112 may be referred to as a "touch screen" for convenience, or simply as a touch-sensing display. The device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more light sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., on a touch-sensing surface such as the touch-sensing display system 112 of the device 100). The device 100 optionally includes one or more haptic output generators 167 for generating haptic output on the device 100 (e.g., generating haptic output on a touch-sensing surface such as the touch-sensing display system 112 of the device 100 or the touch pad 355 of the device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0079] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of a device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensing surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device that will be detected by the user's sense of touch. For example, in a situation where the device or a component of the device is in contact with a touch-sensitive surface of the user (e.g., the finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement is interpreted by the user as a tactile sensation corresponding to a perceived change in the physical characteristics of the device or the component of the device. For example, the movement of a touch-sensing surface (e.g., a touch-sensing display or a trackpad) is optionally interpreted by the user as a "down click" or "up click" of a physical actuator button. In some cases, even when there is no movement of the physical actuator button associated with a touch-sensing surface that has been physically pressed (e.g., displaced) by the user's action, the user will feel a tactile sensation such as a "down click" or "up click". As another example, the movement of a touch-sensing surface is optionally interpreted or perceived by the user as the "roughness" of the touch-sensing surface, even when there is no change in the smoothness of the touch-sensing surface. Such an interpretation of the user's touch is subject to the user's individual sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "up click", "down click", "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of a device or a component of the device that produces the described sensory perception of a typical (or average) user.

[0080] Device 100 is merely an example of a portable multifunctional device. It should be understood that device 100 may optionally have more or fewer components than those shown, and may optionally combine two or more components, or have different configurations or arrangements of those components. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0081] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripheral interface 118, is optionally controlled by memory controller 122.

[0082] Peripheral interface 118 can be used to connect the input and output peripheral devices of this device to CPU(s) 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and to execute data processing.

[0083] In some embodiments, peripheral interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0084] The RF (radio frequency) circuit 108 transmits and receives RF signals, also called electromagnetic signals. The RF circuit 108 converts an electrical signal into an electromagnetic signal or an electromagnetic signal into an electrical signal and communicates with a communication network and other communication devices via the electromagnetic signal. The RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. The RF circuit 108 optionally communicates wirelessly with a network such as the Internet, also called the World Wide Web (WWW), an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN), and other devices.Wireless communication optionally uses any one of a plurality of communication standards, communication protocols, and communication technologies, and these communication standards, communication protocols, and communication technologies include Global System for Mobile Communications (GSM) for mobile communication, Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth (R), Wireless Fidelity (Wi-Fi) (R) (for example, IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (for example, Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (for example, Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed as of the filing date of this document, but not limited thereto.

[0085] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral device interface 118 for processing. The audio data is optionally fetched from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 of FIG. 2). The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device such as an output-only headset or a headset having both an output (e.g., headphones for one or both ears) and an input (e.g., a microphone).

[0086] The I / O subsystem 106 connects input / output peripheral devices on the device 100, such as the touch-sensing display system 112 and other input or control devices 116, to the peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, a light sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from the other input or control devices 116 and transmit electrical signals to the other input or control devices 116. The other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, the input controller(s) 160 are optionally connected to (or not connected to any of) any of a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., 208 of FIG. 2) optionally include up / down buttons for adjusting the volume of the speaker 111 and / or the microphone 113. One or more buttons optionally include push buttons (e.g., 206 of FIG. 2).

[0087] The touch-sensing display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from, and / or transmits electrical signals to, the touch-sensing display system 112. The touch-sensing display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, or other user interface controls.

[0088] The touch-sensing display system 112 has a touch-sensing surface, sensor, or set of sensors that receives input from the user based on tactile and / or haptic contact. The touch-sensing display system 112 and the display controller 156 (along with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of contact) on the touch-sensing display system 112 and convert the detected contact into a two-way interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensing display system 112. In one exemplary embodiment, the point of contact between the touch-sensing display system 112 and the user corresponds to the user's finger or stylus.

[0089] The touch-sensing display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although in other embodiments other display technologies are used. The touch-sensing display system 112 and the display controller 156 use capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensing display system 112, including, but not limited to, any of a plurality of touch-sensing technologies now known or later developed, to optionally detect contact and any movement or interruption thereof. In one exemplary embodiment, projected mutual capacitance sensing technology, such as that found in the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) from Apple Inc. (Cupertino, California), is used.

[0090] The touch-sensing display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the video resolution of the touch screen is greater than 400 dpi (e.g., 500 dpi, 800 dpi, or more). The user optionally contacts the touch-sensing display system 112 using any suitable object or accessory, such as a stylus, finger, etc. In some embodiments, the user interface is designed to function with finger-based contact and gestures, and due to the large contact area of the finger on the touch screen, it may be less accurate than stylus-based input. In some embodiments, the device converts rough finger-based input into a precise pointer / cursor position or a command for performing the user's desired action.

[0091] In some embodiments, in addition to the touch screen, device 100 optionally includes a touch pad for activating or deactivating certain functions. In some embodiments, the touch pad, unlike the touch screen, is a touch-sensitive area of the device that does not display visual output. The touch pad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0092] Device 100 also includes a power system 162 for powering various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator (e.g., light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power within a portable device.

[0093] Device 100 also optionally includes one or more optical sensors 164. FIG. 1A shows an optical sensor connected to an optical sensor controller 158 within the I / O subsystem 106. The optical sensor(s) 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor(s) 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with the imaging module 143 (also referred to as a camera module), the optical sensor(s) 164 optionally captures still images, extended photographs, and / or videos. In some embodiments, the optical sensor is disposed on the back of the device 100, opposite the touch-sensitive display system 112 on the front of the device, such that the touch screen is enabled to be used as a viewfinder for video acquisition of still images and / or videos. In some embodiments, another optical sensor is disposed on the front of the device to obtain an image of the user (e.g., for selfies, for video conferencing while the user views other video conferencing participants on the touch screen, etc.).

[0094] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor connected to an intensity sensor controller 159 within I / O subsystem 106. The contact intensity sensor(s) 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electro-force sensors, piezoelectric force sensors, optical force sensors, capacitive touch sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch sensing surface). The contact intensity sensor(s) 165 receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is disposed on or proximate to a touch sensing surface (e.g., touch sensing display system 112). In some embodiments, at least one contact intensity sensor is disposed on the back of device 100, which is opposite a touch screen display system 112 disposed on the front of device 100.

[0095] Device 100 also optionally includes one or more proximity sensors 166. FIG. 1A shows a proximity sensor 166 connected to a peripheral device interface 118. Alternatively, the proximity sensor 166 is connected to an input controller 160 within I / O subsystem 106. In some embodiments, when a multifunctional device is placed near a user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and the touch sensing display system 112 becomes disabled.

[0096] Device 100 also optionally includes one or more haptic output generators 167. FIG. 1A shows a haptic output generator connected to a haptic feedback controller 161 within I / O subsystem 106. The haptic output generator(s) 167 optionally includes one or more electroacoustic devices such as a speaker or other audio components, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other haptic output generating components (e.g., components that convert an electrical signal into a haptic output) on the device. The haptic output generator(s) 167 receives haptic feedback generation instructions from the haptic feedback module 133 and generates haptic outputs on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is disposed on or proximate to a touch sensing surface (e.g., touch sensing display system 112), and optionally generates haptic outputs by moving the touch sensing surface in a vertical direction (e.g., in / out of the surface of device 100) or a lateral direction (e.g., back / forth within the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is disposed on the back of device 100, which is opposite to the touch sensing display system 112 disposed on the front of device 100.

[0097] Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows an accelerometer 168 connected to the peripheral device interface 118. Alternatively, accelerometer 168 is optionally connected to an input controller 160 within I / O subsystem 106. In some embodiments, information is displayed on a touch screen display in a portrait or landscape orientation based on analysis of data received from one or more accelerometers. Device 100 optionally includes, in addition to the accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the position and orientation (e.g., portrait or landscape) of device 100.

[0098] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Further, in some embodiments, as shown in FIGS. 1A and 3, memory 102 stores a device / global internal state 157. The device / global internal state 157 includes one or more of the following. If there is an application in the currently active state, an active application state indicating which application is active, a display state indicating which application, view, or other information occupies various regions of the touch-sensitive display system 112, a sensor state including information obtained from various sensors of the device and other input or control devices 116, and location and / or position information regarding the location and / or orientation of the device.

[0099] The operating system 126 (e.g., an embedded operating system such as iOS (registered trademark), Darwin (registered trademark), RTXC (registered trademark), LINUX (registered trademark), UNIX (registered trademark), OS X (registered trademark), WINDOWS (registered trademark), or VxWorks (registered trademark)) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0100] The communication module 128 facilitates communication with other devices via one or more external ports 124 and includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE (registered trademark), etc.) are adapted to connect to other devices either directly or indirectly through a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is the same as, similar to, and / or compatible with a multi-pin (e.g., 30-pin) connector used within a portion of the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) devices from Apple Inc. (Cupertino, California). In some embodiments, the external port is the same as, similar to, and / or compatible with a Lightning (registered trademark) connector used within a portion of the iPhone (registered trademark), iPod Touch (registered trademark), and iPad (registered trademark) devices from Apple Inc. (Cupertino, California).

[0101] The contact / motion module 130 optionally detects contact with the touch-sensing display system 112 (in conjunction with the display controller 156) and contact with other touch-sensing devices (such as a touch pad or a physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to the detection of contact (e.g., by a finger or a stylus), such as determining whether contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking of movement across the touch-sensing surface (e.g., detecting one or more finger-drag events), and determining whether the contact has ceased (e.g., detecting a finger-up event or an interruption of the contact). The contact / motion module 130 receives contact data from the touch-sensing surface. Determining the movement of the contact point represented by a series of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These operations are optionally applied to a single contact (e.g., contact by one finger or a stylus) or multiple simultaneous contacts (e.g., "multi-touch" / contact by multiple fingers). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0102] The contact / motion module 130 optionally detects gesture inputs by a user. Different gestures on the touch sensing surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Thus, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event and subsequently detecting a finger up (lift off) event at the same position (or substantially the same position) as that finger down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch sensing surface includes detecting a finger down event, subsequently detecting one or more finger drag events, and then detecting a finger up (lift off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected for a stylus by detecting a specific contact pattern for the stylus.

[0103] The graphic module 132 includes various known software components for rendering and displaying graphics on the touch sensing display system 112 or other display, including components for changing visual effects (e.g., luminance, transparency, chroma, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0104] In some embodiments, the graphic module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphic module 132 receives, from an application or the like, one or more codes specifying the graphics to be displayed, along with coordinate data and other graphic characteristic data as needed, and then generates the image data of the screen to be output to the display controller 156.

[0105] The haptic feedback module 133 includes various software components for generating instructions to be used by the haptic output generator(s) 167 to generate haptic output at one or more locations on the device 100 in response to user interaction with the device 100.

[0106] The text input module 134 is optionally a component of the graphic module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0107] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based dialing, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map / navigation widgets).

[0108] The application 136 optionally includes the following modules (or sets of instructions), or subsets or supersets thereof. ● The contacts module 137 (which may also be referred to as an address book or contacts list), ● The phone module 138, ● Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ● Training support module 142, ● Camera module 143 for still images and / or video footage, ● Image management module 144, ● Browser module 147, ● Calendar module 148, ● Widget module 149 that optionally includes one or more of weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widget 149-6, ● Widget creation module 150 for creating user-created widget 149-6, ● Search module 151, ● Video and music playback module 152 optionally composed of a video playback module and a music playback module, ● Note module 153, ● Map module 154, and / or ● Online video module 155.

[0109] Examples of other applications 136 that are optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA (registered trademark)-compatible applications, encryption, digital rights management, speech recognition, and speech replication.

[0110] In conjunction with the touch sensing display system 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contact module 137 in the memory 102 or memory 370), including adding a name(s) to the address book, deleting a name(s) from the address book, associating a phone number(s), email address(es), address(es), or other information with a name, associating an image with a name, classifying and sorting names, providing a phone number and / or email address to initiate and / or facilitate communication by phone 138, video conference 139, email 140, or IM 141, and the like.

[0111] In conjunction with the RF circuitry 108, the audio circuitry 110, the speaker 111, the microphone 113, the touch sensing display system 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the phone module 138 includes executable instructions for entering a sequence of characters corresponding to a phone number, accessing one or more phone numbers in the address book 137, modifying the entered phone number, dialing each phone number, conducting a conversation, and disconnecting or hanging up the call when the conversation is complete. As described above, wireless communication optionally uses any of a plurality of communication standards, protocols, and technologies.

[0112] Together with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch sensing display system 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphic module 132, text input module 134, contact list 137, and phone module 138, the videoconference module 139 includes executable instructions to start, conduct, and end a videoconference between the user and one or more other participants according to the user's instructions.

[0113] Together with the RF circuit 108, touch sensing display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the email client module 140 includes executable instructions to create, send, receive, and manage emails in response to the user's instructions. In cooperation with the image management module 144, the email client module 140 makes it very easy to create and send emails with still images or video clips captured by the camera module 143.

[0114] In conjunction with the RF circuit 108, the touch sensing display system 112, the display controller 156, the touch module 130, the graphic module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering an instant message and a corresponding character string, modifying the entered characters, sending each instant message (e.g., using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocol for phone-based instant messages, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messages), receiving an instant message, and viewing the received instant message. In some embodiments, the instant messages sent and / or received optionally include graphics, photos (e.g., still images), enhanced photos, audio files, video files, and / or other attachments, as supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" means both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0115] In conjunction with the RF circuit 108, touch sensing display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and music playback module 146, the training support module 142 includes executable instructions to generate a training (e.g., having time, distance, and / or calorie consumption goals), communicate with training sensors (within the sports device and smartwatch), receive training sensor data, calibrate sensors used to monitor the training, select and play music for the training, and display, store, and transmit training data.

[0116] In conjunction with the touch sensing display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphic module 132, and image management module 144, the camera module 143 includes executable instructions to capture still images or videos (including video streams) and store them in the memory 102, modify the characteristics of the still images or videos, and / or delete still images or videos from the memory 102.

[0117] In conjunction with the touch sensing display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 includes executable instructions to arrange, modify (e.g., edit) or otherwise manipulate still images and / or video footage, label, delete, present (e.g., within a digital slide show or album), and store them.

[0118] In combination with the RF circuit 108, the touch sensing display system 112, the display system controller 156, the contact module 130, the graphic module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet in accordance with user instructions, including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attached and other files linked to the web pages.

[0119] In combination with the RF circuit 108, the touch sensing display system 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar items, lists of things to do, etc.) in accordance with user instructions.

[0120] In combination with the RF circuit 108, the touch sensing display system 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, and the browser module 147, the widget module 149 is optionally a mini - application (e.g., weather widget 149 - 1, stock widget 149 - 2, calculator widget 149 - 3, alarm clock widget 149 - 4, and dictionary widget 149 - 5) downloaded and used by the user, or a mini - application created by the user (e.g., user - created widget 149 - 6). In some embodiments, the widget includes HTML (HyperText Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript (registered trademark) files. In some embodiments, the widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., Yahoo! (registered trademark) widget).

[0121] In conjunction with the RF circuit mechanism 108, touch sensing display system 112, display system controller 156, contact module 130, graphic module 132, text input module 134, and browser module 147, the widget creation module 150 includes executable instructions for creating a widget (e.g., changing a user-specified location on a web page to a widget).

[0122] In conjunction with the touch sensing display system 112, display system controller 156, contact module 130, graphic module 132, and text input module 134, the search module 151 includes executable instructions for searching, according to a user's instruction, for text, music, sound, images, videos, and / or other files in the memory 102 that match one or more search criteria (e.g., one or more user-specified search terms).

[0123] In conjunction with the touch sensing display system 112, display system controller 156, contact module 130, graphic module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147, the video and music playback module 152 includes executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing the video (on the touch sensing display system 112 or on an external display connected wirelessly or via the external port 124). In some embodiments, the device 100 optionally has the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0124] In conjunction with the touch sensing display system 112, the display controller 156, the contact module 130, the graphic module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, lists of things to do, etc. according to user instructions.

[0125] In conjunction with the RF circuit 108, the touch sensing display system 112, the display system controller 156, the contact module 130, the graphic module 132, the text input module 134, the GPS module 135, and the browser module 147, the map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving directions, data about stores near a particular location or other points of interest, and other location-based data) according to user instructions.

[0126] In conjunction with the touch sensing display system 112, the display system controller 156, the contact module 130, the graphic module 132, the audio circuit 110, the speaker 111, the RF circuit 108, the text input module 134, the email client module 140, and the browser module 147, the online video module 155 includes executable instructions for enabling a user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touch screen 112 or wirelessly or on an external display connected via the external port 124), send an email including a link to a particular online video, and manage in other ways. In some embodiments, instead of the email client module 140, the instant messaging module 141 is used to send a link to a particular online video.

[0127] Each of the modules and applications identified above corresponds to a set of executable instructions for performing one or more of the above functions and to the methods described in this application (e.g., methods performed by a computer and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules may be optionally combined or rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the identified modules and data structures described above. Further, memory 102 optionally stores additional modules and data structures not described above.

[0128] In some embodiments, device 100 is a device in which the operation of a predetermined set of functions on the device is performed exclusively via a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for the operation of device 100, the number of physical input control devices (push buttons, dials, etc.) on device 100 can be optionally reduced.

[0129] A predetermined set of functions that are performed exclusively via a touch screen and / or a touch pad optionally includes navigation between user interfaces. In some embodiments, when touched by a user, the touch pad navigates device 100 from any user interface displayed on device 100 to a main menu, a home menu, or a root menu. In such embodiments, the "menu button" is implemented using the touch pad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of the touch pad.

[0130] FIG. 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) includes an event sorter 170 (e.g., within operating system 126) and a corresponding application 136-1 (e.g., any of the applications 136, 137-155, and 380-390 described above).

[0131] The event sorter 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display system 112 when the application is active or running. In some embodiments, the device / global internal state 157 is used by the event sorter 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 to which the event information is to be delivered.

[0132] In some embodiments, the application internal state 192 includes additional information such as resume information used when the application 136-1 resumes execution, user interface state information indicating whether information is being displayed or is ready to be displayed by the application 136-1, a state queue for allowing the user to return to a previous state or view of the application 136-1, and a redo / undo queue for actions previously taken by the user.

[0133] The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information about sub-events (e.g., a user's touch on the touch-sensing display system 112 as part of a multi-touch gesture). The peripheral device interface 118 transmits information received from the I / O subsystem 106, or sensors such as the proximity sensor 166, the accelerometer(s) 168, and / or the microphone 113 (via the audio circuit 110). The information that the peripheral device interface 118 receives from the I / O subsystem 106 includes information from the touch-sensing display system 112 or the touch-sensing surface.

[0134] In some embodiments, the event monitor 171 transmits requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 transmits event information. In other embodiments, the peripheral device interface 118 transmits event information only when a significant event (e.g., receiving an input that exceeds a predetermined noise threshold and / or lasts longer than a predetermined duration) exists.

[0135] In some embodiments, the event sorter 170 also includes the hit view determination module 172 and / or the active event recognition unit determination module 173.

[0136] The hit view determination module 172 provides a software procedure for determining where within one or more views a sub-event has occurred when the touch-sensing display system 112 displays more than one view. The views are composed of the control unit and other elements that the user can view on the display.

[0137] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as an application view or user interface window, in which information is displayed and touch-based gestures occur. The application view (for each application) where a touch is detected optionally corresponds to a program level within the program hierarchy or view hierarchy of the application. For example, the lowest level view where a touch is detected is optionally called the hit view, and the set of events recognized as appropriate input is optionally determined based at least in part on the hit view of the first touch that initiates a touch-based gesture.

[0138] The hit view determination module 172 receives information associated with sub-events of a touch-based gesture. When an application has a plurality of views organized as a hierarchy, the hit view determination module 172 identifies the hit view as the lowest level view within the hierarchy in which the sub-event should be processed. In most situations, the hit view is the lowest level view where the originating sub-event (i.e., the first sub-event in the sequence of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events associated with the same touch or input source for which it was identified as the hit view.

[0139] The active event recognition unit determination module 173 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition unit determination module 173 determines that all views including the physical location of the sub-event are views that are actively involved, and thus, all views that are actively involved should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area associated with a particular view, the upper-level views in the hierarchy will still continue to be views that are actively involved.

[0140] The event dispatcher module 174 sends event information to the event recognition unit (e.g., event recognition unit 180). In embodiments including the active event recognition unit determination module 173, the event dispatcher module 174 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information obtained by each event receiving unit module 182 in the event waiting queue.

[0141] In some embodiments, the operating system 126 includes an event sorting unit 170. Alternatively, the application 136-1 includes the event sorting unit 170. In still other embodiments, the event sorting unit 170 is an independent module or part of another module stored in the memory 102, such as the contact / motion module 130.

[0142] In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each including instructions for handling touch events that occur within each view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, each event handler 190 includes one or more of event data 179 received from data update unit 176, object update unit 177, GUI update unit 178, and / or event sorting unit 170. Event handler 190 optionally utilizes or calls data update unit 176, object update unit 177, or GUI update unit 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Also, in some embodiments, one or more of data update unit 176, object update unit 177, and GUI update unit 178 are included in each application view 191.

[0143] Each event recognition unit 180 receives event information (e.g., event data 179) from event sorting unit 170 and identifies an event from the event information. Event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, event recognition unit 180 further includes at least a subset of metadata 183 and event distribution instructions 188 (optionally including sub-event distribution instructions).

[0144] The event receiving unit 182 receives event information from the event sorting unit 170. This event information includes information about sub-events, for example, information about a touch or a movement of a touch. Depending on the sub-event, the event information may also include additional information such as the location of the sub-event. When the sub-event is related to the movement of a touch, the event information may also optionally include the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from a vertical orientation to a horizontal orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the posture of the device).

[0145] The event comparison unit 184 compares event information with the definitions of default events or sub-events, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes, for example, definitions of events such as event 1 (187-1), event 2 (187-2), etc. (e.g., the default sequence of sub-events). In some embodiments, the sub-events within event 187 include, for example, touch start, touch end, touch movement, touch cancellation, and multiple touches. In one example, the definition of event 1 (187-1) is a double-tap on the displayed object. The double-tap includes, for example, a first touch (touch start) at a predetermined stage on the displayed object, a first lift-off (touch end) at a predetermined stage, a second touch (touch start) at a predetermined stage on the displayed object, and a second lift-off (touch end) at a predetermined stage. In another example, the definition of event 2 (187-2) is a drag operation on the displayed object. The drag includes, for example, a touch (or contact) at a predetermined stage on the displayed object, movement of the touch across the touch-sensitive display system 112, and lift-off of the touch (touch end). In some embodiments, an event also includes information regarding one or more associated event handlers 190.

[0146] In some embodiments, event definition 187 includes the definition of events for each user interface object. In some embodiments, event comparison unit 184 performs a hit test to determine the user interface object associated with the sub-event. For example, in an application view where three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparison unit uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparison unit 184 selects the event handler associated with the sub-event and the object that induces the hit test.

[0147] In some embodiments, the definition for each event 187 also includes a delaying action that delays the delivery of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0148] If each event recognition unit 180 determines that a series of sub-events does not match any of the events in event definition 186, each event recognition unit 180 enters a state of event impossible, event failure, or event end, and then ignores the next sub-event of the touch-based gesture. In this situation, if there are other event recognition units that remain active with respect to the hit view, the sub-events of the ongoing touch-based gesture are continuously tracked and processed.

[0149] In some embodiments, each event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or a list indicating how to perform sub - event distribution to the event recognition units in which the event distribution system is actively involved. In some embodiments, the metadata 183 includes configurable properties, flags, and / or a list indicating how event recognition units can interact with each other or become capable of interacting. In some embodiments, the metadata 183 includes configurable properties, flags, and / or a list indicating whether sub - events are distributed to various levels within the view hierarchy or program hierarchy.

[0150] In some embodiments, each event recognition unit 180 activates an event handler 190 associated with an event when one or more specific sub - events of the event are recognized. In some embodiments, each event recognition unit 180 distributes event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferring sending) sub - events to the corresponding hit view. In some embodiments, the event recognition unit 180 throws a flag associated with the recognized event, and the event handler 190 associated with the flag catches the flag and executes a predetermined process.

[0151] In some embodiments, the event distribution command 188 includes a sub - event distribution command that distributes event information regarding sub - events without activating an event handler. Instead, the sub - event distribution command distributes event information to an event handler associated with a series of sub - events or to a view actively involved. The event handler associated with a series of sub - events or a view actively involved receives the event information and executes a predetermined process.

[0152] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates the phone numbers used in the contact module 137 or stores the video files used in the video player module 145. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates a new user interface object or updates the position of a user interface object. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 creates display information and sends the display information to the graphic module 132 for display on the touch-sensitive display.

[0153] In some embodiments, the event handler(s) 190 includes or can access the data update unit 176, the object update unit 177, and the GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0154] The foregoing description regarding event handling of a user's touch on the touch-sensitive display also applies to other forms of user input for operating the multifunctional device 100 using an input device, but it should be understood that not all of them are initiated on the touch screen. For example, movement of a mouse and pressing of a mouse button optionally in conjunction with pressing or holding of one or more keyboards, tapping, dragging, scrolling, etc. of contact movement on a touch pad, pen stylus input, movement of the device, verbally detected eye movement, biometric input, and / or any combination thereof are optionally utilized as input corresponding to sub-events that define the event to be recognized.

[0155] FIG. 2 shows a portable multifunctional device 100 having a touch screen (e.g., the touch sensing display system 112 of FIG. 1A) according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, as well as in other embodiments described hereinafter, a user can select one or more of the graphics by performing a gesture on the graphics using, for example, one or more fingers 202 (not drawn to an exact scale in the figure) or one or more styli 203 (not drawn to an exact scale in the figure). In some embodiments, selection of one or more of the graphics occurs when the user interrupts contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, upward, and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, from left to right, upward, and / or downward). In some implementations or situations, an accidental contact with a graphic does not select the graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture that sweeps over an application icon does not optionally select the corresponding application.

[0156] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As described above, menu button 204 is optionally used to navigate to any application 136 within a set of applications optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0157] In some embodiments, device 100 includes a touch screen display, a menu button 204, a push button 206 for turning the device on / off and locking the device, volume adjustment button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 turns the device on / off by pressing the button and holding it pressed for a predetermined time, locks the device by pressing the button and releasing it before the predetermined time has elapsed, and / or unlocks the device or initiates an unlock process. In some embodiments, device 100 also receives verbal input through microphone 113 to activate or deactivate some functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on touch sensing display system 112 and / or one or more haptic output generators 167 for generating haptic output to the user of device 100.

[0158] FIG. 3 is a block diagram of an exemplary multifunctional device including a display and a touch sensing surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or business controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more networks or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) for interconnecting and controlling communications between system components. Device 300 includes an input / output (I / O) interface 330 that includes a display 340, typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touch pad 355, a haptic output generator 357 (such as haptic output generator 167 described above with reference to FIG. 1A) for generating haptic output on device 300, and sensors 359 (e.g., light sensors, acceleration sensors, proximity sensors, touch sensing sensors, and / or haptic intensity sensors similar to haptic intensity sensor 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices, and also optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310.In some embodiments, the memory 370 stores programs, modules, and data structures, or subsets thereof, similar to those stored in the memory 102 of the portable multifunctional device 100 (FIG. 1A). Further, the memory 370 optionally stores additional programs, modules, and data structures not present in the memory 102 of the portable multifunctional device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while the memory 102 of the portable multifunctional device 100 (FIG. 1A) does not optionally store these modules.

[0159] Each of the elements of FIG. 3 identified above is optionally stored in one or more of the aforementioned memory devices. Each of the identified modules above corresponds to a set of instructions for performing the functions described above. The identified modules or programs (i.e., sets of instructions) above need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules are optionally combined or otherwise rearranged. In some embodiments, the memory 370 optionally stores a subset of the identified modules and data structures above. Further, the memory 370 optionally stores additional modules and data structures not described above.

[0160] Attention is now directed to embodiments of a user interface ("UI") optionally implemented on the portable multifunctional device 100.

[0161] Figure 4A shows an exemplary user interface for a menu of applications on a portable multifunctional device 100, according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or a subset or superset thereof. ● Signal strength indicator(s) 402 for wireless communication(s) such as cellular and Wi-Fi signals, ● Time 404, ● Bluetooth indicator 405, ● Battery status indicator 406, ● Tray 408 including icons for frequently used applications such as the following, ○ Icon 416 for the phone module 138 labeled "Phone", optionally including an indicator 414 of the number of missed calls or voicemail messages, ○ Icon 418 for the email client module 140 labeled "Mail", optionally including an indicator 410 of the number of unread emails, ○ Icon 420 for the browser module 147 labeled "Browser", and ○ Icon 422 for the video and music playback module 152, also referred to as the iPod (trademark of Apple Inc.) module 152, labeled "iPod", and ● Icons for other applications such as the following, ○ Icon 424 for the IM module 141 labeled "Message", ○ Icon 426 for the calendar module 148 labeled "Calendar", ○ Icon 428 for the image management module 144 labeled "Photos", ○ Icon 430 for the camera module 143 labeled "Camera", ○ Icon 432 for the online video module 155 labeled "Online Video", ○ An icon 434 for the stock widget 149-2 labeled with "Stock price", ○ An icon 436 for the map module 154 labeled with "Map", ○ An icon 438 for the weather widget 149-1 labeled with "Weather", ○ An icon 440 for the alarm clock widget 149-4 labeled with "Clock", ○ An icon 442 for the training support module 142 labeled with "Training support", ○ An icon 444 for the memo module 153 labeled with "Memo", and ○ An icon 446 for a settings application or module that provides access to settings related to the device 100 and its various applications 136.

[0162] Note that the labels of the icons shown in FIG. 4A are merely illustrative. For example, in some embodiments, the icon 422 for the video and music player module 152 is labeled with "Music" or "Music player". Other labels are optionally used for the various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0163] FIG. 4B shows an exemplary user interface on a device (e.g., device 300 of FIG. 3) having a touch sensing surface 451 separate from the display 450 (e.g., the tablet or touch pad 355 of FIG. 3). Many of the following examples will be described with reference to inputs on the touch screen display 112 (when the touch sensing surface and the display are combined), but in some embodiments, the device detects inputs on a touch sensing surface separate from the display, as shown in FIG. 4B. In some embodiments, the touch sensing surface (e.g., 451 of FIG. 4B) has a major axis (e.g., 452 of FIG. 4B) corresponding to the major axis (e.g., 453 of FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact (e.g., 460 and 462 in FIG. 4B) with the touch sensing surface 451 at positions (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470) corresponding to each position on the display. Thus, when the touch sensing surface is separate from the display, user inputs (e.g., contacts 460 and 462, and their movements) detected by the device on the touch sensing surface (e.g., 451 of FIG. 4B) are used by the device to operate the user interface on the display (e.g., 450 of FIG. 4B) of the multifunctional device. It should be understood that a similar method is optionally used for other user interfaces described herein.

[0164] Furthermore, although the following examples are mainly described with reference to finger inputs (e.g., finger contact, finger tap gesture, finger swipe gesture, etc.), it should be understood that in some embodiments, one or more of those finger inputs may be replaced with inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced with a mouse click (e.g., instead of contact), and then the cursor is moved along the path of the swipe (e.g., instead of the movement of the contact). As another example, a tap gesture may optionally be replaced with a mouse click while the cursor is positioned over the location of the tap gesture (e.g., instead of stopping the detection of contact following the detection of contact). Similarly, it will be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or a mouse and finger contact may optionally be used simultaneously.

[0165] As used herein, the term "focus selector" refers to an input element that indicates the current part of the user interface with which the user is interacting. In some implementations that include a cursor or other location marker, the cursor functions as a "focus selector" when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 of FIG. 3, or touch-sensitive surface 451 of FIG. 4B) and the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), and the particular user interface element is adjusted according to the detected input. In some implementations that include a touch screen display (e.g., touch-sensitive display system 112 of FIG. 1A, or touch screen of FIG. 4A) that enables direct interaction with user interface elements on the touch screen display, the contact detected on the touch screen serves as the "focus selector", so that when an input (e.g., a press input by contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus is moved from one area of the user interface to another area of the user interface without a corresponding cursor movement or contact movement on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another). In these implementations, the focus selector moves in accordance with the movement of the focus between different areas of the user interface. Regardless of the specific form adopted by the focus selector, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to convey (e.g., by indicating to the device the element of the user interface with which the user intends to interact) the user's intended interaction with the user interface.For example, the location of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button while a press input is detected on a touch-sensing surface (e.g., a touch pad or touch screen) indicates that the user intends to activate that corresponding button (as opposed to other user interface elements shown on the device's display).

[0166] As used in this specification and the claims, the term "intensity" of a contact on a touch-sensing surface refers to the force or pressure (force per unit area) of a contact on the touch-sensing surface (e.g., a finger contact or a stylus contact), or a proxy for the force or pressure of a contact on the touch-sensing surface. The intensity of a contact has a range of values that includes at least four distinct values and, more typically, hundreds of different values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of methods and a variety of sensors, or combinations of sensors. For example, one or more force sensors disposed under or adjacent to the touch-sensing surface are optionally used to measure the force at various points on the touch-sensing surface. In some implementations, the force measurements of multiple force sensors are combined (e.g., weighted average or sum) to determine an estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensing surface. Alternatively, the size and / or change in size of the contact area detected on the touch-sensing surface, the capacitance and / or change in capacitance of the touch-sensing surface proximate to the contact, and / or the resistance and / or change in resistance of the touch-sensing surface proximate to the contact are optionally used as a substitute for the force or pressure of a contact on the touch-sensing surface. In some implementations, the alternative measurement of the force or pressure of a contact is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the alternative measurement). In some implementations, the alternative measurement of the force or pressure of a contact is converted to an estimated force or pressure, and this estimated force or pressure is used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of a user input enables user access to additional device functions that might otherwise be difficult for the user to access in a reduced-size device with a limited area for displaying affordances (e.g., on a touch-sensing display) and / or receiving user input (e.g., via a touch-sensing display, a touch-sensing surface, or a physical / mechanical control such as a knob or button).

[0167] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of a trackpad or touchscreen display can be set to any of a wide range of default threshold values without changing the trackpad or touchscreen display hardware. Further, in some implementations, the user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once by a system-level click "intensity" parameter).

[0168] As used in this specification and the claims, the term "characteristic strength" of a contact refers to a characteristic of that contact based on one or more strengths of the contact. In some embodiments, the characteristic strength is based on a plurality of strength samples. The characteristic strength is optionally based on a set of strength samples collected within a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) for a predetermined number of strength samples, or for a predetermined event (e.g., after detecting the contact, before detecting the lift-off of the contact, before or after detecting the start of movement of the contact, before detecting the end of the contact, before or after detecting an increase in the strength of the contact, and / or before or after detecting a decrease in the strength of the contact). The characteristic strength of a contact is optionally based on one or more of the maximum value of the strength of the contact, the mean value of the strength of the contact, the average value of the strength of the contact, the top 10 percentile value of the strength of the contact, a value that is half of the maximum value of the strength of the contact, a value that is 90% of the maximum value of the strength of the contact, etc. In some embodiments, the duration of the contact is used to determine the characteristic strength (e.g., when the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine whether an operation has been performed by a user. For example, the set of one or more strength thresholds can include a first strength threshold and a second strength threshold. In this example, a first operation is performed as a result of a contact having a characteristic strength that does not exceed the first threshold, a second operation is performed as a result of a contact having a characteristic strength that exceeds the first threshold and does not exceed the second threshold, and a third operation is performed as a result of a contact having a characteristic strength that exceeds the second threshold. In some embodiments, the comparison between the characteristic strength and the one or more strength thresholds is not used to determine which of the first or second operations to perform, but is used to determine whether to perform one or more operations (e.g., to perform each optional selection or to omit the performance of each operation).

[0169] In some embodiments, for the purpose of determining the characteristic intensity, a portion of the gesture is specified. For example, the touch sensing surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a starting position to an ending position as the intensity of the contact increases. In this example, the characteristic intensity of the contact at the ending position may be based on only a portion of the swipe contact (e.g., only the portion of the swipe contact at the ending position), rather than the entire continuous swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm may optionally include one or more of a non-loaded moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate small increases or decreases in the intensity of the swipe contact for the purpose of determining the characteristic intensity.

[0170] The figures of the user interface described in this specification optionally include various intensity diagrams showing the current intensity of contact on the touch-sensing surface relative to one or more intensity thresholds (e.g., a contact detection intensity threshold IT0, a light press intensity threshold ITL, a deep press intensity threshold ITD (e.g., at least initially higher than IL), and / or one or more other intensity thresholds (e.g., an intensity threshold IH that is less than IL)). This intensity diagram is typically not part of the displayed user interface but is provided to aid in the interpretation of those figures. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device will perform an operation typically associated with pressing a button of a physical mouse or clicking a trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device will perform an operation different from that typically associated with pressing a button of a physical mouse or clicking a trackpad. In some embodiments, if contact is detected at a characteristic intensity below the light press intensity threshold (e.g., and above a slight contact detection intensity threshold IT0 below which contact is no longer detected), the device moves the focus selector in accordance with the movement of the contact on the touch-sensing surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface figures.

[0171] In some embodiments, the device's response to an input detected by the device is based on a criterion based on the intensity of contact during the input. For example, for some "light press" inputs, a contact intensity that exceeds a first intensity threshold during the input elicits a first response. In some embodiments, the device's response to an input detected by the device is based on a criterion that includes both the intensity of contact and a criterion based on time during the input. For example, for some "deep press" inputs, a contact intensity that exceeds a second intensity threshold greater than the first intensity threshold for a light press during the input elicits a second response only if a delay time has elapsed between the coincidence with the first intensity threshold and the coincidence with the second intensity threshold. This delay time is typically less than 200 ms in duration (e.g., as the magnitude of the second intensity threshold increases, the delay time increases, to 40 ms, 100 ms, or 120 ms). This delay time helps prevent accidental deep press inputs. As another example, for some "deep press" inputs, there is a period of reduced sensitivity that occurs after the time when the first intensity threshold is satisfied. During the time period when the sensitivity is reduced, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps prevent accidental deep press inputs. For other deep press inputs, the response to the detection of the deep press input is not based on a time-based criterion.

[0172] In some embodiments, one or more of the input intensity thresholds and / or the corresponding outputs vary based on one or more factors such as user settings, movement of the contact, input timing, the application being executed, the rate at which the intensity is applied, the current number of inputs, user history, environmental factors (e.g., ambient noise), the position of the focus selector, and the like. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0173] For example, FIG. 4C shows a dynamic intensity threshold 480 that varies over time, based in part on the intensity 476 of a touch input over time. The dynamic intensity threshold 480 is the sum of two components, namely, a first component 474 that decays over time after a predetermined delay time p1 after the touch input 476 is first detected, and a second component 478 that follows the intensity 476 of the touch input over time. The initial high intensity threshold of the first component 474 suppresses the accidental triggering of a "deep press" response, while enabling a quick "deep press" response if the touch input 476 provides sufficient intensity. The second component 478 suppresses the unintentional triggering of a "deep press" response by smoothing out fluctuations in the intensity of the touch input. In some embodiments, when the touch input 476 satisfies the dynamic intensity threshold 480 (e.g., at point 481 in FIG. 4C), a "deep press" response is triggered.

[0174] FIG. 4D shows another dynamic intensity threshold 486 (e.g., intensity threshold ID). FIG. 4D also shows two other intensity thresholds, namely, a first intensity threshold IH and a second intensity threshold IL. In FIG. 4D, the touch input 484 satisfies the first intensity threshold IH and the second intensity threshold IL before time p2, but does not respond at all until the delay time p2 has elapsed at time 482. FIG. 4D also shows that the dynamic intensity threshold 486 decays over time, where the decay starts at time 488 after a predetermined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold IL was triggered). This type of dynamic intensity threshold suppresses the accidental triggering of a response associated with the dynamic intensity threshold ID immediately after or simultaneous with the triggering of a response associated with a lower intensity threshold such as the first intensity threshold IH or the second intensity threshold IL.

[0175] FIG. 4E shows yet another dynamic intensity threshold 492 (e.g., intensity threshold ID). In FIG. 4E, a response associated with the intensity threshold IL is induced after a delay time p2 has elapsed from the first detected touch input 490. At the same time, the dynamic intensity threshold 492 decays after a predetermined delay time p1 has elapsed from the first detected touch input 490. Thus, even if the intensity of the touch input 490 is below another intensity threshold (e.g., intensity threshold IL), after inducing the response associated with the intensity threshold IL, the intensity of the touch input 490 decreases, and subsequently, if the intensity of the touch input 490 increases without releasing the touch input 490, a response associated with the intensity threshold ID can be induced (e.g., at time 494).

[0176] An increase in the characteristic intensity of contact from an intensity below the light press intensity threshold ITL to an intensity between the light press intensity threshold ITL and the deep press intensity threshold ITD may be referred to as a "light press" input. An increase in the characteristic intensity of contact from an intensity below the deep press intensity threshold ITD to an intensity above the deep press intensity threshold ITD may be referred to as a "deep press" input. An increase in the characteristic intensity of contact from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold ITL may be referred to as the detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 may be referred to as the detection of lift-off of contact from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some figures, shaded circles or ellipses are used to represent the intensity of contact on the touch sensing surface. In some figures, unshaded circles or ellipses are used to represent each contact on the touch sensing surface without specifying the intensity of each contact.

[0177] In some embodiments described herein, one or more operations are performed in response to the detection of a gesture that includes respective pressing inputs, or in response to the detection of respective pressing inputs that are performed at respective contacts (or a plurality of contacts), and those respective pressing inputs are detected based at least in part on the detection of an increase in the intensity of a contact (or a plurality of contacts) that exceeds a pressing input intensity threshold. In some embodiments, each operation is performed in response to the detection of an increase in the intensity of a respective contact that exceeds a pressing input intensity threshold (e.g., each operation is performed by a “downstroke” of a respective pressing input). In some embodiments, a pressing input includes an increase in the intensity of a respective contact that exceeds a pressing input intensity threshold and a subsequent decrease in the intensity of the contact that is below the pressing input intensity threshold, and each operation is performed in response to the detection of the subsequent decrease in the intensity of the respective contact that is below the pressing input threshold (e.g., each operation is performed by an “upstroke” of a respective pressing input).

[0178] In some embodiments, the device employs intensity hysteresis to avoid unpredictable inputs sometimes referred to as "jitter", and the device defines or selects a hysteresis intensity threshold having a predetermined relationship to the press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some other reasonable ratio of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of each contact above the press input intensity threshold and a subsequent decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and each operation is performed in response to detecting a subsequent decrease in the intensity of each contact below its hysteresis intensity threshold (e.g., each operation is performed by the "upstroke" of each press input). Similarly, in some embodiments, a press input is detected only if the device detects an increase in the intensity of a contact from below the hysteresis intensity threshold to above the press input intensity threshold and optionally a subsequent decrease in the intensity of the contact to below the hysteresis intensity, and each operation is performed in response to detecting that press input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, depending on the situation).

[0179] For ease of explanation, the description of an operation performed in response to a pressing input associated with a pressing input strength threshold or in response to a gesture including the pressing input is an increase in the intensity of contact above the pressing input strength threshold, an increase in the intensity of contact from an intensity below a hysteresis strength threshold to an intensity above the pressing input strength threshold, a decrease in the intensity of contact below the pressing input strength threshold, or a decrease in the intensity of contact below a hysteresis strength threshold corresponding to the pressing input strength threshold, optionally triggered in response to detection thereof. Additionally, in an embodiment described as an operation being performed in response to detecting a decrease in the intensity of contact below the pressing input strength threshold, the operation is optionally performed in response to detecting a decrease in the intensity of contact that corresponds to and is below a hysteresis strength threshold lower than the pressing input strength threshold. As described above, in some embodiments, the triggering of these responses is also based on a time-based criterion that matches (e.g., a delay time has elapsed between a first strength threshold that matches and a second strength threshold that matches). User Interface and Related Processes

[0180] Attention is now paid to embodiments of a user interface ("UI") and related processing that can be implemented on an electronic device (such as portable multifunctional device 100 or device 300) comprising a display, a touch sensing surface, and optionally one or more sensors for detecting the intensity of contact with the touch sensing surface.

[0181] Figures 5A-5K illustrate an exemplary user interface for capturing a grouped set of related image sequences, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including the processes in Figures 9A-9G, 10A-10M, 11A-11I, 12A-12B, 24A-24E, 25A-25C, 26A-26D, and 27A-27E. For convenience of explanation, some of the embodiments are described in relation to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally a contact of each finger or stylus, a representative point corresponding to the contact of the finger or stylus (e.g., the centroid of each contact or a point associated with each contact), or the centroid of two or more contacts detected on the touch-sensitive display system 112. However, optionally, on a device having a display 450 and a separate touch-sensitive surface 451, similar operations are performed in response to detecting a contact on the touch-sensitive surface 451 while displaying the illustrated user interface on the display 450 with the focus selector.

[0182] Figure 5A shows a media capture user interface 500 for displaying a live preview on the touch screen 112. The live preview shown in Figure 5A is a preview of an image obtained from a camera of the portable multifunctional device 100 (e.g., a camera module 143 having an optical sensor 164). The live preview on the media capture user interface 500 displays the image obtained from the camera in real time or near real time (e.g., within the processing time required for the portable multifunctional device 100 to generate the displayed image). Thus, in the example shown in Figure 5A, the user is viewing a scene where a seagull 502 is flying in the sky above a tree 504, and the portable multifunctional device 100 is reproducing that scene on the touch screen 112 in real time or near real time. In some embodiments, the live preview displays the image at a first resolution (e.g., less than the upper limit of the camera's resolution).

[0183] In this example, the portable multifunctional device 100 is configured to be in an extended media acquisition mode (e.g., the portable multifunctional device 100 is configured to acquire an extended photograph) or another media acquisition mode (e.g., the portable multifunctional device 100 is configured to capture a still image, a video, a burst image, or any other type of image) during a live preview. In some embodiments, the media capture user interface 500 includes an affordance 506 for enabling the extended media acquisition mode (e.g., toggling the on / off state of the extended media acquisition mode). In some embodiments, the media capture user interface 500 includes a visual indicator that the extended media acquisition mode is off. For example, in FIG. 5A, the affordance 506 displays the word "Off".

[0184] In some embodiments, when the extended media acquisition mode is on, the portable multifunctional device 100 provides a visual indicator that the extended media acquisition mode is on (e.g., to indicate that image and / or audio data is being captured while the media capture user interface 500 is being displayed). For example, as shown in FIGS. 5C-5H, when the extended media acquisition mode is on, the affordance 506 is animated by an animation showing a clock with a continuously advancing pointer face 508.

[0185] In some embodiments, as shown in FIG. 5B, the portable multifunctional device 100 detects a selection of the affordance 506 (e.g., detects a tap gesture 510 on the affordance 506) while the extended media acquisition mode is inactive. In response, the portable multifunctional device 100 activates the extended media acquisition mode (as shown in FIGS. 5C-5H by an animation of the affordance 506).

[0186] When the extended media acquisition mode is on, the portable multifunctional device 100 captures media (e.g., images and / or audio). For example, in FIGS. 5C - 5E, since the extended video mode is on, images 512 - 1 (FIG. 5C), 512 - 2 (FIG. 5D), and 512 - 3 (FIG. 5E) are captured (e.g., stored in the permanent memory). In some embodiments, the audio corresponding to the image is also captured (e.g., by the microphone 113) and associated with the image (for later playback with the image as shown in FIGS. 6E - 6I). In some embodiments, other information (e.g., metadata such as time, location, or event data) is acquired and associated with the captured image (for later display as shown in FIGS. 6J - 6M).

[0187] The media capture user interface 500 includes a shutter button 514 (shown as a shutter release icon). As shown in FIG. 5F, the media capture user interface 500 is configured to detect the activation of the shutter button 514 (e.g., by a tap gesture 518). In response to detecting the activation of the shutter button 514, the portable multifunctional device 100 groups a plurality of images 512 acquired by the camera in the temporal vicinity of the activation of the shutter button 514 into an image sequence (e.g., a so - called "extended photo"). The extended photo includes some of the images 512 taken before the tap gesture 518 (e.g., at least a part of images 512 - 1, 512 - 2, and 512 - 3, stored in the permanent memory as described above), a representative image (e.g., image 512 - 4 in FIG. 5F corresponding to the activation of the shutter), and some of the images taken after the tap gesture 518 (e.g., image 512 - 5 in FIG. 5G, image 512 - 6 in FIG. 5H).

[0188] In some embodiments, the representative image is similar to a single image captured when the shutter button of a conventional digital camera is activated in still image mode. In some embodiments, the representative image 512-4 corresponds to an image acquired at the time when the shutter button 514 is activated by a tap gesture 518. In some embodiments, the representative image 512-4 corresponds to an image acquired immediately after detecting the activation of the shutter button 514 at a time that takes into account the shutter lag (the time delay between detection of the activation of the shutter button and capture / storage of the representative image). In some embodiments, the representative image 512-4 acquired by the camera is used to represent the image sequence, for example, in an image presentation mode (as shown in FIG. 6A).

[0189] As described above, in some embodiments, the live preview displays the image at a first resolution. In some embodiments, the image sequence 512 includes the images displayed in the live preview at the first resolution, while the representative image 512-4 acquired by the camera has a second resolution that is higher than the first resolution. For example, as shown in FIG. 5I, the image sequence 512 includes (in time series) images 512-2, 512-3, 512-4, 512-5, and 512-6, with image 512-4 being the representative image. In some embodiments, the representative image 512-4 is stored at a higher resolution than images 512-2, 512-3, 512-5, or 512-6.

[0190] As shown in FIGS. 5F-5H, in some embodiments, after activation of the shutter button 514, the media capture user interface 500 displays an animation while capturing the remaining images included in the grouped image sequence (e.g., the animation is displayed while the portable multifunctional device 100 captures the representative image 512-4 and the images acquired after the representative image 512-4). In FIGS. 5F-5H, the media capture user interface 500 displays an animation of the shutter button 514 that breaks apart and returns to its original state (e.g., to provide the user with an indication that an image and / or audio is still being captured). In some embodiments, the animation is a loop animation that can be continuously extended if the shutter button 514 remains depressed or is reactivated before the camera finishes acquiring the images for the image sequence.

[0191] In some embodiments, when the capture of the image sequence is complete, the portable multifunctional device 100 returns to the functionality described in connection with FIG. 5A, and as a result, the user can acquire a second image sequence, similar to the capture of the image sequence described above.

[0192] As shown in FIGS. 5I - 5K, in some embodiments, the portable multifunctional device 100 displays a second user interface 520 for editing and / or configuring an image sequence (e.g., the second user interface 520 is a user interface in an image sequence editing mode). In FIG. 5I, the images 512 included in the image sequence are images having solid - line boundaries, namely, images 512 - 2, 512 - 3, 512 - 4, 512 - 5, and 512 - 6, and image 512 - 4 is the representative image. Thus, image 512 - 2 is the first image in the image sequence, and there is one image (image 512 - 3) between the first image 512 - 2 and the representative image 512 - 4 (however, in some embodiments, there are a larger integer number of images, such as 5, 10, or 30 images, between the first image and the representative image). Image 512 - 6 is the last image in the image sequence, and there is one image (image 512 - 5) between the representative image 512 - 4 and the last image 512 - 6 (however, in some embodiments, there are a larger integer number of images, such as 5, 10, or 30 images, between the representative image and the last image, and this number does not have to be the same as the number of images between the first image and the representative image). The thick boundary line around image 512 - 4 in FIG. 5I indicates that image 512 - 4 is the representative image.

[0193] As shown in FIG. 5J, the second user interface 520 is configured to receive a request to change a representative image in an image sequence (e.g., receive a touch gesture 522 on an image other than the current representative image 512-4). As shown in FIG. 5K, the device responds to the touch gesture 522 by changing the representative image to image 512-3 (image 512-3 has a thick border in FIG. 5K, indicating that image 512-3 is the new representative image). In some embodiments, the number of images between the first image and the representative image and the number of images between the representative image and the last image are fixed, such that the portable multifunctional device 100 changes the image sequence by adding an image at one end and removing (e.g., deleting or not including) an image at the other end. For example, in FIG. 5K, image 512-1 is added to the image sequence to fix the number of images between the first image and the representative image, and image 512-6 is removed from the image sequence to fix the number of images between the representative image and the last image.

[0194] FIGS. 6A-6FF are diagrams showing exemplary user interfaces for displaying (or playing back) grouped related image sequences (sometimes referred to as extended photos) according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 9A-9G, FIGS. 10A-10M, FIGS. 11A-11I, FIGS. 12A-12B, FIGS. 24A-24E, FIGS. 25A-25C, FIGS. 26A-26D, and FIGS. 27A-27E. The following examples are described in relation to inputs on a touch screen display (where a touch sensing surface and a display are combined as shown on the portable multifunctional device 100), but in some embodiments, the device detects inputs on a touch sensing surface 451 separate from the display 450, as shown in FIG. 4B.

[0195] FIG. 6A shows a user interface 600. The portable multifunctional device 100 displays a representative image 602-3 in a grouped image sequence 602 in the user interface 600. In some embodiments, the user interface 600 is a user interface in an image presentation mode. As will be described later, the image sequence 602 includes the representative image 602-3, one or more images acquired by the camera after the representative image was acquired (e.g., image 602-4 in FIG. 6C and image 602-5 in FIG. 6D), and one or more images acquired by the camera before the representative image was acquired (e.g., image 602-1 in FIG. 6E and image 602-2 in FIG. 6F).

[0196] In some embodiments, the user interface 600 is a user interface in an image management application (e.g., Photos by Apple Inc. (Cupertino, California)). Thus, in some embodiments, the camera that captured the image sequence 602 is part of the portable multifunctional device 100 (e.g., a camera with an optical sensor 164 in conjunction with the imaging module 143 in FIG. 1A). In some embodiments, the image sequence 602 was captured by a camera that is not part of the portable multifunctional device 100 (e.g., the image sequence 602 was transferred to the portable multifunctional device 100 after being captured by a camera of another device). In some embodiments, the image sequence 602 has been acquired in response to detecting activation of the shutter button at a first time, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600. In some embodiments, the representative image 602-3 corresponds to the representative image acquired by the camera, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600.

[0197] In some embodiments, the portable multifunctional device 100 stores a plurality of grouped image sequences, some of which are acquired using the portable multifunctional device 100, and some of which are transferred to the portable multifunctional device 100 after being captured by a camera on a different device. For example, in some situations, a user may acquire (e.g., capture, record) image sequences on a plurality of devices (e.g., in addition to the portable multifunctional device 100, a tablet, laptop, and / or digital camera) related to method 900 / 2600 and transfer the image sequences to the portable multifunctional device 100 for synchronization or other means.

[0198] In some embodiments, the user interface 600 is a user interface in a messaging application (e.g., Messages by Apple Inc. (Cupertino, California)). In some situations, a user may acquire (e.g., capture, record) each image sequence on the user's own portable multifunctional device 100 and may also receive different image sequences from different users (e.g., in a messaging application). Thus, in some embodiments, each image sequence in the plurality of image sequences stored on the portable multifunctional device 100 includes at least one image sequence acquired using a camera on the portable multifunctional device 100 and at least one image sequence acquired using a camera on a different device that is different from the portable multifunctional device 100.

[0199] In some embodiments, the representative image 602-3 is displayed on the user interface 600 when the portable multifunctional device 100 is in the collection view mode.

[0200] The user interface 600 optionally includes another toolbar. For example, as shown in the illustration, the user interface 600 includes an operation toolbar 604 that includes a plurality of affordances 606 (e.g., a send affordance 606-1 that enables the user to send the image sequence 602 to other users using an email application, a messaging application, or other applications; an edit affordance 606-2 that launches a user interface for editing the image sequence 602; a favorite affordance 606-3 that enables the user to indicate that the image sequence 602 is one of the user's favorites; and a delete affordance 606-4 that enables the user to delete the image sequence 602). As another example, the user interface 600 includes a navigation toolbar 608 that includes another plurality of affordances (e.g., a full photo affordance 610-1 that, when activated, navigates to a user interface for navigating the user's photos; and a "done" affordance 610-2 that, when activated, navigates to a different user interface, such as a user interface for acquiring a photo).

[0201] The image sequence 602 shown in FIGS. 6A - 6V depicts a scene where a cat 612 walks into view, lies down on the ground, and then gets up and walks away. At the same time, a chirping bird 614 perches on a tree branch. In reality, it may take several seconds for such a scene to unfold, but in some embodiments, the image sequence 602 is captured within a short time window. For example, in some embodiments, when the representative image 602-3 is acquired, the image sequence 602 depicts moments around an instant (e.g., within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds). For example, the user may be intrigued when the cat 612 lies down in the grass and may take a picture of the representative image 602-3. In some embodiments, the image sequence 602 includes images immediately before and after the acquisition of the representative image 602-3 so that, as described herein, the image sequence 602 includes an extended photo where the moment "comes to life" when the user performs a specific operation on the representative image 602-3.

[0202] FIG. 6B shows a first portion 616-1 of a first input 616 detected by the portable multifunctional device 100 while the portable multifunctional device 100 is displaying the user interface 600. In particular, while the portable multifunctional device 100 is displaying a representative image 602-3 on the user interface 600 displayed on the touch screen 112, the portable multifunctional device 100 detects a first portion 616-1 of the first input 616. In some embodiments, the operations shown in FIGS. 6B-6O are performed in accordance with a determination that the first input 616 meets a predetermined criterion (e.g., a predetermined extended photo display criterion). For example, in some embodiments, if the first input 616 or the first portion 616-1 is a long press gesture as shown in FIGS. 6B-6O, the operations shown in FIGS. 6B-6O are performed (e.g., induced). In some embodiments, the portable multifunctional device 100 includes one or more sensors that detect the intensity of contact with the touch screen 112, and if the first input 616 has a characteristic intensity that meets (e.g., satisfies) a predetermined intensity criterion, the operations shown in FIGS. 6B-6O are performed (e.g., induced) (e.g., as shown in intensity diagram 618, where the first input 616 exceeds a light press threshold ITL, and the intensity diagram is provided to assist in the interpretation of the drawings but is not part of the user interface 600 being displayed). In some embodiments, if the first input 616 or the first portion 616-1 has a predetermined path characteristic (e.g., substantially linear like a long press gesture or a swipe / drag gesture) and meets (e.g., exceeds) a predetermined intensity criterion, the operations shown in FIGS. 6B-6O are performed (e.g., induced). For the purposes of the following explanation, the operations shown in FIGS. 6B-6O will be described as being induced by a long press gesture that exceeds a light press threshold IL, as shown in intensity diagram 618.

[0203] FIG. 6C shows the response of the portable multifunctional device 100 to the continuation of the first portion 616-1 of the first input 616 (from FIG. 6B) detected while the portable multifunctional device 100 is displaying the user interface 600. In particular, as shown in FIGS. 6B-6D, in response to detecting the first portion 616-1 of the first input 616, the portable multifunctional device 100 replaces the display of the representative image within the user interface 600 with the display of one or more images acquired by the camera after acquiring the representative image 602-3 within the user interface 600. According to some embodiments, the one or more images acquired by the camera after acquiring the representative image 602-3 are displayed in sequence while the first portion 616-1 of the first input 616 is detected. Thus, FIG. 6C shows the display of image 602-4, which is the next image acquired after the representative image 602-3 in the image sequence 602. In image 602-4, the cat 612 has risen after lying on the ground and is about to start walking away. The bird 614 remains perched on the tree. Thus, image 602-4 is an image taken after each image 602-3.

[0204] FIG. 6D shows the response of the portable multifunctional device 100 to the continuation of the first portion 616-1 of the first input 616 (from FIG. 6C) detected while the portable multifunctional device 100 is displaying the user interface 600. In FIG. 6D, the portable multifunctional device 100 replaces the display of image 602-4 within the user interface 600 with the display of image 602-5 within the user interface 600, where image 602-5 is the last image acquired by the camera after acquiring the representative image 602-3 in the image sequence 602. Thus, FIGS. 6A-6D show an example where there are two images in the image sequence 602 acquired after the representative image 602-3. However, it should be understood that in various embodiments and / or situations, the image sequence may include various numbers (e.g., integers) of images acquired by the camera after acquiring the representative image 602-3 (e.g., 2, 5, 10, or 20 images).

[0205] In image 602-5, the cat 612 walks away partially from the field of view, and the bird 614 remains stopped on the tree (e.g., image 602-5 is an image captured after each image 602-4). Thus, FIGS. 6B to 6D show an example according to some embodiments in which an extended photo is displayed in the forward direction starting from a representative image by a sufficiently deep long-press gesture, thereby creating an impression that the image has started to move. In some embodiments, unless the first input 616 is terminated during the first portion 616-1, the first portion 616-1 of the first input 616 continues (e.g., has a duration) for the time it takes for all of the images in the image sequence 602 acquired by the camera after the representative image 602-3 to be replaced in order. In such embodiments, the portion of the first input 616 that occurs after the time it takes for all of the images in the image sequence 602 acquired by the camera after the representative image 602-3 to be replaced in order is not considered part of the first portion 616-1, but is considered a subsequent portion of the first input 616, as described below.

[0206] In some embodiments, after obtaining the representative image 602-3, one or more images obtained by the camera are displayed in order at a rate based on the intensity of contact in the first portion 616-1 of the first input 616, as shown in the intensity diagram 618, in response to detecting the first portion 616-1 of the first input 616 (e.g., the display rate increases as the intensity of contact in the first portion 616-1 of the first input 616 increases, and the display rate decreases as the intensity of contact in the first portion 616-1 of the first input 616 decreases). In some embodiments, while sequentially displaying the image sequence 602 between the first portions 616-1 of the first input 616, the portable multifunctional device 100 stays on each image in the image sequence 602 for a duration proportional (or inversely proportional) to the characteristic intensity of the first input 616 while each image is being displayed. Thus, for example, in such embodiments, as shown in the intensity diagram 618 in each figure, since the intensity of the first input 616 is higher while displaying the representative image 602-3 (FIG. 6B) and the image 602-4 (FIG. 6C) than while displaying the image 602-5 (FIG. 6D), the portable multifunctional device 100 stays on the representative image 602-3 (FIG. 6B) and the image 602-4 (FIG. 6C) for a shorter period than the image 602-5 (FIG. 6D).

[0207] In some embodiments, after the display rate thus initially depends on the contact intensity in the first input 616, the subsequent display of the image sequence 602 (in response to detecting subsequent portions of the first input 616, as described below) is performed at a fixed display rate independently of the contact intensity of the subsequent portions of the first input 616. Thus, for example, the portable multifunctional device 100 stays on the images 602-1 (FIG. 6E) and 602-2 (FIG. 6F) for the same duration, despite the difference in intensity in the first input 616 as shown by the intensity diagram 618 in each figure.

[0208] In some embodiments, as described below with reference to FIGS. 6E-6I, in response to detecting a first portion 616-1 of a first input 616, after obtaining a representation 602-3 and displaying one or more images obtained by a camera, the device 100 loops back in response to a second portion 616-2 of the first input 616 and displays the entire image sequence 602 (or displays the entire image sequence 602 as long as the first input 616 and / or its intensity is maintained). In some embodiments, when the image sequence 602 is looped or redisplayed, a cross-fade animation is displayed from the end of the image sequence 602 to the beginning of the image sequence 602.

[0209] FIG. 6E shows a situation where, after detecting a first portion 616-1 of a first input 616, the portable multifunctional device 100 detects a second portion 616-2 of the first input 616 (e.g., the portable multifunctional device 100 continues to detect sufficient contact and / or intensity in a finger gesture). In response to detecting the second portion 616-2 of the first input 616, as shown in FIGS. 6E-6I, the portable multifunctional device 100 displays, within the user interface 600, one or more images obtained by the camera before obtaining a representative image 616-3 (e.g., image 616-1 in FIG. 6E and image 616-2 in FIG. 6F), the representative image 602-3 (FIG. 6G), and one or more images obtained by the camera after obtaining the representative image (e.g., image 602-4 in FIG. 6H and image 602-5 in FIG. 6I), in that order. Thus, in some embodiments, in response to detecting the second portion 616-2 of the first input 616, the entire image sequence 602 is displayed from the first image to the last image in the sequence (e.g., as long as the first input 616 is not interrupted).

[0210] In some embodiments, the second portion 616-2 of the first input 616 is consecutive with and immediately following the first portion 616-1 of the first input 616. In some embodiments, the second portion 616-2 of the first input 616 continues (e.g., has a duration) for the time it takes for all of the images in the image sequence 602 to be replaced in order, as long as the first input 616 does not end between the second portions 616-2.

[0211] In image 602-1 (FIG. 6E), the cat 612 is beginning to enter the field of view and the bird 614 has not yet landed on the tree. In image 602-2 (FIG. 6F), the cat 612 is fully in the field of view and the bird 614 is on the tree. Thus, image 602-2 is an image taken after 602-1, and both images 602-1 and 602-2 are taken before the representative image 602-3 (FIG. 6G). (Each image is the same in the various drawings in which it is shown. For example, image 602-4 is the same in FIGS. 6C and 6H. For the sake of brevity, aspects of these drawings that are the same as aspects described in connection with other drawings are not repeatedly described.)

[0212] In some embodiments, after obtaining the representative image 602-3, during the first portion 616-1 of the first input 616, the sequential display of one or more images (as shown in FIGS. 6B-6D) obtained by the camera, and during the second portion 616-2 of the first input 616, the sequential display of the entire image sequence 602 (as shown in FIGS. 6E-6I), one difference is that in response to detecting the second portion 616-2 of the first input 616, the portable multifunctional device 100 presents the audio 620 corresponding to the image sequence 602 (e.g., via the speaker 111). This is shown in FIGS. 6F-6I by the word "chunchun" uttered by the bird 614. (In this example, the word "chunchun" does not appear in the image but is provided in the drawing to indicate the audio generated by the speaker 111.) In some embodiments, in response to detecting the second portion 616-2 of the first input 616, the entire image sequence 602 is displayed along with the corresponding audio 620 recorded when the image sequence 602 was obtained. In some embodiments, in response to detecting the first portion 616-1 of the first input 616, no audio is presented. In some embodiments, if the first input 616 is maintained after the first complete playback of the image sequence 602 (e.g., as described with reference to FIGS. 6J-6M showing the second playback of the entire image sequence 602), during subsequent sequence playback in response to continuously detecting the first input 616, no audio is presented again. In some embodiments, for a given input, audio is presented only during the first complete playback of the image sequence 602. In some embodiments, for a given input, audio is presented only during different subsequent playbacks of the image sequence 602 (e.g., the second complete playback of the image sequence 602), or during a certain number of predefined playbacks (e.g., the first and second complete playbacks of the image sequence 602).

[0213] In some embodiments, the image sequence 602 is displayed in order at a fixed rate (e.g., at the same rate at which the images were acquired, also referred to as the "1x" rate) in response to detecting a second portion 616-2 of the first input 616 (e.g., during an initial full playback). For example, in some embodiments, during an initial full playback, audio is presented at a 1x rate and the corresponding image sequence 602 is displayed at a 1x rate to make the playback look natural and have natural sound. In some embodiments, the 1x rate means that the portable multifunctional device 100 stays on each image for substantially the same amount of time as the time elapsed while the portable multifunctional device 100 acquired each respective image and the next image.

[0214] In some embodiments, the images in the image sequence 602 are displayed in order at a fixed rate independent of the intensity of the contact of the first input 616. For example, the portable multifunctional device 100 stays on image 602-1 (FIG. 6E) and image 602-2 (FIG. 6F) for the same length of time despite the different input intensities shown in the intensity diagrams 618 of the respective figures. In some embodiments, during a second portion 616-2 of the first input 616, the rate at which the images in the image sequence 602 are displayed in order depends on the intensity of the contact of the first input 616. For example, the rate increases as the intensity of the contact increases.

[0215] In some embodiments, as described below with reference to FIGS. 6J-6M, after the portable multifunctional device 100 displays the image sequence 602 in response to detecting a second portion 616-2 of the first input 616 (e.g., after the device completes an initial full playback of the image sequence 602), the device 100 loops back in response to a third portion 616-3 of the first input 616 and displays the entire image sequence 602 (e.g., as long as the first input 616 and / or its intensity is maintained). In some embodiments, when the image sequence 602 is looped or redisplayed, a cross-fade animation is displayed from the end of the image sequence 602 to the beginning of the image sequence 602.

[0216] Figures 6J through 6M illustrate a situation where, after detecting a second portion 616-2 of a first input 616, a portable multifunctional device 100 detects a third portion 616-3 of the first input 616 (e.g., the portable multifunctional device 100 continues to detect sufficient contact and / or intensity in a finger gesture). In response to detecting the third portion 616-3 of the first input 616, the portable multifunctional device 100 displays, within the user interface 600, in sequence, one or more images (e.g., image 602-1 of FIG. 6J and image 602-2 of FIG. 6K) that a camera acquired before obtaining a representative image 602-3, a representative image 616-3 (FIG. 6L), and one or more images (e.g., image 602-4 of FIG. 6M) that the camera acquired after obtaining the representative image. However, in the illustrated example, the first input 616 ends during the third portion 616-3 and assumes a different function, as detailed below. Thus, in some embodiments, in response to detecting the third portion 616-3 of the first input 616, the entire image sequence 602, from the first image to the last image in the sequence, is displayed until the first input 616 is interrupted (e.g., aborted) before completion of the display of the entire image sequence 602. In some embodiments, the loop continues as long as the first input 616 is maintained, but different functions and / or operations are optionally available (or performed) on different loops. For example, as described above, the portable multifunctional device 100 provides audio during the first full playback. Similarly, in some embodiments, in response to detecting the third portion 616-3 of the first input 616 and displaying a second full playback, the portable multifunctional device 100 displays metadata 622 corresponding to the image sequence 602 (e.g., indicating a date, time, location, or any other information associated with the image sequence 602).

[0217] As described above, FIGS. 6J - 6O show an example where the first input 616 is aborted (e.g., by lift - off or the intensity dropping below a predetermined threshold IT0 as shown in the intensity graph 618 of FIG. 6N) between the third part 616 - 3. FIGS. 6N - 6O show operations performed in response to the end (e.g., abortion or stop) of the first input 616 between the third part 616 - 3 according to some embodiments. In some embodiments, similar operations are performed when the first input 616 ends between the second part 616 - 2 or the first part 616 - 1 of the first input 616. In some embodiments, when the first input 616 ends, the portable multifunctional device 100 determines whether the current display image occurred before or after the representative image 602 - 3. If the current display image occurred after (e.g., was taken later than) the representative image 602 - 3 as shown in FIGS. 6N - 6O, the portable multifunctional device 100 displays the image sequence 602 in reverse chronological order from the current display image (e.g., the image 602 - 4 in FIG. 6N) to the representative image 602 - 3 (e.g., the portable multifunctional device 100 goes back to the representative image 602 - 3). Conversely, if the current display image occurred before (e.g., was taken earlier than) the representative image 602 - 3, the portable multifunctional device 100 displays the image sequence 602 in chronological order from the current display image to the representative image 602 - 3 (e.g., the portable multifunctional device 100 advances the loop in the forward direction to the representative image 602 - 3).

[0218] In some situations, the grouped image sequence is asymmetric with respect to its representative image, which means that the number of images occurring before the representative image is not equal to the number of images occurring after it. In some embodiments, the portable multifunctional device 100 determines which of the time - series direction and the reverse - time - series direction has fewer images between the current display image and the representative image. Then, the portable multifunctional device 100 displays (e.g., moves) the image sequence in the direction with fewer images between the current display image and the representative image.

[0219] Figures 6P to 6V illustrate embodiments in which a user controls the display of images in a grouped image sequence by controlling the intensity of the long - press gesture 636. Figure 6P is similar to Figure 6A and is provided as a starting point for explaining the functions in Figures 6Q to 6V. In some embodiments, when the long - press gesture 636 meets a predetermined criterion, a playback function for the image sequence 602 is triggered. For example, if each long - press input remains below the pressing threshold ITL, the portable multifunctional device 100 does not replace the display of the representative image 602 - 3 in response to the long - press gesture (e.g., instead, the portable multifunctional device 100 performs a different function). In contrast, when the long - press gesture 636 exceeds the light pressing threshold ITL as shown in Figure 6Q, the portable multifunctional device 100 associates the intensity of the long - press gesture 636 (shown in intensity diagram 618) with at least some of the images in the image sequence 602. For example, since the playback function for the image sequence 602 is triggered in Figure 6Q, the portable multifunctional device 100 displays the representative image 602 - 3 when the intensity of the long - press gesture 636 is within the intensity range 618 - 3 (Figures 6Q and 6U). Similarly, the portable multifunctional device 100 displays image 602 - 1 when the intensity of the long - press gesture 636 is within the intensity range 618 - 1, displays image 602 - 2 when the intensity of the long - press gesture 636 is within the intensity range 618 - 2 (Figure 6V), displays image 602 - 4 when the intensity of the long - press gesture 618 is within the intensity range 618 - 4 (Figures 6T and 6R), and displays image 602 - 5 when the intensity of the long - press gesture 636 is within the intensity range 618 - 5 (Figure 6S). Thus, Figures 6Q to 6V show that a user can scrub (e.g., directly control the displayed image in the grouped image sequence) the images in the grouped image sequence in reverse and forward directions based on the intensity of the user input (e.g., as a result, a smooth animation for replacing the images in the grouped image sequence plays forward or backward).

[0220] Figure 6W shows an embodiment in which a user controls the display of an image obtained after a representative image in a grouped image sequence 656 by controlling an input intensity 654. In the example shown in Figure 6W, intensity values between a light press threshold ITL and a deep press threshold ITD are associated with each image obtained after the representative image in the grouped image sequence 656. The intensity diagram shown in Figure 6W shows the input intensity 654 associated with a particular image obtained after the representative image in the grouped image sequence 656, as indicated by those arrows. When the input exceeds the light press threshold ITL, the user can scrub forward and then backward between images obtained after the representative image in the grouped image sequence by controlling the intensity of the input. In some embodiments, when the input exceeds the deep press threshold ITD, the grouped image sequence 656 is replaced at a fixed rate (e.g., advanced) (e.g., the device plays the grouped image sequence 656 at a fixed rate and loops back to the start image after the last image in the grouped image sequence 656 is displayed). Figure 6W also shows an audio 658 and metadata 660 associated with the grouped image sequence 656 (e.g., and provided with the grouped image sequence 658 as described above).

[0221] FIG. 6X shows an embodiment that is mostly similar to the embodiments described with reference to FIGS. 6A-6O, but where the response of device 100 to the first part of the user input is different from the embodiments described with reference to FIGS. 6A-6O. In particular, in the embodiment shown in FIG. 6X, in response to detecting a first part of the user input (e.g., a user input similar to the user input described with reference to FIGS. 6A-6O), device 100 either transitions directly to the first image in the image sequence (e.g., as shown in chart 656), or starts playback by either cross-fading to the first image after playing the image sequence forward briefly (e.g., by playing a few images forward as shown in chart 650) instead of first playing the image sequence forward to the last image in the image sequence.

[0222] In FIG. 6X, playback during user input 648 is represented by one or more curves (e.g., curve 662 and / or curve 664). According to some embodiments, the solid line portions of the curves representing playback during user input 648 represent the images that are played, while the dotted line portions represent the images that are not played.

[0223] Therefore, for example, in Chart 650, device 100 first displays representative image 652. In response to user input 648, device 100 plays three images (e.g., or one image or ten images, etc.) forward to image 660, and then replaces the display of image 660 with the display of the first image 654. According to any of the embodiments described with reference to FIGS. 6A - 6O, device 100 then plays the image sequence forward from the first image 654 (e.g., in subsequent loops, loop the extended photo from start to finish along with sound, metadata, etc.). In this way, device 100 transitions from displaying representative image 652 to displaying the first image 654 (or any other respective previous image) by displaying one or more images acquired after representative image 652. In some embodiments, device 100 cross - fades and / or blurs representative image 652 and / or one or more of the images acquired after the representative image to the first image 654.

[0224] As another example, in Chart 656, device 100 first displays representative image 652. In response to user input 648, device 100 replaces the display of representative image 652 with the display of the first image 654 (or any other respective previous image). According to any of the embodiments described with reference to FIGS. 6A - 6O, device 100 then plays the image sequence forward from the first image 654 (e.g., in subsequent loops, loop the extended photo from start to finish along with sound, metadata, etc.). In this way, device 100 transitions from displaying representative image 652 to directly displaying the first image 654. In some embodiments, device 100 cross - fades and / or blurs representative image 652 to the first image 654.

[0225] In some embodiments, as shown in Chart 656, transitioning from displaying the representative image 652 to displaying the first image 654 (e.g., each previous image) does not include displaying any of the one or more images acquired by the camera after the representative image 652 was acquired (e.g., the device immediately returns to the first image 654).

[0226] In some embodiments, device 100 determines which transition (e.g., the transition shown in Chart 650 or the transition shown in Chart 656) to apply based on the characteristics of the user input 648 (e.g., the contact strength of the first portion of the first input 648). For example, as shown in Intensity Chart 668-2, if the first portion of the first input 648 exceeds the deep press threshold ITD, device 100 transitions according to Chart 656. As shown in Intensity Chart 668-1, if the first portion of the first input 648 does not exceed the deep press threshold ITD, device 100 transitions according to Chart 650.

[0227] In some embodiments, certain images acquired during the acquisition of an image sequence are removed or merged when generating the image sequence. For example, blurry images are removed from (e.g., not included in) the image sequence and / or one or more dark images are combined to enhance the quality of the images in the image sequence. In some situations, the removed and / or merged images result in an image sequence that is not evenly spaced in time. For example, if the camera acquires 10 images per second but 3 images are merged to form each single image in the image sequence, the time elapsed for each single image is greater than that of the other images in the image sequence. Thus, in some embodiments, the playback time of the image sequence is adjusted according to the removal and / or merging of images within the image sequence (e.g., in the above example, when playing the image sequence at 1x playback, device 100 stays at each single image for 0.3 seconds, i.e., three times longer than normal).

[0228] According to some embodiments, FIGS. 6Y-6BB show a user interface that first displays a first image in an image sequence (e.g., an extended photograph). The user interface associates a rate of forward movement through the image sequence with an intensity range above a threshold and a rate of reverse movement through the image sequence with an intensity range below the threshold according to the intensity of the user input contact, and plays the image sequence forward and backward. In some embodiments, the user interface does not loop the image sequence. Thus, when the first image is displayed, contact with an intensity above the threshold causes the image to play forward at a rate proportional to the contact intensity and stop when the last image is reached. If the user lightens the contact so that the contact intensity drops below the threshold, the device plays the image backward at a rate based on the contact intensity and stops when the first image is reached.

[0229] FIG. 6Y shows a user interface 640. In some embodiments, the user interface 640 is a lock screen user interface. For example, the user can lock the device 100 so that the user can put the device 100 in the user's pocket without accidentally operating the device 100 (e.g., accidentally calling someone). In some embodiments, when the user wakes up the device 100 (e.g., by pressing some button), the lock screen user interface 640 is displayed. In some embodiments, a process of unlocking the device 100 is initiated by a swipe gesture on the touch screen 112.

[0230] The portable multifunctional device 100 displays a representative image 602-1 in the grouped image sequence 602 at the user interface 640. In some embodiments, the image sequence 602 is an extended photo selected by the user for the lock screen (e.g., selected in the settings user interface). In the example shown in FIGS. 6Y to 6BB, the image sequence is an extended photo depicting a scene where a cat 612 walks into view and lies down on the ground. At the same time, a bird 614 perches on a tree branch. In some embodiments, the image sequence includes one or more images acquired after the representative image was acquired (e.g., the representative image 602-1 is the first image in the image sequence).

[0231] In some embodiments, the user interface 640 also includes quick access information 642 such as time and date information.

[0232] While the representative image 602-1 is being displayed on the touch screen 112, the device 100 detects an input 644 (e.g., a long press gesture) on the touch screen 112 where the characteristic intensity of the contact exceeds an intensity threshold. In this example, the intensity threshold is a light press threshold ITL. As shown in the intensity diagram 618 (FIG. 6Y), the input 644 includes a contact that exceeds the light press threshold ITL.

[0233] In response to detecting an increase in the characteristic intensity of the contact, the device advances in time series at a rate determined at least in part based on the characteristic intensity of the contact of the input 644 between one or more images obtained after obtaining the representative image 602-1. Thus, for example, the display of the representative image 602-1 (FIG. 6Y) is replaced by the display of the image 602-2 (FIG. 6Z) at the rate indicated by the rate diagram 646 (FIG. 6Y) based on the contact intensity shown in the intensity diagram 618 (FIG. 6Y). The image 602-2 is an image in the image sequence 602 obtained after the representative image 602-1. The display of the image 602-2 (FIG. 6Z) is replaced by the display of the image 602-3 (FIG. 6AA) at a faster rate indicated by the rate diagram 646 (FIG. 6Z) based on the contact intensity shown in the intensity diagram 618 (FIG. 6Z). The image 602-3 is an image in the image sequence 602 obtained after the image 602-2.

[0234] In FIG. 6AA, the intensity of the contact of the input 644 decreases until it falls below the ITL, which is a threshold for playing the image sequence 602 in the reverse or forward direction in this example. As a result, the image 602-3 (FIG. 6AA) is replaced with the previous image 602-2 (FIG. 6BB) at a backward rate based on the current contact intensity of the input 644.

[0235] In some embodiments, the rate shown in the rate diagram 646 (FIGS. 6Y-6AA) is proportional to the absolute value of the difference between the ITL and the current contact intensity of the input 644 as shown in the intensity diagram 618 (FIGS. 6Y-6AA). The direction of movement is based on whether the current contact intensity is above (e.g., forward movement) or below (e.g., backward movement) the ITL (or any other suitable threshold).

[0236] In some embodiments, the forward or reverse rate is determined in real-time or near real-time, such that the user can accelerate or decelerate the movement between images (either forward or backward) by varying the intensity of the contact. Thus, in some embodiments, the user can scrub forward and backward between image sequences 602 (e.g., between the first and last images in an image sequence) by increasing and decreasing the intensity of the contact of user input 644.

[0237] According to some embodiments, FIGS. 6CC-6DD are graphs showing how the movement rate V is related to the current contact intensity I of input 644.

[0238] As shown in FIG. 6CC, the threshold for forward / reverse movement is, in this example, the light press threshold ITL. If the current contact intensity of input 644 is equal to the light press threshold ITL, device 100 does not advance between image sequences in either chronological or reverse chronological order. Thus, device 100 maintains the current displayed image from image sequence 602 (e.g., the movement rate is equal to 0x, where 1x is the rate at which the images in image sequence 602 were acquired). If the current contact intensity of input 644 slightly exceeds the light press threshold ITL, device 100 advances between image sequences at a first rate (e.g., 0.2x) in chronological order. If the current contact intensity of input 644 is below the light press threshold ITL by the same amount, device 100 advances between image sequences at a first rate (e.g., -0.2x, where the minus sign indicates reverse chronological or reverse playback) in reverse chronological order.

[0239] In this example, device 100 has a maximum rate Vmax (e.g., plus or minus 2x) that is reached when the current contact intensity of input 644 reaches the deep press threshold ITD (or any other upper threshold) and the hint threshold ITH (or any other suitable lower threshold), respectively. The movement rate between image sequences is constrained by the maximum reverse rate while the contact is detected on the touch sensing surface.

[0240] FIG. 6DD shows an exemplary response curve where the movement rate exponentially increases from 0x to Vmax between a light press threshold ITL and a deep press threshold ITD. When the deep press threshold ITD is exceeded, the movement rate is constant.

[0241] According to some embodiments, in certain situations, optionally, device 100 will deviate from the movement rate based solely on the current contact intensity of input 644. For example, as device 100 approaches the last image while advancing forward between image sequences 602, device 100 slows the movement rate below what it would be if based solely on the current contact intensity of input 644 (e.g., device 100 slightly "applies the brakes" as it approaches the end of the image sequence). Similarly, in some embodiments, as device 100 approaches the first image while moving backward between image sequences 602, device 100 slows the movement rate below what it would be if based solely on the current contact intensity of input 644 (e.g., device 100 slightly "applies the brakes" as it moves backward and approaches the beginning of the image sequence).

[0242] Figures 6EE - 6FF illustrate embodiments in which the image sequence 602 is displayed and / or played in a user interface 680 for a messaging application (e.g., Messages by Apple Inc., Cupertino, California). In some embodiments, the image sequence 602 is a message in a message conversation displayed in a scrollable region 682 of the messaging application (e.g., the user can scroll up and down to view earlier or later messages in region 682). In some embodiments, the representative image 602 - 3 is first displayed in the messaging application 680. In some embodiments, the image sequence 602 is displayed (e.g., played) in response to a swipe / drag gesture. In some embodiments, the display of the images in the image sequence 602 is controlled by the position of a drag gesture (e.g., the user can scrub forward or backward in the image sequence 602 by moving the drag gesture right or left, respectively). For example, in Figures 6EE - 6FF, a contact 686 moves from position 686 - 1 (Figure 6EE) to position 686 - 2 (Figure 6FF), resulting in the image sequence 602 advancing from the representative image 602 - 3 (Figure 6EE) to the image 602 - 4 (Figure 6FF).

[0243] In some embodiments, a swipe gesture causes the playback of the image sequence 602 to be triggered when the swipe gesture ends (e.g., lifts off). In some embodiments, the image sequence 602 is not played during a drag gesture but is played when the drag gesture ends (e.g., lifts off). In some embodiments, the image sequence 602 is played in response to a long - press gesture (e.g., the image sequence 602 in the messaging application 680 is played according to any of the embodiments described with reference to Figures 6A - 6DD). In some embodiments, the image sequence 602 in the messaging application 680 is played according to any of the embodiments described with reference to Figures 7A - 7CC.

[0244] In some embodiments, the image sequence 602 is displayed (e.g., played back) when the scrollable area of the messaging application is scrolled, and the image may have, in some situations, a text message 684 or other messages sent and received via the messaging application added thereto (e.g., in a speech bubble). In some situations, the user may obtain (e.g., take, capture) each image sequence on the user's own portable multifunction device 100 and may receive different image sequences from different users (e.g., in a messaging application). Thus, in some situations, the plurality of image sequences stored on the portable multifunction device 100 includes at least one image sequence obtained using the camera on the portable multifunction device 100 and at least one image sequence obtained using a camera on another device different from the portable multifunction device 100.

[0245] Figures 7A-7CC illustrate an exemplary user interface for navigating between related image sequences (also referred to as extended photos) in some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 9A-9G, 10A-10M, 11A-11I, 12A-12B, 24A-24E, 25A-25C, 26A-26D, and 27A-27E. The following examples are described in relation to inputs on a touch screen display (where a touch sensing surface is combined with a display as shown on the portable multifunction device 100), but in some embodiments, the device detects inputs on a touch sensing surface 451 separate from the display 450, as shown in Figure 4B.

[0246] The portable multifunctional device 100 displays a user interface 700. The user interface 700 optionally includes another toolbar. For example, as shown in the figure, the user interface 700 includes an operation toolbar 704 that includes a plurality of affordances 706 (e.g., a send affordance 706-1 that enables the user to send a first image sequence 702 to another user using email, messaging, or other applications; an edit affordance 706-2 that launches a user interface for editing the first image sequence 702; a favorite affordance 706-3 that allows the user to indicate that the first image sequence 702 is one of the user's favorites; and a delete affordance 706-4 that enables the user to delete the first image sequence 702). As another example, the user interface 700 includes a navigation toolbar 706 that includes another plurality of affordances (e.g., an all photos affordance 710-1 that navigates to a user interface for navigating the user's photos; a "done" affordance 710-2 that navigates to a different user interface such as a user interface for acquiring photos).

[0247] Figures 7A - 7CC show an example where the portable multifunctional device 100 stores a plurality of image sequences (e.g., the first image sequence 702, the second image sequence 724, the third image sequence 726, and the fourth grouped image sequence 760 in Figures 7A - 7CC). The first grouped image sequence 702 includes a first representative image 702-3 acquired by the camera (Figure 7A), one or more images acquired by the camera after acquiring the first representative image 702-3 (e.g., image 702-4 in Figure 7C and image 702-5 in Figure 7D), and one or more images acquired by the camera before acquiring the first representative image 702-3 (e.g., image 702-2 in Figure 7H and image 702-1 in Figure 7I). Thus, the time series of the first image sequence 702 (e.g., the order in which the camera took the images) is image 702-1, image 702-2, image 702-3, image 702-4, and image 702-5.

[0248] The first image sequence 702 depicts a scene where a cat 712 walks into view, lies down on the ground, and then gets up and walks away. At the same time, a bird 714 perches on a tree branch. In reality, it may take several seconds for such a scene to unfold, but in some embodiments, the first image sequence 702 is captured within a short time window. For example, in some embodiments, any of the image sequences described herein can depict an instant around the moment when each representative image was acquired (e.g., within half a second or one second). For example, the user may be intrigued when the cat 712 lies down on the grass and may decide to take the first representative image 702-3. In some embodiments, the first image sequence 702 includes images immediately before and after the first representative image 702-3 was acquired, such that the first image sequence 702 includes an extended photograph where the moment can "come to life" when the user performs a specific operation on the first representative image 702-3, as described herein.

[0249]

[0250] The second grouped image sequence 724 includes the second representative image 724-3 (FIG. 7F) and at least one or more images that the camera acquired before obtaining the second representative image 724-3 (e.g., image 724-1 in FIG. 7C and image 724-2 in FIG. 7D). The second image sequence 724 includes one or more images that the camera acquired after obtaining the second representative image 724-3. Thus, the time series of the second image sequence 724 (e.g., the order in which the camera captured the images) is image 724-1, image 724-2, and image 724-3. The second image sequence 724 depicts a scene where a seagull 728 is flying far away (image 724-1 in FIG. 7C), flying towards the foreground (image 724-2 in FIG. 7D), and starting to fly away again (image 724-3 in FIG. 7F).The third grouped image sequence 726 includes a third representative image 726-1 and at least one or more images acquired by the camera after acquiring the third representative image 726-1 (for example, image 726-3 in FIG. 7H and image 724-2 in FIG. 7I). The third image sequence 726 includes one or more images acquired by the camera before acquiring the third representative image 726-1. Thus, the time series of the third image sequence 726 (for example, the order in which the camera captured the images) is image 726-1, image 726-2, and image 726-3. The third image sequence 726 depicts a scene where the whale 730 jumps out of the water (image 726-1 in FIG. 7K), swims alongside the boat 732 within the field of view (image 726-2 in FIG. 7I), dives into the sea, and disappears from the field of view (image 726-3 in FIG. 7H).

[0251] The fourth grouped image sequence 760 includes a fourth representative image 760-3 and at least one or more images acquired by the camera before acquiring the fourth representative image 760-1 (for example, image 760-1 in FIG. 7V and image 760-2 in FIG. 7W). Thus, the time series of the fourth image sequence 760 (for example, the order in which the camera captured the images) is image 760-1, image 760-2, and image 760-3. The fourth image sequence 760 depicts a scene where the firework tube 762 is launched (image 760-1 in FIG. 7V), flies through the air (image 760-2 in FIG. 7W), and the firework explodes (image 760-3 in FIG. 7X).

[0252] In some embodiments, the first image sequence 702 is acquired by the camera before the second image sequence 724, and the first image sequence 702 is acquired by the camera after the third image sequence 726.

[0253] In some embodiments, the user interface 700 is a user interface in an image management application (e.g., Photos by Apple Inc. (Cupertino, California)). Thus, in some embodiments, the camera that captured the first image sequence 702 (and / or the second image sequence 724, the third image sequence 726, etc.) is part of the portable multifunctional device 100 (e.g., a camera with an optical sensor 164 in conjunction with the imaging module 143 of FIG. 1A). In some embodiments, the first image sequence 702 is captured by a camera that is not part of the portable multifunctional device 100 (e.g., the first image sequence 702 is captured by a camera of another device and then transferred to the portable multifunctional device 100). In some embodiments, the first image sequence 702 is obtained in response to detecting activation of a shutter button at a first time, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600. In some embodiments, the first representative image 702-3 corresponds to a representative image acquired by the camera, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600.

[0254] In some embodiments, some of the plurality of image sequences are acquired using the portable multifunctional device 100, and some of the plurality of image sequences are transferred to the portable multifunctional device 100 after being captured by a camera on a different device. For example, in some situations, the user may acquire (e.g., take, capture) image sequences described in connection with method 900 / 2600 on multiple devices (e.g., in addition to the portable multifunctional device 100, a tablet, a laptop, and / or a digital camera), and synchronize or otherwise transfer the image sequences to the portable multifunctional device 100.

[0255] In some embodiments, the user interface 700 is the user interface in a messaging application (e.g., Messages by Apple Inc. (Cupertino, California)). In some embodiments, the first image sequence 702 is a message in a message conversation displayed in a scrollable area of the messaging application. The first image sequence 702 is displayed when the scrollable area of the messaging application is scrolled, and in some situations, text messages or other messages sent and received via the messaging application may be added to the images (e.g., in speech bubbles). In some situations, the user may obtain each image sequence on the user's own portable multifunctional device 100 (e.g., take, capture), and may receive different image sequences from different users (e.g., in a messaging application). Thus, in some situations, the plurality of image sequences stored on the portable multifunctional device 100 includes at least one image sequence obtained using the camera on the portable multifunctional device 100 and at least one image sequence obtained using the camera on another device different from the portable multifunctional device 100.

[0256] In some embodiments, the representative image 702-3 is displayed in the user interface 700 (e.g., displayed in an image management application or a messaging application when the user scrolls between the user's images or messages).

[0257] FIG. 7A shows the user interface 700. The portable multifunctional device 100 displays a first representative image 702 in a movable first area 734 on the touch screen 112 in the user interface 700. It should be understood that the boundaries of the movable first area 734 are not always displayed on the touch screen 112 and are provided to assist in the description of the drawings.

[0258] FIG. 7B shows a portable multifunctional device 100 that detects a drag gesture 736 (starting at position 736-1 on the touch screen 112). In some embodiments, the operations shown in FIGS. 7B-7K are performed according to a determination that the drag gesture 736 meets a predefined criterion (e.g., a predefined next expanded photo navigation criterion). For example, in some embodiments, the operations shown in FIGS. 7B-7F are performed (e.g., induced) when the drag gesture 736 has predefined path characteristics (e.g., the drag gesture 736 is characterized by a horizontal (or vertical) speed, i.e., the drag gesture is more horizontal (or vertical) than vertical (or horizontal) in the orientations shown in FIGS. 7A-7CC). In some embodiments, the portable multifunctional device 100 includes one or more sensors that detect the intensity of contact with the touch screen 112, and the operations shown in FIGS. 7B-7F are performed (e.g., induced) when the drag gesture 736 has a characteristic intensity that meets (e.g., satisfies) a predefined intensity criterion (e.g., when exceeding a light press threshold ITL as described elsewhere herein). In some embodiments, the operations shown in FIGS. 7B-7F are performed (e.g., induced) when the drag gesture 736 has predefined path characteristics (e.g., the drag gesture 736 is characterized by a horizontal speed) and meets (e.g., exceeds) a predefined intensity criterion.

[0259] As shown in FIG. 7A, the user interface 700 shows the display of the first representative image 702-3 in the image presentation mode. In some embodiments, as shown in FIG. 7A, the movable first area 734 is an area that displays images in the first image sequence without displaying images from image sequences other than the first image sequence.

[0260] In FIG. 7B, the drag gesture 736 is in the left direction. Therefore, as shown in FIGS. 7C - 7D, the portable multifunctional device 100 moves the first area 734 to the left. In addition, in the movable first area 734 of the user interface 700, the portable multifunctional device 100 replaces the display of the first representative image 702 - 3 with a time - series display of at least a part of one or more images for the first image sequence 702 acquired by the camera after the first representative image is acquired (i.e., image 702 - 4 in FIG. 7C and image 702 - 5 in FIG. 7D). That is, the portable multifunctional device 100 displays an animated display of the first image sequence 702 in the first area.

[0261] In some embodiments, the time - series display of at least a part of one or more images for the first image sequence 702 acquired by the camera after the first representative image is acquired in the first area 734 is performed according to the movement of the contact in the drag gesture 736. Thus, when the leftward movement of the drag gesture 736 accelerates, the display of the time - series progression of the images in the first area 734 also accelerates. When the leftward movement of the drag gesture 736 decelerates, the display of the time - series progression of the images in the first area 734 also decelerates. When the leftward movement of the drag gesture 736 stops, the display of the time - series progression of the images in the first area 734 also stops. And when the movement of the drag gesture 736 reverses direction (e.g., from a left - direction drag gesture to a right - direction drag gesture), the display of the progression of the images in the first image sequence 702 in the first area 734 also reverses, and the images are shown in reverse time - series according to the movement of the reverse - direction drag gesture 736. More generally, in some embodiments, in each image sequence, the display of the progression of the images in each area for each image sequence is performed according to the movement of the contact in the drag gesture.

[0262] In some embodiments, the user induces the operations shown in FIGS. 6A-6FF by changing one or more characteristics of the drag gesture 736 in a predefined manner. For example, in some embodiments, if the user stops the drag gesture 736 and presses deeper on the touch screen 112, the portable multifunctional device 100 plays the first image sequence 702 as described with reference to FIGS. 6A-6FF even if only a portion of the first area 734 is on the display. In some embodiments, the portable multifunctional device 100 is configured to detect a change or modification of one or more characteristics of the drag gesture 736.

[0263] FIGS. 7C-7D also show, in some embodiments, the portable multifunctional device 100 moving the second area 738 to the left in accordance with a leftward drag gesture 736. In some embodiments, moving the second area 738 to the left includes moving at least a portion of the second area 738 from the left into the touch screen 112. In some embodiments, the movable second area 738 is an area that displays the images in the second image sequence 724 without displaying images from image sequences other than the second image sequence 724 (e.g., the first image sequence 702 and the third image sequence 726 are not displayed in the movable second area 738). In some embodiments, as shown in FIG. 7C, the movable second area 738 is adjacent to the movable first area 734 (e.g., to the right of the movable first area 734). In some embodiments, while moving the second area 738 to the left, the portable multifunctional device 100 displays in chronological order at least a portion of one or more images for the second image sequence 724 that the camera acquired before acquiring the second representative image 724-3 in the second area 738.

[0264] In some embodiments, the movement of the first area 734 corresponds to the movement of the drag gesture 736. For example, in some embodiments, the movement of the first area 734 between FIGS. 7B and 7C is proportional to the distance between positions 736-1 (FIG. 7B) and 736-2 (FIG. 7C). Similarly, in some embodiments, the movement of the first area 734 between FIGS. 7C and 7D is proportional to the distance between positions 736-2 (FIG. 7C) and 736-3 (FIG. 7D), thereby giving the user the impression of pulling the movable first area 734. In some embodiments, as shown in FIGS. 7C-7D, moving the first area 734 to the left includes moving at least a portion of the first area 734 outside the touch screen 112 to the left.

[0265] In some embodiments, as shown in FIGS. 7B-7D, the first area 734 and the second area 738 move at the same rate in the touch screen 112 (e.g., the movement distance of the first area 734 and the movement distance of the second area 738 correspond to the movement distance of the drag gesture 736). In some embodiments, as shown in FIGS. 7L-7P, the first area 734 and the second area 738 move at different rates. For example, in FIGS. 7L-7P, the movement of the second area 738 in response to the drag gesture 752 is less than the movement of the first area 734 in response to the drag gesture 752 (e.g., the movement distance of the first area 734 coincides with the movement distance of the drag gesture 752, and the movement distance of the second area 738 is a portion such as 50% of the movement distance of the drag gesture 752).

[0266] In some embodiments, the time-series display of at least a portion of one or more images of the second image sequence 724 acquired by the camera before acquiring the second representative image 724-3 in the second area 738 is performed in accordance with the movement of the contact in the drag gesture 736 (e.g., in a manner similar to that described above with respect to the first image sequence 702). For example, during the drag gesture 736, the images in the first area 734 and the images in the second area 738 both advance at the same rate based on the movement of the drag gesture 736. In some embodiments, for example, as will be described below with reference to FIGS. 7L-7P, during the drag gesture 752, the images in the first area 734 and the images in the second area 738 are advanced at different rates. For example, in FIGS. 7L-7P, the rate at which the images advance in the second area in response to the drag gesture 752 is less than the rate at which the images advance in the first area 734 in response to the drag gesture 752 (e.g., 50% of the rate at which the images advance in the first area 734 in response to the drag gesture 752).

[0267] In some embodiments, as an alternative to the example shown in FIGS. 7B-7D, while moving the second area 738 to the left, the second area 738 displays only the second representative image 724-3 for the second image sequence without displaying other images in the second image sequence 724.

[0268] In some embodiments, user interface 700 includes a next icon 750-1 (e.g., FIG. 7A) and a previous icon 750-2. In some embodiments, similar to detecting a drag gesture in a first direction, detecting the activation of the next icon 750-1 also animates an image from the first image sequence 702 to the first area 734 and animates an image from the second image sequence 724 to the second area 738. In some embodiments, detecting the activation of the next icon 750-1 causes the display of the second representative image 724-3 to replace the display of the first representative image 702-3 without animating an image from the first image sequence 702 to the first area 734 and without animating an image from the second image sequence 724 to the second area 738. In some embodiments, detecting the activation of the next icon 750-1 causes the display of the second representative image 724-3 to replace the display of the first representative image 702-3 without displaying other images in the first image sequence or the second image sequence.

[0269] The operations performed in FIGS. 7B-7D are described in relation to left / right movement of the drag gesture 736, but similar operations are envisioned in relation to up / down movement of the drag gesture and are intended to be included within the scope of the claims unless otherwise specified. For example, in some embodiments, as an alternative to the examples shown in FIGS. 7B-7D, instead of moving the second area 738 into the touch screen 112 in a left direction, if the drag gesture 736 is a leftward or downward gesture, the second area 738 is below the first area 734 in the Z-layer (front-back) order and appears (e.g., becomes visible) when the first area 734 moves out of the touch screen 112 to the left (or towards the bottom).

[0270] As shown in FIG. 7F, in some embodiments, after moving the second area 738 to the left, as described in connection with FIGS. 7B-7D, the portable multifunctional device 100 displays the second representative image 724-3 on the user interface 700. In some embodiments, the display of the second representative image 724-3 is performed according to the operations described below in connection with FIGS. 7E-7F.

[0271] As shown in FIG. 7E, in some embodiments, while moving the first area 734 and the second area 738, the portable multifunctional device 100 detects the end (e.g., lift-off) of the drag gesture 736. In response, the portable multifunctional device 100 determines whether the drag gesture 736 meets the next sequence navigation criteria. For example, in some embodiments, the next sequence navigation criteria are met when the movement of the first area 734 causes the first area 734 to cross half of the touch screen 112 (e.g., the midpoint of the first area 734 moves outside the touch screen 112). In some embodiments, as shown in FIG. 7E, the next sequence navigation criteria are met when the movement of the first area 734 and the movement of the second area 738 cause the boundary 740 between the first area 734 and the second area 738 to cross the midpoint 742 of the touch screen 112 (or any other suitable predefined point such as one-third or one-fourth across the user interface 700, even if not centered within the touch screen 112). In some embodiments, the next sequence navigation criteria are met when the speed of the drag gesture 736 meets a predefined speed criteria (e.g., when the average speed or instantaneous speed of the drag gesture 736 exceeds a speed threshold). In some embodiments, the next sequence navigation criteria are met when the speed of the drag gesture 736 indicates a "flick" gesture.

[0272] As shown in FIG. 7F, when the next sequence navigation criterion is satisfied, the portable multifunctional device 100 moves the first area 734 completely out of the touch screen 112 (e.g., by moving the first area 734 further to the left until the first area 734 is completely out of the touch screen 112), and moves the second area 738 completely into the touch screen 112 (e.g., by moving the second area 738 further to the left until the second area 738 is completely within the touch screen 112). As a result, the portable multifunctional device 100 displays the second representative image 724-3 (FIG. 7F). In this way, in some embodiments, by ending the drag gesture 736 when the next sequence navigation criterion is satisfied, the user is given the impression that the second representative image 724-3 snaps onto the user interface 700.

[0273] Conversely, in some embodiments, when the next sequence navigation criterion is not satisfied, the portable multifunctional device 100 moves the second area 738 completely out of the touch screen 112 (e.g., by moving the second area 738 to the right until the second area 738 is completely out of the touch screen 112), and moves the first area 734 completely into the touch screen 112 (e.g., by moving the first area 734 to the right and then back until the first area 734 is completely within the touch screen 112). As a result, the portable multifunctional device 100 redisplay the first representative image 702-3 (e.g., returns to the view shown in FIG. 7A). In this way, in some embodiments, by ending the drag gesture 736 when the next sequence navigation criterion is not satisfied, the user is given the impression that the first representative image 702-3 snaps back onto the user interface 700. In some embodiments, while the portable multifunctional device 100 moves the first area 734 completely into the touch screen 112 and moves the second area 738 completely out of the touch screen 112, the first image sequence 702 and the second image sequence 724 are displayed in the reverse direction (e.g., in reverse chronological order).

[0274] Figures 7G - 7K show features similar to those in Figures 7B - 7F, but Figures 7B - 7F show next sequence navigation according to some embodiments, while Figures 7G - 7K show previous sequence navigation according to some embodiments, which is different.

[0275] Figure 7G shows the portable multifunctional device 100 detecting a drag gesture 744 (starting at position 744 - 1 on the touch screen 112). In some embodiments, the operations shown in Figures 7G - 7K are performed according to a determination that the drag gesture 744 meets a predefined criterion (e.g., a predefined previous photo navigation criterion). In some embodiments, the predefined criterion for navigating towards the previous photo (e.g., the previous grouped image sequence) is similar to the predefined criterion for navigating towards the next photo (e.g., the next grouped image sequence) described with reference to Figures 7B - 7F, but the two respective drag gestures are different in that they are generally in opposite directions (or at least in principle in opposite directions).

[0276] Figure 7G is similar to Figure 7B, except that the drag gesture 744 is in the opposite direction of the drag gesture 736 (Figure 7B) while the portable multifunctional device 100 is displaying the first representative image 702-3. That is, in Figure 7G, the drag gesture 744 is in the right direction. Accordingly, the portable multifunctional device 100 moves the first area 734 to the right, as shown in Figures 7H-7I. In addition, the portable multifunctional device 100 replaces the display of the first representative image 702-3 in the movable first area 734 of the user interface 700 with a reverse chronological display of at least a portion of one or more images of the first image sequence 702 that the camera acquired before acquiring the first representative image 702-3 (i.e., image 702-2 in Figure 7H and image 702-1 in Figure 7I). In some embodiments, as described above with reference to Figures 7B-7F, in each image sequence, the display of the progression of the images in each area follows the movement of the contact in the drag gesture (e.g., the movement from position 744-1 in Figure 7G to position 744-2 in Figure 7H and to position 744-3 in Figure 7I).

[0277] Figures 7H - 7I also show that in some embodiments, in accordance with a right - direction drag gesture 744, the portable multifunctional device 100 moves a third area 746 to the right. In some embodiments, moving the third area 746 to the right includes moving at least a portion of the third area 746 into the touch screen 112 in the right - direction. In some embodiments, the movable third area 746 is an area that displays images in the third image sequence 726 without displaying images from image sequences other than the third image sequence 726 (for example, the first image sequence 702 and the second image sequence 724 are not displayed in the movable third area 746). In some embodiments, as shown in FIG. 7H, the movable third area 746 is adjacent to the movable first area 734 (for example, to the left of the movable first area 734, on the opposite side of the movable second area 738). In some embodiments, while moving the third area 746 to the right, the portable multifunctional device 100 displays at least a portion of one or more images for the third image sequence 726 that the camera acquired before acquiring the third representative image 726 - 1 in the third area 746 in reverse chronological order.

[0278] In some embodiments, the reverse chronological display of at least a portion of one or more images for the third image sequence 726 that the camera acquired after acquiring the third representative image 726 - 1 in the third area 746 is performed in accordance with the movement of the contact in the drag gesture 744 (for example, in a manner similar to that described above with respect to the first image sequence 702). For example, during the drag gesture 744, the images in the first area 734 and the images in the third area 746 both reverse at the same rate based on the movement of the drag gesture 744.

[0279] In some embodiments, similar to detecting the drag gesture 744 in the second direction, by detecting the activation of the forward icon 750-2 (FIG. 7A), an image from the first sequence 702 is animatedly displayed in the first area 734, and an image from the third sequence 726 is animatedly displayed in the third area 744. In some embodiments, by detecting the activation of the forward icon 750-2, the display of the third representative image 726-1 replaces the display of the first representative image 702-3 without animatedly displaying the image from the first sequence 702 in the first area 734 and without animatedly displaying the image from the third sequence 726 in the third area 744. In some embodiments, by detecting the activation of the forward icon 750-2, the display of the third representative image 726-1 replaces the display of the first representative image 702-3 without displaying other images in the first sequence 702 or the third sequence 726.

[0280] In some embodiments, as an alternative to the examples shown in FIGS. 7G-7I, while the third area 746 is moved to the right in response to the drag gesture 744, the third area 746 displays only the third representative image 726-1 for the third image sequence 726 without displaying other images in the third image sequence 726.

[0281] The operations performed in FIGS. 7G-7I are described in relation to the left / right movement of the drag gesture 744, but similar operations are envisioned in relation to the up / down movement of the drag gesture and are intended to be included within the scope of the claims unless otherwise specified. For example, in some embodiments, as an alternative to the examples shown in FIGS. 7G-7I, instead of moving the third area 746 into the touch screen 112 to the left, if the drag gesture 744 is a rightward or upward gesture, the third area 746 is below the first area 734 in the Z-layer (fore-aft) order, and when the first area 734 moves out of the touch screen 112 to the right (or upward), the third area 746 appears (e.g., becomes visible).

[0282] As shown in FIG. 7K, in some embodiments, after moving the third area 746 to the right, as described in connection with FIGS. 7G-7I, the portable multifunctional device 100 displays a third representative image 726-1 on the user interface 700. In some embodiments, the display of the third representative image 726-1 is performed according to the operations described below in connection with FIGS. 7J-7K.

[0283] As shown in FIG. 7J, in some embodiments, while moving the first area 734 and the third area 746, the portable multifunctional device 100 detects the end (e.g., lift-off) of the drag gesture 744. In response, the portable multifunctional device 100 determines whether the drag gesture 744 meets the previous sequence navigation criteria. For example, in some embodiments, the previous sequence navigation criteria are met when the movement of the first area 734 causes the first area 734 to cross half of the touch screen 112 (e.g., the midpoint of the first area 734 moves outside the touch screen 112). In some embodiments, as shown in FIG. 7J, the previous sequence navigation criteria are met when the movement of the first area 734 and the movement of the third area 746 cause the boundary 748 between the first area 734 and the third area 746 to cross the midpoint 742 of the touch screen 112 (or a midpoint of the user interface 700 that is not the center within the touch screen 112, or any other suitable predefined point).

[0284] As shown in FIG. 7K, when the previous sequence navigation criteria are met, the portable multifunctional device 100 moves the first area 734 completely outside the touch screen 112 (e.g., by moving the first area 734 further to the right until the first area 734 is completely outside the touch screen 112), and moves the third area 746 completely inside the touch screen 112 (e.g., by moving the third area 746 further to the right until the third area 746 is completely inside the touch screen 112). As a result, the portable multifunctional device 100 displays the third representative image 726-1 (FIG. 7K). In this way, in some embodiments, by ending the drag gesture 744 when the previous sequence navigation criteria are met, the user is given the impression that the third representative image 726-1 snaps onto the user interface 700.

[0285] Conversely, in some embodiments, when the previous sequence navigation criteria are not met, the portable multifunctional device 100 moves the third area 746 completely outside the touch screen 112 (e.g., by moving the third area 746 to the left until the third area 746 is completely outside the touch screen 112), and moves the first area 734 completely inside the touch screen 112 (e.g., by moving the first area 734 to the left and then back until the first area 734 is completely inside the touch screen 112). As a result, the portable multifunctional device 100 redisplays the first representative image 702-3 (e.g., returns to the view shown in FIG. 7A). In this way, in some embodiments, by ending the drag gesture 744 when the previous sequence navigation criteria are not met, the user is given the impression that the first representative image 702-3 snaps back onto the user interface 700. In some embodiments, while the portable multifunctional device 100 moves the first area 734 completely inside the touch screen 112 and moves the third area 746 completely outside the touch screen 112, the first image sequence 702 and the third image sequence 726 are displayed in a forward direction (e.g., in time series).

[0286] Figures 7L to 7P show an embodiment in which the first area 734 and the second area 738 move on the touch screen 112 at different rates in response to the drag gesture 752. Figure 7L is similar to Figure 7A and is provided as a starting point for the functions shown in Figures 7M to 7P. As shown in Figures 7M to 7O, the drag gesture 752 moves from position 752-1 (Figure 7M) to position 752-2 (Figure 7N) and then to position 752-3 (Figure 7O). In some embodiments, the movement of the first area 734 corresponds to the movement of the drag gesture 752. For example, when the drag gesture 752 moves 1 centimeter (cm), the first area 734 moves 1 cm. In some embodiments, the second area 738 moves based on the movement of the drag gesture 752, but the movement distance of the second area 738 is smaller than the movement distance of the drag gesture 752. For example, when the drag gesture 752 moves 1 centimeter (cm), the second area 738 moves 0.5 cm.

[0287] In this example, the first area 734 is on top of the second area 738 (e.g., in the Z layer), and as the first area 734 moves out of the touch screen 112 in response to the drag gesture 752, the second area 738 gradually appears. At the beginning of the drag gesture 752, the second area 738 is partially within the touch screen 112 but not completely (e.g., half or three-quarters of the touch screen 112). When the user slides the first area 734 out of the touch screen 112 in the right direction, the second area 738 slides the remaining part into the touch screen 112, and sliding the first area 734 completely out of the touch screen 112 corresponds to sliding the second area 738 completely into the touch screen 112. By moving the first area 734 out of the touch screen 112 at a rate different from the rate at which the second 738 moves into the touch screen 112, an intuitive visual cue regarding the direction in which the user navigates in the layer of the enlarged photo (e.g., the Z layer) is provided to the user.

[0288] In some embodiments, during the drag gesture 752, the images in the first area 734 and the images in the second area 738 proceed at different rates. In some embodiments, the respective rates at which the images in the first area 734 and the images in the second area 738 proceed are both based on the movement of the drag gesture 752. In some embodiments, the images in the first area 734 proceed at a higher rate than the rate at which the images in the second area 738 proceed. In FIGS. 7M - 7P, the images in the first area 734 proceed at twice the rate of the images in the second area 738 in response to the drag gesture 752. For example, as shown in FIGS. 7N - 7O, during the same period, the first area 734 proceeds between two images (702 - 4 in FIG. 7N and 702 - 5 in FIG. 7O) in the first image sequence 702, while the second area 738 maintains the display of a single image (724 - 2 in FIGS. 7N and 7O) in the second image sequence. FIG. 7P shows that the second area 738 has advanced to the display of the representative image 724 - 3.

[0289] FIGS. 7Q - 7CC show embodiments in which the device 100 slides an enlarged photograph into the display. When the enlarged photograph slides into the display, the device 100 plays back an image sequence including the enlarged photograph (e.g., from the first image to the representative image) in time series. In some embodiments, as described above, sliding a new enlarged photograph into the display displaces the currently displayed enlarged photograph. For example, sliding an enlarged photograph into the display slides the currently displayed enlarged photograph out of the display. As another example, sliding an enlarged photograph into the display covers the currently displayed enlarged photograph (e.g., in the Z direction). In some embodiments, the currently displayed enlarged photograph does not play back while being displaced (e.g., the device maintains the display of the representative image rather than the currently displayed enlarged photograph). In some embodiments, the enlarged photograph plays back in the forward direction regardless of whether it is the next enlarged photograph or the previous enlarged photograph (e.g., in the camera roll).

[0290] Therefore, FIG. 7Q shows a device 100 that displays a first representative image 702-3 of a first image sequence 702 in a first movable area 734 on a touch screen 112.

[0291] As shown in FIG. 7Q, the device detects a gesture 740 (e.g., a swipe gesture) on the touch screen 112 that includes a leftward movement by contact. As shown in FIG. 7R, in response to the gesture 740, the device 100 moves the first area 734 out of the display to the left and moves a second movable area 738 into the display to the left (e.g., the second area 738 is a movable area adjacent to the first area 734). The arrow in FIG. 7R indicates the inertia resulting from the gesture 740 that continues to move the second area 738 into the display. In this example, since the gesture 740 is to the left, the image sequence that slides into the display (e.g., the second image sequence 724) is the next image sequence (e.g., in the camera roll).

[0292] As shown in FIGS. 7R-7T, in accordance with a determination that the sequence display criteria are met, while moving the second area 738 to the left, the device 100 plays at least a portion of one or more images of the second image sequence 724 that the camera acquired before obtaining the second representative image 724-3 in a forward direction. For example, the device 100 starts by displaying the first image 724-1 from the second image sequence 724 (FIG. 7R). The device 100 plays the second image sequence 724 from the first image 724-1 to the representative image 724-3 (e.g., in time series) (FIGS. 7S-7T). In some embodiments, the playback of the second image sequence 724 is timed so that the representative image 724-3 appears when the movable area 738 has just finished moving into the touch screen 112 (FIG. 7T).

[0293] In some embodiments, the sequence display criteria include navigation criteria (e.g., criteria indicating that device 100 should finish transitioning to the next or previous photo without further user input). For example, if the user flicks and / or drags far enough and quickly enough for device 100 to transition to the next image sequence (e.g., the second image sequence 724), device 100 simply plays between the second image sequences 724. The navigation criteria have already been described in detail with reference to FIG. 7E.

[0294] In some embodiments, when the first area 734 slides out of the display, device 100 maintains the display of the first representative image 702-3 (e.g., statically without replacing the display of the first representative image 702-3), and at that time, plays at least a portion of the second image sequence 724 (e.g., the first image sequence 702 does not play while the second image sequence 724 is playing). In this way, the representative images 702-3 are displayed in the first area 734 in each of FIGS. 7Q-7S.

[0295] In some embodiments, a similar function occurs with a rightward swipe / drag gesture, except that instead of the next expanded photo (e.g., in the camera roll), the previous expanded photo (e.g., in the camera roll) slides into the display. For example, in FIG. 7U, device 100 detects a swipe gesture 742 that is similar to the swipe gesture 740 except that the swipe gesture 742 is to the right. In FIGS. 7V-7X, the fourth image sequence 760 (which is the previous expanded photo in the camera roll in this example) plays in the forward direction from the first image 760-1 to the representative image 760-3. That is, in some embodiments, regardless of whether the swipe / drag gesture is a previous photo navigation gesture or a next photo navigation gesture, device 100 plays the expanded photo in the forward direction (e.g., not playing the previous photo in the camera roll in reverse as described above with reference to FIGS. 7G-7K). FIGS. 7U-7X are similar to FIGS. 7Q-7T in other respects.

[0296] As shown in FIGS. 7Y to 7CC, in some embodiments, the sequence display criteria include criteria that are met when device 100 detects a lift-off of a gesture (e.g., device 100 starts playing a new enlarged photo only when the user lifts off the user's finger from the touch screen 112). For example, in FIG. 7Y, device 100 detects the start of drag gesture 764 at position 764-1. In FIG. 7Z, the user has moved drag gesture 764 to position 764-2, and accordingly, device 100 has moved the first image 760-1 of the fourth image sequence 760 partially into the display. In FIG. 7AA, the user has further moved drag gesture 764 to position 764-3, and accordingly, the device has moved the first image 760-1 of the fourth image sequence 760 further into the display. However, device 100 does not start playing the fourth image sequence 760 until the user lifts off drag gesture 764 (FIG. 7BB). This criterion allows the user to drag the enlarged photo in and out of the display without being overly stimulated. As shown in FIG. 7CC, in some embodiments, the playback and / or movement of the movable area following the lift-off of the gesture is timed such that the representative image of the new enlarged photo (e.g., the fourth representative image 760-3) is displayed during playback when the new enlarged photo has just finished sliding into the display.

[0297] Figures 8A-8L are diagrams showing exemplary user interfaces for performing operations different from individual images on related image sequences according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 9A-9G, 10A-10M, 11A-11I, 12A-12B, 24A-24E, 25A-25C, 26A-26D, and 27A-27E. The following examples are described in relation to inputs on a touch screen display (where a touch sensing surface and a display are combined as shown on the portable multifunctional device 100), but in some embodiments, the device detects inputs on a touch sensing surface 451 separate from the display 450, as shown in Figure 4B.

[0298] Figures 8A-8L show an example where the portable multifunctional device 100 stores a plurality of image sequences, one of which is a grouped image sequence 802 (displayed on the user interface 800). Some features of the user interface 800 are similar to the user interface 600 (Figures 6A-6W) and the user interface 700 (Figures 7A-7CC) and will not be repeated here for brevity. The image sequence 802 includes a representative image 802-3 (Figure 8A) acquired by the camera, one or more images acquired by the camera after acquiring the representative image 802-3 (e.g., image 802-4 in Figure 8C and image 802-5 in Figure 8D), and one or more images acquired by the camera before acquiring the representative image 802-3 (e.g., image 802-1 in Figure 8E and image 802-2 in Figure 8F). Thus, the time series of the image sequence 802 (e.g., the order in which the camera captured the images) is image 802-1, image 802-2, image 802-3, image 802-4, and image 802-5.

[0299] The image sequence 802 depicts a scene where a cat 812 walks into view, lies down on the ground, and then stands up and walks away. At the same time, a bird 814 perches on a tree branch. In reality, it may take several seconds for such a scene to unfold, but in some embodiments, the image sequence 802 is captured within a short time window. For example, in some embodiments, any of the image sequences described herein can depict moments around the instant when each representative image was obtained (e.g., within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds). For example, the user may be intrigued when the cat 812 lies down in the grass and may decide to take the representative image 802-3. In some embodiments, the image sequence 802 includes images immediately before and after obtaining the first representative image 802-3, such that, as described herein, when the user performs a specific operation on the representative image 802-3, the first image sequence 802 includes an extended photograph in which the moment can "come to life".

[0300] In the example shown in FIGS. 8A-8L, the portable multifunctional device 100 also stores a plurality of images that are different from the images in the plurality of grouped image sequences. For example, the portable multifunctional device 100 stores an image 824 (FIG. 8I), and the image 824 is not part of an image sequence in the plurality of image sequences (e.g., the image 824 is a still image).

[0301] In some embodiments, user interface 800 is the user interface in an image management application (e.g., Photos by Apple Inc. (Cupertino, California)). Thus, in some embodiments, the camera that captured image sequence 802 is part of portable multifunctional device 100 (e.g., a camera with optical sensor 164 in conjunction with imaging module 143 of FIG. 1A). In some embodiments, image sequence 802 was captured by a camera that is not part of portable multifunctional device 100 (e.g., image sequence 802 was captured by a camera of another device and then transferred to portable multifunctional device 100). In some embodiments, image sequence 802 was obtained in response to detecting activation of a shutter button at a first time, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600. In some embodiments, representative image 802-3 corresponds to a representative image acquired by the camera, as described herein in connection with FIGS. 5A-5K and method 900 and / or FIGS. 22A-22D and method 2600.

[0302] In some embodiments, some of the still images and / or some of the plurality of image sequences were acquired using portable multifunctional device 100, and some of the plurality of image sequences were transferred to portable multifunctional device 100 after being captured by a camera on a different device. For example, in some situations, a user may acquire (e.g., take, capture) image sequences described in connection with method 900 / 2600 on multiple devices (e.g., in addition to portable multifunctional device 100, a tablet, laptop, and / or digital camera), and synchronize or otherwise transfer the image sequences to portable multifunctional device 100, which stores additional still images.

[0303] In some embodiments, user interface 800 is a user interface in a messaging application (e.g., Messages by Apple Inc., Cupertino, California). In some embodiments, image sequence 802 and / or still image 824 are messages in a message conversation displayed in a scrollable region of the messaging application, and image sequence 802 is displayed as the scrollable region of the messaging application is scrolled. In some situations, the user may obtain (e.g., take, capture) each image sequence on the user's own portable multifunction device 100 and receive different image sequences or different still images from different users (e.g., in a messaging application). Thus, in some embodiments, the plurality of image sequences stored on portable multifunction device 100 includes at least one image sequence or still image obtained using the camera on portable multifunction device 100 and at least one image sequence obtained using the camera on another device different from portable multifunction device 100.

[0304] As shown in FIGS. 8A-8L, portable multifunction device 100 detects two similar inputs, namely, a first input 816 (FIGS. 8B-8F) and a second input 836 (FIGS. 8K-8L). The first input 816 and the second input 836 are similar because they share (e.g., meet) a common set of characteristics such as intensity characteristics (as shown in intensity diagram 818) and path characteristics (e.g., both the first input 816 and the second input 836 are long-press gestures). The first input 816 and the second input 836 are the same except that the first input 816 is detected on an image (e.g., representative image 802-3) that is part of an image sequence, while the second input 836 is detected on an image (e.g., still image 824) that is not part of the image sequence.

[0305] As a result, the portable multifunctional device 100 performs a first operation when a first input 816 is detected while displaying the representative image 802-3, and performs a different second operation when a second input 836 is detected while displaying the still image 824. In the examples shown in FIGS. 8B-8F, the first operation includes displaying at least a portion of the image sequence 802 in the manner described with reference to FIGS. 6A-6FF and methods 1000 / 10000 / 10050. That is, during the first portion 816-1 of the first input 816, after the portable multifunctional device 100 acquires the image 802-3, it plays back the images acquired by the camera (e.g., displays the image 802-4 in FIG. 8C and the image 802-5 in FIG. 8D), and during the second portion 816-2 of the first input 816, the portable multifunctional device 100 plays back the images acquired by the camera before acquiring the image 802-3 (e.g., displays the image 802-1 in FIG. 8E and the image 802-2 in FIG. 8F). In the examples shown in FIGS. 8K-8L, the second operation includes displaying an animation showing different portions of the still image 824. For example, as shown in FIG. 8L, the second operation includes an animation of zooming in on a portion of the still image 824 (e.g., the portion below or near the second input 836). In addition to or instead of zooming in, in some embodiments, the second operation includes displaying information (e.g., metadata 822) regarding the still image 824.

[0306] Figures 8G - 8I show a navigation gesture 844 (e.g., a drag gesture). The navigation gesture is a left - hand gesture that starts at position 844 - 1, moves to position 844 - 2, and then moves to position 844 - 3. Thus, in some embodiments, the portable multifunctional device 100 transitions from displaying an image sequence 802 (e.g., by displaying a sequence as described with reference to FIGS. 7A - 7CC) to displaying a still image 824 (e.g., since image 824 is a still image rather than an extended photo, it slides across the touch screen 112 without animating between image sequences). In some embodiments, when an input similar to the navigation gesture 844 is detected on a still image, the portable multifunctional device 100 transitions to a different image without displaying (e.g., because they do not exist) between images associated with the still image.

[0307] Figures 9A - 9G show a flowchart of a method 900 for capturing grouped related image sequences according to some embodiments. The method 900 is executed on an electronic device having a display and a camera (e.g., device 300 of FIG. 3 or portable multifunctional device 100 of FIG. 1A). In some embodiments, the device includes one or more sensors for detecting the intensity of contact with a touch - sensitive surface. In some embodiments, the display is a touch - screen display and the touch - sensitive surface is on or integrated with the display. In some embodiments, the display is separated from the touch - sensitive surface. Some of the operations in method 900 are optionally combined and / or the order of some operations is optionally changed.

[0308] While in a first media acquisition mode for the camera, the device displays a live preview on the display (902) (e.g., as shown in FIG. 5A) (e.g., the device displays the image in real - time or near - real - time while acquiring the image with the camera). For example, the first media acquisition mode is a mode named such as an extended photo mode, a moment mode, etc.

[0309] In some embodiments, the first media acquisition mode is configured to be enabled or disabled by the user of the device (e.g., via a setting interface for the camera) (904). In some embodiments, the device has at least three media acquisition modes, namely, (1) a first media acquisition mode that groups image sequences in response to detecting activation of a shutter button, where the image sequences include images acquired before and after activation of the shutter button, and stores them as a group of images (which may be regarded as an "extended still image" acquisition mode); (2) a second media acquisition mode that stores a single image in response to detecting activation of the shutter button, such as in the still image mode of a conventional digital camera (e.g., a conventional still image acquisition mode); and (3) a third media acquisition mode that stores a video acquired after detecting activation of the shutter button and continues to record the video until the shutter button is activated again (e.g., a video acquisition mode). In some embodiments, the user can select whether to enable the media acquisition mode via a setting interface for the camera, a mode selection button, a mode selection dial, etc.

[0310] In some embodiments, a live preview is displayed (906) as part of a media capture user interface that includes affordances (e.g., affordance 506 in FIGS. 5A - 5H) for enabling a first media acquisition mode. While the first media acquisition mode is enabled (e.g., while displaying the media capture user interface to indicate that image and / or audio data is being captured), the affordance is animated, and while the first media acquisition mode is disabled, the affordance is not animated. In some embodiments, in response to detecting a selection of the affordance (e.g., a tap gesture on the affordance) while the first media acquisition mode is disabled, the device enables the first media acquisition mode, starts capturing media (e.g., image and / or audio), and starts animating the affordance. In some embodiments, media capture includes recording of images and / or audio. In some embodiments, media capture includes storage of images and / or audio (e.g., to persistent memory).

[0311] While displaying the live preview, the device detects (908) activation of a shutter button at a first time (e.g., the device detects a physical button press at the first time, or detects a gesture on a virtual shutter button on a touch - sensitive display at the first time, such as a tap gesture on a shutter release icon (as shown in FIG. 5F) or a tap gesture on the live preview (where the live preview acts as a virtual shutter button)). In some embodiments, the detected activation is a single activation of the shutter button (e.g., similar to a single activation used to capture a single image in a still - image mode of a conventional digital camera).

[0312] In response to detecting the activation of the shutter button at the first time, the device groups (910) a plurality of images acquired by the camera in the temporal vicinity of the activation of the shutter button at the first time (e.g., as shown in FIGS. 5I-5K) into a first image sequence. The first image sequence includes a plurality of images acquired by the camera before detecting the activation of the shutter button at the first time, a representative image that represents the first image sequence and is acquired by the camera after one or more of the other images in the first image sequence, and a plurality of images acquired by the camera after acquiring the representative image.

[0313] In some embodiments, the representative image is similar to a single image acquired by the camera at the first time and captured when activating its shutter button in the still image mode of a conventional digital camera. In some embodiments, the representative image acquired by the camera corresponds to the image acquired at the first time. In some embodiments, the representative image acquired by the camera corresponds to an image acquired immediately after detecting the activation of the shutter button at the first time at a time that takes into account the shutter lag (the time delay between detecting the activation of the shutter button and capturing / storing the representative image). In some embodiments, the representative image acquired by the camera is used, for example, in an image presentation mode to represent the image sequence.

[0314] In some embodiments, the first image sequence includes a predetermined number of images, such as 5, 10, 15, 20, 25, or 30 images, acquired after the representative image is obtained. In some embodiments, the images acquired after the representative image is obtained are images within a predetermined time after the representative image is obtained (e.g., within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds after the representative image is obtained). In some embodiments, the first image sequence includes a predetermined number of images, such as 5, 10, 15, 20, 25, or 30 images, acquired after the activation of the shutter button at a first time. In some embodiments, the images acquired after detecting the activation of the shutter button at the first time are images within a predetermined time after the first time (e.g., images within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds after the first time). In some embodiments, a plurality of images in the first image sequence acquired after the representative image is obtained meet a predetermined grouping criterion. In some embodiments, the predetermined grouping criterion includes selecting a predetermined number of images after the representative image. In some embodiments, the predetermined grouping criterion includes selecting images within a predetermined time range immediately after detecting the activation of the shutter button. In some embodiments, the predetermined grouping criterion includes selecting images within a predetermined time range immediately after the time when the representative image is obtained.

[0315] In some embodiments, the first image sequence is stored in memory (912) as a first separate set of images (e.g., stored together in a data structure in non-volatile memory). In some embodiments, the representative image acquired by the camera is used, for example, in an image presentation mode, to represent the first separate set of images (e.g., FIGS. 6A - 6FF, FIGS. 7A - 7CC, and FIGS. 8A - 8L).

[0316] In some embodiments, the live preview displays the image at a first resolution, and the first image sequence includes images at the first resolution that are displayed in the live preview (914) (e.g., the first resolution is a resolution lower than the upper limit of the camera's resolution). In some embodiments, the representative image acquired by the camera has a second resolution that is higher than the first resolution (916). In some embodiments, the representative image acquired by the camera has a higher resolution than the resolution of the other images in the first image sequence. For example, the representative image acquired by the camera is an image of 12, 18, or 24 megapixels, and the other images in the first image sequence have a lower resolution corresponding to the resolution displayed in the live preview (e.g., the first resolution). In some embodiments, the representative image acquired by the camera has the same resolution as the resolution of the other images in the first image sequence.

[0317] In some embodiments, parameters for each image sequence grouped in response to detecting each activation of the shutter button are configurable by a user of the device (918). For example, via a settings interface for the camera, the user can select the number of images in each sequence, which image to use as the representative image of the sequence (e.g., as shown in FIGS. 5I-5K), and / or other acquisition or display parameters for the image sequence (e.g., the resolution of each image, the resolution of the other images, the frame rate, the filter effect, etc.).

[0318] In some embodiments, a plurality of images acquired by the camera before detecting the activation of the shutter button at a first time are stored in a first form in a memory (e.g., program memory, volatile memory, ring buffer, etc.) before detecting the activation of the shutter button at the first time, and in response to detecting the activation of the shutter button at the first time, are stored in a second form in a memory (e.g., non-volatile memory / storage) (920).

[0319] In some embodiments, the plurality of images acquired before detecting the activation of the shutter button at the first time are a predetermined number of images (e.g., 5, 10, 15, 20, 25, or 30 images) (922).

[0320] In some embodiments, the plurality of images acquired before detecting the activation of the shutter button at the first time are images within a predetermined time before the first time (e.g., images within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds before the first time) (924).

[0321] In some embodiments, the plurality of images acquired before detecting the activation of the shutter button at the first time are images within a predetermined time before the time when the representative image was acquired (e.g., images within 0.5, 1.0, 1.5, 2.0, or 2.5 seconds before the time when the representative image was acquired) (926).

[0322] In some embodiments, the plurality of images acquired before detecting the activation of the shutter button at the first time are from a time range between the first time and a second time that is before the first time, and acquiring the plurality of images before detecting the activation of the shutter button at the first time is independent of detecting a two-way action with the shutter button that is (other than detecting the activation of the shutter at the first time) in the temporal vicinity of the second time (928). For example, the plurality of images acquired before detecting the activation of the shutter button at the first time are not acquired in response to detecting a two-way action with the shutter button that is (other than detecting the activation of the shutter button at the first time) in the temporal vicinity of the second time. For example, the plurality of images acquired before detecting the activation of the shutter button at the first time are not acquired in response to detecting a partial (or complete) activation of the shutter button at the second time or in the vicinity thereof.

[0323] In some embodiments, a plurality of images in a first image sequence, acquired before detecting activation of a shutter button at a first time, satisfy one or more predefined grouping criteria (930). In some embodiments, the predefined grouping criteria include selecting a predefined number of images before detecting activation of the shutter button (932). In some embodiments, the predefined grouping criteria include selecting a predefined number of images before a representative image (934). In some embodiments, the predefined grouping criteria include selecting images within a predefined time range immediately before detecting activation of the shutter button (936). In some embodiments, the predefined grouping criteria include selecting images within a predefined time range immediately before the time at which the representative image was acquired (938).

[0324] In some embodiments, a live preview is displayed as part of a media capture user interface that includes an affordance for enabling a first media capture mode, and the shutter button is a software button displayed in the media capture user interface (e.g., shutter button 514 in FIGS. 5A-5H) (940). In response to detecting activation of the shutter button (e.g., tap gesture 518 in FIG. 5F), the device displays an animation associated with the shutter button (e.g., an animation of a portion of the shutter button that falls apart and returns to its original state, as shown in FIGS. 5F-5H) for a time corresponding to a time after activation of the shutter button during which the camera acquires images for the first image sequence (e.g., to provide the user with an indication that media is still being captured) (942). In some embodiments, the animation is a loop animation that can be continuously extended if the shutter button is re-activated before the camera finishes acquiring images for the first image sequence.

[0325] In some embodiments, the device starts acquiring and storing images when it enters a first media acquisition mode (independently of detecting activation of the shutter button) (944). While the device is in the first media acquisition mode, it deletes (or marks for deletion) images that are not grouped into a plurality of images in the temporal vicinity of the activation of the shutter button at each respective time (946).

[0326] In some embodiments, the device starts acquiring and storing images when it displays a live preview (independently of detecting activation of the shutter button) (948). While the device is in the first media acquisition mode, it deletes (or marks for deletion) images that are not grouped into a plurality of images in the temporal vicinity of the activation of the shutter button at each respective time (950).

[0327] In some embodiments, the device acquires and stores images while displaying a live preview, independently of detecting activation of the shutter button (952). While the device is in the first media acquisition mode, it deletes (or marks for deletion) acquired and stored images that are not grouped into a plurality of images in the temporal vicinity of the activation of the shutter button at each respective time (954).

[0328] In some embodiments, the user can select the length of time to hold an image before discarding it if the image is not grouped into an image sequence. For example, the user can set the device to hold an image displayed in live preview mode for 5, 10, or 20 seconds. For example, assuming the user selects a length of time of 5 seconds, an image displayed in live preview is held for 5 seconds after being displayed in live preview and then discarded (e.g., deleted or marked for deletion) if it is not grouped into an image sequence by activation of the shutter button.

[0329] In some embodiments, in response to detecting activation of the shutter button at a first time, the device associates (956) audio corresponding to a first image sequence (e.g., including audio recorded before detecting activation of the shutter button and audio recorded after detecting activation of the shutter button) with the first image sequence. In some embodiments, the device includes (or communicates with) a microphone, and the audio detected when the image sequence is acquired is stored in memory and linked (or otherwise associated) with the stored first image sequence. For example, FIGS. 6E-6I illustrate playback of an image sequence with corresponding audio.

[0330] In some embodiments, in response to detecting activation of the shutter button at a first time, the device associates (958) metadata corresponding to the first image sequence with the first image sequence (e.g., FIGS. 6J-6M illustrate playback of an image sequence with corresponding metadata). In some embodiments, metadata for the image sequence, such as time, date, location (e.g., via GPS), weather, music being played when the image sequence was acquired (e.g., music identified by the device's music identification software such as Shazam®, SoundHound®, or Midomi®), local event information (such as a sports game being played at the time and location the first image sequence was acquired), post-event information (such as the final score), etc., is stored in memory and linked (or otherwise associated) with the stored image sequence.

[0331] In some embodiments, the device automatically excludes (960) (or deletes or does not display as part of the sequence) blurry images from the first image sequence.

[0332] In some embodiments, after detecting activation of the shutter button at a first time, the device detects a next activation of the shutter button at a second time (962) (without detecting any activation of the shutter button between the first time and the second time). In response to detecting the next activation of the shutter button at the second time, the device groups a plurality of images captured by the camera in the temporal vicinity of the activation of the shutter button at the second time into a second image sequence (964). The second image sequence includes a plurality of images captured by the camera before detecting the activation of the shutter button at the second time, and a representative image captured by the camera after one or more of the other images in the second image sequence and representative of the second image sequence. In some embodiments, the capture of the image sequence is done in a manner similar to the capture of individual images by a conventional digital camera, so that the capture of such an image sequence is simple and intuitive even for novice users. In a conventional digital camera, an individual image is captured each time the shutter button is activated. Here, an image sequence is captured each time the shutter button is activated. The way this image sequence is captured is different from the way a video is captured by a conventional digital camera. In the capture of a video by a conventional digital camera, the recording of the video starts with the first activation of the shutter button and stops with the next activation of the shutter button.

[0333] In some embodiments, the first and / or last frames in the sequence are changed according to a change in the representative image (e.g., as shown in FIGS. 5I - 5K). Thus, in some embodiments, the first image sequence includes the first image in the first image sequence, a first number of images obtained between the first image and the representative image, the last image in the first image sequence, and a second number of images obtained between the representative image and the last image (966). The device detects an input corresponding to a request to change the representative image in the first image sequence (968). In some embodiments, while in the image sequence editing mode, the device detects a gesture (e.g., a drag gesture or a tap gesture) (e.g., touch gesture 522) that moves the representative image selector from the current representative image to another image in the first image sequence. In some embodiments, while in the image sequence editing mode, the device detects a gesture (e.g., a drag gesture or a tap gesture) that moves the current representative image out of the representative image selection area and moves another image in the first image sequence into the representative image selection area. In response to detecting an input corresponding to a request to change the representative image in the first image sequence, the device changes the representative image to a corrected representative image according to the detected input, and adds an image to one end of the first image sequence and deletes an image at the other end of the first image sequence so that the first image sequence has the corrected first and last images, thereby changing the plurality of grouped images in the first image sequence (970).

[0334] In some embodiments, the number of images between the first image and the representative image is the same as the number of images between the corrected first image and the corrected representative image. In some embodiments, the number of images between the representative image and the last image is the same as the number of images between the corrected representative image and the corrected last image. In some embodiments, the added images are in the temporal vicinity of one end of the first image sequence. For example, if the corrected representative image is 3 images before the first image in the first image sequence, 3 images (acquired just before the first image, with the earliest of the 3 images being the corrected first image) are added to the beginning of the first sequence, and 3 images are deleted f...

Claims

1. In an electronic device having a display, Displaying an image on the display, the image is an image in a sequence of images taken by a camera; the sequence of images includes a representative image; the image sequence includes one or more images captured by the camera after capturing the representative image; the image sequence includes one or more images captured by the camera prior to capturing the representative image; and detecting a first input while displaying the images in the sequence of images on the display; In response to detecting the first input, displaying an image editing user interface for editing the image sequence, the image editing user interface including an affordance for turning on animated playback of the image sequence or turning off animated playback of the image sequence while holding the image sequence; detecting a second input to the affordance while displaying the image editing user interface; in response to detecting the second input, turning on animated playback of the image sequence or turning off animated playback of the image sequence while maintaining the image sequence, wherein the representative image is displayed to represent the image sequence when playback of the image sequence is not animated.

2. The method of claim 1 , wherein the image displayed on the display is the representative image from the sequence of images.

3. 2. The method of claim 1, wherein the image displayed on the display is a currently selected image from the sequence of images.

4. 2. The method of claim 1 , in response to detecting a third input, displaying within the image editing user interface an area including a representation of an image within the sequence of images; selecting a new representative image for the image sequence in response to detecting a fourth input within the area that includes a representation of an image in the image sequence.

5. 2. The method of claim 1 , In response to detecting a fifth input corresponding to a request to edit the representative image, a user of the electronic device is continuing editing of the representative image with the animated playback of the image sequence disabled; and canceling the editing of the representative image.

6. 2. The method of claim 1 , presenting an affordance for deleting images in the sequence of images other than the representative image; in response to detecting a sixth input, deleting the one or more images captured by the camera after capturing the representative image and the one or more images captured by the camera before capturing the representative image.

7. 2. The method of claim 1 , In response to detecting a seventh input, an area containing a representation of an image in the sequence of images; a user-adjustable start crop icon that defines a start image within a subset of the sequence of images via a position of the user-adjustable start crop icon within the area that includes a representation of an image within the sequence of images; and displaying within the image editing user interface a user-adjustable end crop icon that defines an end image in the subset within the image sequence via a position of the user-adjustable end crop icon within the area including a representation of an image within the image sequence.

8. 8. The method of claim 7, wherein the area including a representation of an image in the image sequence is displayed simultaneously with the affordance for turning on animated playback of the image sequence or turning off animated playback of the image sequence while retaining the image sequence.

9. 2. The method of claim 1 , wherein the image editing user interface includes an affordance for switching between a first editing mode in which edits are applied to each image in the image sequence and a second editing mode in which edits are applied only to individual images in the image sequence.

10. 1. An electronic device comprising: A display and one or more processors; Memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: Displaying an image on the display, the image is an image in a sequence of images taken by a camera; the sequence of images includes a representative image; the image sequence includes one or more images captured by the camera after capturing the representative image; the image sequence includes one or more images captured by the camera prior to capturing the representative image; and detecting a first input while displaying the images in the sequence of images on the display; In response to detecting the first input, displaying an image editing user interface for editing the image sequence, the image editing user interface including an affordance for turning on animated playback of the image sequence or turning off animated playback of the image sequence while holding the image sequence; detecting a second input to the affordance while displaying the image editing user interface; in response to detecting the second input, turning on animated playback of the image sequence or turning off animated playback of the image sequence while holding the image sequence, the representative image being displayed to represent the image sequence when playback of the image sequence is not animated.

11. 11. The electronic device of claim 10, wherein the one or more programs further comprise instructions for carrying out the method of any one of claims 2 to 9.

12. When executed by an electronic device having a display, the electronic device: Displaying an image on the display, the image is an image in a sequence of images taken by a camera; the sequence of images includes a representative image; the image sequence includes one or more images captured by the camera after capturing the representative image; the image sequence includes one or more images captured by the camera prior to capturing the representative image; and detecting a first input while displaying the images in the sequence of images on the display; In response to detecting the first input, displaying an image editing user interface for editing the image sequence, the image editing user interface including an affordance for turning on animated playback of the image sequence or turning off animated playback of the image sequence while holding the image sequence; detecting a second input to the affordance while displaying the image editing user interface; responsive to detecting the second input, turning on animated playback of the image sequence or turning off animated playback of the image sequence while maintaining the image sequence, wherein the representative image is displayed to represent the image sequence when playback of the image sequence is not animated.

13. 13. The computer program of claim 12, further comprising instructions which, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 2 to 9.

Images