Device, method, and graphical user interface for depth-based annotation
The computer system addresses inefficiencies in augmenting media by using depth data for intuitive virtual object interaction, improving user interface efficiency and power conservation.
Patent Information
- Application Number
- JP2024113573
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-09-17
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2039-09-20
AI Technical Summary
Existing methods for augmenting media with virtual objects and annotations are cumbersome, inefficient, and place a significant cognitive burden on users, particularly in battery-operated devices, due to the need for multiple inputs and time-consuming processes.
A computer system with improved methods and interfaces that utilize depth data to efficiently add, relocate, and display virtual objects or annotations on images or videos, allowing for intuitive interaction and reduced user input, especially in battery-operated devices.
Enhances user interface efficiency and power conservation by reducing the need for multiple inputs, making it easier to rearrange annotations and add virtual objects, while conserving battery life.
Smart Images

Figure 2024156710000001_ABST
Abstract
Description
[Technical field]
[0001] This application relates generally to electronic devices that display images of a physical environment, including but not limited to electronic devices that display annotations at spatial locations within the images that correspond to spatial locations in the physical environment captured in the images. [Background technology]
[0002] The development of computer systems for augmented media has progressed significantly in recent years. Examples of augmented media include augmented reality environments that include at least some virtual elements that replace or augment the physical world, and augmented stored media that includes at least some virtual elements that replace or augment stored media, such as image and video content. Input devices such as touch-sensitive surfaces of computer systems and other electronic computing devices are used to augment the media. Exemplary touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touchscreen displays. These surfaces are used to manipulate the user interface and objects therein on the display. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons and other graphics).
[0003] However, methods and interfaces for augmenting media are cumbersome, inefficient, and limited. For example, an augmentation, such as a user-input annotation that has a fixed spatial location relative to a portion of the physical environment, may be difficult for a user to locate if the current camera view of the user's device does not correspond to that portion of the physical environment. Searching for the augmentation places a significant cognitive burden on the user, detracting from the experience of using the augmented media. In addition, providing augmented input for stored media (e.g., a previously captured video) is time intensive when augmented input must be provided separately for various portions of the stored media. In addition, the methods take longer than necessary, thereby wasting energy. The latter problem is particularly acute in battery-operated devices. Summary of the Invention
[0004] Thus, there is a need for computer systems with improved methods and interfaces for enhancing media data. Such methods and interfaces optionally complement or replace conventional methods for enhancing media data. Such methods and interfaces reduce the number, range, and / or type of inputs from a user, creating a more efficient human-machine interface. For battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.
[0005] The above deficiencies and other problems associated with interfaces for enhancing media data with virtual objects and / or annotation input are reduced or eliminated by the disclosed computer system. In some embodiments, the computer system comprises a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system comprises a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has (and / or is in communication with) a touchpad. In some embodiments, the computer system has (and / or is in communication with) a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI, in part, through contacts and gestures with a stylus and / or finger on a touch-sensitive surface. In some embodiments, the functionality optionally includes game playing, image editing, drawing, presenting, word processing, spreadsheet creation, telephony, video conferencing, email, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functionality are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.
[0006] According to some embodiments, a method is performed on a computer system having a display generation element, one or more input devices, and one or more cameras. The method includes displaying, via the display generation element, a first user interface region including a representation of the field of view of the one or more cameras that is updated with changes in the field of view of the one or more cameras over time. The method further includes receiving, via the one or more input devices, a first request to add an annotation to the displayed representation of the field of view of the one or more cameras while displaying the first user interface region including the representation of the field of view of the one or more cameras. In response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras, the method further includes replacing the display of the representation of the field of view of the one or more cameras in the first user interface region with a still image of the field of view of the one or more cameras captured at a time corresponding to receiving the first request to add the annotation. The method further includes receiving, via the one or more input devices, a first annotation on a first portion of the still image while displaying the still image in the first user interface region, the first portion of the still image corresponding to a first portion of the physical environment captured in the still image. The method further includes receiving, via the one or more input devices, a first request to redisplay a representation of the field of view of the one or more cameras in the first user interface region while displaying the first annotation on the first portion of the still image in the first user interface region. The method further includes replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface region in response to receiving the first request to redisplay the representation of the field of view of the one or more cameras in the first user interface region. The method further includes displaying an indication of a current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image concurrently with the representation of the field of view of the one or more cameras pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, and withholding display of the indication pursuant to a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
[0007] In some embodiments, a method is executed in a computer system having a display generation element and one or more input devices. The method includes displaying, via the display generation element, a user interface on a display including a video playback area. The method further includes receiving, via the one or more input devices, a request to add an annotation to the video playback while displaying a playback of a first portion of the video in the video playback area. The method further includes, in response to receiving the request to add the annotation, pausing the playback of the video at a first location in the video and displaying a still image corresponding to the first paused location of the video. The method further includes, while displaying the still image, receiving, via the one or more input devices, an annotation on a first portion of the physical environment captured in the still image. The method further includes, after receiving the annotation, displaying, within the video playback area, a second portion of the video corresponding to a second location of the video different from the first location in the video, the first portion of the physical environment being captured in the second portion of the video, and the annotation being displayed in the second portion of the video.
[0008] In some embodiments, a method is implemented in a computer system having a display generating element and one or more input devices. The method includes displaying, via the display generating element, a first previously captured media object including one or more first images, the first previously captured media object stored with first depth data corresponding to a first physical environment captured in each of the one or more first images. The method further includes receiving a first user request via the one or more input devices while displaying the first previously captured media object, and adding a first virtual object to the first previously captured media object. The method further includes displaying the first virtual object over at least a portion of each image in the first previously captured media object in response to the first user request to add the first virtual object to the first previously captured media object, the first virtual object being displayed at at least a first position or orientation determined using the first depth data corresponding to each image in the first previously captured media object.
[0009] According to some embodiments, a method is performed on a computer system having a display generating element, a first set of one or more input devices, and a first set of one or more cameras. The method includes sending a request to a remote device to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras. The method further includes receiving an indication of acceptance of the request to initiate the shared annotation session in response to sending the request to initiate the shared annotation session with the second device. The method further includes displaying, via the first display generating element, a first prompt that causes the first device to move towards the second device in response to receiving the indication of acceptance of the request to initiate the shared annotation session. The method further includes, after displaying the first prompt, displaying a representation of a field of view of the first camera set in a shared annotation session with the second device in accordance with a determination that a connectivity criterion for the first device and the second device is satisfied, the connectivity criterion requiring that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to a same portion of a physical environment surrounding the first and second devices. The method further includes, during the shared annotation session, displaying one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device via the first display generation element and displaying one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device via the first display generation element if the respective positions are included within the field of view of the first camera set.
[0010] According to some embodiments, the electronic device includes a display generating element, optional one or more input devices, optional one or more touch-sensitive surfaces, optional one or more cameras, optional one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors for detecting changes in posture, one or more processors, and a memory having one or more programs stored thereon, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing or causing to be performed any of the operations of the methods described herein. According to some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by an electronic device comprising a display generating element, optional one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more posture sensors, cause the device to perform or cause the performance of any of the operations of the methods described herein. According to some embodiments, a graphical user interface of an electronic device having a display generating element, optional one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors, memory, and one or more processors executing one or more programs stored in the memory, includes one or more of the elements displayed in any of the methods described herein that are updated to include input as described in any of the methods described herein.According to some embodiments, an electronic device includes a display generating element, optional one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors for detecting changes in posture, and means for performing or causing the performance of any of the operations of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device comprising a display generating element, optional one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more posture sensors for detecting changes in posture includes means for performing or causing the performance of any of the operations of the methods described herein.
[0011] Thus, electronic devices comprising a display generating element, optional one or more input devices, optional one or more touch-sensitive surfaces, optional one or more cameras, one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more posture sensors are provided with improved methods and interfaces for displaying virtual objects in a variety of contexts, thereby increasing the effectiveness, efficiency, and user satisfaction of these devices. These methods and interfaces can complement or replace conventional methods of displaying virtual objects in a variety of contexts. [Brief description of the drawings]
[0012] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:
[0013] [Figure 1A] 1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.
[0014] [Figure 1B] FIG. 1 is a block diagram illustrating example components for event handling, according to some embodiments.
[0015] [Figure 1C] FIG. 1 is a block diagram illustrating a tactile output module, according to some embodiments.
[0016] [Diagram 2] FIG. 1 illustrates a portable multifunction device having a touch screen according to some embodiments.
[0017] [Diagram 3] FIG. 1 is a block diagram illustrating an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.
[0018] [Figure 4A] 1 illustrates an exemplary user interface for a menu of applications on a portable multifunction device in accordance with some embodiments.
[0019] [Figure 4B] 1A-1C illustrate example user interfaces of a multifunction device with a touch-sensitive surface separate from a display in accordance with some embodiments.
[0020] [Figure 4C] FIG. 1 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4D] FIG. 1 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4E] FIG. 1 illustrates an example of a dynamic intensity threshold, according to some embodiments.
[0021] [Figure 5A] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5B] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5C] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5D] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5E] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5F] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5G] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5H] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5I] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5J] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5K] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5L] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5M]1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5N] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5O] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5P] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5Q] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5R] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5S] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5T] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5U] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5V] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5W] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5X] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5Y] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5Z] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AA]1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AB] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AC] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AD] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AE] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments. [Figure 5AF] 1 illustrates an example of a user interface for relocating an annotation according to some embodiments.
[0022] [Figure 6A] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6B] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6C] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6D] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6E] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6F]1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6G] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6H] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6I] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6J] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6K] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6L] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6M] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 6N] 1 illustrates an exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments.
[0023] [Figure 7A]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7B] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7C] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7D] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7E] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7F] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7G] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7H] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7I] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7J] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7K]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7L] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7M] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7N] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7O] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7P] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7Q] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7R] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7S] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7T] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7U]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7V] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7W] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7X] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7Y] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7Z] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AA] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AB] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AC] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7A-D] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AE]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AF] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7A-G] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AH] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AI] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AJ] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AK] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AL] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AM] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AN] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AO]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AP] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AQ] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AR] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AS] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AT] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AU] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AV] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AW] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AX] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AY]1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7AZ] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BA] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BB] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BC] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BD] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BE] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments. [Figure 7BF] 1 illustrates an exemplary user interface for adding a virtual object to a previously captured media object according to some embodiments.
[0024] [Figure 8A] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8B] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8C]1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8D] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8E] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8F] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8G] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8H] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8I] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8J] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8K] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8L] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8M]1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8N] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8O] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8P] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8Q] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8R] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8S] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8T] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8U] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8V] 1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments. [Figure 8W]1 illustrates an exemplary user interface for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments.
[0025] [Figure 9A] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments. [Figure 9B] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments. [Figure 9C] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments. [Figure 9D] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments. [Figure 9E] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments. [Figure 9F] FIG. 1 is a flow diagram of a process for relocating an annotation according to some embodiments.
[0026] [Figure 10A] FIG. 1 is a flow diagram of a process for receiving an annotation on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments. [Figure 10B] FIG. 1 is a flow diagram of a process for receiving an annotation on a portion of a physical environment captured in a still image corresponding to a pause point in a video, according to some embodiments.
[0027] [Figure 11A] FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments. [Figure 11B] FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments. [Figure 11C]FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments. [Figure 11D] FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments. [Figure 11E] FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments. [Figure 11F] FIG. 1 is a flow diagram of a process for adding a virtual object to a previously captured media object according to some embodiments.
[0028] [Figure 12A] 1 is a flow diagram of a process for initiating a shared annotation session according to some embodiments. [Figure 12B] 1 is a flow diagram of a process for initiating a shared annotation session according to some embodiments. [Figure 12C] 1 is a flow diagram of a process for initiating a shared annotation session according to some embodiments. [Figure 12D] 1 is a flow diagram of a process for initiating a shared annotation session according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] Conventional methods of augmenting media often require multiple separate inputs (e.g., individual annotations of multiple frames and / or placement of augmentations relative to objects in the media) to achieve an intended result (e.g., annotating a portion of a stored video or live video feed and / or displaying a virtual object at a location that corresponds to the surface of a physical object in the stored media). Embodiments herein provide an intuitive way for a user to augment media, such as stored content, still images, and / or live video captured by one or more cameras of a device (e.g., using depth data stored and / or captured along with the image data to position the augmentations and maintain a constant spatial relationship between the augmentations and portions of the physical environment within the camera's field of view).
[0030] The systems, methods, and GUIs described herein improve user interface interaction with augmented media in several ways: for example, they make it easier to rearrange annotations, annotate videos, add virtual objects to previously captured media, and initiate shared annotation sessions.
[0031] Below, Figures 1A-1C, 2, and 3 provide a description of an exemplary device. Figures 4A-4B, 5A-5AF, 6A-6N, 7A-7BF, and 8A-8W show exemplary user interfaces displaying virtual objects in various contexts. Figures 9A-9F show a process for repositioning annotations. Figures 10A-10B show a process for receiving annotations on a portion of a physical environment captured in a still image corresponding to a paused position of a video. Figures 11A-11F show a process for adding a virtual object to a previously captured media object. Figures 12A-12D show a process for initiating a shared annotation session. The user interfaces of Figures 5A-5AF, 6A-6N, 7A-7BF, and 8A-8W are used to illustrate the processes of Figures 9A-9F, 10A-10B, 11A-11F, and 12A-12D. Exemplary Devices
[0032] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
[0033] In this specification, terms such as first, second, etc. are used in some examples to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the various embodiments described. Although a first contact and a second contact are both contacts, they are not the same contact unless the context clearly indicates otherwise.
[0034] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. In the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that as used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0035] As used herein, the term "if" is optionally interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.
[0036] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad).
[0037] In the following discussion, electronic devices are described that include a display and a touch-sensitive surface, however, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.
[0038] The device typically supports a variety of applications, such as one or more of note-taking applications, drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.
[0039] The various applications that run on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface, and corresponding information displayed on the device, are optionally adjusted and / or changed for each application and / or within each application. In this manner, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports a variety of applications with user interfaces that are intuitive and transparent to the user.
[0040] Attention is now directed to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating a portable multifunction device 100 having a touch-sensitive display system 112, according to some embodiments. Touch-sensitive display system 112 may be conveniently referred to as a "touch screen" or simply a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more light sensors 164. Device 100 optionally includes one or more intensity sensors 165 that detect the intensity of a contact on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 that generate tactile output on device 100 (e.g., generate tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate over one or more communication buses or signal lines 103.
[0041] It is understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of its components. The various components shown in FIG. 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.
[0042] Memory 102 optionally includes high-speed random access memory, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripherals interface 118, is optionally controlled by a memory controller 122.
[0043] A peripheral interface 118 may be used to couple input and output peripherals of the device with the CPU(s) 120 and memory 102. The one or more processors 120 operate or execute various software programs and / or instruction sets stored in the memory 102 to perform various functions and process data for the device 100.
[0044] In some embodiments, peripherals interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.
[0045] The radio frequency (RF) circuitry 108 transmits and receives RF signals, also referred to as electromagnetic signals. The RF circuitry 108 converts electrical signals to or from electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuitry 108 optionally includes well-known circuits for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and the like. The RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and with other devices, by wireless communication. Radio communication options include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPADA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), and code division multiple access (CDMA).Wireless technology such as IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n, voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE)), instant messaging and presence services (e.g., IEEE 802.11b), instant messaging and presence services (e.g., IEEE 802.11c), instant messaging and presence services (e.g., IEEE 802.11b ... The present invention may use any of a number of communications standards, protocols, and technologies, including, but not limited to, Intermediate System (IS) Intermediate Message Service (IMPS), and / or Short Message Service (SMS), or any other suitable communications protocols, including communications protocols not yet developed as of the filing date of this document.
[0046] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripherals interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives an electrical signal converted from the sound wave by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits the audio data to the peripherals interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripherals interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212, FIG. 2). The headset jack provides an interface between audio circuitry 110 and a detachable audio input / output peripheral such as an output-only headphone or a headset with both an output (e.g., one-ear or binaural headphones) and an input (e.g., a microphone).
[0047] I / O subsystem 106 couples input / output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, with peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from and send electrical signals to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 160 are optionally coupled to any (or none) of a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. The one or more buttons (e.g., 208, FIG. 2) optionally include up / down buttons for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., 206, FIG. 2).
[0048] Touch-sensitive display system 112 provides an input and output interface between the device and a user. Display controller 156 receives electrical signals from and / or transmits electrical signals to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed towards the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, a button, a slider, an icon, a selectable menu item, a switch, a hyperlink, or other user interface control.
[0049] Touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touch-sensitive display system 112 and display controller 156 (along with any associated modules and / or instruction sets in memory 102) detect contacts (and any movement or breaking of contact) on touch-sensitive display system 112 and translate the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touch-sensitive display system 112. In some embodiments, the point of contact between touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.
[0050] Touch-sensitive display system 112 optionally uses liquid crystal display (LCD), light emitting polymer display (LPD), or light emitting diode (LED) technology, although other display technologies are used in other embodiments. Touch-sensitive display system 112 and display controller 156 optionally use any of a number of now known or later developed touch sensing technologies to detect contact and any movement or disruption thereof, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.
[0051] Touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen video resolution is greater than 400 dpi (e.g., 500 dpi, 800 dpi, or higher). A user optionally contacts touch-sensitive display system 112 using any suitable object or appendage, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may be less precise than stylus-based inputs due to the larger contact area of a finger on a touchscreen than that of a stylus. In some embodiments, the device translates the rough finger input into precise pointer / cursor positions or commands to perform the actions desired by the user.
[0052] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touchscreen.
[0053] Device 100 also includes a power system 162 that provides power to the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power within a portable device.
[0054] Device 100 also optionally includes one or more light sensors 164. FIG. 1A shows a light sensor coupled to light sensor controller 158 in I / O subsystem 106. Light sensor(s) 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Light sensor(s) 164 receive light from the environment, projected through one or more lenses, and convert the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), light sensor(s) 164 optionally captures still images and / or video. In some embodiments, the light sensor is located on the back of device 100, opposite touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for still and / or video image acquisition. In some embodiments, another light sensor is placed on the front of the device so that an image of the user is captured (e.g., for a selfie, for a video conference while the user views other video conference participants on the touch screen, etc.).
[0055] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled with intensity sensor controller 159 in I / O subsystem 106. Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100, opposite touchscreen display system 112, which is located on the front of device 100.
[0056] Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near a user's ear (e.g., when the user is conducting a phone call).
[0057] Device 100 also optionally includes one or more tactile output generators 167. FIG. 1A shows a tactile output generator coupled to haptic feedback controller 161 in I / O subsystem 106. In some embodiments, tactile output generator(s) 167 include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Tactile output generator(s) 167 receive tactile feedback generation instructions from haptic feedback module 133 and generate a tactile output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one tactile output generator is juxtaposed with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and generates a tactile output, optionally by moving the touch-sensitive surface vertically (e.g., in / out of the surface of device 100) or horizontally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, the at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display system 112, which is located on the front of device 100.
[0058] Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. In addition to accelerometer(s) 168, device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information regarding the location and orientation (e.g., portrait or landscape) of device 100.
[0059] In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and applications (or instruction set) 136. Additionally, in some embodiments, as shown in Figures 1A and 3, memory 102 stores device / global internal state 157. Device / global internal state 157 includes one or more of: active application state, which indicates which applications, if any, are currently active; display state, which indicates what applications, views, or other information are occupying various areas of touch-sensitive display system 112; sensor state, which includes information obtained from the device's various sensors and other input or control devices 116; and position and / or location information regarding the device's position and / or orientation.
[0060] Operating system 126 (e.g., an embedded operating system such as iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.
[0061] The communications module 128 also includes various software components for facilitating communication with other devices via one or more external ports 124 and for processing data received by the RF circuitry 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) are adapted to couple directly to other devices or indirectly via a network (e.g., Internet, wireless LAN, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors identical to or similar to and / or compatible with the 30-pin connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external ports are Lightning connectors identical to or similar to and / or compatible with the Lightning connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California.
[0062] Contact / motion module 130 optionally detects contact with touch-sensitive display system 112 (in cooperation with display controller 156), and with other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes software components for performing various operations related to detecting contact (e.g., by a finger or stylus), such as determining if contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a proxy for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining if the contact has ended (e.g., detecting a finger up event or an interruption of the contact). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact, as represented by the set of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact. These operations are optionally applied to a single contact (e.g., one finger contact or stylus contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contacts on a touchpad.
[0063] Contact / motion module 130 optionally detects gesture input by a user. Different gestures on a touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or strength of detected contacts). Thus, gestures are optionally detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger down event, followed by a finger up (lift off) event at the same location (or substantially the same location) as the finger down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by one or more finger drag events, followed by a finger up (lift off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected by detecting a particular contact pattern with respect to the stylus with respect to the stylus.
[0064] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger down event and detecting a finger up event, but is not related to the strength of the finger contact between detecting a finger down event and detecting a finger up event. In some embodiments, a tap gesture is detected according to determining that the length of time between a finger down event and a finger up event is less than a predefined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a shallow or deep pressure intensity threshold. Thus, a finger tap gesture may satisfy certain input criteria that do not require that the characteristic intensity of the contact meet a given intensity threshold in order for the particular input criterion to be met. For clarity, a finger contact in a tap gesture generally needs to meet a nominal contact detection intensity threshold below which the contact is not detected in order to detect a finger down event. A similar analysis applies to detecting a tap gesture or other contact with a stylus. In cases where the device is capable of detecting contact of a finger or stylus hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.
[0065] In a similar manner, the same concepts are applied to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or long press gestures are optionally detected based on meeting criteria that are either unrelated to the intensity of the contacts included in the gesture or do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, swipe gestures are detected based on the amount of movement of one or more contacts, pinch gestures are detected based on the movement of two or more contacts towards each other, de-pinch gestures are detected based on the movement of two or more contacts away from each other, and long press gestures are detected based on a duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. Thus, a statement that a particular gesture recognition criterion does not require that the intensity of a contact(s) meet a corresponding intensity threshold in order for the particular gesture recognition criterion to be satisfied means that the particular gesture recognition criterion can be satisfied when the contact(s) in the gesture do not reach the corresponding intensity threshold, and can also be satisfied in situations where one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on a determination that a finger down event and a finger up event are detected within a predetermined time period, regardless of whether the contacts are above or below the respective intensity thresholds during the predetermined time period, and a swipe gesture is detected based on a determination that a movement of the contact is greater than a predetermined magnitude, even if the contacts exceed the respective intensity thresholds at the end of the movement of the contact. Even in implementations in which detection of gestures is affected by the strength of the contact performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact exceeds an intensity threshold, or the device is slow to detect a tap input when the intensity of the contact is higher), detection of those gestures does not require that the contact reach a particular intensity threshold, so long as the criteria for recognizing the gesture can be met in situations in which the contact does not reach a particular intensity threshold (e.g., even if the amount of time it takes to recognize the gesture varies).
[0066] The contact intensity threshold, duration threshold, and movement threshold are combined in various different combinations in some situations to create heuristics for distinguishing between two or more different gestures directed at the same input element or region, thereby enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. A statement that a particular set of gesture recognition criteria does not require the intensity of the contact(s) to meet the respective intensity threshold for that particular gesture recognition criterion to be met does not preclude simultaneously evaluating other intensity-dependent gesture recognition criteria for identifying other gestures having criteria that are met when the gesture includes a contact having an intensity that exceeds the respective intensity threshold. For example, in some situations, a first gesture recognition criterion for a first gesture that does not require the intensity of the contact(s) to meet the corresponding intensity threshold for the first gesture recognition criterion to be met is in competition with a second gesture recognition criterion for a second gesture that is dependent on the contact(s) reaching the corresponding intensity threshold. In such a competition, a gesture is optionally not recognized as satisfying the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are satisfied first. For example, if the contact reaches the corresponding intensity threshold before it moves the predetermined amount of movement, a deep press gesture is detected rather than a swipe gesture. Conversely, if the contact moves the predetermined amount of movement before it reaches the corresponding intensity threshold, a swipe gesture is detected rather than a deep press gesture. Even in such a situation, the first gesture recognition criteria for the first gesture still do not require the intensity of the contact(s) to meet the corresponding intensity threshold in order for the first gesture recognition criteria to be satisfied, because if the contact remains below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact that does not increase in intensity to above the corresponding intensity threshold), the gesture will have been recognized by the first gesture recognition criteria as a swipe gesture.In that way, a particular gesture recognition criterion that does not require that the intensity of the contact(s) meet a corresponding intensity threshold in order for the particular gesture recognition criterion to be satisfied may still be dependent on the intensity of the contact with respect to an intensity threshold in the sense that (A) in some circumstances it ignores the intensity of the contact with respect to an intensity threshold (e.g., for a tap gesture) and / or (B) in some circumstances the particular gesture recognition criterion (e.g., for a long press gesture) will not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes an input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes a gesture corresponding to the input (e.g., for a long press gesture that competes with a deep press gesture for recognition).
[0067] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other display, including components for modifying the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, and animations.
[0068] In some embodiments, graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, including coordinate data and other graphic characteristic data, from an application or the like, as appropriate, and then generates screen image data for output to display controller 156.
[0069] The haptic feedback module 133 includes various software components that generate instructions (e.g., instructions used by the haptic feedback controller 161) that use the tactile output generator(s) 167 to create a tactile output at one or more locations on the device 100 in response to user interaction with the device 100.
[0070] Text input module 134 is optionally a component of graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).
[0071] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based calling, to the camera 143 as photo / video metadata, and to applications that provide location-based services, such as a weather widget, a local yellow pages widget, and a maps / navigation widget).
[0072] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof: ● a contacts module 137 (sometimes called an address book or contact list); ●Telephone module 138, ● Videoconferencing module 139, ● an email client module 140; ● Instant messaging (IM) module 141; ● Training support module 142, A camera module 143 for still and / or video images, ● Image management module 144, ● Browser module 147, ● Calendar module 148, A widget module 149, optionally including one or more of a weather widget 149-1, a stock price widget 149-2, a calculator widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; ● a widget creation module 150 for creating user-created widgets 149-6; ● Search module 151, ● a video and music player module 152, optionally consisting of a video player module and a music player module; ● Memo module 153, Map module 154, and / or ●Online video module 155.
[0073] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA® enabled applications, encryption, digital rights management, voice recognition, and voice duplication.
[0074] Contacts module 137, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, includes executable instructions (e.g., stored in memory 102 or in the application internal state 192 of contacts module 137 in memory 370) for managing an address book or contact list, including adding a name(s) to the address book, deleting a name(s) from the address book, associating phone number(s), email address(es), physical address(es), or other information with a name, associating an image with a name, categorizing and sorting names, providing phone numbers and / or email addresses to initiate and / or facilitate communication by telephone 138, video conference 139, email 140, or IM 141, and the like.
[0075] In cooperation with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for entering a sequence of characters corresponding to a telephone number, accessing one or more telephone numbers in address book 137, modifying an entered telephone number, dialing each telephone number, conducting a conversation, and disconnecting or hanging up when the conversation is completed. As described above, the wireless communication optionally uses any of a number of communication standards, protocols, and technologies.
[0076] In cooperation with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, videoconferencing module 139 contains executable instructions for initiating, conducting, and terminating a videoconference between a user and one or more other participants according to the user's commands.
[0077] In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 contains executable instructions for creating, sending, receiving, and managing emails in response to user commands. In cooperation with image management module 144, email client module 140 greatly facilitates the creation and sending of emails with still or video images captured by camera module 143.
[0078] In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and Internet-based messaging (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).
[0079] In cooperation with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the training support module 142 contains executable instructions to create workouts (e.g., having time, distance, and / or calorie burn goals), communicate with training sensors (in the sports device and the smart watch), receive training sensor data, calibrate sensors used to monitor the workouts, select and play music for the workouts, and display, store, and transmit training data.
[0080] Camera module 143, along with touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, contain executable instructions to capture still images or videos (including video streams) and store them in memory 102, change characteristics of the still images or videos, and / or delete still images or videos from memory 102.
[0081] Image management module 144, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, contain executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and / or video images.
[0082] Browser module 147, together with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, contains executable instructions for browsing the Internet according to a user's commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
[0083] Calendar module 148, along with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, contain executable instructions that create, display, modify, and store calendars and data associated with the calendars (e.g., calendar entries, to-do lists, etc.) according to a user's instructions.
[0084] Together with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally a mini-application downloaded and used by a user (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or a mini-application created by a user (e.g., user-created widget 149-6). In some embodiments, a widget includes a Hypertext Markup Language (HTML) file, a Cascading Style Sheets (CSS) file, and a JavaScript file. In some embodiments, a widget includes an Extensible Markup Language (XML) file and a JavaScript file (e.g., Yahoo! Widgets).
[0085] In cooperation with the RF circuitry 108, the touch-sensing display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the widget creation module 150 contains executable instructions for creating widgets (e.g., turning user-specified portions of a web page into widgets).
[0086] In cooperation with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 contains executable instructions to search for text, music, sound, images, video, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's instructions.
[0087] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 contains executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions to display, present, or otherwise play videos (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).
[0088] In cooperation with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the notes module 153 contains executable instructions for creating and managing notes, to-do lists, and the like according to a user's instructions.
[0089] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 can be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data) in accordance with a user's instructions.
[0090] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 contains executable instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), and play (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124) online videos in one or more file formats, such as H.264, and send and otherwise manage emails with links to particular online videos. In some embodiments, instant messaging module 141 is used rather than email client module 140 to send links to particular online videos.
[0091] Each of the above-identified modules and applications corresponds to an executable instruction set that performs one or more of the functions described above, as well as the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). The modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of the modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.
[0092] In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively via a touchscreen and / or touchpad. Using the touchscreen and / or touchpad as the primary input control device for operation of device 100 optionally reduces the number of physical input control devices (push buttons, dials, etc.) on device 100.
[0093] The set of predefined functions performed only through the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 to a main menu, a home menu, or a root menu from any user interface displayed on device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device rather than a touchpad.
[0094] 1B is a block diagram illustrating example components for event processing, according to some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g., within operating system 126) and a respective application 136-1 (e.g., any of applications 136, 137-155, 380-390 described above).
[0095] Event sorter 170 receives the event information and determines which application 136-1 to deliver the event information to and application view 191 for application 136-1. Event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) that are displayed on touch-sensitive display system 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which application(s) is currently active, and application internal state 192 is used by event sorter 170 to determine which application view 191 to deliver the event information to.
[0096] In some embodiments, application internal state 192 includes additional information such as one or more of resume information to be used when application 136-1 resumes execution, user interface state information indicating or ready to display information being displayed by application 136-1, state cues that allow a user to return to a previous state or view of application 136-1, and redo / undo cues of previous actions taken by the user.
[0097] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 168, and / or microphone 113 (via audio circuitry 110). Information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.
[0098] In some embodiments, event monitor 171 sends a request to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or for longer than a predetermined period of time).
[0099] In some embodiments, event sorter 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173.
[0100] Hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when touch-sensitive display system 112 displays more than one view. A view consists of the controls and other elements that a user can see on the display.
[0101] Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which the touch is detected is optionally referred to as a hit view, and the set of events that are recognized as suitable inputs is optionally determined based at least in part on the hit view of the initial touch that initiates the touch gesture.
[0102] Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest level view in which the initiating sub-event occurs (i.e., the first sub-event in a series of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.
[0103] Active event recognizer determination module 173 determines which view or views in the view hierarchy should receive the particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive the particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that contain the physical location of the sub-event are actively participating views, and therefore all actively participating views should receive the particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with one particular view, views higher in the hierarchy still remain actively participating views.
[0104] Event dispatcher module 174 dispatches the event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information obtained by each event receiver module 182 in an event queue.
[0105] In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In yet other embodiments, event sorter 170 is a stand-alone module or is part of another module stored in memory 102, such as contact / motion module 130.
[0106] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each including instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes multiple event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, each event handler 190 includes one or more of a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Instead, one or more of the application views 191 include one or more respective event handlers 190. Also, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in each application view 191.
[0107] Each event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including sub-event delivery instructions).
[0108] Event receiver 182 receives event information from event sorter 170. The event information includes information about a sub-event, e.g., a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves a movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device's attitude).
[0109] The event comparator 184 compares the event information to predefined event or sub-event definitions and determines the event or sub-event or determines or updates the state of the event or sub-event based on the comparison. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a sequence of predefined sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, the sub-events in event 187 include, for example, touch start, touch end, touch movement, touch stop, and multiple touches. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap includes, for example, a first touch (touch start) for a predefined stage on the displayed object, a first lift-off (touch end) for a predefined stage, a second touch (touch start) for a predefined stage on the displayed object, and a second lift-off (touch end) for a predefined stage. In another example, a definition of event 2 (187-2) is a drag on a displayed object. Drag includes, for example, a touch (or contact) of a predetermined stage on a displayed object, a movement of the touch across the touch-sensitive display system 112, and a lift-off of the touch (end of the touch). In some embodiments, the event also includes information regarding one or more associated event handlers 190.
[0110] In some embodiments, event definition 187 includes a definition of the event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, event comparator 184 uses the results of the hit test to determine which event handler 190 to activate. For example, event comparator 184 selects the event handler associated with the sub-event and object that triggers the hit test.
[0111] In some embodiments, each event 187 definition also includes a delay action that delays delivery of the event information until it is determined whether a set of sub-events corresponds to the event type of the event recognizer.
[0112] If the respective event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the respective event recognizer 180 enters an event disabled, event failed, or event finished state and thereafter ignores the next sub-event of the touch-based gesture. In this situation, any other event recognizers that remain active for the hit view continue to track and process sub-events of the ongoing touch gesture.
[0113] In some embodiments, each event recognizer 180 includes metadata 183 with configurable properties, flags, and / or lists that indicate to actively participating event recognizers how the event delivery system should perform sub-event delivery. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact with each other or are enabled to interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are delivered to various levels in a view or programmatic hierarchy.
[0114] In some embodiments, each event recognizer 180 activates an event handler 190 associated with an event when one or more particular sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and postponing sending) the sub-events to the respective hit view. In some embodiments, the event recognizer 180 pops a flag associated with the recognized event, and the event handler 190 associated with the flag catches the flag and performs a predefined process.
[0115] In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with a set of sub-events or to an actively participating view. The event handler associated with the set of sub-events or the actively participating view receives the event information and performs the predetermined processing.
[0116] In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, data updater 176 updates phone numbers used by contacts module 137 or stores video files used by video and music player module 152. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.
[0117] In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
[0118] It should be understood that the foregoing description of event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 with input devices, not all of which are initiated on a touch screen. For example, mouse movements and mouse button presses, contact movements such as tapping, dragging, scrolling on a touchpad, optionally coordinated with single or multiple presses or holds of a keyboard, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof, are optionally utilized as inputs corresponding to sub-events that define the event to be recognized.
[0119] 1C is a block diagram illustrating a tactile output module, according to some embodiments. In some embodiments, I / O subsystem 106 (e.g., haptic feedback controller 161 (FIG. 1A) and / or other input controller(s) 160 (FIG. 1A) includes at least some of the example components shown in FIG. 1C. In some embodiments, peripherals interface 118 includes at least some of the example components shown in FIG. 1C.
[0120] In some embodiments, the tactile output module includes a haptic feedback module 133. In some embodiments, the haptic feedback module 133 aggregates and combines tactile output to user interface feedback from software applications on the electronic device (e.g., user input corresponding to a display user interface, as well as feedback responsive to alerts and other notifications indicating the performance of an action or the occurrence of an event in the user interface of the electronic device). The haptic feedback module 133 includes one or more of a waveform module 123 (providing a waveform used to generate the tactile output), a mixer 125 (mixing waveforms, such as waveforms in different channels), a compressor 127 (reducing or compressing the dynamic range of the waveform), a low pass filter 129 (filtering high frequency signal components in the waveform), and a thermal controller 131 (adjusting the waveform according to thermal conditions). In some embodiments, the haptic feedback controller 161 (FIG. 1A) includes the haptic feedback module 133. In some embodiments, a separate unit of haptic feedback module 133 (or a separate implementation of haptic feedback module 133) is also included in an audio controller (e.g., audio circuit 110 of FIG. 1A) and used to generate audio signals. In some embodiments, a single haptic feedback module 133 is used to generate waveforms for audio signals and tactile outputs.
[0121] In some embodiments, the haptic feedback module 133 also includes a trigger module 121 (e.g., a software application, an operating system, or other software module that determines a tactile output to be generated and initiates the process of generating a corresponding tactile output). In some embodiments, the trigger module 121 generates a trigger signal that initiates the generation of a waveform (e.g., by the waveform module 123). For example, the trigger module 121 generates the trigger signal based on a pre-set timing criteria. In some embodiments, the trigger module 121 receives trigger signals from outside the haptic feedback module 133 (e.g., in some embodiments, the haptic feedback module 133 receives trigger signals from a hardware input processing module 146 located outside the haptic feedback module 133) based on activation of a user interface element (e.g., an application icon or affordance within an application) or a hardware input device (e.g., an intensity-sensing input surface such as a home button or an intensity-sensing touchscreen) and relays those trigger signals to other components within the haptic feedback module 133 (e.g., the waveform module 123) or software applications that trigger actions (e.g., by the trigger module 121). In some embodiments, trigger module 121 also receives tactile feedback generation instructions (e.g., from tactile feedback module 133, FIGS. 1A and 3). In some embodiments, trigger module 121 generates a trigger signal in response to tactile feedback module 133 (or trigger module 121 within tactile feedback module 133) receiving a tactile feedback instruction (e.g., from tactile feedback module 133, FIGS. 1A and 3).
[0122] Waveform module 123 receives a trigger signal as an input (e.g., from trigger module 121) and, in response to the received trigger signal, provides a waveform for generation of one or more tactile outputs (e.g., a waveform selected from a predetermined set of waveforms designed for use by waveform module 123, such as the waveforms described in more detail below with reference to Figures 4F-4G).
[0123] Mixer 125 receives waveforms (e.g., from waveform module 123) as inputs and mixes these waveforms together. For example, when mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps with the first waveform), mixer 125 outputs a combined waveform that corresponds to the sum of the two or more waveforms. In some embodiments, mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or decreasing the scale of the rest of the waveforms). In some circumstances, mixer 125 selects one or more waveforms to be removed from the combined waveform (e.g., when there are waveforms from four or more sources requested to be output simultaneously by tactile output generator 167, the waveform from the oldest source is dropped).
[0124] The compressor 127 receives as input a waveform (e.g., a composite waveform from mixer 125) and modifies the waveform. In some embodiments, the compressor 127 reduces the waveform (e.g., according to the physical specifications of tactile output generator 167 (FIG. 1A) or 357 (FIG. 3)) such that the tactile output corresponding to the waveform is reduced. In some embodiments, the compressor 127 limits the waveform, such as by enforcing a predetermined maximum amplitude for the waveform. For example, the compressor 127 reduces the amplitude of portions of the waveform that exceed a predetermined amplitude threshold while maintaining the amplitude of portions of the waveform that do not exceed the predetermined amplitude threshold. In some embodiments, the compressor 127 reduces the dynamic range of the waveform. In some embodiments, the compressor 127 dynamically reduces the dynamic range of the waveform so that the combined waveform remains within the performance specifications (e.g., force and / or displacement limits of a moving mass) of the tactile output generator 167.
[0125] Low pass filter 129 receives waveforms (e.g., compressed waveforms from compressor 127) as inputs and filters (e.g., smoothes) these waveforms (e.g., removes or reduces high frequency signal components within the waveforms). For example, in some instances, when a tactile output is generated according to a compressed waveform, compressor 127 includes extraneous signals (e.g., high frequency signal components) within the compressed waveform that interfere with the generation of the tactile output and / or exceed the performance specifications of tactile output generator 167. Low pass filter 129 reduces or removes such extraneous signals within the waveform.
[0126] Thermal controller 131 receives waveforms (e.g., filtered waveforms from low pass filter 129) as inputs and adjusts these waveforms according to the thermal state of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies with temperature (e.g., haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and generates a second tactile output when haptic feedback controller 161 is at a second temperature different from the first temperature, in response to receiving the same waveform). For example, the magnitude (or amplitude) of the tactile output may vary with temperature. The waveforms are modified (e.g., the amplitude of the waveform is increased or decreased based on temperature) to reduce the effects of temperature variations.
[0127] In some embodiments, haptic feedback module 133 (e.g., trigger module 121) is coupled to hardware input processing module 146. In some embodiments, other input controller(s) 160 of FIG. 1A includes hardware input processing module 146. In some embodiments, hardware input processing module 146 receives input from hardware input device 145 (e.g., other input or control device 116 of FIG. 1A, such as a home button, or an intensity-sensitive input surface such as an intensity-sensitive touchscreen). In some embodiments, hardware input device 145 is either one of touch-sensitive display system 112 (FIG. 1A), keyboard / mouse 350 (FIG. 3), touchpad 355 (FIG. 3), other input or control device 116 (FIG. 1A), or an input device such as an intensity-sensitive home button. In some embodiments, hardware input device 145 consists of an intensity-sensitive home button and is not touch-sensitive display system 112 (FIG. 1A), keyboard / mouse 350 (FIG. 3), or touchpad 355 (FIG. 3). In some embodiments, in response to input from hardware input device 145 (e.g., an intensity-sensitive home button or a touch screen), hardware input processing module 146 provides one or more trigger signals to haptic feedback module 133 to indicate that a user input has been detected that satisfies a predetermined input criteria, such as an input corresponding to a home button "click" (e.g., a "down click" or an "up click") of the home button. In some embodiments, haptic feedback module 133, in response to an input corresponding to a home button "click," provides a waveform corresponding to a home button "click" to simulate the haptic feedback of pressing a physical home button.
[0128] In some embodiments, the tactile output module includes a haptic feedback controller 161 (haptic feedback controller 161 in FIG. 1A ) that controls generation of the tactile output. In some embodiments, the haptic feedback controller 161 is coupled to a plurality of tactile output generators, selects one or more of the plurality of tactile output generators, and transmits waveforms to the selected one or more tactile output generators that generate the tactile output. In some embodiments, the haptic feedback controller 161 consolidates tactile output requests corresponding to activation of the hardware input device 145 and tactile output requests corresponding to software events (e.g., tactile output requests from the haptic feedback module 133), modifies one or more of the two or more waveforms, emphasizes a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of a particular waveform and / or decreasing the scale of the rest of the waveforms, such as to prioritize tactile output corresponding to activation of the hardware input device 145 over tactile output corresponding to the software event).
[0129] In some embodiments, as shown in FIG. 1C, the output of haptic feedback controller 161 is coupled to an audio circuit of device 100 (e.g., audio circuit 110, FIG. 1A) and provides an audio signal to the audio circuit of device 100. In some embodiments, haptic feedback controller 161 provides both a waveform used to generate the tactile output and an audio signal used to provide an audio output in conjunction with generating the tactile output. In some embodiments, haptic feedback controller 161 modifies the audio signal and / or waveform (used to generate the tactile output) such that the audio output and the tactile output are synchronized (e.g., by delaying the audio signal and / or waveform). In some embodiments, haptic feedback controller 161 includes a digital-to-analog converter used to convert a digital waveform to an analog signal, which is received by amplifier 163 and / or tactile output generator 167.
[0130] In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the tactile feedback controller 161) and amplifies the waveform before transmitting the amplified waveform to a tactile output generator 167 (e.g., either of the tactile output generators 167 (FIG. 1A) or 357 (FIG. 3)). For example, the amplifier 163 amplifies the received waveform to a signal level according to the physical specifications of the tactile output generator 167 (e.g., the voltage and / or current required by the tactile output generator 167 to generate a tactile output such that the signal transmitted to the tactile output generator 167 generates a tactile output corresponding to the waveform received from the tactile feedback controller 161) and transmits the amplified waveform to the tactile output generator 167. In response, the tactile output generator 167 generates a tactile output (e.g., by shifting the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).
[0131] In some embodiments, the tactile output module includes a sensor 169 coupled to the tactile output generator 167. The sensor 169 detects a state or state change (e.g., mechanical position, physical displacement, and / or movement) of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving parts such as a membrane used to generate the tactile output). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or movement sensor. In some embodiments, the sensor 169 provides information (e.g., position, displacement, and / or movement of one or more parts within the tactile output generator 167) to the tactile feedback controller 161, which adjusts the waveform output from the tactile feedback controller 161 (e.g., a waveform sent to the tactile output generator 167, optionally via the amplifier 163) according to the information provided by the sensor 169 regarding the state of the tactile output generator 167.
[0132] FIG. 2 illustrates portable multifunction device 100 having a touch screen (e.g., touch-sensitive display system 112, FIG. 1A ), according to some embodiments. The touch screen optionally displays one or more graphics in user interface (UI) 200. In these embodiments, as well as those described below, a user is enabled to select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of one or more graphics is performed when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upwards and / or downwards), and / or rolling of a finger in contact with device 100 (right to left, left to right, upwards and / or downwards). In some implementations or situations, accidental contact with a graphic does not select the graphic: for example, a swipe gesture sweeping over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.
[0133] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As mentioned above, menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on a touch screen display.
[0134] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and locking the device, a volume control button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external port 124 for docking / charging. Push button 206 is optionally used to power the device on / off by pressing and holding the button down for a predefined period of time, to lock the device by pressing and releasing the button before the predefined time has elapsed, and / or to unlock the device or initiate an unlocking process. In some embodiments, device 100 also accepts verbal input through microphone 113 to activate or deactivate some features. Device 100 also optionally includes one or more contact intensity sensors 165 that detect the intensity of a contact on touch-sensitive display system 112 and / or one or more tactile output generators 167 that generate a tactile output to a user of device 100.
[0135] FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. Device 300 includes input / output (I / O) interfaces 330, including a display 340, which is typically a touchscreen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, as well as a touchpad 355, a tactile output generator 357 for generating tactile outputs on device 300 (e.g., similar to tactile output generator(s) 167 described above with reference to FIG. 1A), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to contact intensity sensor(s) 165 described above with reference to FIG. 1A). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A). Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disk authoring module 388, and / or spreadsheet module 390, while memory 102 of portable multifunction device 100 (FIG. 1A) optionally does not store those modules.
[0136] Each of the elements in FIG. 3 identified above is optionally stored in one or more of the memory devices mentioned above. Each of the modules identified above corresponds to an instruction set that performs the functions described above. The modules or programs (i.e., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules, and thus various subsets of those modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Additionally, memory 370 optionally stores additional modules and data structures not described above.
[0137] Attention is now directed to embodiments of a user interface (“UI”) that is optionally implemented on portable multifunction device 100.
[0138] 4A shows an exemplary user interface 400 for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof: ● signal strength indicator(s) for wireless communication(s), such as cellular signals and Wi-Fi signals; ●Time, ●Bluetooth (registered trademark) indicator, ● Battery status indicator, Tray 408 with icons of frequently used applications, such as: ○ An icon 416 for the phone module 138, labeled “Phone”, optionally including an indicator 414 of the number of missed calls or voicemail messages; ○ An icon 418 of the email client module 140, labeled “Mail”, optionally including an indicator 410 of the number of unread emails; ○ An icon 420 of the Browser module 147, labeled "Browser"; and ○ An icon 422 of the video and music player module 152 labeled "Music", and Icons of other applications, such as: ○ Icon 424 of IM module 141, labeled "Messages"; ○ Icon 426 of the Calendar module 148, labeled "Calendar", ○ Icon 428 of the image management module 144, labeled "Photos"; ○ An icon 430 of the camera module 143, labeled “camera”; ○ Icon 432 of the online video module 155, labeled "Online Video"; Icon 434 of Stock Price Widget 149-2, labeled "Stock Price" ○ Icon 436 of the map module 154, labeled "Map"; ○ Icon 438 of weather widget 149-1, labeled "Weather"; ○ Icon 440 of alarm clock widget 149-4, labeled "Clock"; ○ Icon 442 of Training Support Module 142, labeled "Training Support"; ○ Icon 444 of the Notes module 153, labeled "Notes"; and ○ An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.
[0139] 4A are merely examples. For example, other labels are optionally used for the various application icons. In some embodiments, the label for each application icon includes the name of the application that corresponds to the respective application icon. In some embodiments, the label for a particular application icon is different from the name of the application that corresponds to that particular application icon.
[0140] FIG. 4B illustrates an exemplary user interface on a device (e.g., device 300, FIG. 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3) separate from a display 450. Although many of the following examples are given with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in FIG. 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) that corresponds to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to those embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B) at locations that correspond to respective locations on the display (e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462, and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B ) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B ) of the multifunction device. It is to be understood that similar methods are optionally used for the other user interfaces described herein.
[0141] In addition, while the following examples are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture, etc.), it should be understood that in some embodiments, one or more of those finger inputs are replaced with input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch) followed by movement of a cursor along the path of the swipe (e.g., instead of a movement of the touch). As another example, a tap gesture is optionally replaced by a mouse click (e.g., instead of detecting a touch and subsequently terminating detection of the touch) while the cursor is located over the location of the tap gesture. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally used simultaneously, or mice and finger touches are optionally used simultaneously.
[0142] The term "focus selector" as used herein refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations involving a cursor or other location marker, the cursor functions as a "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations involving a touchscreen display (e.g., touch-sensitive display system 112 in FIG. 1A or touchscreen in FIG. 4A) that enables direct interaction with user interface elements on the touchscreen display, a contact detected on the touchscreen functions as a "focus selector" such that when an input (e.g., a press input by contact) is detected on the touchscreen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of the user interface to another area of the user interface (e.g., by using the tab or arrow keys to move focus from one button to another) without a corresponding cursor movement or contact movement on the touchscreen display. In these implementations, the focus selector moves to follow the movement of focus between various areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device which element of the user interface the user intends to interact with).For example, positioning of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that corresponding button (and not other user interface elements shown on the device's display).
[0143] As used herein and in the claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force, or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or a proxy for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values, and more typically includes hundreds (e.g., at least 256) distinct values. The intensity of a contact is optionally determined (or measured) using a variety of techniques and a variety of sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or sum) to determine an estimated force of the contact. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensitive surface, the capacitance and / or change in the capacitance of the touch-sensitive surface proximate the contact, and / or the resistance and / or change in the capacitance of the touch-sensitive surface proximate the contact, are optionally used as a surrogate for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the surrogate measure of the force or pressure of the contact is used directly to determine whether the intensity threshold is exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measure). In some implementations, the surrogate measure for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).Using the intensity of contact as an attribute of user input enables a user to access additional device functionality that may not otherwise be easily accessible to the user on a device of reduced size with limited assets for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).
[0144] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has “clicked” on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of a particular physical actuator and may be adjusted without modifying the physical hardware of the device 100). For example, the mouse “click” threshold of a trackpad or touchscreen display may be set to any of a wide range of predefined thresholds without modifying the hardware of the trackpad or touchscreen display. Furthermore, in some implementations, a user of the device is provided with software settings to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click “intensity” parameter).
[0145] As used herein and in the claims, the term "characteristic intensity" of a contact refers to a characteristic of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples, i.e., a set of intensity samples collected during a predefined time (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predefined event (e.g., after detecting a contact, before detecting lift-off of the contact, before or after detecting the start of movement of the contact, before detecting an end of the contact, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact). The characteristic intensity of the contact is optionally based on one or more of a maximum intensity of the contact, a mean value of the intensity of the contact, an average value of the intensity of the contact, a top 10% value of the intensity of the contact, a half value of the maximum intensity of the contact, a 90% value of the maximum intensity of the contact, a value generated by low-pass filtering the intensity of the contact starting over a predefined period or at a predefined time, and the like. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is an average of the intensity of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action is performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a first action is performed as a result of a contact having a characteristic intensity that does not exceed the first threshold, a second action is performed as a result of a contact having a characteristic intensity that exceeds the first intensity threshold but does not exceed the second intensity threshold, and a third action is performed as a result of a contact having a characteristic intensity that exceeds the second intensity threshold. In some embodiments, the comparison between the characteristic intensity and one or more intensity thresholds is not used to determine whether to perform a first operation or a second operation, but rather is used to determine whether to perform one or more operations (e.g., to perform a respective option or to refrain from performing a respective operation).
[0146] In some embodiments, a portion of the gesture is identified for purposes of determining the characteristic intensity. For example, a touch-sensitive surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a start position to an end position, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end position may be based on only a portion of the continuous swipe contact (e.g., only the portion of the swipe contact at the end position) rather than the entire swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact prior to determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some circumstances, these smoothing algorithms eliminate narrow spikes or dips in the swipe contact intensity for purposes of determining the characteristic intensity.
[0147] The user interface diagrams described herein optionally include various intensity diagrams showing the current intensity of a contact on the touch-sensitive surface relative to one or more intensity thresholds (e.g., a contact-detection intensity threshold ITO, a light press intensity threshold ITL, a deep press intensity threshold ITD (e.g., at least initially higher than ITL), and / or one or more other intensity thresholds (e.g., an intensity threshold ITH lower than ITL). The intensity diagrams are typically not part of the displayed user interface, but are provided to aid in interpretation of the diagrams. In some embodiments, the light press intensity threshold is set to the intensity at which the device performs the action normally associated with clicking a button on a physical mouse or a trackpad. In some embodiments, the deep pressure intensity threshold corresponds to an intensity at which the device performs an action different than that typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact is detected having a characteristic intensity below the shallow pressure intensity threshold (e.g., above a nominal contact-detection intensity threshold ITO below which contact is no longer detected), the device moves a focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the shallow or deep pressure intensity thresholds. In general, unless otherwise specified, these intensity thresholds are consistent among various sets of values for a user interface.
[0148] In some embodiments, the device's response to an input detected by the device depends on criteria based on the contact intensity during the input. For example, for some "shallow press" inputs, the contact intensity during the input exceeding a first intensity threshold triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria including both the contact intensity during the input and time-based criteria. For example, for some "deep press" inputs, the contact intensity during the input exceeding a second intensity threshold that is greater than the first intensity threshold for shallow presses triggers a second response only if a delay time has elapsed between meeting the first intensity threshold and meeting the second intensity threshold. This delay time is typically less than 200 ms (milliseconds) in time (e.g., 40 ms, 100 ms, or 120 ms depending on the magnitude of the second intensity threshold, with the delay time increasing as the second intensity threshold increases). This delay time helps to avoid accidental recognition of deep press inputs. As another example, for some "deep press" inputs, there is a period of reduced sensitivity that occurs after the time when the first intensity threshold is met. During the period of reduced sensitivity, the second intensity threshold is increased. This temporary increase in the second intensity threshold also serves to avoid accidental deep press inputs. For other deep press inputs, the response to the detection of a deep press input does not depend on a time-based criterion.
[0149] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors, such as user settings, contact movement, input timing, running application, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), position of focus selector, etc. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entireties.
[0150] For example, FIG. 4C illustrates a dynamic intensity threshold 480 that varies over time based in part on the intensity of touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components, a first component 474 that decays over time after a predefined delay time p1 from when touch input 476 is first detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if touch input 476 provides sufficient intensity. The second component 478 reduces unintended triggering of a "deep press" response due to gradual intensity variations of the touch input. In some embodiments, a "deep press" response is triggered when touch input 476 meets the dynamic intensity threshold 480 (e.g., at point 481 in FIG. 4C).
[0151] FIG. 4D illustrates another dynamic intensity threshold 486 (e.g., intensity threshold ID). FIG. 4D also illustrates two other intensity thresholds: a first intensity threshold IH and a second intensity threshold IL. In FIG. 4D, the touch input 484 meets the first intensity threshold IH and the second intensity threshold IL before time p2, but no response is provided until the delay time p2 has elapsed at time 482. Also in FIG. 4D, the dynamic intensity threshold 486 decays over time, beginning at time 488 after the predefined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold IL is triggered). This type of dynamic intensity threshold reduces accidental triggering of a response associated with the dynamic intensity threshold ID immediately or simultaneously with triggering a response associated with a lower intensity threshold, such as the first intensity threshold IH or the second intensity threshold IL.
[0152] 4E illustrates yet another dynamic intensity threshold 492 (e.g., Intensity Threshold ID). In FIG. 4E, the response associated with Intensity Threshold IL is triggered after a delay time p2 from when touch input 490 is first detected. At the same time, dynamic intensity threshold 492 decays after a predefined delay time p1 from when touch input 490 is first detected. As such, a decrease in the intensity of touch input 490 after triggering a response associated with Intensity Threshold IL, followed by an increase in the intensity of touch input 490 without releasing touch input 490, can trigger a response associated with Intensity Threshold ID (e.g., at time 494) even when the intensity of touch input 490 falls below another intensity threshold, e.g., Intensity Threshold IL.
[0153] An increase in the characteristic intensity of a contact from an intensity below a shallow pressure intensity threshold ITL to an intensity between the shallow pressure intensity threshold ITL and the deep pressure intensity threshold ITD may be referred to as a "shallow press" input. An increase in the characteristic intensity of a contact from an intensity below a deep pressure intensity threshold ITD to an intensity above the deep pressure intensity threshold ITD may be referred to as a "deep press" input. An increase in the characteristic intensity of a contact from an intensity below a contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the shallow pressure intensity threshold ITL may be referred to as a detection of a contact on the touch surface. A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 may be referred to as a detection of a lift-off of the contact from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, a shaded circle or ellipse is used to represent the intensity of a contact on the touch-sensitive surface. In some examples, an unshaded circle or ellipse is used to represent each contact on the touch-sensitive surface without specifying the intensity of each contact.
[0154] In some embodiments described herein, one or more actions are performed including upon detecting a gesture including a respective press input or including upon detecting a respective press input performed on a respective contact (or contacts), where the respective press input is detected based at least in part on detecting an increase in intensity of the contact (or contacts) above a press input intensity threshold. In some embodiments, the respective actions are performed including upon detecting an increase in intensity of the respective contact above a press input intensity threshold (e.g., the respective actions are performed on a "downstroke" of the respective press input). In some embodiments, the press input includes an increase in intensity of the respective contact above a press input intensity threshold and a subsequent decrease in intensity of the contact below the press input intensity threshold, where the respective actions are performed including upon detecting a subsequent decrease in intensity of the respective contact below the press input threshold (e.g., the respective actions are performed on an "upstroke" of the respective press input).
[0155] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, sometimes referred to as "jitter," and the device defines or selects a hysteresis intensity threshold that has a predefined relationship to the press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units below the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, the press input includes an increase in the intensity of the respective contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a respective action is performed in response to detecting the subsequent decrease in intensity of the respective contact below the hysteresis intensity threshold (e.g., the respective action is performed on the "upstroke" of the respective press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the press input intensity threshold, and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and a respective action is performed in response to detecting the press input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, as the case may be).
[0156] For ease of explanation, the description of the operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is optionally triggered in response to detecting an increase in the intensity of the contact above the press input intensity threshold, an increase in the intensity of the contact from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, a decrease in the intensity of the contact below the press input intensity threshold, or a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the press input intensity threshold. Furthermore, in examples where an operation is described as being performed in response to detecting a decrease in the intensity of the contact below a press input intensity threshold, the operation is optionally performed in response to detecting a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold. As discussed above, in some embodiments, the triggering of these responses also depends on a time-based criterion being met (e.g., a delay time has elapsed between the first intensity threshold being met and the second intensity threshold being met).
[0157] As used herein and in the claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, that will be detected by a user with the user's sense of touch. For example, in a situation where a device or a component of a device is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of a user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, a user feels a tactile sensation such as a "down-click" or "up-click" even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's action. As another example, movement of a touch-sensitive surface is optionally interpreted or sensed by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. Such user interpretation of touch depends on the user's personal sensory perception, but there are many sensory perceptions of touch that are common to a majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an "up click," a "down click," "roughness"), unless otherwise noted, the generated tactile output corresponds to a physical displacement of the device, or a component of a device, that produces the described sensory perception of a typical (or average) user.Using tactile output to provide haptic feedback to the user improves usability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs when operating / interacting with the device and reducing user errors), as well as reducing power usage and improving the battery life of the device by enabling the user to use the device more quickly and efficiently. User Interface and Related Processes
[0158] Attention is now directed to embodiments of user interfaces ("UI") and associated processes that may be implemented on an electronic device, such as portable multifunction device 100 or device 300, that includes a display and a touch-sensitive surface, (optionally) one or more tactile output generators for generating tactile output, and (optionally) one or more sensors for detecting the intensity of contact with the touch-sensitive surface.
[0159] 5A-5AF show examples of user interfaces for relocating an annotation, according to some embodiments. The user interfaces in those figures are used to illustrate processes described below, including those in FIGS. 9A-9F, 10A-10B, 11A-11F, and 12A-12D. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., an enantiomeric center of each contact or a point associated with each contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, similar operations are optionally performed on a device having display 450 and separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying the user interface shown in the figures on display 450, along with a focus selector.
[0160] 5A shows an annotation user interface 5003 displayed on the touch screen 112 of the device 100. The user interface displays the field of view of the camera of the device 100 as it captures the physical environment 5000 of the device 100. A table 5001a and a mug 5002a are located within the physical environment 5000. The camera view includes a visual representation 5001b of the physical table 5001a as displayed in the user interface 5003. The user interface 5003 includes controls 5004 for toggling between still and video modes, controls 5006 for controlling camera flash settings, controls 5008 for accessing camera settings, and a mode indicator 5010 that indicates the current mode.
[0161] From Figures 5A to 5B, the device 100 is moved relative to the physical environment such that the portion of the table 5001b visible within the camera's field of view changes and the visual representation 5002b of the physical mug 5002a becomes visible within the camera's field of view.
[0162] In FIG. 5C , stylus 5012 is touched on touch screen display 112 at a point in annotation user interface 5003 indicated by contact 5014. In response to detection of contact 5014, a still image of the camera's field of view is captured and displayed in user interface 5003. The state of mode control 5004 and mode indicator 5010 are changed to indicate that the active mode of the user interface has changed from video mode to still image mode. The transition to still image mode, which occurs in response to a touchdown of the stylus (and / or in response to other types of input, as described further below), allows annotation input to be received relative to the view of the physical environment captured by camera 143 without being affected by changes in the spatial relationship between device 100 and physical environment 5000 caused by movement of device 100.
[0163] 5C to 5D, device 100 has been moved relative to physical environment 5000. The image displayed by touchscreen display 112 does not change from Figure 5C to Figure 5D because the active mode of the user interface has changed from video mode to still image mode.
[0164] From FIG. 5E to FIG. 5G, contact 5014 moves along a path indicated by arrow 5016, generating a first annotation 5018 on a portion of the still image that includes visual representation 5002b of physical mug 5002a.
[0165] In Figure 5H, stylus 5012 provides input at a location corresponding to mode control 5004, as indicated by contact 5019. In Figure 5I, in response to the input by stylus 5012, the state of mode control 5004 and mode indicator 5010 have changed to indicate that the active mode of annotation user interface 5003 has changed from a still image mode to a video mode.
[0166] From Figures 5I-5L, the still image displayed in Figure 5I gradually transitions from a full-sized view of the still image (as shown in user interface 5003 of Figure 5I) to a miniature view 5020 (as shown in user interface 5003 of Figures 5J-5K) up to an indicator dot 5022 (as shown in user interface 5003 of Figure 5L). In Figure 5J, a miniature view 5020 of the still image shown in Figure 5I is shown covering the camera's current field of view. The size of the miniature view 5020 is reduced from Figure 5J to Figure 5K (e.g., to provide an indication of the correspondence between the still image displayed by device 100 in Figure 5I and the indicator dot 5022 displayed over the video corresponding to the device camera's field of view in Figure 5L).
[0167] 5L-5M, the position of device 100 is changed relative to physical environment 5000. As device 100 moves, the field of view of the camera of device 100 changes, causing the position of indicator 5022 in user interface 5003 to change. The movement of indicator 5022 in user interface 5003 provides an indication of the virtual spatial location of annotation 5018 relative to the current location of device 100. In this manner, the user is provided with visual feedback indicating that movement of device 100 in the direction indicated by indicator 5022 is required to redisplay annotation 5018.
[0168] 5M-5N, the position of device 100 continues to change relative to physical environment 5000. As a result of device 100's movement, the field of view of the camera of device 100 is updated such that the portion of physical environment 5000 captured in the annotated still image of FIG. 5I is visible in the field of view of the camera. Annotation 5018 is displayed in a location in the video (e.g., in each image frame including a visual representation of physical mug 5002a) that corresponds to the location in the still image where annotation 5018 was received (e.g., in a location that corresponds to the location of visual representation 5002b of physical mug 5002a (e.g., as shown in the still image of FIG. 5I)). In some embodiments, as device 100 moves closer to physical mug 5002a, visual representation 5002b of physical mug 5002a appears larger in the video image, and annotation 5018 also appears larger according to the changed size of visual representation 5002b. In some embodiments, as device 100 moves around physical mug 5002a, visual representation 5002b of physical mug 5002a is updated to reflect the different viewing angles of physical mug 5002a, and the appearance of annotations 5018 in the video image is also updated according to the change in viewing angle of the physical mug (e.g., as viewed from a different angle).
[0169] 5O, stylus 5012 is touched down on touch screen display 112 at a point in annotation user interface 5003 indicated by contact 5024. In response to detecting contact 5024, a second still image of the camera's field of view is captured and displayed in user interface 5003.
[0170] 5P through 5R, while a second still image is displayed, input by stylus 5012 is received at a location indicated by contact 5026. Movement of contact 5026 creates a second annotation 5028 on a portion of the still image that includes visual representation 5002b of physical mug 5002a. In FIG. 5S, contact 5012 has lifted off of touch screen 112.
[0171] In Figure 5T, an input is detected at a location on touch screen display 112 corresponding to control 5004 for toggling between still image mode and video mode, as indicated by contact 5030. In Figure 5U, in response to the input, the state of mode control 5004 and mode indicator 5010 change to indicate that the active mode of annotation user interface 5003 has changed from still image mode to video mode. Because the portion of physical environment 5000 captured in the annotated still image of Figure 5T is already visible within the field of view of the camera displayed in the video mode of user interface 5003 of Figure 5U, annotations 5018 and 5028 are displayed at locations in the video that correspond to the corresponding locations in the still image where the annotations were received (e.g., locations that correspond to the corresponding locations of visual representation 5002b of physical mug 5002a shown in Figures 5G and 5R).
[0172] 5U through 5V, the position of device 100 is altered relative to physical environment 5000 such that the field of view of the device camera displayed in user interface 5003 does not include the portion of physical environment 5000 that includes mug 5002a. Indicator dot 5022 corresponding to annotation 5018 and indicator dot 5032 corresponding to annotation 5028 are displayed (e.g., indicator dots 5022 and 5032 are displayed in positions that indicate the off-screen virtual spatial locations of annotations 5018 and 5028, respectively, relative to physical environment 5000 (e.g., the sides of physical mug 5002a shown in FIGS. 5G and 5R).
[0173] From Figure 5V to Figure 5W, the position of device 100 continues to change relative to physical environment 5000. As device 100 moves downward, the field of view of the camera of device 100 changes and indicators 5022 and 5032 move upward in user interface 5003 (e.g., indicating the virtual spatial positions of annotations 5018 and 5028 relative to the current position of device 100). In Figure 5W, stylus 5012 is touched down on touch screen display 112 at a point in annotation user interface 5003 indicated by contact 5034. In response to detecting contact 5034, a third still image of the camera's field of view is captured and displayed in user interface 5003.
[0174] From Figure 5W to Figure 5X, while the third still image is displayed, contact 5034 moves along the touch screen display 112 to create a third annotation 5036 in a portion of the third still image including the lower right surface of visual representation 5001b of physical table 5001a.
[0175] In Figure 5Y, stylus 5012 provides input at a location corresponding to mode control 5004, as indicated by contact 5038. In Figure 5Z, in response to the input by stylus 5012, the state of mode control 5004 and mode indicator 5010 have changed to indicate that the active mode of annotation user interface 5003 has changed from still image mode to video mode. Because the lower right surface of table 5001a is visible within the field of view of the camera displayed in the video mode of user interface 5003 of Figure 5Z, annotation 5036 is displayed at a location in the video (e.g., in an image frame that includes the location of the table surface shown in Figure 5X) that corresponds to the location in the still image where annotation 5036 was received (e.g., a location corresponding to the lower right surface of visual representation 5001b of physical table 5001a). The portion of physical environment 5000 visible within the field of view of the camera displayed in video mode of user interface 5003 does not include the portion of the physical environment that corresponds to the spatial locations of annotations 5018 and 5028 (e.g., mug 5002a is not visible within the camera's field of view), and indicator dots 5022 and 5032 are displayed in positions relative to physical environment 5000 that indicate the off-screen virtual spatial locations of annotations 5018 and 5028.
[0176] 5Z through 5AA, the position of device 100 changes relative to physical environment 5000. As device 100 moves upward, the field of view of the device 100's camera changes and indicators 5022 and 5032 move downward in user interface 5003. Because the lower right face of table 5001a is no longer visible in the field of view of the camera displayed in the video mode of the user interface, indicator dot 5040 is displayed at a location in the video that corresponds to the location in the still image where annotation 5036 was received.
[0177] 5AA to 5AB, the position of device 100 changes relative to physical environment 5000. As device 100 continues to move upward, the field of view of the camera of device 100 changes and indicators 5022, 5032, and 5040 move downward in user interface 5003 to indicate the off-screen virtual spatial locations of annotations 5018, 5028, and 5036 relative to physical environment 5000.
[0178] 5AB to 5AC, the position of device 100 changes, causing a change in the field of view of the camera of device 100 displayed in user interface 5003. The positions of indicators 5022, 5032, and 5040 are updated based on the off-screen virtual spatial positions of annotations 5018, 5028, and 5036.
[0179] 5AC to 5AD, the position of device 100 is changed relative to physical environment 5000 such that the field of view of the device camera displayed in user interface 5003 includes partial physical environment 5000, including mug 5002a. Annotations 5018 and 5028 are displayed in user interface 5003, and indicator dots 5022 and 5032 are no longer displayed.
[0180] 5AD through 5AE, the position of device 100 is changed relative to physical environment 5000 such that the field of view of the device camera displayed in user interface 5003 includes partial physical environment 5000 that includes the lower right surface of table 5001a. Annotation 5036 is displayed in user interface 5003 and indicator dot 5040 is no longer displayed.
[0181] From Figures 5AE to 5AF, device 100 is moved around the periphery and over table 5001a such that the position and perspective of annotations 5018, 5028 and 5036 change.
[0182] 6A-6N show exemplary user interfaces for receiving annotations on a portion of a physical environment captured in a still image corresponding to a paused position of a video, according to some embodiments. The user interfaces in those figures are used to illustrate processes described below, including those in FIGS. 9A-9F, 10A-10B, 11A-11F, and 12A-12D. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., an enantiomeric center of the respective contact or a point associated with the respective contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, similar operations are optionally performed on a device having display 450 and separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying the user interface shown in the figures on display 450, along with a focus selector.
[0183] 6A illustrates a user interface 6000 that includes a video playback area 6002. In some embodiments, the user interface 6000 is accessed via a list of media content objects (e.g., in an image and / or video browsing application). In some embodiments, the user interface 6000 also includes a timeline 6004 (e.g., a set of sample frames 6006 that correspond to successive segments of a video). The timeline 6004 includes a current position indicator 6008 that indicates a position on the timeline 6004 that corresponds to a frame displayed in the video playback area 6002 through a video displayed in the video playback area 6002. In some embodiments, the user interface includes a markup control 6010 (e.g., for initiating a markup mode for marking the video displayed in the video playback area 6002), a rotate control 6012 (e.g., for rotating the video displayed in the video playback area 6002), an edit control 6014 (e.g., for editing the video displayed in the video playback area 6002), a cancel control 6020 (e.g., for canceling a current operation), a rotate control 6022 (e.g., for rotating the video displayed in the video playback area 6002), and a play / pause toggle control 6016 (e.g., for playing and pausing the video displayed in the video playback area 6002). Contact 6018 (e.g., input by a user's finger) with the touch screen display 112 is detected at a location corresponding to the play / pause toggle control 6016.
[0184] In FIG. 6B, in response to an input detected at a location corresponding to play / pause toggle control 6016, video playback area 6002 begins playing the video.
[0185] 6C , as the video continues to play, input is detected at a location corresponding to markup control 6010, as indicated by contact 6024. In response to input by contact 6024 at a location corresponding to markup control 6010, playback of the video is paused, a still image corresponding to the paused location of the video is displayed, a markup mode is initiated in which input received at a location corresponding to video playback area 6002 marks the video, and the state of markup control 6010 is changed to display the text "Done" (e.g., to indicate that input into selection control 6010 terminates markup mode).
[0186] In Figures 6D-6F, annotation input is detected at a location within the video playback region 6002, as indicated by contact 6026. As the contact moves along the path shown in Figures 6D-6F, annotation 6030 is received. In some embodiments, the annotation is received at a location corresponding to an object in the video (e.g., kite 6028). In Figure 6G, while the markup mode is active and the text "Done" is displayed, input by contact 6031 is detected at a location corresponding to markup control 6010. In response to input by contact 6031, the markup session is ended (and / or the time since the last input was received increases above a threshold amount of time) and the play / pause toggle control 6016 is redisplayed as shown in Figure 6H.
[0187] In FIG. 6I, input is detected at a location corresponding to play / pause toggle control 6016, as indicated by contact 6032.
[0188] In Figures 6I-6K, in response to an input detected at a location corresponding to play / pause toggle control 6016, the video resumes playback in video playback area 6002. As shown in Figure 6J, sample frames of timeline 6004 include a marking at a location in the respective image that corresponds to kite 6028. For example, sample frame 6006 includes marking 6034 at a location that corresponds to kite 6036. Annotation 6030 applied to kite object 6028 in video playback area 6002 has been applied to sample frames in timeline 6004 that include frames of video occurring before the point in the video at which annotation 6030 was received (e.g., frames to the left of scrub control 6008 in timeline 6004, such as sample frame 6006) and frames of video occurring after the point in the video at which annotation 6030 was received (e.g., frames to the right of scrub control 6008 in timeline 6004). In each sample frame of the timeline 6004 in which the kite object 6036 (corresponding to the kite object 6028 for which annotation 6030 was applied to the video playback area 6002) is visible, a marking (e.g., marking 6034 corresponding to annotation 6030) is shown at a position corresponding to the changing position of the kite object 6036 in the sample frame. The marking is displayed at a size and orientation that is scaled corresponding to the change in size and orientation of the kite object 6036 in the sample frame. As the video displayed in the video playback area 6002 plays forward beyond the frame at which the video was stopped to receive annotation input later in the video, the annotation 6030 is displayed at a size and orientation that is scaled corresponding to the change in size and orientation of the kite object 6026, as shown in FIG. 6K. In this manner, annotations received in the video playback area 6002 are applied to the object (e.g., such that the annotation tracks the object) as the object moves, changes size (e.g., due to a change in distance from the camera), and changes orientation (e.g., in any direction in three-dimensional space).
[0189] In FIG. 6L, input is detected at a location corresponding to timeline 6004, as indicated by contact 6038. As contact 6038 moves along the path indicated by arrows 6040 and 6044, the video displayed in video playback area 6002 is rewound, as shown in FIGS. 6L-6N. For example, as contact 6038 moves across timeline 6004, time indication 6042 associated with the currently displayed frame in video playback area 6002 decreases. When the video displayed in video playback area 6002 rewinds to a frame prior to the frame at which the video was stopped to receive annotation input, annotation 6030 is displayed with a scaled size and orientation corresponding to the change in size and orientation of kite object 6026.
[0190] 7A-7BF show exemplary user interfaces for adding a virtual object to a previously captured media object, according to some embodiments. The user interfaces in those figures are used to illustrate processes described below, including those in FIGS. 9A-9F, 10A-10B, 11A-11F, and 12A-12D. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., an enantiomeric center of the respective contact or a point associated with the respective contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, similar operations are optionally performed on a device having display 450 and separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying the user interface shown in the figures on display 450, along with a focus selector.
[0191] 7A illustrates a user interface 7000 displayed by the touch screen display 112 of device 100 including a media object display area 7002 and a navigation area 7004. In some embodiments, the user interface 7000 is accessed (e.g., in an image and / or video viewing application) via a list of media content objects. Previously captured images are displayed in the media object display area. Information corresponding to the previously captured images (e.g., the location where the image was captured, "Cupertino") is displayed in the information area 7003. The navigation area 7004 includes a previous media object control 7006 (e.g., for navigating to a previous media object), a current media object indicator 7008 (e.g., indicating the location of the previously captured image (represented by an enlarged dot) relative to other media objects (represented by non-enlarged dots) stored by device 100), and a later media object control 7010 (e.g., for navigating to a later media object). The user interface 7000 includes controls 7012-7024 for adding various virtual objects to previously captured images displayed within the media object display area 7002, as discussed further below with respect to Figures 7B-7BG. As described in accordance with various embodiments herein, the manner in which virtual objects are displayed relative to physical objects in previously captured images provides an indication to the user that depth data has been stored for the previously captured images and that the virtual objects can interact with various surfaces in the various images. Interaction of the virtual objects with physical objects in the physical environment captured in the images provides the user with an indication of the presence of detectable surfaces in the images.
[0192] 7B-7L show how a virtual ball object interfaces with the surface of a physical object depicted in a previously captured image displayed in media object display area 7002. For example, the captured image is stored along with depth data that is used to determine the location of surfaces (e.g., horizontal and / or vertical planes) that correspond to the physical object in the physical environment captured in the image.
[0193] In Figure 7B, input by contact 7026 (e.g., tap input) is received at a location corresponding to ball control 7012 for adding a virtual ball to a previously captured image displayed within media object display area 7002. In Figure 7C, in response to detecting input selecting ball control 7012, the mode of user interface 7000 is changed to a ball generation mode, as indicated by the "Ball" label in information area 7003 and by the changed visual state of ball control 7012. Input (e.g., tap input) is received at a location indicated by contact 7028. In response to detecting the contact, a virtual ball 7030 is added to a previously captured image displayed within media object display area 7002. For example, adding virtual ball 7030 to a previously captured image includes launching virtual ball 7030 having an upward trajectory along the path indicated by dotted line 7032 from the point where contact 7028 is detected (e.g., virtual ball 7030 is influenced by simulated gravity to bounce off the surface of chair object 7036 in the previously captured image, bounce off the surface of table object 7038 in the previously captured image, fall onto floor surface 7040 in the previously captured image, and roll along floor surface 7040).
[0194] 7D , an input (e.g., a tap input) is received at a location indicated by contact 7042. In response to detecting the contact, a virtual ball 7044 is added to the previously captured image displayed within media object display area 7002, and virtual ball 7044 travels along a path 7046 (e.g., bouncing off of floor 7040 and landing on floor 7040).
[0195] In FIG. 7E, several additional virtual balls have been added to the previously captured image displayed in the media object display area 7002, some of which have landed and anchored on the surfaces of the chair object 7036 and the table object 7038. An input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the subsequent media object control 7010, indicated by contact 7048. In response to the input, the display of the first previously captured image shown in FIG. 7E is replaced by a second previously captured image, as shown in FIG. 7F. The virtual balls added to the previously captured image displayed in the media object display area 7002 are added to the second previously captured image, as shown in FIG. 7F (e.g., the virtual balls are animated as if they are under the influence of pseudo gravity from the floor surface 7040 in the first previously captured image of FIG. 7E to the top of the physical environment of the second previously captured image, as shown in FIG. 7F-FIG. 7I). As the virtual ball falls, it settles on surfaces in the physical environment captured in the second previously captured image, such as the surfaces of ramp 7052 and table 7054, and the arm grooves of people 7056 and 7058. In some embodiments, the virtual object is compared to depth data corresponding to physical objects in the previously captured image to determine the placement of the virtual object relative to the physical objects (e.g., to determine whether the physical object occludes the virtual object, or vice versa). For example, in FIG. 7G, the virtual ball 7045 is partially occluded by the physical table 7054.
[0196] In FIG. 7J, an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the subsequent media object control 7010 indicated by contact 7050. In response to the input, the display of the second previously captured image shown in FIG. 7J is replaced by a third previously captured image, as shown in FIG. 7K. The virtual ball added to the second previously captured image displayed within the media object display area 7002 is added to the first previously captured image, as shown in FIG. 7K (e.g., the virtual ball is animated as if under the influence of pseudo gravity from a surface in the second previously captured image of FIG. 7J to the top of the physical environment of the third previously captured image, as shown in FIG. 7K-FIG. 7L). As the virtual ball falls, it settles onto a surface in the physical environment captured in the third previously captured image, e.g., the surface of the couch 7060.
[0197] In Figure 7L, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to text insertion control 7014, indicated by contact 7062. In response to the input, the virtual balls (e.g., balls 7034 and 7044) are no longer displayed and a text object 7064 is added to the third previously captured image, as shown in Figure 7M.
[0198] 7M-7T show how a virtual text object 7064 interfaces with the surface of a physical object depicted in a previously captured image displayed in the media object display area 7002.
[0199] In FIG. 7M, a tap input by contact 7066 is detected at a location on touch screen 112 corresponding to text object 7064. From FIG. 7M through FIG. 7P, contact 7066 moves along a path indicated by arrows 7068, 7070, and 7072. As contact 7066 moves, text object 7064 is "dragged" by contact 7066 such that the movement of text object 7064 corresponds to the movement of contact 7066. As text object 7064 is dragged to a position corresponding to couch 7060, text object 7064 interacts with the detected surface of couch 7060 by "marching" across the arms of couch 7060, as shown in FIG. 7N-7O. For example, in FIG. 7O, when the text object 7064 is dragged into the area of the third previously captured image corresponding to the sofa 7060, a first portion of the text object 7064 is adjacent to a first surface of the sofa 7060 (e.g., above the arms of the sofa) and a second portion of the text object 7064 is adjacent to a second surface of the sofa 7060 (e.g., above the seat of the sofa 7060).
[0200] In Figure 7Q, input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to text object 7064, as indicated by contact 7062. In response to the input, a text editing mode is initiated for text object 7064, as indicated by the display of cursor 7080 and keyboard 7078 in Figure 7R. In Figure 7S, the input provided via keyboard 7078 has changed the text of text object 7064 from the word "text" to the word "chills."
[0201] In Figure 7T, input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to insert decal control 7016, as indicated by contact 7064. In response to the input, text object 7064 disappears and decal object 7084 is added to the third previously captured image, as shown in Figure 7U.
[0202] 7U-7Y show how the virtual decal object 7084 interfaces with the surface of a physical object depicted in a previously captured image displayed in the media object display area 7002.
[0203] In Figure 7U, a tap input by contact 7086 is detected at a location on touch screen 112 corresponding to decal object 7084. From Figure 7U to Figure 7X, contact 7086 moves along the path indicated by arrows 7088, 7090, and 7092. As the contact 7086 moves, the decal object 7084 is "dragged" by the contact 7086 such that the movement of the decal object 7084 corresponds to the movement of the contact 7086. As the decal object 7084 is dragged onto the surface of the couch 7060, the decal object 7084 conforms to the detected horizontal and vertical surfaces of the couch 7060 and the floor 7094, as shown in Figures 7U-7X. For example, in Figure 7V, when the decal object 7084 is dragged onto the couch 7060, a first portion of the decal object 7084 is adjacent to a first surface of the couch 7060 (flat on the seat of the couch) and a second portion of the decal object 7084 is adjacent to a second surface of the couch 7060 (e.g., dragged in front of the couch 7060). In Figure 7X, when the decal object 7084 is dragged onto the floor 7094, the decal object 7084 is partially occluded by the couch 7060.
[0204] In Figure 7Y, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to emoji insertion control 7018 as indicated by contact 7096. In response to the input, decal object 7084 disappears and emoji object 7098 is added to the third previously captured image, as shown in Figure 7Z.
[0205] 7Z-7AE show how a virtual emoji object 7098 interfaces with the surface of a physical object depicted in a previously captured image displayed in the media object display area 7002.
[0206] In FIG. 7AA, a tap input by contact 7100 is detected at a location on touch screen 112 corresponding to emoji object 7098. In FIGS. 7AA and 7AB, contact 7100 moves along a path indicated by arrow 7102. As contact 7100 moves, it "drags" emoji object 7098 such that the movement of emoji object 7098 corresponds to the movement of contact 7100. In FIG. 7AC, contact 7100 lifts off of touch screen display 112 while emoji object 7098 is suspended in space above surfaces (e.g., couch 7060 and floor 7094) in the third previously captured image. In response to the lift-off of contact 7100, emoji object 7098 descends under the influence of simulated gravity, as shown in FIGS. 7AC-7AE. In FIG. 7AC, the glyph object 7098 encounters the physical object couch 7060, causing the glyph object's orientation to change as the glyph object 7098 rolls up the arm of the couch 7060 and continues to descend to the floor 7094, as shown in FIG. 7AD.
[0207] In Figure 7AF, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to confetti insertion control 7020, as indicated by contact 7104. In response to the input, emoji object 7098 is no longer displayed and a confetti object (e.g., confetti object 7106) is added to the third previously captured image, as shown in Figure 7AG.
[0208] 7AG-7AT show how virtual confetti objects interface with the surfaces of physical objects depicted in previously captured images and videos displayed within media object display area 7002. In FIG.
[0209] In Figures 7AG-7AJ, virtual confetti objects are continually added (e.g., as shown in Figures 7AH, 7AI, and 7AJ) and, under the influence of pseudo gravity, collect on detected surfaces (e.g., substantially horizontal surfaces, such as horizontal surfaces on couch 5060 and floor 7094) in the third previously captured image.
[0210] In Figure 7AJ, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to the previous media object control 5006 as indicated by contact 7104. In response to the input, the display of the third previously captured image shown in Figure 7AJ is replaced by a display of the second previously captured image, as shown in Figure 7AK. The confetti added to the third previously captured image displayed within media object display area 7002 is added to the second previously captured image as shown in Figure 7AK (e.g., displayed in the same location where the confetti was displayed in Figure 7AJ).
[0211] In Figures 7AK-7AL, virtual confetti objects are continually added and, under the influence of pseudo gravity, gather onto detected surfaces in the second previously captured image.
[0212] In Figure 7AJ, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to the previous media object control 5006 as indicated by contact 7104. In response to the input, the display of the third previously captured image shown in Figure 7AJ is replaced by a display of the second previously captured image, as shown in Figure 7AK. The confetti added to the third previously captured image displayed within media object display area 7002 is added to the second previously captured image as shown in Figure 7AK (e.g., displayed in the same location where the confetti was displayed in Figure 7AJ).
[0213] In Figures 7AK-7AL, virtual confetti objects are continually added and, under the influence of pseudo gravity, gather onto detected surfaces in the second previously captured image.
[0214] In Figure 7AL, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to subsequent media object control 7010 as indicated by contact 7110 (e.g., multiple taps are received to advance the currently displayed media object twice). In response to the input, the display of the second previously captured image shown in Figure 7AL is replaced by a display of the previously captured video as shown in Figure 7AM. The confetti added to the second previously captured image displayed within media object display area 7002 is added to the video as shown in Figure 7AM (e.g., displayed in the same location where the confetti was displayed in Figure 7AL).
[0215] In Figures 7AM-7AT, virtual confetti objects are continually added and, under the influence of pseudo gravity, collect at detected surfaces in the video (e.g., at the edges of kite object 7112). For example, as video playback progresses through Figures 7AN, 7AO, and 7AP, virtual confetti objects are continually added and, under the influence of pseudo gravity, collect at the edges of kite object 7112 and the bottom edge of the video frame.
[0216] In Figure 7AP, an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to playback control 7114 as indicated by contact 7116. In response to the input, playback of the video is repeated. The confetti that was added to the video of Figures 7AM-7AT is added to the video, at which point playback begins in Figure 7AQ. For example, when playback of the video begins, the confetti that has accumulated on kite object 7112 falls from the position shown in Figure 7AP and kite object 7112 is shown in a different position within the video of Figure AQ.
[0217] In Figures 7AQ-7AT, virtual confetti objects are continually added and, under the influence of pseudo gravity, collect at detected surfaces in the video, such as the edges of kite object 7112. For example, as video playback progresses through Figures 7AR, 7AS, and 7AT, virtual confetti objects are continually added and, under the influence of pseudo gravity, collect at the edges of kite object 7112 and the bottom edge of the video frame.
[0218] In some embodiments, the displayed virtual confetti objects are faded out and / or removed from view (e.g., when the time elapsed since the virtual confetti objects were displayed exceeds a threshold time).
[0219] 7AU-7AX show how a virtual spotlight object 7118 interfaces with physical objects depicted in a previously captured image displayed in the media object display area 7002.
[0220] In FIG. 7AU, a second previously captured image is displayed within the media object display area 7002, and a spotlight mode has been activated (e.g., in response to input received at the spotlight control 7022). In the spotlight mode, a spotlight virtual object 7118 is shown illuminating a portion of the image (e.g., person 7058) and an area of the image beyond the spotlight virtual object 7118. In this manner, the spotlight virtual object 7118 allows for attention to be focused on a particular portion of the image. In some embodiments, the first physical object to be illuminated is selected automatically (e.g., based on a determination of the closest physical object to the foreground). The spotlight virtual object 7118 includes a simulated light beam 7122 and a simulated illumination spot 7124 that illuminates a portion of the floor in the previously captured image. In some embodiments, the light beam 7122 illuminates at least a portion of a representation of the physical object in the previously captured image. In some embodiments, the illumination spot 7124 illuminates a portion of the image that corresponds to a horizontal plane detected in the image, such as the floor 7124.
[0221] In Figure 7AV, a tap input by contact 7128 is detected at a location on touch screen 112 corresponding to spotlight object 7118. In Figures 7AV and 7AW, contact 7128 moves along the path indicated by arrow 7102. As contact 7128 moves, spotlight object 7118 is "dragged" by contact 7128 such that the movement of spotlight object 7118 corresponds to the movement of contact 7128. In Figure 7AW, the position of spotlight object 7118 has been shifted such that person 7056 is illuminated by spotlight object 7118. As spotlight object 7118 moves, the size of illumination spot 7124 has changed.
[0222] 7AX, an input (eg, a tap input) is detected at a location on touch screen display 112 corresponding to measurement control 7020, as indicated by contact 7026. In response to the input, spotlight object 7118 is no longer displayed.
[0223] In Figure 7AY, input is detected at a location on touch screen display 112 indicated by contacts 7132 and 7134. In Figure 7AZ, in response to detecting contacts 7132 and 7134, virtual measurement object 7136 is displayed at a location that spans the distance between contacts 7132 and 7134 (e.g., corresponding to the height of person 7058). Measurement indicator 7138 indicates that the distance between points corresponding to contacts 7132 and 7134 in the physical environment captured in the image is 1.8 m (e.g., determined using depth data stored with a previously captured image).
[0224] 7AZ to 7BA, contact 7132 moves along the path indicated by arrow 7140. As contact 7132 moves, the size of virtual measurement object 7136 is adjusted to span the adjusted distance between contacts 7132 and 7134, and the measurement indicated by measurement indicator 7138 is updated.
[0225] In FIG. 7BB, contacts 7132 and 7134 have been lifted off of the touch screen 112. The virtual measurement object 7136 and measurement indicator 7138 remain displayed. An input is detected at a location on the touch screen display 112 indicated by contact 7142. From FIG. 7BB to FIG. 7BC, contact 7142 moves along a path indicated by arrows 7144 and 7146. In response to movement of contact 7142 (e.g., beyond a threshold amount of movement), a virtual measurement object 7148 and measurement indicator 7150 are displayed. From FIG. 7BC to FIG. 7BD, as contact 7142 continues to move, the size of virtual measurement object 7148 is adjusted and the measurement indicated by measurement indicator 7150 is updated. The dotted portion of virtual measurement object 7148 indicates a portion of virtual measurement object 7148 passing through a physical object (e.g., person 7056).
[0226] In Figure 7BE, contact 7142 has lifted off of touch screen 112. Virtual measurement object 7148 and measurement indicator 7148 remain displayed. In Figure 7BF, an input is detected at a location on touch screen display 112 indicated by contact 7152. In response to the input, an end of virtual measurement object 7148 (e.g., the end closest to the received input) moves to a position corresponding to the location of contact 7152.
[0227] 8A-8W show exemplary user interfaces for initiating a shared annotation session, according to some embodiments. The user interfaces in those figures are used to illustrate the processes described below, including the processes in FIGS. 9A-9F, 10A-10B, 11A-11F, and 12A-12D. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., an enantiomeric center of the respective contact or a point associated with the respective contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, similar operations are optionally performed on a device having display 450 and separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying the user interface shown in the figures on display 450, along with a focus selector.
[0228] 8A-8G illustrate the establishment of a shared annotation session between two devices.
[0229] 8A illustrates a physical environment 8000 in which a first user operates a first device 100-1 (e.g., device 100) and a second user operates a second device 100-2 (e.g., device 100). A collaborative user interface 8002 displayed by device 100-1 is shown in inset 8004, which corresponds to device 100-1. Inset 8006 illustrates a web browser user interface currently displayed by device 100-2. A prompt 8008 displayed in collaborative user interface 8002 includes instructions to initiate a shared annotation session.
[0230] In Figure 8B, input by contact 8012 is received at a location corresponding to a control 8010 displayed by device 100-1 to initiate a shared annotation session. In response to the input, a request is sent from device 100-1 to a remote device (e.g., device 100-2) to initiate a shared annotation session. After the request has been sent and no response indicating acceptance of the request has been received, a notification 8014 is displayed by device 100-1, as shown in Figure 8C.
[0231] 8C, in response to receiving a request to initiate a shared annotation session, device 100-2 displays prompt 8016 including instructions to accept the request for the shared annotation session. Input by contact 8020 is received at a location corresponding to control 8018 displayed by device 100-2 to accept the request for the shared annotation session. In response to the input, an acceptance of the request is transmitted from device 100-2 to a remote device (e.g., device 100-1).
[0232] In Figure 8D, an indication of acceptance of the request to initiate a shared annotation session has been received by device 100-1. Device 100-1 displays prompt 8022 including instructions to move device 100-1 toward device 100-2. Prompt 8022 includes a representation 8026 of device 100-1 and a representation 8028 of device 100-2. Device 100-2 displays prompt 8024 including instructions to move device 100-2 toward device 100-1. Prompt 8024 includes a representation 8030 of device 100-1 and a representation 8032 of device 100-2.
[0233] 8D-8E show animations displayed in prompts 8022 and 8024. In prompt 8022, representation 8026 of device 100-1 is animated to move toward representation 8028 of device 100-2. In prompt 8024, representation 8032 of device 100-2 is animated to move toward representation 8030 of device 100-1.
[0234] 8F, the connection criteria are met (e.g., the first device 100-1 and the second device 100-2 move towards each other and / or at least a portion of the physical space 8000 captured within the field of view of one or more cameras of the first device 100-1 corresponds to at least a portion of the physical space 8000 captured within the field of view of one or more cameras of the device 100-2). The notification 8034 displayed by the first device 100-1 and the notification 8036 displayed by the second device 100-2 each include an indication that a shared annotation session has begun. The first device 100-1 displays a representation of the field of view of one or more cameras of the first device 100-1 (e.g., overlaid by the notification 8034). A poster 8038a in the physical environment 8000 is visible within the field of view of one or more cameras of the first device 100-1, as indicated by a representation 8038b of the poster 8038a displayed by the first device 100-1. The second device 100-2 displays a representation of the field of view of one or more cameras of the second device 100-2 (e.g., overlaid by a notification 8036). Devices 100-1 and 100-2 display a shared field of view (e.g., at least a portion of the physical space 8000 captured within the field of view of one or more cameras of the first device 100-1 corresponds to at least a portion of the physical space 8000 captured within the field of view of one or more cameras of device 100-2). For example, poster 8038a in physical environment 8000 is visible within the field of view of one or more cameras of second device 100-2, as indicated by representation 8038c of poster 8038a displayed by second device 100-2. In FIG. 8G, the respective fields of view of the cameras are displayed by first device 100-1 and second device 100-2 without notifications 8034 and 8036 (e.g., notifications 8034 and 8036 are no longer displayed).
[0235] Figures 8H-8M show annotation input received during a shared annotation session, Figures 8H-8J show annotation input provided to second device 100-2, and Figures 8K-8M show annotation input provided to first device 100-1.
[0236] In FIG. 8H, input by contact 8044 (e.g., input received at a touchscreen display of second device 100-2) is detected by second device 100-2. While the input is being detected by second device 100-2, first device 100-1 displays avatar 8048 at a location in the shared field of view that corresponds to the location in the shared field of view at which the input is received at second device 100-2. As contact 8044 moves along the path indicated by arrow 8046 as shown in FIGS. 8H-I, annotations corresponding to the movement of contact 8044 are displayed by first device 100-1 (as annotation 8050-1) and second device 100-2 (as annotation 8050-2). In FIG. 8J, further annotation input is provided via further movement of contact 8044.
[0237] In FIG. 8K, input by contact 8052 (e.g., input received on a touchscreen display of first device 100-1) is detected by first device 100-1. While the input is being detected by first device 100-1, second device 100-2 displays avatar 8054 at a location in the shared field of view that corresponds to the location in the shared field of view where the input is received at first device 100-1. As contact 8052 moves, an annotation corresponding to the movement of contact 8052 is displayed by first device 100-1 (as annotation 8056-1) and second device 100-2 (as annotation 8056-2), as shown in FIG. 8K-8M.
[0238] In Figures 8M-8P, the movement of first device 100-1 increases the distance between first device 100-1 and second device 100-2. In Figure 8N, as first device 100-1 moves away from second device 100-2, the representation of the field of view of the camera of first device 100-1 displayed by first device 100-1 is adjusted (e.g., such that the portion of representation 8038b of physical poster 8038a displayed by first device 100-1 is decreased).
[0239] In some embodiments, one or more annotations (e.g., 8050-1, 8050-2, 8056-1, and / or 8056-2) have a fixed spatial relationship with respect to a portion of the physical environment 8000 (e.g., such that movement of a device camera with respect to the physical environment 8000 changes the display location of the annotations). In FIG. 8O, as the first device 100-1 continues to move away from the second device 100-2 such that annotation 8056-1 is no longer displayed by the first device 100-1, a visual indication 8058 corresponding to annotation 8056-1 is displayed by the first device 100-1 (e.g., to indicate the direction of movement of the first device 100-1 required to redisplay annotation 8056-1).
[0240] In FIG. 8P, movement of first device 100-1 away from second device 100-2 increases the distance between first device 100-1 and second device 100-2 to an extent (e.g., above a threshold distance) that causes the shared annotation session to be disconnected. Device 100-1 displays prompt 8060 including instructions to move device 100-1 toward device 100-2. Device 100-2 displays prompt 8062 including instructions to move device 100-2 toward device 100-1. In some embodiments, prompt 8060 includes an animated element (e.g., as described with respect to FIGS. 8D-8E).
[0241] From Figure 8P to Figure 8Q, the movement of the first device 100-1 reduces the distance between the first device 100-1 and the second device 100-2. In Figure 8Q, the distance between the first device 100-1 and the second device 100-2 has decreased sufficiently for the shared annotation session to be restored. When devices 100-1 and 100-2 finish displaying their respective prompts 8060 and 8062, the respective fields of view of the respective device cameras are redisplayed.
[0242] 8R-8W illustrate a gaming application using a shared session between first device 100-1 and second device 100-2 (eg, a shared session established as described above with respect to FIGS. 8A-8G).
[0243] 8R illustrates a physical environment 8068 in which a first user 8064a operates a first device 100-1 (e.g., device 100) and a second user 8066a operates a second device 100-2 (e.g., device 100). A game application user interface 8063-1 displayed by the first device 100-1 is shown in inset 8070, which corresponds to the first device 100-1. A game application user interface 8063-2 displayed by the second device 100-2 is shown in inset 8072, which corresponds to the second device 100-2. User 8064a faces user 8066a such that a representation 8066b of user 8066a is visible within the field of view of one or more cameras (e.g., rear-facing cameras) of device 100-1, as shown in user interface 8063-1. Representation 8064b of user 8064a is similarly visible within the field of view of one or more cameras (eg, rear-facing cameras) of device 100-2, as shown in user interface 8063-2.
[0244] Game application user interfaces 8063-1 and 8063-2 display basketball rings 8074 and 8076, respectively. By providing input within the respective game application user interfaces, the user activates a virtual basketball object in the displayed representation of the respective device camera's field of view to create a basket within the respective basketball ring. In some embodiments, the respective basketball rings are fixed to the spatial location of the respective device such that the respective users can move the devices to create challenges to opponents. Game application user interfaces 8063-1 and 8063-2 also display game data areas 8078 and 8080, respectively. Game data displayed in the game data areas may include, for example, a record of points scored by successful baskets, as well as the distance between devices 100-1 and 100-2 (e.g., for use as a criterion for assigning a score to a given shot).
[0245] In FIG. 8S, an input (e.g., a tap input) by contact 8082 is detected by device 100-2 for activating a virtual basketball object. In response to detecting the input by contact 8082, a virtual basketball object 8084 is added to the field of view of the camera of device 100-2 displayed within game application user interface 8063-2, as shown in FIG. 8T. FIG. 8T further illustrates an input (e.g., a tap input) by contact 8086 detected by device 100-2 for activating a virtual basketball object. In response to detecting the input by contact 8086, a virtual basketball object 8088 is added to the field of view of the camera of device 100-1 displayed within game application user interface 8063-1, as shown in FIG. 8V. In FIG. 8V, an input (e.g., a tap input) by contact 8090 is detected by device 100-2 for activating a virtual basketball object. In response to detecting input by contact 8090, a virtual basketball object 8092 is added to the field of view of the camera of device 100-2 displayed within game application user interface 8063-2, as shown in Figure 8W. From Figure 8V to Figure 8W, user 8064a has lowered device 100-1 such that the displayed position of ring 8076 and representation 8064b of user 8064 is changed within user interface 8063-2.
[0246] 9A-9F are flow diagrams illustrating a method 900 for repositioning an annotation, according to some embodiments. Method 900 is performed on an electronic device (e.g., device 300 of FIG. 3 or portable multifunction device 100 of FIG. 1A ) having a display generating element (e.g., a display, a projector, a heads-up display, etc.), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also serves as a display generating element, a mouse, a joystick, a wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hand), and one or more cameras (e.g., one or more rear cameras on a side of the device opposite the display and the touch-sensitive surface). In some embodiments, the display is a touchscreen display and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations of method 900 are optionally combined and / or the order of some operations is optionally changed.
[0247] The device, via a display generating element, displays (902) a first user interface region (e.g., user interface 5003) including a representation of the field of view of one or more cameras that is updated with changes in the field of view of the one or more cameras over time (e.g., the representation of the field of view is continuously updated (e.g., with a pre-set frame rate, such as 24, 48, or 60 fps) according to changes occurring in the physical environment around the cameras and according to movement of the cameras relative to the physical environment). For example, as shown in FIGS. 5A-5B, a view of the physical environment 5000 within the field of view of one or more cameras is updated according to changes in the position of the cameras of device 100 as device 100 is moved.
[0248] While displaying a first user interface region including a representation of the field of view of the one or more cameras, the device receives (904) a first request to add an annotation (e.g., text or a drawing generated and / or placed by movement of a contact (e.g., a contact with a finger or a stylus) on the touch-sensitive surface) to the displayed representation of the field of view of the one or more cameras via one or more input devices (e.g., the first request is an input by a contact detected within the first user interface region on the touch screen display (e.g., at a location corresponding to a control for starting an annotation, or at a location within the first user interface region (e.g., the location where an annotation is to be started)). For example, the request to add an annotation is an input by a stylus 5012 (e.g., an input to start an annotation as described with respect to FIG. 5C and / or an input by a contact at a location corresponding to a control (e.g., a control 5004 for toggling between still and video modes) as described with respect to FIG. 5H).
[0249] In response to a first request to add an annotation to the displayed representation of the field of view of the one or more cameras (e.g., including in response to detecting a finger contact or touchdown or movement of a stylus on the touch-sensitive surface at a location corresponding to a portion of the physical environment captured in the representation of the camera's field of view, or a user interface object (e.g., a button that activates an AR annotation mode)), the device replaces (906) the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to receiving the first request to add the annotation (e.g., pausing a live feed of the field of view of the one or more cameras (e.g., displaying a still image of the current field of view) while the field of view continues to change with movement of the device) and displays within the first user interface area a still image corresponding to the paused live feed of the field of view of the one or more cameras. For example, in response to input by stylus 5012 (e.g., as described with respect to FIG. 5C), the representation of the device camera's field of view (e.g., as described with respect to FIG. 5A-5B) is replaced by a display of a still image of one or more camera's field of view captured at a time corresponding to receipt of a first request to add an annotation (e.g., as described with respect to FIG. 5C-5D).
[0250] While displaying the still image in the first user interface region, the device receives (908) a first annotation (e.g., drawing input) via one or more input devices on a first portion of the still image, the first portion of the still image corresponding to a first portion of the physical environment captured in the still image. For example, annotation 5018 is received on a portion of the still image (e.g., a portion including representation 5002b of physical mug 5002a) that corresponds to a portion of the physical environment captured in the still image (e.g., a portion of physical environment 5000 including physical mug 5002a), as described with respect to FIGS. 5D-5G. In some embodiments, while displaying the still image and receiving annotation input on the still image, the device continues to track the position of the camera relative to the surrounding physical environment (e.g., based on changes in the camera's field of view and input from other sensors (e.g., motion sensors, gravity sensors, gyroseconds, etc.)). In some embodiments, the device determines whether a physical location or object corresponding to the location of the annotation in the still image has moved out of the camera's field of view, and if so, the device also determines the spatial relationship between the portion of the physical environment currently in the camera's field of view and the physical location or object that is the target of the annotation. For example, mug 5002 is the target of annotation 5018.
[0251] While displaying the first annotation on the first portion of the still image in the first user interface region (e.g., after receiving the first annotation on the first portion of the still image), the device receives a first request via one or more input devices to redisplay the representation of the field of view of the one or more cameras in the first user interface region (910). For example, the request to redisplay the representation of the field of view of the one or more cameras in the first user interface region is an input by a stylus 5012 on a control 5004 for toggling between still image mode and video mode, as described with respect to FIG.
[0252] In response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in the first user interface region (e.g., in response to detecting an absence of contact on the touch-sensitive surface for a threshold amount of time (e.g., a drawing session is deemed to have ended) or in response to detecting a tap on a user interface object (e.g., a button for deactivating an AR annotation mode)), the device replaces the display of the still image with a representation of the field of view of the one or more cameras in the first user interface region (910) (e.g., the representation of the field of view is continuously updated at a preset frame rate (e.g., 24, 48, or 60 fps, etc.) in accordance with changes occurring in the physical environment around the cameras and in accordance with movement of the cameras relative to the physical environment). For example, as described with respect to FIG. 5H, in response to input with stylus 5012 in control 5004, the still image displayed in FIGS. 5C-5H is replaced by a representation of the field of view of the one or more cameras in first user interface region 5003, as described with respect to FIGS. 5I-5N. Pursuant to a determination that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object about which the first annotation was received) is currently outside the field of view of one or more cameras (e.g., as a result of movement of the device that occurred after the live feed of the camera views was stopped), the device displays (e.g., as part of the computing system) an indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image, concurrently with the representations of the field of view of the one or more cameras (e.g., displaying a visual indication such as a dot or other shape on the edge of the displayed camera's field of view and at the location of the edge closest to the annotated first portion of the physical environment, or displaying a simplified map of the physical environment concurrently with the representations of the cameras' field of view and marking the relative position of the first portion of the physical environment and the device on the map).For example, as described with respect to FIG. 5L, a portion of the physical environment captured in the still image (e.g., including mug 5002 for which annotation 5018 was received) is currently outside the field of view of one or more cameras (e.g., in FIGS. 5J-5L, physical mug 5002a is outside the field of view of the cameras displayed in user interface 5003), and an indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image (e.g., indicator dots 5022) is displayed concurrently with a representation of the field of view of the one or more cameras (e.g., as shown in FIG. 5L). Following a determination that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object for which the first annotation was received) is currently within the field of view of the one or more cameras, the device refrains from displaying the indication. For example, in response to an input to redisplay the camera's field of view (e.g., an input by stylus 5012 on control 5004 for toggling between still image and video mode, as described with respect to FIG. 5T), in response to a determination that a portion of the physical environment captured in the still image (e.g., including mug 5002 for received annotations 5018 and 5028) is currently within the field of view of one or more cameras (e.g., in FIG. 5T, physical mug 5002a is visible (as visual representation 5002b) within the field of view of the cameras displayed in user interface 5003), an indication of the current spatial relationship of the one or more cameras to a first portion of the physical environment captured in the still image is not displayed (e.g., as shown in FIG. 5T). Displaying an indication of the current spatial relationship of one or more cameras to a portion of the physical environment captured in the still image pursuant to a determination that the portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras provides improved visual feedback to a user (e.g., indicating that movement of the camera is required to view the physical environment captured in the still image).Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling the user to quickly and accurately locate a portion of the physical environment that corresponds to the annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0253] In some embodiments, displaying (912) an indication of the current spatial relationship of the one or more cameras with respect to a first portion of the physical environment captured in the still image includes displaying the indicator proximate an edge of the representation of the field of view of the one or more cameras and moving the indicator along the edge in accordance with movement of the one or more cameras with respect to the physical environment. For example, as described with respect to FIG. 5L, indicator 5022 is displayed proximate a left edge of the field of view of the camera displayed in user interface 5003, and as described with respect to FIGS. 5L-5M, the indicator is moved along the edge in accordance with movement of the camera of device 100. In some embodiments, in the case of a rectangular representation of the field of view of the one or more cameras, the indicator is a visual indication such as a dot or other shape that moves along an edge of the rectangular representation of the field of view, where the visual indication can move along one straight edge in accordance with a first movement of the one or more cameras, and the visual indication may also hop from one straight edge to another straight edge in accordance with a second movement of the one or more cameras. Moving the indicator along the edge of the camera view provides visual feedback to the user (e.g., indicating the direction of camera movement required to view the portion of the physical environment captured in the still image). Providing improved visual feedback to the user enhances usability of the device (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), makes the user-device interface more efficient, and additionally reduces device power usage and improves battery life by allowing the user to use the device more quickly and efficiently.
[0254] In some embodiments, while displaying an indication of the current spatial relationship of the one or more cameras to a first portion of the physical environment captured in the still image, the device detects (914) a first movement of the one or more cameras (e.g., as part of a computing system). In response to detecting the first movement of the one or more cameras, the device updates the representation of the field of view of the one or more cameras according to a change in the field of view of the one or more cameras caused by the first movement. In response to determining that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including an object for which a first annotation was received) is currently outside the field of view of the one or more cameras, the device (e.g., as part of a computing system) updates the indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image according to the first movement of the one or more cameras (e.g., moves a visual indication, such as a dot or other shape along an edge of the representation of the field of view according to the first movement of the cameras). Pursuant to a determination that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object for which the first annotation was received) is now within the field of view of one or more cameras, the device ceases displaying the indication. For example, as the camera of device 100 moves, the indication 5022 is updated (e.g., moved upward in user interface 5003) while the portion of the physical environment captured in the still image is outside the field of view of the camera (e.g., mug 5002a is outside the field of view of the camera, as described with respect to FIGS. 5L-5M), and the indication 5022 is no longer displayed when the portion of the physical environment captured in the still image is within the field of view of the camera (e.g., as described with respect to FIGS. 5L-5N). In some embodiments, a scaled-down representation of the still image with the first annotation (e.g., scaled-down representation 5020) is displayed adjacent to a location on an edge of the representation of the field of view where the visual indicator was last displayed, and the still image view scales down and moves toward the location of the first annotation shown in the representation of the field of view of the camera).In some embodiments, the indication is re-displayed when the first portion of the physical environment moves out of the camera's field of view with further movement of the camera relative to the physical environment (e.g., as described with respect to FIGS. 5U-5V). Terminating the display of the indication of the one or more cameras' current spatial relationship to the portion of the physical environment pursuant to determining that the first portion of the physical environment captured in the still image is now within the field of view of the one or more cameras provides visual feedback to the user (e.g., indicating that no further movement is required to view the portion of the physical environment captured in the still image). Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotation portion of the still image), provides a more efficient user-device interface, and additionally reduces device power usage and improves battery life by allowing the user to use the device more quickly and efficiently.
[0255] In some embodiments, in response to receiving the first request to redisplay the representation of the field of view of the one or more cameras in the first user interface region (916), the device displays the first annotation on the first portion of the physical environment captured in the representation of the field of view of the one or more cameras in accordance with a determination that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object for which the first annotation was received) is currently within the field of view of the one or more cameras. For example, as described with respect to FIG. 5N, the device displays the annotation 5018 on the portion of the physical environment captured in the representation of the field of view of the one or more cameras (e.g., annotation 5018 is displayed at a position corresponding to visual representation 5002b of physical mug 5002a in accordance with a determination that the first portion of the physical environment captured in the still image (e.g., a portion of physical environment 5000 including physical mug 5002a) is currently within the field of view of the one or more cameras). Displaying the annotation of the portion of the physical environment captured in the annotated still image provides visual feedback to the user (e.g., indicating that the portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras). Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling the user to quickly and accurately locate a portion of the physical environment that corresponds to the annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0256] In some embodiments, the first annotation is displayed (918) as a two-dimensional object (e.g., annotation 5018) on a first depth plane within a first portion of the physical environment captured within a representation of the field of view of the one or more cameras. In some embodiments, the first depth plane is detected according to detection of a physical object (e.g., physical mug 5002a) or object feature at the first depth plane in the first portion of the physical environment. Displaying the annotation on a depth plane within the portion of the physical environment within the field of view of the one or more cameras provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship to the physical environment). Providing improved visual feedback to the user enhances device usability (e.g., by allowing the user to imbue physical world objects with additional information contained in the annotation), makes the user device interface more efficient, and further reduces power usage and improves device battery life by allowing the user to use the device more quickly and efficiently.
[0257] In some embodiments, the first annotation (e.g., annotation 5018) is displayed (920) at a location in space within a first portion of the physical environment captured within the representation of the field of view of one or more cameras. In some embodiments, the first annotation floats in space separated from any physical objects detected in the first portion of the physical environment. Displaying the annotation at a location in space within a portion of the physical environment captured in the still image provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship to the physical environment). Providing improved visual feedback to the user enhances device usability (e.g., by allowing the user to imbue physical world objects with additional information contained in the annotation), makes the user device interface more efficient, and further reduces power usage and improves device battery life by allowing the user to use the device more quickly and efficiently.
[0258] In some embodiments, a first annotation (e.g., annotation 5018) is displayed (922) at a location on a physical object (e.g., physical mug 5002a) detected in a first portion of the physical environment captured in a representation of the field of view of one or more cameras. In some embodiments, the first annotation is attached to a physical object (or a feature of a physical object) detected in the first portion of the physical environment. Displaying the annotation of a portion of the physical environment at a location on a physical object detected in the portion of the physical environment captured in the still image provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship to the physical environment). Providing improved visual feedback to the user enhances device usability (e.g., by allowing the user to imbue physical world objects with additional information contained in the annotation), makes the user device interface more efficient, and further reduces power usage and improves device battery life by allowing the user to use the device more quickly and efficiently.
[0259] In some embodiments, in response to receiving a first request to re-display the representation of the field of view of the one or more cameras in the first user interface area (924), in accordance with a determination that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object for which the first annotation was received) is not currently within the field of view of the one or more cameras, the device displays a visual representation of the first annotation (e.g., drawing input) drawn on the first portion of the still image simultaneously with the representation of the field of view of the one or more cameras in the first user interface area (e.g., a reduced-size representation of the still image with the first annotation is displayed adjacent to a location on an edge of the representation of the field of view that is closest to the first portion of the physical space currently represented within the field of view of the camera). For example, in response to a request to redisplay a representation of the field of view of one or more cameras in the first user interface area (e.g., input by stylus 5012 on control 5004 for toggling between still image mode and video mode, as described with respect to FIG. 5H), and following a determination that a first portion of the physical environment captured in the still image is currently within the field of view of one or more cameras (e.g., a portion of the physical environment captured in the still image (e.g., including mug 5002 for which annotation 5018 was received) is currently within the field of view of one or more cameras), the device displays annotation 5018 drawn on the first portion of the still image simultaneously with the representation of the field of view of the one or more cameras in the first user interface area (e.g., as described with respect to FIG. 5N). In some embodiments, in response to receiving a first request to re-display a representation of the field of view of the one or more cameras in the first user interface area, in accordance with a determination that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object about which the first annotation was received) is currently outside the field of view of the one or more cameras, the computing system refrains from displaying a visual representation of the first annotation drawn on the first portion of the still image (e.g., as shown in Figures 5L and 5M, annotation 5018 is not displayed).Displaying a visual representation of the annotation of the still image simultaneously with the representation of the field of view of the camera pursuant to determining that the first portion of the physical environment is not within the field of view of the camera provides improved visual feedback to the user (e.g., an indication of the current spatial relationship (e.g., a dot) corresponds to the received annotation input). Providing improved visual feedback to the user enhances usability of the device (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), provides a more efficient user-device interface, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0260] In some embodiments, the first annotation shown in the representation of the field of view has a first viewpoint based on the current spatial relationship of one or more cameras to a first portion of the physical environment captured in the still image (926), which is different from a second viewpoint of the first annotation displayed on the still image (e.g., a viewpoint of the first annotation shown in a scaled-down representation of the still image displayed adjacent to the representation of the field of view). In some embodiments, an animated transition is displayed showing a scaled-down representation of the still image being transformed into a representation of the current field of view. Displaying the annotation at a different viewpoint from the viewpoint shown in the still image provides improved visual feedback to the user (e.g., indicating that the annotation is anchored to a portion of the physical environment captured in the still image). Figures 5AE and 5AF provide an example of an annotation 5018 shown in the representation of the field of view at a different viewpoint. Providing improved visual feedback to the user enhances device operability (e.g., by enabling the user to quickly and accurately locate a portion of the physical environment that corresponds to the annotation portion of the still image), makes the user-device interface more efficient, and also reduces device power usage and improves battery life by allowing the user to use the device more quickly and efficiently.
[0261] In some embodiments, in response to receiving a first request to re-display the representation of the field of view of the one or more cameras in the first user interface region (928), pursuant to a determination that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including the object for which the first annotation was received) is currently outside the field of view of the one or more cameras, a visual representation of a first annotation drawn on the first portion of the still image (e.g., drawing input) is displayed concurrently with the representation of the field of view of the one or more cameras in the first user interface region (e.g., a scaled-down representation of the still image with the first annotation is displayed adjacent to a location on an edge of the representation of the field of view that is closest to the first portion of the physical space currently displayed in the field of view of the camera), and the visual representation of the first annotation drawn on the first portion of the still image (e.g., scaled-down representation 5020) is translated into an indication (e.g., indication 5022) of the current spatial relationship of the one or more cameras (e.g., as part of the computing system) to the first portion of the physical environment captured in the still image (e.g., as described with respect to FIGS. 5I-5L ). For example, the indication may be a visual indication such as a dot or other shape displayed on an edge of the camera's displayed field of view at a location on that edge closest to the first portion of the physical environment where the annotation was made, and before the visual indication is displayed, a miniature view of the still image with the first annotation is displayed at that location and converted into the visual indication. Converting the visual representation of the annotation (e.g., a scaled-down representation of the still image) into an indication (e.g., a dot) of the camera's current spatial relationship to the portion of the physical environment captured in the still image provides improved visual feedback to the user (e.g., indicating that the indication (e.g., dot) and the visual representation of the annotation (e.g., a reduced-scale image) are different representations of the same annotation).Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling the user to quickly and accurately locate a portion of the physical environment that corresponds to the annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0262] In some embodiments, while displaying a first user interface region including a representation of the field of view of the one or more cameras, and prior to detecting a first request to add an annotation to the displayed view of the displayed view, the device displays (930) an indication (e.g., indication 5022) of the current spatial relationship (e.g., as part of a computing system) of the one or more cameras relative to a previously received second portion of the physical environment to which the second annotation was previously added (e.g., displaying a visual indication such as a dot or other shape on an edge of the displayed camera's field of view and at a location on the edge closest to the second portion of the physical environment where the second annotation was made, or displaying a simplified map of the physical environment simultaneously with the representation of the camera's field of view and marking the relative location of the physical environment and the second portion of the device on the map). In some embodiments, the second annotation is added to the second portion of the physical environment shown in the representation of the field of view of the one or more cameras in the same manner as the first annotation is added to the first portion of the physical environment shown in the representation of the camera's field of view. Displaying an indication of the camera's current special relationship to the second portion of the physical environment to which the previously received annotation was added provides improved visual feedback to the user (e.g., indicating that camera movement is necessary to view the previously received annotation for the portion of the physical environment captured in the still image). For example, indication 5022 is displayed prior to detecting a request to add annotation 5028 as described with respect to FIGS. 5P-5R. Providing improved visual feedback to the user enhances device usability (e.g., by enabling the user to quickly and accurately locate the portion of the physical environment that corresponds to the previously annotated portion of the still image), provides a more efficient user-device interface, and reduces device power usage and improves battery life by allowing the user to use the device more quickly and efficiently.
[0263] In some embodiments, after receiving a first request to redisplay a representation of the field of view of one or more cameras in the first user interface region, in accordance with a determination that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras, the device simultaneously displays (932) an indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment and an indication of the current spatial relationship of the one or more cameras to the second portion of the physical environment. For example, if multiple annotations (e.g., annotations 5018 and 5036) have been added to different portions of the physical environment, indicators corresponding to the different annotations (e.g., indicators 5018 and 5040, FIG. 5AA) are simultaneously displayed around the edge of the representation of the field of view of the camera at their respective positions closest to the corresponding portions of the physical environment. Simultaneously displaying an indication of the current spatial relationship of the camera to the first portion of the physical environment and an indication of the current spatial relationship of the camera to the second portion of the physical environment provides improved visual feedback to the user (e.g., indicating the direction of camera movement required to view one or more of the multiple received annotations). Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling a user to quickly and accurately locate a portion of the physical environment that corresponds to a previously annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0264] In some embodiments, while displaying the indication of the current spatial relationship of the one or more cameras relative to a first portion of the physical environment and the indication of the current spatial relationship of the one or more cameras relative to a second portion of the physical environment, the device detects (934) a second movement of the one or more cameras relative to the physical environment, and in response to detecting the second movement of the one or more cameras relative to the physical environment, in accordance with a determination that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras, the device updates the indication of the current spatial relationship of the one or more cameras relative to the first and second portions of the physical environment, respectively, in accordance with the second movement of the one or more cameras relative to the physical environment (e.g., moving a virtual indicator in a different direction and / or at a different speed along an edge of the representation of the field of view). For example, respective visual indicators corresponding to different annotations directed to different portions of the physical environment are displayed at different locations on the edge of the representation of the camera's field of view, and movement of the device causes the respective visual indicators to move in different directions and at different speeds according to changes in the current spatial relationship of their corresponding annotations to the device (e.g., indicators 5018 and 5040 move according to movement of device 100 as described with respect to FIGS. 5AA-5AD). The visual indicators may move together or separately and / or at different speeds depending on the actual spatial relationship between the different portions of the physical environment marked with the different annotations. Updating the indication of the camera's current spatial relationship to the first and second portions of the physical environment according to camera movement provides improved visual feedback to the user (e.g., camera movement moves the camera closer to or further away from portions of the physical environment corresponding to the annotated portions of the still image).Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling a user to quickly and accurately locate a portion of the physical environment that corresponds to a previously annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0265] In some embodiments, pursuant to a determination that the first and second portions of the physical environment are within a predetermined range of the one or more cameras, an indication of a current spatial relationship of the one or more cameras to the first portion of the physical environment and an indication of a current spatial relationship of the one or more cameras to the second portion of the physical environment are displayed (936). In some embodiments, the user interface provides a method for selecting a subset of annotations from all annotations added to various portions of the physical environment, and only an indication corresponding to the selected subset of annotations is displayed along with the representation of the camera's field of view. Displaying an indication for the first portion of the physical environment and for the second portion of the physical environment pursuant to a determination that the first and second portions of the physical environment are within a predetermined range of the one or more cameras provides improved visual feedback to the user (e.g., by reducing cluttering of the user interface with indicators when the first and second portions are outside of the predetermined range). Providing improved visual feedback to the user enhances usability of the device (e.g., by enabling a user to quickly and accurately locate a portion of the physical environment that corresponds to a previously annotated portion of a still image), makes the user-device interface more efficient, and additionally reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.
[0266] It should be understood that the particular order described for the operations in Figures 9A-9F is merely an example, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. Furthermore, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 1000, 1100, and 1200) may also be applied in a manner similar to method 9000 described above with respect to Figures 9A-9F. For example, the contact, input, annotation, physics object, user interface area, field of view, movement, and / or animation described above with reference to method 9000, optionally have one or more of the characteristics of the contact, input, annotation, physics object, user interface area, field of view, movement, and / or animation described herein with reference to other methods described herein (e.g., methods 1000, 1100, and 1200). For the sake of brevity, those details will not be repeated here.
[0267] 10A-10B are flow diagrams illustrating a method 1000 of receiving an annotation on a portion of a physical environment captured in a still image corresponding to a paused position of a video, according to some embodiments. Method 1000 is performed on an electronic device (e.g., device 300 of FIG. 3 or portable multifunction device 100 of FIG. 1A) having a display generating element (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touchscreen display that also functions as a display generating element). In some embodiments, the display is a touchscreen display and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations of method 1000 are optionally combined and / or the order of some operations is optionally changed.
[0268] The device displays, via a display generation element, a user interface including a video playback area (1002). For example, the device 100 displays, via the touch screen display 112, a user interface 6000 including a video playback area 6002, as described with respect to FIG. 6A.
[0269] While displaying a playback of a first portion of the video in the video playback area, the device receives (1004) a request to add an annotation to the video playback via one or more input devices (e.g., the request is input by a contact detected within a video playback user interface on a touch screen display (e.g., a location corresponding to a control to begin the annotation or a location within the video playback area (e.g., the location where the annotation is to begin)). For example, the request to add an annotation to the video playback is input by contact 6024 detected at a location corresponding to markup control 6010.
[0270] In response to receiving the request to add the annotation, the device pauses (1006) playing the video at a first position in the video (e.g., identifies a current frame of the video (e.g., a pause position) and stops playing the video at the current frame).
[0271] The device displays (1008) a still image (e.g., a frame of the video) corresponding to the first pause position of the video (e.g., displaying the current frame of the video shown at the time the request was received). For example, as described with respect to Figures 6C-6D, in response to input by contact 6024 at a location corresponding to markup control 6010 of the video, playback of the video is paused and a still image corresponding to the pause position of the video is displayed.
[0272] While displaying the still image (e.g., in a video playback area), the device receives (1008) annotations (e.g., touch drawing input) via one or more input devices at a first portion of the physical environment captured in the still image. For example, as described with respect to FIGS. 6E-6F, annotations 6030 are received at a portion of the physical environment corresponding to a kite object 6028 in the displayed still image. When a "physical environment" is referred to herein, it will be understood that a non-physical environment (e.g., a computer-generated environment) may be included in the still image. For example, annotations are received at a portion of the image (e.g., a portion of a computer-generated video) that corresponds to a pause position of the video. In some embodiments, the computer-generated image includes depth data and / or objects for which annotations and / or virtual objects are suitable.
[0273] After receiving the annotation, the device displays (1010) in the video playback area (e.g., during continued playback of the video or while receiving input (e.g., on a timeline) such as scrubbing the video back and forth) a second portion of the video that corresponds to a second location in the video that is different from the first location in the video (e.g., before or after a paused location in the video), where the first portion of the physical environment is captured in the second portion of the video, and the annotation is displayed in the second portion of the video. For example, after the annotation 6030 is received as described with respect to FIGS. 6E-6F, the annotation 6030 is displayed in the second portion of the video (e.g., as described with respect to FIG. 6J). In some embodiments, while the second portion of the video is displayed, the annotation is displayed in a second location in the video playback area that is different from the first location of the annotation received while the still image was displayed (e.g., the annotation is "stitched" to a location (e.g., a physical object) in the physical environment that was captured in the video clip, such that the location (e.g., the physical object) moves at the same time as the annotation moves as the video progresses). Displaying annotations on a portion of the video that is different from the portion of the video to which the annotations were applied without requiring further input (e.g., to identify the surface to which the annotations should be applied) enhances usability of the device. Performing operations without requiring further user input enhances usability of the device and makes the user device interface more efficient (e.g., by allowing a user to add information to a previously captured video via direct annotation of the video without having to re-record the video or annotate multiple portions of the video), as well as reducing power usage and improving battery life of the device by allowing a user to use the device more quickly and efficiently.
[0274] In some embodiments, a video is captured (1012) by the camera during relative movement of the camera and the physical environment (e.g., during the video capture, camera movement data and depth data of the physical environment are simultaneously captured and stored with the simultaneously captured image data), and a third portion of the video is captured between the first and second portions of the video, not including the first portion of the physical environment during the relative movement of the camera and the physical environment. In some embodiments, an annotation (e.g., annotation 6030) received on a still image (e.g., as shown in Figures 6E-6F) is directed to a first object (e.g., kite 6028) located in the first portion of the physical environment and is displayed at a position corresponding to the first object in the second portion of the video (e.g., the annotation is not made directly on the still image of any frame of the second portion of the video). For example, in Figure 6J, annotation 6030 is displayed in the second portion of the video at a position corresponding to kite 6028. In some embodiments, the annotation is not displayed in the third portion of the video that does not include the first object (e.g., the annotation is not persistently displayed and is only shown when the current frame includes the first object). In some embodiments, the annotation is rotated and scaled to appear at the position of the first object according to the distance and field of view of the first object. For example, in Figures 6J-6N, the annotation 6030 is rotated and scaled to appear at the position of the kite 5026 in the video. In some embodiments, the first, third, and second portions of the video are consecutively captured portions of the video, or the second, third, and first portions of the video are consecutively captured portions of the video. Because there is a discontinuity in the captured subject matter in the field of view of the camera (e.g., while capturing the third portion of the video), the first portion of the physical environment captured in the first portion of the video cannot be recognized as the same first portion of the physical environment captured in the third portion of the video based solely on the image data of the video (e.g., identifying tracking points across consecutive frames via frame-to-frame comparison).The camera movement and / or depth data is used (optionally along with the image data) to create a three-dimensional or quasi-three-dimensional model of the physical environment captured in the video, so that a particular location in the physical environment can be recognized in each frame of the video, regardless of its appearance in the frame or the viewpoint of the display. Displaying the annotations at locations corresponding to the object to which they are pointed, and not displaying the annotations in portions of the video that do not include the object, provides improved visual feedback to the user (e.g., by providing an indication that the annotation is anchored at a location corresponding to the object). Performing operations without requiring further user input improves device usability, makes the user device interface more efficient (e.g., by allowing a user to add information to a video without re-recording the video or annotating multiple portions of the video), and additionally reduces device power usage and improves battery life by allowing a user to use the device more quickly and efficiently.
[0275] In some embodiments, the device displays (1014) a timeline of the video (e.g., a scrub bar having a position indicator indicating the position of the currently displayed frame, a scrollable sequence of reduced-scale images of sample frames from successive segments of the video) (e.g., concurrently with the display of the video (e.g., while the video is playing and / or while the video is paused)), and displaying a second portion of the video is performed in response to a user input that scrubs through the video's timeline to a second position in the video (e.g., a user input that drags the position indicator along the scrub bar, or a user input that scrolls the sequence of reduced-scale images of the sample frames past a still marker of the currently displayed frame). For example, input by contact 6038 is received at a position corresponding to a timeline 6004 that includes a series of sample frames (e.g., sample frame 6006 as described with respect to FIG. 6A), and the video displayed in the video playback area 6002 is played in response to the input (e.g., as described with respect to FIGS. 6L-6N). Displaying a timeline for scrubbing through the video without requiring further user input enhances the usability of the device (e.g., by allowing a user to access annotations displayed in a second portion of the video using existing scrubbing controls without requiring a separate control or input). Reducing the number of user inputs required to perform an action enhances the usability of the device and makes the user device interface more efficient. This further reduces power usage and improves the battery life of the device by allowing a user to use the device more quickly and efficiently.
[0276] In some embodiments, displaying the second portion of the video is performed as a result of rewinding (1016) the video from a first position to a second position (e.g., the second position precedes the first position on the video's timeline). For example, as described with respect to FIGS. 6L-6N, an input by contact 6038 is received to rewind the video displayed within the video playback area 6002. Displaying the annotations in a portion of the video different from the portion of the video to which the annotations were applied in response to the rewind input without requiring further user input enhances usability of the device (e.g., by allowing a user to access the annotations displayed in the second portion of the video using existing scrub controls without requiring a separate control or input). Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing a user to use the device more quickly and efficiently.
[0277] In some embodiments, displaying the second portion of the video is performed as a result of fast-forwarding (1018) the video from a first position to a second position (e.g., the second portion precedes the first portion on the video's timeline and is played at a faster than normal playback speed). Displaying the annotations in a portion of the video different from the portion of the video to which the annotations were applied in response to the fast-forward input without requiring further user input enhances usability of the device (e.g., by allowing a user to access the annotations displayed in the second portion of the video using existing scrub controls without requiring a separate control or input). Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing a user to use the device more quickly and efficiently.
[0278] In some embodiments, displaying the second portion of the video is performed as a result of normal playback (1020) of the video from the first position to the second position (e.g., the second portion precedes the first portion on the video's timeline and is played at a faster rate than normal playback). In some embodiments, when the user is finished providing annotations on the still image, the user exits the annotation mode by selecting a "Done" button displayed with the still image (e.g., as described with respect to FIG. 6G). As a result, the device continues playing the video from the first position, and annotations are displayed in each subsequent frame that includes the first portion of the physical environment at the same physical location (e.g., physical object), even if the first portion of the physical environment is captured at a different distance and / or a different perspective compared to the still image. Displaying annotations in a portion of the video that is different from the portion of the video to which the annotations were applied in response to normal playback of the video without requiring further user input enhances usability of the device (e.g., by allowing the user to access annotations displayed in the second portion of the video using existing scrub controls without requiring a separate control or input). Performing actions without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.
[0279] In some embodiments, the device displays (1022) a list of media content objects including the video via a display generating element (e.g., displaying a representation of the video in a media library). The device receives an input to select a video from the list of media content objects, and in response to receiving the input to select the video, displays a user interface object (e.g., a "markup" button for adding annotations) having a representation of the video in a video playback area, the user interface object configured to receive a request to add annotations to the video while the video is being played (e.g., a tap input to activate the button to add annotations). Displaying the user interface object configured to receive a request to add annotations to the video while the video is being played provides improved feedback (e.g., showing an option to add annotations to a portion of the video while the video is being played). In some embodiments, the markup button provides improved feedback (e.g., along with other playback controls) when touch input is detected over the video playback area during video playback, and when the markup button is activated, the currently displayed frame is shown in a markup enabled state and is ready to receive drawing annotations directly onto the image of the currently displayed frame. Providing improved feedback enhances usability of the device (e.g., by allowing a user to access annotations displayed in the second portion of the video using existing scrub controls without requiring a separate control or input). Performing actions without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing a user to use the device more quickly and efficiently.
[0280] It should be understood that the particular order described for the operations in Figures 10A-10B is merely an example, and the order described is not intended to indicate that the operations are the only order in which they may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. Furthermore, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 900, 1100, and 1200) may also be applied in a manner similar to method 1000 described above with respect to Figures 10A-10B. For example, the contact, input, annotation, physical object, user interface area, field of view, movement, and / or animation described above with reference to method 1000, optionally have one or more of the characteristics of the contact, input, annotation, physical object, user interface area, field of view, movement, and / or animation described herein with reference to other methods described herein (e.g., methods 900, 1100, and 1200). For the sake of brevity, those details will not be repeated here.
[0281] 11A-11F are flow diagrams illustrating a method 1100 of adding a virtual object to a previously captured media object. Method 1100 is performed on an electronic device (e.g., device 300 of FIG. 3 or portable multifunction device 100 of FIG. 1A) having a display generating element (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generating element, a mouse, a joystick, a wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hand). In some embodiments, the display is a touchscreen display and the touch-sensitive surface is on or integrated into the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations of method 1100 are optionally combined and / or the order of some operations is optionally changed.
[0282] The device displays (1102), via a display generating element, a first previously captured media object including one or more first images (e.g., still photos, live photos, or a video including a sequence of image frames), where the first previously captured media object was recorded and stored along with first depth data corresponding to the first physical environment captured in each of the one or more first images (e.g., the first depth data generated by one or more depth sensors of the device (e.g., an emitter / detector system such as infrared, sonar, and / or lidar, and / or an image analysis system (e.g., video segment analysis and / or stereo image / video analysis)) at a time corresponding to the time the first media object was captured by one or more cameras. For example, as described with respect to FIG. 7A , the device 100 displays the previously captured images within the media object display area 7002 via the touchscreen display 112.
[0283] While displaying the first previously captured media object (e.g., while displaying a still image, while displaying a representative image of a live photo, while displaying a frame of a video while the video is playing, or while displaying a frame of a video when the video is in a paused or stopped state), the device receives a first user request via one or more input devices to add a first virtual object (e.g., a falling ball, confetti, text, spotlight, emoji, paint, measurement graphic) to the first previously captured media object (1104). For example, the request to add a virtual object to the previously captured media object is a tap input received on the touch screen display 112 to add the virtual object to the previously captured image, as described with respect to FIG. 7C.
[0284] In response to adding the first virtual object to the first previously captured media object in response to the first user request, the device displays (1106) the first virtual object over at least a portion of the respective image in the first previously captured media object, the first virtual object being displayed with at least a first position or orientation (or movement path) determined using the first depth data corresponding to the respective image in the first previously captured media object. For example, as described with respect to FIG. 7C, in response to a tap input received on the touch screen display 112, a virtual object (e.g., a virtual ball object 7030) is added to the previously captured image. Adding a virtual object to a previously captured media object using depth data from a previously captured media object without requiring user input (e.g., indicating a planar location of the media object) enhances the usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the battery life of the device by allowing a user to use the device more quickly and efficiently.
[0285] In some embodiments, displaying (1108) the first virtual object over at least a portion of the respective image in the first previously captured media object includes displaying a first movement of the first virtual object relative to a first physical surface captured in the first previously captured media object after the first virtual object is positioned over a corresponding one of the one or more first images, the first movement of the first virtual object being constrained by a first pseudo surface corresponding to the first physical surface determined based on the depth data and a pseudo gravity direction (e.g., the pseudo gravity direction is optionally determined based on a direction of gravity recorded in the depth data at the time the first image was captured or is the same as an actual gravity direction for a current orientation of the device displaying the respective image). For example, as described with respect to FIG. 7C, the virtual ball object 7030 is constrained by a pseudo gravity direction by a pseudo surface corresponding to a physical surface determined based on the depth data (e.g., floor surface 7040). In some embodiments, different types of virtual objects have different simulated physical properties (e.g., shape, size, weight, elasticity, etc.) that interact with the simulated surface in different ways. In one example, if a still image captures a couch with curved armrests and a flat seating area, a virtual rubber ball is shown falling from the top of the image, landing on the curved surface of the armrests, bouncing off the curved surface of the armrests, landing on the flat seating area, and then rolling onto the floor. In contrast, confetti flakes are shown fluttering down from the top of the image, landing on the curved surface of the armrests, sliding off the curved surface of the armrests onto the flat seating area, and remaining on the flat seating area. In another example, a 3D letter "A" is placed on the curved surface of the arm by a user's finger, and the 3D letter "A" falls sideways and lands on the flat surface of the seating area when the user's finger is lifted off the touch screen.In some embodiments, a surface mesh corresponding to the physical environment captured in the still image is generated based on the depth data, and the virtual object inserted into the still image is animated during insertion and / or after initial placement to show movement and final position / orientation that conforms to basic physics, such as laws related to gravity, forces, and physical interactions between objects. By moving the virtual object onto the physical plane of a previously captured media object (e.g., according to simulated gravity) to show the planar location of the media object, the movement of the virtual object can be performed automatically without requiring further input (e.g., without the need to provide the user with input to indicate the path of movement of the virtual object), enhancing the usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.
[0286] In some embodiments, displaying 1110 the first virtual object over at least a portion of the respective image in the first previously captured media object displays a change in shape of the first virtual object according to a first physical surface captured in the first previously captured media object after the first virtual object is placed over a corresponding one of the one or more first images, the change in shape of the first virtual object being constrained by a first pseudo-surface corresponding to the first physical surface determined based on the first depth data. For example, as described with respect to FIGS. 7U-7X, the shape of the virtual decal object 7084 changes as the object moves over the surface of the couch 7060 and onto the floor 7004 depicted in the previously captured image. In some embodiments, if a still image captures a couch with curved armrests and a flat seating area, a virtual paintball is shown shooting in the image, with the virtual paintball spreading over the curved surface of the armrest. In contrast, if a virtual paintball is shown squirting into an image and lands on a flat surface of the seating area, the virtual paint spreads over the flat surface of the seating area. In another example, a long virtual streamer falls on the armrest, over the curved surface of the armrest, while a long virtual streamer that fell on a flat seating area lies flat on the flat surface of the seating area. By displaying the change in shape of the virtual object according to a surface in a previously captured media object, the change in shape of the virtual object can be made automatically without requiring further input (e.g., without requiring the user to provide input that directs the change in shape of the virtual object), enhancing the usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.
[0287] In some embodiments, while displaying the first virtual object over at least a portion of a respective image in the first previously captured media object, the device detects (1112) a second user request to switch from displaying the first previously captured media object to displaying a second previously captured media object (e.g., a horizontal swipe input on the first virtual object to indicate a previous or next item in the horizontally arranged list of media objects, a vertical swipe on the first virtual object to indicate a previous or next item in the vertically arranged list of media objects, or a vertical swipe on the first virtual object to switch to the next or previous media object). a tap on a front or rear button for displaying a second previously captured media object, the second previously captured media object including one or more second images, the second previously captured media object recorded and stored along with second depth data corresponding to the second physical environment captured in each of the one or more second images (e.g., second depth data generated by one or more depth sensors of the device (e.g., emitter / detector systems such as infrared, ultrasonic, and / or lidar, and / or image analysis systems (e.g., video segment analysis and / or stereo image / video analysis)) at a time corresponding to the time the second media object was captured by the one or more cameras). For example, as described with respect to FIGS. 7E-7F, while a virtual ball object is displayed over the first previously captured image, as shown in FIG. 7E, a request (e.g., an input on a subsequent media object control 7010) to switch from displaying the first previously captured image to displaying the second previously captured image (as shown in FIG. 7F) is detected.In response to receiving a second user request to switch from displaying a first previously captured media object to displaying a second previously captured media object (e.g., a second user request to switch from displaying a first previously captured media object to displaying a second previously captured media object), the device may replace the display of the first previously captured media object with a display of the second previously captured media, slide out the first previously captured media object and slide in the second previously captured media in the direction of the swipe input (e.g., a horizontal swipe or a vertical swipe input). For example, in response to an input received as described with respect to FIG. 7E, the device may switch from displaying a first previously captured media object in media object display region 7002 as shown in FIG. 7E to displaying a second previously captured media object in media object display region 7002 as shown in FIG. 7F. 7E and switches to displaying the second previously captured image in area 7002. The device displays a first virtual object over at least a portion of each image in the second previously captured media object, with the first virtual object displayed with at least a second position or orientation (or path of movement) determined based on a first position or orientation (or path of movement) of the first virtual object in each image of the first previously captured media object and based on second depth data corresponding to each image in the second previously captured media object. For example, virtual ball objects 7034 and 7044 added to the first previously captured image displayed in FIG. 7E are displayed over the second previously captured image displayed in FIG. 7F.In some embodiments, if a first virtual object is a piece of virtual confetti or a virtual ball that falls in a first previously captured image of a media object and lands on a first surface of the image (e.g., a flat surface of a core seating area), when the user switches to display a second image by swiping horizontally on the first image, the second image slides horizontally and the virtual confetti or virtual ball begins to move from its position in the first image (e.g., falls downward from a position corresponding to the surface of the seating area of the coffe) and lands on a second surface (e.g., falls downward from the surface of the floor, or the surface of a cushion on the floor, etc.). In other words, the virtual object persists when switching between media objects, and the position, orientation, and movement path of the virtual object in the next image is influenced by the position, orientation, and movement path of the virtual object in the previous image. Switching from displaying a virtual object over a first previously captured media object to displaying a virtual object over a second previously captured media object at a position determined based on the position or orientation of the virtual object in the first previously captured media object without requiring user input (e.g., to indicate the position of the virtual object in the second previously captured media object) enhances the usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing a user to use the device more quickly and efficiently.
[0288] In some embodiments, the first user request is a request (1114) to add multiple instances of a first type of virtual object (e.g., virtual ball objects 7034 and 7044 described with respect to FIGS. 7C-7E) to a previously captured media object over time (e.g., add virtual confetti or virtual balls falling to an image over time), e.g., the first virtual object is one of multiple instances of the first type of virtual object added to the first previously captured media object. In response to receiving the second user request of switching to displaying a second previously captured media object, the device displays a second virtual object over at least a portion of the respective images in the second previously captured media object, the second virtual object being an instance of a first type of virtual object that is distinct from the first virtual object and not added to the first previously captured media object, the second virtual object being displayed with at least a third position or orientation or movement path that is determined using second depth data corresponding to the respective images in the second previously captured media object. For example, in some embodiments, the first user request is a request to add a series of virtual objects of the same type over time (e.g., in a sequential manner) to generate effects in the images, such as falling confetti, rain drops, or fireworks. If the user switches to a next image or video while an effect is being applied to a first image or video (e.g., when multiple instances of virtual confetti, raindrops, or fireworks have been added to the first image), (e.g., by swiping horizontally or vertically across the first image or video), the effect is also automatically applied to the next image or video (e.g., adding new instances of virtual confetti, raindrops, or fireworks to the image) without the user explicitly invoking the effect for the next image or video (e.g., activating a control for that effect). Switching from displaying a first virtual object over a first previously captured media object to displaying a second virtual object over a second previously captured media object at a position determined based on the position or orientation of the virtual object in the first previously captured media object without requiring user input (e.g., to indicate the position of the virtual object in the second previously captured media object) enhances usability of the device.Performing actions without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.
[0289] In some embodiments, the first previously captured media object and the second previously captured media object (1116) are two separate still images (e.g., a first previously captured image as shown in the media object display area 7002 of FIG. 7E and a second previously captured image as shown in the media object display area 7002 of FIG. 7F) previously recorded and stored with different depth data corresponding to different physical environments and / or corresponding to different views of the same physical environment. For example, the second still image does not need to have any connection with the first still image in terms of subject matter captured within the image to have the same effects (e.g., falling confetti, virtual balls, fireworks, virtual block letters, etc.) that continue to be applied to the second still image. Switching from displaying a virtual object over a first previously captured media object to displaying a second virtual object over a second previously captured media object having different depth data than the first previously captured media object without requiring further user input (e.g., to indicate the location of the virtual object within the second previously captured media object) enhances the operability of the device. Performing actions without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.
[0290] In some embodiments, the first previously captured media object is a video including a sequence of consecutive image frames (e.g., as described with respect to FIGS. 7AM-7AT), and displaying the first virtual object over at least a portion of a respective image in the first previously captured media object includes displaying (1118) the first virtual object over a first portion of the first image frame while displaying the first image frame of the first previously captured media object during playback of the first previously captured media object, where the first virtual object is displayed with a position or orientation (or path of movement) determined according to a portion of the first depth data corresponding to the first image frame of the first previously captured media object, and immediately after displaying the first image frame, displaying (1119) the first virtual object over a first portion of the first image frame of the first previously captured media object. and while displaying a second image frame of the first previously captured media object (e.g., the second image frame immediately precedes the first image frame in the media object in normal or fast forward play of the media object, the second image frame immediately precedes the first image frame in the media object in reverse play of the media object, or the second image is an initial frame of the media object and the first image is a last frame of the media object during looped play of the media object), displaying the first virtual object over the second portion of the second image frame, wherein the first virtual object is displayed with a position or orientation (or movement path) determined according to a position or orientation (or movement path) of the first virtual object in the first image frame and according to a portion of the first depth data corresponding to the first previously captured second image frame of the media object.For example, if the first virtual object is virtual confetti or a virtual ball falling on a surface (e.g., a surface of a moving or stationary object) in a first image frame of the video, when the video continues to play and the surface is shown in the next image frame, the position and / or orientation and / or path of movement of the virtual confetti or virtual ball will change depending on the position and orientation of the surface in the new image frame. For example, if the surface is a stationary table surface, the virtual confetti will appear to be located at the same position on the stationary table surface, and the virtual ball will appear to roll along the stationary table surface, even if the surface is now viewed from a different perspective and occupies a different area on the second image frame compared to the first image frame. Similarly, if the surface is the top of a trap door that is suddenly seen in the video, the virtual confetti will begin to fall gradually from its stationary position on the top of the trap door, and the virtual ball will appear to descend with acceleration from its position on the top of the trap door due to pseudo gravity. In some embodiments, the first user request is a request to add a series of virtual objects of the same type over time (e.g., in a continuous manner) to create an effect on the image, such as falling confetti, or fireworks. For example, the virtual confetti object placed on the edge of the physical kite object 7112 is displayed with a changed position, orientation, and path of movement as video playback occurs in FIGS. 7AM-7AT. While the effect is being applied to the first image frame (e.g., as multiple instances of virtual confetti or fireworks are added to the first image frame), as video playback continues, the effect is also automatically applied to the next image frame (e.g., new instances of virtual confetti or fireworks are also added to the next image frame). In some embodiments, at the end of the video, the virtual object added to the last image frame includes a virtual object that was added to multiple previous image frames and settled into a final position and orientation in the final image frame based on prior interactions with a pseudo surface corresponding to the physical environment depicted in the previous image frames and a pseudo surface corresponding to the physical environment depicted in the last image frame.Displaying a virtual object over a second frame of video displayed immediately after displaying a first frame of the video, where the position or orientation of the virtual object in the second image frame is determined using depth data from the second image frame without requiring further input (e.g., without requiring a user to provide input indicating the location of the virtual object in each frame of the video), improves usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the battery life of the device by allowing a user to use the device more quickly and efficiently.
[0291] In some embodiments, displaying the first previously captured media object includes playing the video according to a first timeline including at least one of looping, fast-forwarding, or reversing the sequence of consecutive image frames (1120), and displaying the first virtual object over at least a portion of each image in the first previously captured media object includes displaying, during playback of the video according to the first timeline, changes in the position or orientation (or path of movement) of the first virtual object according to a forward timeline associated with the actual order of the sequence of image frames displayed during playback of the video (including, for example, looping from the end of the video to the beginning, switching frames at a non-uniform rate during video playback, playing the video backwards from a later frame to an earlier frame, etc.) (e.g., the previous position and orientation of the virtual object in each previously displayed image frame affects the position and orientation of the currently displayed image frame). In other words, the timeline of the movement of the virtual object in the displayed image frames is independent of the timeline in which the media object is played. Displaying changes in the position or orientation of a virtual object according to a timeline associated with the order of a sequence of image frames without requiring further input (e.g., without requiring a user to provide input indicating the change in the virtual object's position in each frame of a video) improves usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the device's battery life by allowing a user to use the device more quickly and efficiently.
[0292] In some embodiments, displaying the first virtual object over at least a portion of the respective image in the first previously captured media object (1122) includes displaying a shadow of the first virtual object according to a first physical surface captured in the first previously captured media object while the first virtual object is positioned over a corresponding one of the one or more first images, where the shadow of the first virtual object is constrained by a first pseudo surface corresponding to the first physical surface determined based on the first depth data. For example, if the still image captures a couch with curved armrests and a flat seating area, a virtual character A that is positioned on the curved surface of the armrest and then falls sideways onto the flat surface of the seating area has a shading that changes according to the surface that virtual character A is currently on and a current orientation of virtual character A relative to the surface. In some embodiments, a three-dimensional or quasi-three-dimensional mesh is generated based on depth data associated with the image or video, the mesh surface assumes geometric characteristics of the physical environment captured in the image or video, and shadows are cast on the mesh surface based on the simulated light source and the position and orientation of the virtual object relative to the mesh surface. Displaying a virtual object with shadows constrained by pseudo surfaces corresponding to physical surfaces based on the depth data of a previously captured media object without requiring further input (e.g., to identify surfaces in the previously captured media object) enhances the usability of the device. Performing operations without requiring further user input makes the user device interface more efficient, further reducing power usage and improving the battery life of the device by allowing the user to use the device more quickly and efficiently.
[0293] In some embodiments, the first user request is a user request to place a virtual first text object at a first location in the respective image in the first previously captured media object (1124). For example, the user request is an input provided at the virtual text object 7064 to initiate an edit mode of the virtual text object 7064, as described with respect to FIGS. 7Q-7S. The device receives a user input to update the virtual first text object (1126), including adding a first virtual character to the virtual first text object (e.g., editing the text input area by typing a character at the end of an existing text input), and in response to receiving the user input, the device displays the first virtual character at a second location in the respective image in the first previously captured media object adjacent to a preceding virtual character in the virtual first text object and according to a portion of the first depth data corresponding to the second location in the respective image. In some embodiments, the text has lighting and shadows that are generated based on a surface mesh of the environment captured in the respective image. Displaying a text object at a location within a previously captured media object without requiring further input (e.g., to identify depth data within the previously captured media object) and positioning characters of the text object according to the depth data within the media object enhances usability of the device. Performing operations without requiring further user input makes the user device interface more efficient and also reduces power usage and improves battery life of the device by allowing a user to use the device more quickly and efficiently.
[0294] In some embodiments, displaying the first virtual object over at least a portion of the respective image in the first previously captured media object (1128) includes displaying the first virtual object above a horizontal surface (e.g., rather than below the horizontal surface) in accordance with a determination that the pseudo surface proximate the current location of the first virtual object in the respective image is a horizontal surface, and displaying the first virtual object in front of a vertical surface in accordance with a determination that the pseudo surface proximate the current location of...
Claims
1. 1. A method comprising:
1. A computer system having a display generating element, one or more input devices, and one or more cameras, comprising: displaying, via the display generation element, a first user interface region including a representation of a field of view of the one or more cameras that is updated as the field of view of the one or more cameras changes over time; receiving, while displaying the first user interface region including the representation of the field of view of the one or more cameras, a first request via the one or more input devices to add an annotation to the displayed representation of the field of view of the one or more cameras; in response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras; replacing a display of the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to the receipt of the first request to add the annotation; receiving, while displaying the still image in the first user interface area, via the one or more input devices, a first annotation on a first portion of the still image, where the first portion of the still image corresponds to a first portion of a physical environment captured in the still image; receiving, via the one or more input devices, a first request to re-display the representation of the field of view of the one or more cameras in the first user interface area while displaying the first annotation on the first portion of the still image in the first user interface area; in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface area; pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, displaying, concurrently with the representation of the field of view of the one or more cameras, an indication of a current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; and refraining from displaying the indication in accordance with a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
2. Displaying the indication of the current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image includes: displaying indicators proximate an edge of the representation of the field of view of the one or more cameras; and moving the indicator along the edge in accordance with movement of the one or more cameras relative to the physical environment.
3. detecting a first movement of the one or more cameras while displaying the indication of the current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; in response to detecting the first movement of the one or more cameras, updating the representation of the field of view of the one or more cameras according to a change in the field of view of the one or more cameras caused by the first movement; updating the indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image according to the first movement of the one or more cameras in accordance with a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras; and ceasing to display the indication in accordance with a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
4. in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; 4. The method of claim 1, further comprising: displaying the first annotation on the first portion of the physical environment captured within the representation of the field of view of the one or more cameras in accordance with the determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
5. The method of claim 4 , wherein the first annotation is displayed as a two-dimensional object on a first depth plane within the first portion of the physical environment captured within the representation of the field of view of the one or more cameras.
6. The method of claim 5 , wherein the first annotation is displayed at a location in space within the first portion of the physical environment captured within the representation of the field of view of the one or more cameras.
7. The method of claim 5 , wherein the first annotation is displayed at a location on a physical object detected in the first portion of the physical environment captured within the representation of the field of view of the one or more cameras.
8. in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; 8. The method of claim 1, further comprising: displaying a visual representation of the first annotation drawn on the first portion of the still image simultaneously with the representation of the field of view of the one or more cameras in the first user interface area in accordance with the determination that the first portion of the physical environment captured in the still image is not currently within the field of view of the one or more cameras.
9. 9. The method of claim 8, wherein the first annotation shown on the representation of the field of view has a first perspective based on the current spatial relationship of the one or more cameras to the first portion of the physical environment captured in the still image, the first perspective being different from a second perspective of the first annotation shown on the still image.
10. in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; 10. The method of claim 1, further comprising: displaying, in accordance with the determination that the first portion of the physical environment captured in the still image is outside the field of view of the one or more cameras, a visual representation of the first annotation drawn on the first portion of the still image simultaneously with the representation of the field of view of the one or more cameras in the first user interface area, wherein the visual representation of the first annotation drawn on the first portion of the still image is converted into an indication of the current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image.
11. 11. The method of claim 1, further comprising: while displaying the first user interface region including the representation of the field of view of the one or more cameras, displaying an indication of a current spatial relationship of the one or more cameras relative to a second portion of the physical environment to which a second annotation was previously added, which was previously received prior to detecting the first request to add an annotation to the displayed representation of the field of view.
12. after receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; 12. The method of claim 11, comprising: simultaneously displaying the indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment and the indication of the current spatial relationship of the one or more cameras to the second portion of the physical environment pursuant to a determination that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras.
13. detecting a second movement of the one or more cameras relative to the physical environment while simultaneously displaying the indication of the current spatial relationship of the one or more cameras relative to the first portion of the physical environment and the indication of the current spatial relationship of the one or more cameras relative to the second portion of the physical environment; in response to detecting the second movement of the one or more cameras relative to the physical environment; and updating the indication of the current spatial relationship of the one or more cameras to the first and second portions of the physical environment in accordance with the second movement of the one or more cameras relative to the physical environment in accordance with a determination that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras, respectively.
14. 14. The method of claim 11, wherein the indication of the current spatial relationship of the one or more cameras to the first portion of the physical environment and the indication of the current spatial relationship of the one or more cameras to the second portion of the physical environment are displayed pursuant to a determination that the first portion and the second portion of the physical environment are within a predetermined range of the one or more cameras.
15. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the display generation element, a first user interface region including a representation of a field of view of the one or more cameras that is updated with changes in the field of view of the one or more cameras over time; receiving, while displaying the first user interface region including the representation of the field of view of the one or more cameras, a first request via the one or more input devices to add an annotation to the displayed representation of the field of view of the one or more cameras; in response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras; replacing a display of the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to the receipt of the first request to add the annotation; receiving, while displaying the still image in the first user interface region, via the one or more input devices, a first annotation on a first portion of the still image, the first portion of the still image corresponding to a first portion of a physical environment captured in the still image; receiving a first request, via the one or more input devices, to re-display the representation of the field of view of the one or more cameras in the first user interface area while displaying the first annotation on the first portion of the still image in the first user interface area; in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface area; pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, displaying, concurrently with the representation of the field of view of the one or more cameras, an indication of a current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; 5. A computer system comprising: instructions for withholding display of the indication pursuant to a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
16. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs being executed by a computer system having a display generating element, one or more input devices, and one or more cameras, the computer system being configured to: displaying, via the display generation element, a first user interface region including a representation of a field of view of the one or more cameras that is updated with changes in the field of view of the one or more cameras over time; receiving, while displaying the first user interface region including the representation of the field of view of the one or more cameras, a first request to add an annotation to the displayed representation of the field of view of the one or more cameras via the one or more input devices; in response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras; replacing a display of the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to the receipt of the first request to add the annotation; receiving, while displaying the still image in the first user interface region, via the one or more input devices, a first annotation on a first portion of the still image, the first portion of the still image corresponding to a first portion of a physical environment captured in the still image; receiving, via the one or more input devices, a first request to re-display the representation of the field of view of the one or more cameras in the first user interface area while displaying the first annotation on the first portion of the still image in the first user interface area; in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface region; pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, displaying, concurrently with the representation of the field of view of the one or more cameras, an indication of a current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; 11. A computer-readable storage medium comprising: instructions for causing display of the indication to be withheld pursuant to a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
17. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; means for displaying, via the display generation element, a first user interface region including a representation of the field of view of the one or more cameras that is updated as the field of view of the one or more cameras changes over time; means for receiving, while displaying the first user interface region including the representation of the field of view of the one or more cameras, a first request to add an annotation to the displayed representation of the field of view of the one or more cameras via the one or more input devices; in response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras; means for replacing a display of the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to the receipt of the first request to add the annotation; means for receiving, while displaying the still image in the first user interface area, via the one or more input devices, a first annotation on a first portion of the still image, the first portion of the still image corresponding to a first portion of a physical environment captured in the still image; means for receiving, via the one or more input devices, a first request to re-display the representation of the field of view of the one or more cameras in the first user interface area while displaying the first annotation on the first portion of the still image in the first user interface area; in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface area; pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, displaying, concurrently with the representation of the field of view of the one or more cameras, an indication of a current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; means for withholding display of the indication pursuant to a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
18. 1. An information processing apparatus for use in a computer system comprising a display generating element, one or more input devices, and one or more cameras, comprising: means for displaying, via the display generation element, a first user interface region including a representation of the field of view of the one or more cameras that is updated as the field of view of the one or more cameras changes over time; means for receiving, while displaying the first user interface region including the representation of the field of view of the one or more cameras, a first request to add an annotation to the displayed representation of the field of view of the one or more cameras via the one or more input devices; in response to the first request to add an annotation to the displayed representation of the field of view of the one or more cameras; means for replacing a display of the representation of the field of view of the one or more cameras in the first user interface area with a still image of the field of view of the one or more cameras captured at a time corresponding to the receipt of the first request to add the annotation; means for receiving, while displaying the still image in the first user interface area, via the one or more input devices, a first annotation on a first portion of the still image, the first portion of the still image corresponding to a first portion of a physical environment captured in the still image; means for receiving, via the one or more input devices, a first request to re-display the representation of the field of view of the one or more cameras in the first user interface area while displaying the first annotation on the first portion of the still image in the first user interface area; in response to receiving the first request to re-display the representation of the field of view of the one or more cameras in the first user interface area; replacing the display of the still image with the representation of the field of view of the one or more cameras in the first user interface area; pursuant to a determination that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, displaying, concurrently with the representation of the field of view of the one or more cameras, an indication of a current spatial relationship of the one or more cameras with respect to the first portion of the physical environment captured in the still image; means for withholding display of the indication in accordance with a determination that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras.
19. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; one or more processors; 15. A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 14.
20. 15. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system comprising a display generating element, one or more input devices, and one or more cameras, cause the computer system to perform the method of any one of claims 1 to 14.
21. A graphical user interface of a computer system comprising a display generating element, one or more input devices, one or more cameras, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 1 to 14.
22. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; A computer system comprising: means for executing the method according to any one of claims 1 to 14.
23. 1. An information processing apparatus for use in a computer system comprising a display generating element, one or more input devices, and one or more cameras, comprising: An information processing device comprising means for carrying out the method according to any one of claims 1 to 14.
24. 1. A method comprising:
1. A device having a display generation element and one or more input devices, comprising: displaying, via the display generation element, a user interface including a video playback area; receiving, while displaying a playback of a first portion of a video in the video playback area, a request via the one or more input devices to add an annotation to the video playback; In response to receiving the request to add the annotation, pausing playback of the video at a first position in the video; displaying a still image corresponding to the first pause point of the video; and receiving, while displaying the still image, an annotation on a first portion of the physical environment captured in the still image via the one or more input devices; and after receiving the annotation, displaying within the video playback area a second portion of the video corresponding to a second location of the video that is different from the first location within the video, wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed on the second portion of the video.
25. the video is captured by the camera during relative movement of the camera and the physical environment; a third portion of the video is captured between the first and second portions of the video and does not include the first portion of the physical environment during the relative movement of the camera and the physical environment; the annotation received on the still image is directed to a first object located in the first portion of the physical environment and is displayed at a position corresponding to the first object in the second portion of the video; 25. The method of claim 24, wherein the annotation is not displayed in the third portion of the video that does not include the first object.
26. 26. The method of claim 24, comprising displaying a timeline of the video, wherein displaying the second portion of the video is performed in response to user input scrubbing through the timeline of the video to the second position within the video.
27. 27. The method of any one of claims 24 to 26, wherein displaying the second portion of the video is performed as a result of rewinding the video from the first position to the second position.
28. 27. The method of any one of claims 24 to 26, wherein displaying the second portion of the video is performed as a result of fast-forwarding the video from the first position to the second position.
29. 27. The method of any one of claims 24 to 26, wherein displaying the second portion of the video is performed as a result of normal playback of the video from the first position to the second position.
30. displaying via the display generation element a list of media content objects including the video; receiving an input selecting the video from the list of media content objects; 30. The method of claim 24, comprising: in response to receiving the input selecting the video, displaying a user interface object having the representation of the video within the video playback area, the user interface object configured to receive a request to add an annotation to the video while the video is being played.
31. 1. A computer system comprising: A display generating element; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via said display generation element, a user interface including a video playback area; while displaying a playback of a first portion of a video in the video playback area, receiving a request via the one or more input devices to add an annotation to the video playback; In response to receiving the request to add the annotation, pausing playback of the video at a first location within the video; displaying a still image corresponding to the first pause point of the video; receiving, while displaying the still image, an annotation on a first portion of the physical environment captured in the still image via the one or more input devices; and after receiving the annotation, display within the video playback area a second portion of the video corresponding to a second location in the video that is different from the first location in the video, wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed in the second portion of the video.
32. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs being executed by a computer system having a display generating element and one or more input devices, the computer system being configured to: displaying, via said display generation element, a user interface including a video playback area; receiving, while displaying a playback of a first portion of a video in the video playback area, a request to add an annotation to the video playback via the one or more input devices; In response to receiving the request to add the annotation, pausing the playing of the video at a first location within the video; displaying a still image corresponding to the first pause point of the video; receiving, while displaying the still image, an annotation on a first portion of the physical environment captured in the still image via the one or more input devices; a computer-readable storage medium comprising instructions for, after receiving the annotation, causing a second portion of the video to be displayed within the video playback area corresponding to a second location of the video that is different from the first location in the video, wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed in the second portion of the video.
33. 1. A computer system comprising: A display generating element; one or more input devices; means for displaying, via said display generation element, a user interface including a video playback area; means for receiving, while displaying a playback of a first portion of a video in the video playback area, a request to add an annotation to the video playback via the one or more input devices; In response to receiving the request to add the annotation, pausing playback of the video at a first location within the video; means for displaying a still image corresponding to the first pause point of the video; means for receiving, while displaying the still image, an annotation on a first portion of the physical environment captured in the still image via the one or more input devices; and means for displaying within the video playback area a second portion of the video corresponding to a second location of the video that is different from the first location within the video after receiving the annotation, wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed on the second portion of the video.
34. 1. An information processing apparatus for use in a computer system having a display generating element and one or more input devices, comprising: means for displaying, via said display generation element, a user interface including a video playback area; means for receiving, while displaying a playback of a first portion of a video in the video playback area, a request to add an annotation to the video playback via the one or more input devices; In response to receiving the request to add the annotation, pausing playback of the video at a first location within the video; means for displaying a still image corresponding to the first pause point of the video; means for receiving, while displaying the still image, an annotation on a first portion of the physical environment captured in the still image via the one or more input devices; and means for displaying within the video playback area a second portion of the video corresponding to a second location of the video that is different from the first location within the video after receiving the annotation, wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed on the second portion of the video.
35. 1. A computer system comprising: A display generating element; one or more input devices; one or more processors; 31. A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 24 to 30.
36. 31. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system comprising a display generating element and one or more input devices, cause the computer system to perform the method of any one of claims 24 to 30.
37. A graphical user interface of a computer system comprising a display generating element, one or more input devices, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 24 to 30.
38. 1. A computer system comprising: A display generating element; one or more input devices; A computer system comprising means for carrying out the method according to any one of claims 24 to 30.
39. 1. An information processing apparatus for use in a computer system having a display generating element and one or more input devices, comprising:
31. An information processing device comprising means for carrying out the method according to any one of claims 24 to 30.
40. 1. A method comprising:
1. A device having a display generation element and one or more input devices, comprising: displaying, via the display generation element, a first previously captured media object including one or more first images, the first previously captured media object having been recorded and stored along with first depth data corresponding to a first physical environment captured in each of the one or more first images; receiving, while displaying the first previously captured media object, a first user request via the one or more input devices to add a first virtual object to the first previously captured media object; in response to the first user request to add the first virtual object to the first previously captured media object, displaying the first virtual object over at least a portion of a respective image in the first previously captured media object, wherein the first virtual object is displayed in at least a first position or orientation determined using the first depth data corresponding to the respective image in the first previously captured media object.
41. Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; 41. The method of claim 40, comprising displaying a first movement of the first virtual object relative to a first physical surface captured in the first previously captured media object after the first virtual object is placed on a respective one of the one or more first images, wherein the first movement of the first virtual object is constrained by a first pseudo surface corresponding to the first physical surface determined based on the depth data and a pseudo gravity direction.
42. Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; 42. The method of claim 40, comprising displaying a change in a shape of the first virtual object according to a first physical surface captured in the first previously captured media object after the first virtual object is placed on a respective one of the one or more first images, wherein the change in shape of the first virtual object is constrained by a first pseudo surface that corresponds to the first physical surface determined based on the first depth data.
43. detecting a second user request to switch from displaying the first previously captured media object to displaying a second previously captured media object while displaying the first virtual object over at least a portion of a respective image within the first previously captured media object, the second previously captured media object including one or more second images, the second previously captured media object being stored along with second depth data corresponding to a second physical environment captured in each of the one or more second images; in response to receiving the second user request to switch from displaying the first previously captured media object to displaying the second previously captured media object. replacing the first previously captured representation of the media object with the second previously captured representation of the media; 43. The method of any one of claims 40 to 42, comprising: displaying the first virtual object over at least a portion of a respective image in the second previously captured media object, wherein the first virtual object is displayed in at least a second position or orientation determined based on the first position or orientation of the first virtual object in the respective image of the first previously captured media object and based on the second depth data corresponding to the respective image in the second previously captured media object.
44. the first user request is a request to add multiple instances of a first type of virtual object to the previously captured media object over time, the first virtual object being one of the multiple instances of the first type of virtual object added to the first previously captured media object; The method further comprises: in response to receiving the second user request to switch from displaying the first previously captured media object to displaying the second previously captured media object. displaying a second virtual object over at least a portion of each image in the second previously captured media object; the second virtual object is an instance of the first type of virtual object that is separate from the first virtual object and has not been added to the first previously captured media object; 44. The method of claim 43, wherein the second virtual object is displayed at at least a third position or orientation determined using the second depth data corresponding to the respective image within the second previously captured media object.
45. 45. The method of claim 43 or 44, wherein the first previously captured media object and the second previously captured media object are two separate still images recorded and stored with different depth data corresponding to different physical environments and / or different views of the same physical environment.
46. the first previously captured media object is a video comprising a sequence of consecutive image frames; Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; During playback of the first previously captured media object, While displaying a first image frame of the first previously captured media object, displaying the first virtual object over a first portion of the first image frame, the first virtual object being displayed at a position or orientation determined according to a portion of the first depth data corresponding to the first image frame of the first previously captured media object; 43. The method of any one of claims 40 to 42, comprising: while displaying a second image frame of the first previously captured media object immediately after displaying the first image frame, displaying the first virtual object over a second portion of the second image frame, the first virtual object being determined according to the position or orientation of the first virtual object in the first image frame and displayed according to a portion of the first depth data corresponding to the second image frame of the first previously captured media object.
47. displaying the first previously captured media object includes playing the video according to a first timeline that includes at least one of looping, fast-forwarding, or reversing the sequence of successive image frames; Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; and displaying, during playback of the video according to the first timeline, changes in position or orientation of the first virtual object according to a forward timeline associated with an actual order of the sequence of image frames displayed during the playback of the video.
48. Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; 48. The method of any one of claims 40 to 47, comprising displaying a shadow of the first virtual object according to a first physical surface captured in the first previously captured media object while the first virtual object is positioned on a corresponding one of the one or more first images, wherein the shadow of the first virtual object is constrained by a first pseudo surface that corresponds to the first physical surface determined based on the first depth data.
49. the first user request is a user request to place a virtual first text object at a first location within a respective image within the first previously captured media object, and the method further comprises: receiving a user input for updating the virtual first text object, the user input including adding a first virtual character to the virtual first text object; 49. The method of claim 40, further comprising: in response to receiving the user input, displaying the first virtual character at a second location in the respective image in the first previously captured media object adjacent to a preceding virtual character in the virtual first text object according to a portion of the first depth data corresponding to the second location in the respective image.
50. Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; displaying the first virtual object above a horizontal plane in accordance with a determination that a pseudo-surface proximate a current position of the first virtual object in the respective image is a horizontal plane; and displaying the first virtual object in front of a vertical surface in accordance with a determination that a pseudo surface proximate to the current location of the first virtual object in the respective image is a vertical surface.
51. Displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; 51. The method of any one of claims 40 to 50, comprising, pursuant to a determination that the respective image includes a first pseudo surface and a second pseudo surface having different depths proximate a current position of the first virtual object in the respective image, displaying the first virtual object at a depth between the first pseudo surface and the second pseudo surface.
52. the one or more input devices include a touch-sensitive surface, the method comprising: detecting an object location input due to a contact on the touch-sensitive surface directed at the first virtual object, the object location input specifying a placement location of the first virtual object on the respective image corresponding to a final location of the contact on the touch-sensitive surface; In response to detecting the object positioning input, placing the first virtual object at the position on the respective image according to the object positioning input; detecting an end of the object positioning input, the end including detecting a lift-off of the contact from the touch-sensitive surface while the first virtual object is displayed at the placement location on the respective image; 52. The method of claim 40, further comprising: in response to detecting the end of the object positioning input, moving the first virtual object from the placement location to a final location in accordance with the depth data corresponding to a portion of the physical environment surrounding the placement location and in accordance with one or more pseudo-physical properties of the physical environment and the portion of the first virtual object.
53. the first virtual object includes a simulated spotlight, and displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; displaying, in each of the images, a pseudo light beam having a pseudo three-dimensional shape, the pseudo three-dimensional shape remaining constant with movement of the pseudo light beam within the respective images; 53. A method according to any one of claims 40 to 52, comprising displaying in each of the images a pseudo-illuminated spot having a two-dimensional shape, the two-dimensional shape changing according to the movement of the pseudo-light beam in the respective image and according to a relative spatial relationship between the pseudo-light beam and a pseudo intersection surface corresponding to a physical surface in the physical environment determined based on the first depth data.
54. the first virtual object comprises a graphical object, and displaying the first virtual object over at least a portion of a respective image in the first previously captured media object; 53. A method according to any one of claims 40 to 52, comprising displaying the graphical objects at locations in the respective images which correspond to free space in the physical environment.
55. the one or more input devices include a touch-sensitive surface, the first virtual object includes a measurement object, and the method further comprises: Detecting two simultaneous contacts on the touch-sensitive surface; and in response to detecting the two simultaneous contacts, displaying a first measurement object having two respective ends located at respective positions on the respective images corresponding to the respective positions of the two simultaneous contacts on the touch-sensitive surface.
56. Displaying the first measurement object comprises:
56. The method of claim 55, comprising: displaying a first portion of the first measurement object located on a first side of the pseudo foreground surface having a first appearance in accordance with a determination that the two respective ends of the first measurement object are located on two sides of a pseudo foreground surface corresponding to a physical surface captured in the respective images; and displaying a second portion of the first measurement object, different from the first portion, located on a second side different from the first side of the pseudo foreground surface having a second appearance, wherein the second appearance is different from the first appearance.
57. Displaying the first measurement object comprises: detecting a positioning input by contact on the touch-sensitive surface while displaying the first measurement object at a first end and a second end in the respective images, the positioning input specifying a new end position of the first measurement object in the respective images; 57. A method according to claim 55 or 56, comprising in response to detecting the positioning input by contact, moving one of the first end and the second end of the first measurement object which is closer to the new end position to the new end position in the respective image specified by the positioning input.
58. 1. A computer system comprising: A display generating element; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the display generation element, a first previously captured media object including one or more first images, the first previously captured media object being recorded and stored along with first depth data corresponding to a first physical environment captured in each of the one or more first images; receiving, while displaying the first previously captured media object, a first user request via the one or more input devices to add a first virtual object to the first previously captured media object; responsive to the first user request to add the first virtual object to the first previously captured media object, displaying the first virtual object over at least a portion of a respective image in the first previously captured media object, the first virtual object being displayed at at least a first position or orientation determined using the first depth data corresponding to the respective image in the first previously captured media object.
59. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs being executed by a computer system having a display generating element and one or more input devices, the computer system being configured to: displaying, via the display generation element, a first previously captured media object including one or more first images, the first previously captured media object being recorded and stored along with first depth data corresponding to a first physical environment captured in each of the one or more first images; receiving, while displaying the first previously captured media object, a first user request via the one or more input devices to add a first virtual object to the first previously captured media object; 12. A computer-readable storage medium comprising: instructions for, in response to the first user request to add the first virtual object to the first previously captured media object, causing a first virtual object to be displayed over at least a portion of a respective image in the first previously captured media object, the first virtual object being displayed in at least a first position or orientation determined using the first depth data corresponding to the respective image in the first previously captured media object.
60. 1. A computer system comprising: A display generating element; one or more input devices; means for displaying, via the display generating element, a first previously captured media object comprising one or more first images, the first previously captured media object having been recorded and stored along with first depth data corresponding to a first physical environment captured in each of the one or more first images; means for receiving, while displaying the first previously captured media object, a first user request via the one or more input devices to add a first virtual object to the first previously captured media object; and means for displaying, in response to the first user request to add the first virtual object to the first previously captured media object, the first virtual object over at least a portion of a respective image in the first previously captured media object, wherein the first virtual object is displayed in at least a first position or orientation determined using the first depth data corresponding to the respective image in the first previously captured media object.
61. 1. An information processing apparatus for use in a computer system having a display generating element and one or more input devices, comprising: means for displaying, via the display generating element, a first previously captured media object comprising one or more first images, the first previously captured media object having been recorded and stored along with first depth data corresponding to a first physical environment captured in each of the one or more first images; means for receiving, while displaying the first previously captured media object, a first user request via the one or more input devices to add a first virtual object to the first previously captured media object; and means for displaying, in response to the first user request to add the first virtual object to the first previously captured media object, the first virtual object over at least a portion of a respective image within the first previously captured media object, wherein the first virtual object is displayed in at least a first position or orientation determined using the first depth data corresponding to the respective image within the first previously captured media object.
62. 1. A computer system comprising: A display generating element; one or more input devices; one or more processors; 58. A computer system comprising: a memory having stored thereon one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 40 to 57.
63. 58. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a display generating element and one or more input devices, cause the computer system to perform the method of any one of claims 40 to 57.
64. 58. A graphical user interface of a computer system comprising a display generating element, one or more input devices, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 40 to 57.
65. 1. A computer system comprising: A display generating element; one or more input devices; 58. A computer system comprising means for carrying out the method of any one of claims 40 to 57.
66. 1. An information processing apparatus for use in a computer system having a display generating element and one or more input devices, comprising:
58. An information processing device comprising means for carrying out a method according to any one of claims 40 to 57.
67. 1. A method comprising:
1. A first device having a first display generating element, a first set of one or more input devices, and a first set of one or more cameras, sending a request to the remote device to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras; in response to sending the request to initiate the shared annotation session with the second device, receiving an indication of acceptance of the request to initiate the shared annotation session; displaying, via the first display generating element, a first prompt to move the first device towards the second device in response to receiving the indication of acceptance of the request to initiate the shared annotation session; after displaying the first prompt, in accordance with a determination that a connection criterion for the first device and the second device is satisfied, displaying a representation of a field of view of the first camera set in the shared annotation session with the second device; the connectivity criterion requires that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of a physical environment surrounding the first and second devices; The method, wherein during the shared annotation session, one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device are displayed via the first display generation element if the respective positions are included within the field of view of the first camera set, and one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device are displayed via the first display generation element.
68. 68. The method of claim 67, wherein during the shared annotation session, if the respective positions are included within the field of view of the second camera set, the one or more first virtual annotations corresponding to the annotation input targeted to the respective positions in the physical environment by the first device are displayed via the second display generation element, and the one or more second virtual annotations corresponding to the annotation input targeted to the respective positions in the physical environment by the second device are displayed via the second display generation element.
69. 69. The method of claim 68, wherein during the shared annotation session, if the respective positions are within the field of view of the second camera set but not within the field of view of the first camera set, the one or more first virtual annotations corresponding to the annotation input targeted to the respective positions in the physical environment by the first device are displayed via the second display generation element but not via the first display generation element.
70. 70. A method according to any one of claims 67 to 69, comprising, after displaying the first prompt, displaying within the representation of the field of view of the first camera set a notification that the shared annotation session with the second device has been established in accordance with a determination that the connection criteria for the first device and the second device have been met.
71. the first prompt includes an animation showing representations of the first and second devices moving towards each other, and the method further comprises:
71. The method of any one of claims 67 to 70, comprising, after displaying the first prompt, replacing the display of the first prompt with the representation of the field of view of the first camera set in accordance with a determination that the connection criteria for the first device and the second device are satisfied.
72. During the shared annotation session, 72. The method of any one of claims 67 to 71, comprising, while displaying the representation of the field of view of the first camera set via the first display generation element, pursuant to a determination that first annotation input directed toward a portion of the physical environment captured within the field of view of the first camera set has been received from the second device, displaying an avatar of the second device at a position within the representation of the field of view of the first camera set corresponding to a position at which the first annotation input was directed.
73. During the shared annotation session, displaying a virtual object within the representation of the field of view of the first camera set at a position corresponding to a current position of the second device while displaying the representation of the field of view of the first camera set via the first display generation element; Detecting movement of the second device from a first location to a second location within the physical environment; and displaying the movement of the virtual object in the representation of the field of view of the first camera set corresponding to the movement of the second device from the first position to the second position in the physical environment if the first position and the second position are within the field of view of the first camera set during the movement of the second device from the first position to the second position in the physical environment.
74. 74. A method according to any one of claims 67 to 73, comprising, during the shared annotation session, the first device displays the representation of the field of view of the first camera set via the first display generating element, and the second device displays the representation of the field of view of the first camera set via the second display generating element.
75. 74. A method according to any one of claims 67 to 73, comprising, during the shared annotation session, the first device displays the representation of the field of view of the first camera set via the first display generating element, and the second device displays the representation of the field of view of the second camera set via the second display generating element.
76. 76. The method of any one of claims 67 to 75, comprising terminating the shared annotation session pursuant to determining during the shared annotation session that the connection criteria is no longer satisfied.
77. 77. The method of claim 67, further comprising: generating, via the first display generating element, a second prompt to move the first device toward the second device in accordance with a determination that an amount of overlap between the field of view of the first camera set and the field of view of the second camera set during the shared annotation session falls below a first threshold amount.
78. 77. The method of claim 67, further comprising: displaying, via the first display generating element, an indication of a current relative spatial relationship between the first device and the second device in accordance with a determination that an amount of overlap between the field of view of the first camera set and the field of view of the second camera set during the shared annotation session falls below a first threshold amount.
79. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: sending a request to the remote device to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras; in response to sending the request to initiate the shared annotation session with the second device, receiving an indication of acceptance of the request to initiate the shared annotation session; displaying, via the first display generating element, a first prompt to move the first device towards the second device in response to receiving the indication of acceptance of the request to initiate the shared annotation session; after displaying the first prompt, in accordance with a determination that a connection criterion for the first device and the second device is satisfied, display a representation of a field of view of the first camera set in the shared annotation session with the second device; the connectivity criterion requires that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of a physical environment surrounding the first and second devices; A computer system, wherein during the shared annotation session, one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device are displayed via the first display generation element when the respective positions are included within the field of view of the first camera set, and one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device are displayed via the first display generation element.
80. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs being executed by a computer system having a display generating element, one or more input devices, and one or more cameras, the computer system being configured to: causing the remote device to send a request to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras; receiving, in response to sending the request to initiate the shared annotation session with the second device, an indication of acceptance of the request to initiate the shared annotation session; displaying, via the first display generating element, a first prompt to move the first device towards the second device in response to receiving the indication of acceptance of the request to initiate the shared annotation session; instructions for causing, after displaying the first prompt, in accordance with a determination that a connection criterion for the first device and the second device is satisfied, to display a representation of a field of view of the first camera set in the shared annotation session with the second device; the connectivity criterion requires that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of a physical environment surrounding the first and second devices; A computer-readable storage medium, wherein during the shared annotation session, one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device are displayed via the first display generation element if the respective positions are included within the field of view of the first camera set, and one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device are displayed via the first display generation element.
81. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; means for transmitting to the remote device a request to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras; means for receiving, in response to sending the request to initiate the shared annotation session with the second device, an indication of acceptance of the request to initiate the shared annotation session; means for displaying, in response to receiving the indication of acceptance of the request to initiate the shared annotation session, a first prompt via the first display generating element to move the first device towards the second device; means for displaying a representation of a field of view of the first camera set in the shared annotation session with the second device in accordance with a determination that a connection criterion for the first device and the second device has been satisfied after displaying the first prompt; the connectivity criterion requires that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of a physical environment surrounding the first and second devices; A computer system, wherein during the shared annotation session, one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device are displayed via the first display generation element when the respective positions are included within the field of view of the first camera set, and one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device are displayed via the first display generation element.
82. 1. An information processing apparatus for use in a computer system comprising a display generating element, one or more input devices, and one or more cameras, comprising: means for transmitting to the remote device a request to initiate a shared annotation session with a second device including a second display generating element, a second set of one or more input devices, and a second set of one or more cameras; means for receiving, in response to sending the request to initiate the shared annotation session with the second device, an indication of acceptance of the request to initiate the shared annotation session; means for displaying, in response to receiving the indication of acceptance of the request to initiate the shared annotation session, a first prompt via the first display generating element to move the first device towards the second device; means for displaying a representation of a field of view of the first camera set in the shared annotation session with the second device in accordance with a determination that a connection criterion for the first device and the second device has been satisfied after displaying the first prompt; the connectivity criterion requires that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of a physical environment surrounding the first and second devices; An information processing device, wherein during the shared annotation session, when the respective positions are included within the field of view of the first camera set, one or more first virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the first device are displayed via the first display generation element, and one or more second virtual annotations corresponding to annotation input targeted to the respective positions in the physical environment by the second device are displayed via the first display generation element.
83. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; one or more processors; 80. A computer system comprising: a memory having stored thereon one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 67 to 78.
84. 80. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system comprising a display generating element, one or more input devices, and one or more cameras, cause the computer system to perform the method of any one of claims 67 to 78.
85. A graphical user interface of a computer system comprising a display generating element, one or more input devices, one or more cameras, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 67 to 78.
86. 1. A computer system comprising: A display generating element; one or more input devices; One or more cameras; 79. A computer system comprising means for carrying out the method of any one of claims 67 to 78.
87. 1. An information processing apparatus for use in a computer system comprising a display generating element, one or more input devices, and one or more cameras, comprising:
79. An information processing device comprising means for carrying out a method according to any one of claims 67 to 78.
Citation Information
Patent Citations
Information processing device, information processing method, and program
CN102194050A
Terminal device, server device, and image processing method
JP2004145448A
Image display device, method, and program
JP2011070495A
Information processing device, information processing method and program
JP2011197777A
Display device and control method
JP2014071499A
