Systems, methods, and graphical user interfaces for annotating, measuring, and modeling an environment

Abstract

To provide a computer system which displays an annotation placement user interface that includes a representation of a field of view of one or more cameras that is updated over time based on changes in the field of view, and a placement user interface element indicating a virtual annotation placement location.SOLUTION: In the case where a placement user interface element is over a representation of a first type of physical feature, an annotation placement user interface operates in a first annotation mode for adding annotations of a first type. In response to detecting movement of the one or more cameras relative to a physical environment, a system updates the representation of the field of view. If the placement user interface element is over a representation of a second, different type of physical feature, the annotation placement user interface operates in a second, different annotation mode for adding annotations of a second, different type.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (Related Applications) This application claims priority to U.S. Patent Application No. 17 / 018,994, filed September 11, 2020, and U.S. Provisional Patent Application No. 62 / 969,647, filed February 3, 2020. [Technical field]

[0002] The present invention relates generally to electronic devices for annotating, measuring, and modeling environments, such as physical environments, and / or computer systems for augmented reality and / or virtual reality, including, but not limited to, objects within the devices, using augmented reality and / or virtual reality environments. [Background technology]

[0003] The development of computer systems for augmented reality and / or virtual reality has increased significantly in recent years. Augmented reality environments are useful for annotating and modeling physical environments and objects therein. However, conventional annotation and measurement methods using augmented reality and / or virtual reality are cumbersome, inefficient, and limited. In some cases, conventional measurement methods using augmented reality are limited in functionality by requiring the user to specify the type of measurement to make a particular feature. In some cases, conventional annotation methods using augmented reality do not provide guidance to help the user add annotations, or provide guidance in a static manner that does not consider the current context as the annotation progresses. In some cases, conventional annotation methods using augmented reality do not keep track of annotations made, and users cannot efficiently review past annotation actions. In some cases, conventional annotation and measurement methods using augmented reality are limited to annotation and measurement of straight lines that do not consider the shape or curvature of the physical surface. In some cases, conventional annotation methods using augmented reality require separate input for each annotation the user adds. In some cases, conventional methods of modeling physical environments do not provide the user with sufficient feedback about the progress of the modeling process. In some cases, conventional methods of identifying models of an environment, such as sketches, do not maintain relevant aspects of the model within the field of view. Furthermore, conventional methods take longer than necessary and therefore waste energy. This latter problem is particularly acute in battery-operated devices. Summary of the Invention

[0004] Therefore, there is a need for a computer system having improved methods and interfaces for annotating, measuring, and modeling an environment using an augmented reality and / or virtual reality environment. Such methods and interfaces optionally complement or replace conventional methods for annotating, measuring, and modeling an environment using an augmented reality and / or virtual reality environment. Such methods and interfaces reduce the number, range, and / or type of inputs from a user, creating a more efficient human-machine interface. For battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.

[0005] The above-mentioned drawbacks and other problems associated with augmented reality and / or virtual reality user interfaces are reduced or eliminated by the disclosed computer system. In some embodiments, the computer system comprises a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system comprises a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has (and / or is in communication with) a touchpad. In some embodiments, the computer system has (and / or is in communication with) a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI, in part, through contacts and gestures with a stylus and / or fingers on a touch-sensitive surface. In some embodiments, in addition to the augmented reality-based measurement functionality, the functionality optionally includes game playing, image editing, drawing, presenting, word processing, spreadsheet creation, telephony, video conferencing, email, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functionality are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0006] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying, via the display device, an annotation placement user interface including a representation of a field of view of one or more cameras, the representation including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras. The method includes updating the representation of the field of view over time based on changes in the field of view of the one or more cameras; and a placement user interface element indicating a location where a virtual annotation is to be placed within the representation of the field of view in response to receiving an annotation placement input. The method includes detecting, while displaying the annotation placement user interface, a first movement of the one or more cameras relative to the physical environment, and updating the representation of the field of view based on the first movement of the one or more cameras in response to detecting the first movement of the one or more cameras relative to the physical environment. The method also includes, in response to determining that the placement user interface element is over at least a portion of a representation of a physical feature within the measurable physical environment, modifying an appearance of the placement user interface element according to one or more aspects of the representation of the physical feature. The method includes receiving, while displaying an annotation placement user interface, an annotation placement input including a request to perform one or more measurements of the physical feature. The method also includes, in response to receiving the input corresponding to the request to perform the one or more measurements of the physical feature, displaying a representation of a first set of one or more measurements of a first measurement type over the representation of the physical feature in accordance with a determination that the physical feature is a first type of physical feature, and displaying a representation of a second set of one or more measurements of a second measurement type different from the first measurement type over the representation of the physical feature in accordance with a determination that the physical feature is a second type of physical feature different from the first type of physical feature.

[0007] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying an annotation placement user interface via the display device. The annotation placement user interface includes a representation of a field of view of one or more cameras including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras, the representation of the field of view being updated over time based on changes in the field of view of the one or more cameras, and a placement user interface element indicating a location where a virtual annotation will be placed within the representation of the field of view in response to receiving annotation placement input. The method includes detecting movement of the one or more cameras relative to the physical environment while displaying the annotation placement user interface. The method also includes, in response to detecting movement of the one or more cameras relative to the physical environment, updating the representation of the field of view based on the movement of the one or more cameras, and, in response to a determination that the placement user interface element is over at least a portion of a representation of a first type of feature in the physical environment, modifying an appearance of the placement user interface element in response to the annotation placement input to indicate an annotation to be placed at an anchor point corresponding to the first type of feature and displaying a first set of one or more guides corresponding to at least a portion of the representation of the first type of feature. The method also includes, in response to a determination that the placement user interface element is over at least a portion of a representation of a second type of feature in the physical environment that is different from the first type of feature, modifying an appearance of the placement user interface element in response to the annotation placement input to indicate an annotation to be placed at an anchor point corresponding to the second type of feature and displaying a second set of one or more guides that are different from the first set of one or more guides corresponding to at least a portion of the representation of the second type of feature.

[0008] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying an annotation placement user interface via the display device. The method also includes the annotation placement user interface including a representation of a field of view of one or more cameras, the representation including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras. The method also includes the representation of the field of view being updated over time based on changes in the field of view of the one or more cameras. The method also includes the annotation placement user interface also including a placement user interface element indicating a location where a virtual annotation is to be placed within the representation of the field of view in response to receiving an annotation placement input. The method includes receiving a first annotation placement input while the placement user interface element indicates the first location of the representation of the field of view, and in response to receiving the first annotation placement input, displaying an annotation at the first location of the representation of the field of view and displaying one or more first guides extending from the first location of the representation of the field of view.

[0009] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying, via the display device, an annotation placement user interface, the annotation placement user interface including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras, the representation of the field of view being updated over time based on changes in the field of view of the one or more cameras, and a placement user interface element indicating a location where a virtual annotation will be placed within the representation of the field of view in response to receiving annotation placement input. The method also includes detecting movement of the one or more cameras relative to the physical environment while displaying the annotation placement user interface, and updating the representation of the field of view based on the movement of the one or more cameras in response to detecting the movement of the one or more cameras relative to the physical environment, and displaying a first mode indication indicating that the annotation placement user interface is operating in a first annotation mode associated with adding a first type of annotation in response to determining that the placement user interface element is over at least a portion of a representation of a first type of physical feature within the physical environment. The method also includes, while displaying the first mode indication, detecting subsequent movement of the one or more cameras relative to the physical environment; and, in response to detecting the subsequent movement of the one or more cameras relative to the physical environment, updating the representation of the field of view based on the subsequent movement of the one or more cameras; and, pursuant to determining that the placement user interface element is over at least a portion of a representation of a second type of physical feature within the physical environment, displaying a second mode indication indicating that the annotation placement user interface is operating in a second annotation mode different from the first annotation mode associated with adding a second type of annotation different from the first type.

[0010] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes receiving a first input corresponding to a request to display an annotation placement user interface. The method also includes displaying, via the display device, the annotation placement user interface in response to receiving the first input, a representation of the field of view of the one or more cameras, the representation including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras, the representation of the field of view being updated over time based on changes in the field of view. The annotation placement user interface also includes a placement user interface element indicating a location where a virtual annotation will be placed within the representation of the field of view in response to receiving the annotation placement input. The annotation placement user interface includes a session history user interface element. The method includes, while displaying the annotation placement user interface, receiving a set of inputs corresponding to multiple requests to annotate the representation of the field of view, and adding a first annotation and a second annotation to the representation of the field of view of the one or more cameras in response to receiving the set of inputs. The method also includes, after adding the first annotation and the second annotation to the representation of the field of view of the one or more cameras, receiving input corresponding to activating a session history user interface element, and displaying at least a portion of a list of annotations including the first annotation and the second annotation, including simultaneously displaying the representation of the first annotation and the representation of the second annotation in response to receiving the input corresponding to activating the session history user interface element.

[0011] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying, via the display device, an annotation placement user interface, the annotation placement user interface including a representation of a field of view of one or more cameras, the representation of the field of view including a representation of a portion of a three-dimensional physical environment within the field of view of the one or more cameras. The representation of the field of view is updated over time based on changes in the field of view of the one or more cameras. The method includes receiving one or more first inputs corresponding to a request to measure from a first position of the representation of the field of view to a second position different from the first position of the representation of the field of view. The first position of the representation of the field of view corresponds to a first physical location on a physical surface of the physical environment, and the second position of the representation of the field of view corresponds to a second physical location on the physical surface different from the first physical location. In response to receiving the one or more first inputs, the method also includes displaying, via the display device, a representation of a first measurement value from the first position of the representation of the field of view to the second position of the representation of the field of view, including accounting for the shape of the physical surface when displaying the representation of the first measurement value in accordance with a determination that the physical surface is not a flat surface.

[0012] According to some embodiments, a method is implemented in a computer system including a display device. The method includes displaying a first user interface of an application executing on the computer system via the display device. The method includes, while displaying the first user interface of the application, receiving one or more inputs corresponding to a request to display a second user interface of the application to display an annotated representation of first previously captured media. The method includes displaying a representation of the first previously captured media in response to receiving the one or more inputs corresponding to the request to display the second user interface to display the annotated representation of the first previously captured media, the first previously captured media representation including a representation of a first portion of a three-dimensional physical environment from which the first media was captured, and displaying one or more representations of measurements of the one or more first physical features in accordance with a determination that the first previously captured media representation includes one or more regions within the measurable physical environment corresponding to the one or more first physical features.

[0013] According to some embodiments, a method is implemented in a computer system including a display device and one or more cameras. The method includes displaying representations of fields of view of one or more cameras in a first region of a user interface. The one or more cameras are present in a three-dimensional physical environment, and the representations of fields of view include representations of first fields of view of first respective portions of the physical environment that are within the fields of view of the one or more cameras. The method includes capturing depth information indicative of a first subset of the first respective portions of the physical environment. The method also includes displaying, over the representations of fields of view, a first indication indicative of the extent to which the depth information was captured of the first respective portions of the physical environment, including displaying a first indication overlaid over at least a first portion of the representation of fields of view that includes the representation of the first subset, and displaying at least a second portion of the representation of fields of view without the first indication overlaid. The method includes detecting movement of one or more cameras that move their fields of view to include a second respective portion of the physical environment, and updating, in response to detecting the movement of the one or more cameras, representations of the fields of view of the one or more cameras to include representations of the first field of view of the second respective portion of the physical environment. The method also includes, in response to detecting the movement of the one or more cameras, capturing depth information indicative of a second subset of the second respective portion of the physical environment. The method also includes, in response to detecting the movement of the one or more cameras, updating a first indication displayed on top of the updated representation of the fields of view to display a first indication superimposed on the second portion of the representation of the field of view, where the second portion of the representation of the field of view includes a representation of the second subset, indicating the extent to which the depth information was captured of the second respective portion of the physical environment.

[0014] According to some embodiments, a method is implemented in a computer system including a display device. The method includes displaying a first portion of a schematic diagram of an environment. The first portion of the schematic diagram includes a first representation of a first feature within the environment and a representation of a first metric corresponding to the first feature and displayed at a first location within the first portion of the schematic diagram of the environment. The method also includes receiving a first input corresponding to a request to display a second portion of the schematic diagram of the environment. The method includes, in response to receiving the first input, displaying the second portion of the schematic diagram of the environment. In response to receiving the first input, the method includes displaying a portion of the first representation of the first feature in the second portion of the schematic diagram of the environment, and, pursuant to a determination that the second portion of the schematic diagram of the environment does not include the first location, displaying the representation of the first metric at a second location within the second portion of the schematic diagram of the environment.

[0015] According to some embodiments, a computer system includes (and / or is in communication with) a display generation component (e.g., also referred to as a display device, such as a display, projector, head-mounted display, head-up display, etc.); one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of content within the camera's field of view, and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view); one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or one or more cameras that track the position of one or more features of a user, such as the user's hands); optionally one or more depth sensors; optionally one or more orientation sensors; optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface; According to some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, cause the computer system to perform or cause the performance of any of the operations of the methods described herein.According to some embodiments, a graphical user interface on a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, memory, and one or more processors running one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, and the elements are updated in response to input according to any of the methods described herein. According to some embodiments, the computer system includes (and / or is in communication with) the display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein. According to some embodiments, an information processing device for use in a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, includes means for performing or causing to be performed the operations of any of the methods described herein.

[0016] Thus, a computer system having (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, is provided with improved methods and interfaces for annotating, measuring, and modeling an environment, such as a physical environment, and / or objects therein using an augmented reality and / or virtual reality environment, thereby increasing the effectiveness, efficiency, and user satisfaction with such computer systems. Such methods and interfaces can complement or replace conventional methods for annotating, measuring, and modeling an environment, such as a physical environment, and / or objects therein using an augmented reality and / or virtual reality environment. [Brief explanation of the drawings]

[0017] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0018] [Figure 1A] 1 is a block diagram illustrating a portable multifunction device having a touch-sensitive display in accordance with some embodiments.

[0019] [Figure 1B] FIG. 2 is a block diagram illustrating exemplary components for event processing according to some embodiments.

[0020] [Figure 2A] 1 illustrates a portable multifunction device with a touch screen in accordance with some embodiments.

[0021] [Figure 2B]FIG. 1 illustrates a portable multifunction device having an optical sensor and a time-of-flight sensor in accordance with some embodiments.

[0022] [Figure 3A] FIG. 1 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface in accordance with some embodiments.

[0023] [Figure 3B] FIG. 1 is a block diagram of an exemplary computer system according to some embodiments. [Figure 3C] FIG. 1 is a block diagram of an exemplary computer system according to some embodiments.

[0024] [Figure 4A] 1 illustrates an exemplary user interface for presenting a menu of applications on a portable multifunction device in accordance with some embodiments.

[0025] [Figure 4B] 1 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from a display in accordance with some embodiments.

[0026] [Figure 5A] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5B] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5C] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5D] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5E] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5F] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5G] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5H] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5I] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5J] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5K] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5L] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5M] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5N] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5O] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5P]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5Q] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5R] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5S] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5T] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5U] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5V] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5W] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5X] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5Y] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5Z] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AA]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AB] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AC] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AD] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AE] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AF] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AG] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AH] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AI] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AJ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AK] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AL]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AM] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AN] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AO] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AP] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AQ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AR] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AS] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AT] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AU] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AV] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AW]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AX] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AY] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5AZ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BA] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BB] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BC] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BD] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BE] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BF] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BG] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BH]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BI] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BJ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BK] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BL] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BM] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BN] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BO] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BP] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BQ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BR] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BS]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BT] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BU] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BV] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BW] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BX] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BY] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5BZ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CA] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CB] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CC] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CD]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CE] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CF] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CG] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CH] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CI] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CJ] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CK] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CL] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CM] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments. [Figure 5CN] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments.

[0027] [Figure 6A] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6B] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6C] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6D] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6E] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6F] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6G] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6H] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6I]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6J] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6K] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6L] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6M] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6N] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6O] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6P] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6Q]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6R] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6S] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6T] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6U] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6V] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6W] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6X] FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments. [Figure 6Y]FIG. 1 illustrates an exemplary user interface for annotating and measuring a physical environment using augmented reality, according to some embodiments, and retrieving measurement information from stored media items, according to some embodiments.

[0028] [Figure 7A] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7B] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7C] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7D] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7E] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7F] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7G] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7H] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7I]FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7J] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7K] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7L] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7M] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7N] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7O] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7P] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7Q] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7R] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7S]FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7T] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7U] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7V] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7W] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7X] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7Y] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7Z] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AA] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AB] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AC]FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AD] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AE] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AF] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AG] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AH] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AI] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AJ] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AK] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AL] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AM]FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AN] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AO] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AP] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AQ] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AR] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AS] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments. [Figure 7AT] FIG. 1 illustrates an exemplary user interface for scanning and modeling an environment and interacting with the resulting schematic representation, according to some embodiments.

[0029] [Figure 8A] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. [Figure 8B] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. [Figure 8C] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. [Figure 8D] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. [Figure 8E] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. [Figure 8F] 1 is a flow diagram of a process for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments.

[0030] [Figure 9A] 1 is a flow diagram of a process for providing alignment guides based on automatically determined anchor points within an augmented reality environment, according to some embodiments. [Figure 9B] 1 is a flow diagram of a process for providing alignment guides based on automatically determined anchor points within an augmented reality environment, according to some embodiments. [Figure 9C] 1 is a flow diagram of a process for providing alignment guides based on automatically determined anchor points within an augmented reality environment, according to some embodiments.

[0031] [Figure 10A] 1 is a flow diagram of a process for providing dynamic positioning guides in response to user input within an augmented reality environment according to some embodiments. [Figure 10B] 1 is a flow diagram of a process for providing dynamic positioning guides in response to user input within an augmented reality environment according to some embodiments. [Figure 10C]1 is a flow diagram of a process for providing dynamic positioning guides in response to user input within an augmented reality environment according to some embodiments.

[0032] [Figure 11A] 1 is a flow diagram of a process for automatically changing annotation modes based on the type of physical features in view in an augmented reality environment, according to some embodiments. [Figure 11B] 1 is a flow diagram of a process for automatically changing annotation modes based on the type of physical features in view in an augmented reality environment, according to some embodiments. [Figure 11C] 1 is a flow diagram of a process for automatically changing annotation modes based on the type of physical features in view in an augmented reality environment, according to some embodiments. [Figure 11D] 1 is a flow diagram of a process for automatically changing annotation modes based on the type of physical features in view in an augmented reality environment, according to some embodiments. [Figure 11E] 1 is a flow diagram of a process for automatically changing annotation modes based on the type of physical features in view in an augmented reality environment, according to some embodiments.

[0033] [Figure 12A] 1 is a flow diagram of a process for displaying a session history, including a list of annotations added using an annotation placement user interface during an annotation session, according to some embodiments. [Figure 12B] 1 is a flow diagram of a process for displaying a session history, including a list of annotations added using an annotation placement user interface during an annotation session, according to some embodiments. [Figure 12C] 1 is a flow diagram of a process for displaying a session history, including a list of annotations added using an annotation placement user interface during an annotation session, according to some embodiments.

[0034] [Figure 13A] 1 is a flow diagram of a process for measuring a path along a physical surface and taking into account the shape and curvature of the physical surface, according to some embodiments. [Figure 13B] 1 is a flow diagram of a process for measuring a path along a physical surface and taking into account the shape and curvature of the physical surface, according to some embodiments. [Figure 13C] 1 is a flow diagram of a process for measuring a path along a physical surface and taking into account the shape and curvature of the physical surface, according to some embodiments. [Figure 13D] 1 is a flow diagram of a process for measuring a path along a physical surface and taking into account the shape and curvature of the physical surface, according to some embodiments.

[0035] [Figure 14A] 1 is a flow diagram of a process for measuring physical characteristics represented in previously captured media, according to some embodiments. [Figure 14B] 1 is a flow diagram of a process for measuring physical characteristics represented in previously captured media, according to some embodiments. [Figure 14C] 1 is a flow diagram of a process for measuring physical characteristics represented in previously captured media, according to some embodiments. [Figure 14D] 1 is a flow diagram of a process for measuring physical characteristics represented in previously captured media, according to some embodiments.

[0036] [Figure 15A] 1 is a flow diagram of a process for scanning a physical environment and capturing depth information for modeling the physical environment, according to some embodiments. [Figure 15B]1 is a flow diagram of a process for scanning a physical environment and capturing depth information for modeling the physical environment, according to some embodiments. [Figure 15C] 1 is a flow diagram of a process for scanning a physical environment and capturing depth information for modeling the physical environment, according to some embodiments. [Figure 15D] 1 is a flow diagram of a process for scanning a physical environment and capturing depth information for modeling the physical environment, according to some embodiments.

[0037] [Figure 16A] 1 is a flow diagram of a process for viewing and interacting with a schematic representation of an environment, according to some embodiments. [Figure 16B] 1 is a flow diagram of a process for viewing and interacting with a schematic representation of an environment, according to some embodiments. [Figure 16C] 1 is a flow diagram of a process for viewing and interacting with a schematic representation of an environment, according to some embodiments. [Figure 16D] 1 is a flow diagram of a process for viewing and interacting with a schematic representation of an environment, according to some embodiments. [Figure 16E] 1 is a flow diagram of a process for viewing and interacting with a schematic representation of an environment, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0038] As described above, augmented reality environments are useful for annotating and modeling physical environments and objects therein by providing different views of the physical environment and the objects therein, allowing users to overlay annotations, such as measurements, on the physical environment and the objects therein, and visualizing the interaction between the annotations and the physical environment and the objects therein. Conventional annotation and modeling methods using augmented reality and / or virtual reality often have limited functionality. In some cases, conventional measurement methods using augmented reality are limited in functionality by requiring the user to specify the type of measurement to make up a particular feature. In some cases, conventional annotation methods using augmented reality do not provide guidance to help the user add annotations, or provide guidance in a static manner that does not consider the current context as the annotation progresses. In some cases, conventional annotation methods using augmented reality do not keep track of annotations made, preventing users from efficiently reviewing past annotation activity. In some cases, conventional annotation and measurement methods using augmented reality are limited to annotation and measurement of straight lines, which do not consider the shape or curvature of the physical surface. In some cases, conventional annotation methods using augmented reality require a separate input for each annotation the user adds. In some cases, conventional methods of modeling physical environments do not provide users with sufficient feedback about the progress of the modeling process. In some cases, conventional methods of reviewing a model of an environment, such as a schematic, do not maintain relevant aspects of the model within the field of view. Embodiments disclosed herein provide an intuitive way for users to annotate, measure, and model an environment using an augmented reality and / or virtual reality environment (e.g., by providing smarter and more sophisticated functionality, by allowing users to perform different operations within the augmented reality environment with less input, and / or by simplifying the user interface). Furthermore, embodiments herein provide users with improved feedback that provides additional information about the physical object being measured or modeled and about the operations being performed within the virtual / augmented reality environment.

[0039] The systems, methods, and GUIs described herein improve user interface interaction with augmented reality and / or virtual reality environments in several ways: for example, they facilitate annotating and modeling physical environments by providing automatic detection of features in physical space and appropriate measurements for different types of detected features, improved labeling and guidance (e.g., for improved annotation placement), by allowing users to interact with and manage measurement information, by keeping relevant annotations and measurements visible on a representation of the environment, and by providing users with improved feedback about the progress of the modeling process while modeling the environment.

[0040] Below, Figures 1A-1B, 2A-2B, and 3A-3C provide descriptions of exemplary devices. Figures 4A-4B, 5A-5CN, 6A-6Y, and 7A-7AT illustrate exemplary user interfaces for interacting with environments, such as augmented reality environments, and media items to annotate, measure, and model them. Figures 8A-8F show a flow diagram of a method for displaying automatically determined measurements of a physical environment using augmented reality. Figures 9A-9C show a flow diagram of a method for providing positioning guides based on automatically determined anchor points within an augmented reality environment. Figures 10A-10C show a flow diagram of a method for providing dynamic positioning guides in response to user input within an augmented reality environment. Figures 11A-11E show a flow diagram of a method for automatically changing annotation modes based on the type of physical feature within view within an augmented reality environment, according to some embodiments. Figures 12A-12C show a flow diagram of a method for displaying a session history, including a list of annotations made using an annotation placement user interface during an annotation session, according to some embodiments. Figures 13A-13D show a flow diagram of a method for measuring a path along a physical surface and taking into account the shape and curvature of the physical surface, according to some embodiments. Figures 14A-14D show a flow diagram of a method for measuring physical features depicted in previously captured media, according to some embodiments. Figures 15A-15D show a flow diagram of a method for scanning a physical environment and capturing depth information for modeling the physical environment, according to some embodiments. Figures 16A-16E show a flow diagram of a method for viewing and interacting with a schematic representation of an environment, according to some embodiments.

[0041] The user interfaces of Figures 5A-5CN, 6A-6Y, and 7A-7AT are used to illustrate the processes of Figures 8A-8F, 9A-9C, 10A-10C, 11A-11C, 12A-12C, 13A-13D, 14A-14D, 15A-15D, and 16A-16E. Exemplary Devices

[0042] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0043] In this specification, terms such as "first," "second," etc. are used to describe various elements in some examples, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the various embodiments being described. Although a first contact and a second contact are both contacts, they are not the same contact unless the context clearly dictates otherwise.

[0044] The terminology used in the description of the various embodiments set forth herein is for the purpose of describing particular embodiments only and is not intended to be limiting. In the description of the various embodiments set forth and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and / or" should be understood to refer to and include any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] As used herein, the term "if" is optionally interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

[0046] A computer system for augmented and / or virtual reality includes an electronic device that generates an augmented and / or virtual reality environment. Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad) and also including or in communication with one or more cameras.

[0047] In the following description, a computer system is described that includes an electronic device having (and / or in communication with) a display and a touch-sensitive surface. However, it should be understood that the computer system optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, a joystick, a wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hands.

[0048] The device typically supports a variety of applications, such as one or more of gaming applications, note-taking applications, drawing applications, presentation applications, word processing applications, spreadsheet applications, phone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0049] Various applications running on the device optionally use at least one common physical user-interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed by the device are optionally adjusted and / or changed for each application and / or within each application. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports various applications with user interfaces that are intuitive and transparent to the user.

[0050] Attention now turns to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating portable multifunction device 100 having touch-sensitive display system 112, according to some embodiments. Touch-sensitive display system 112 may conveniently be referred to as a "touch screen" or simply a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more light sensors 164 (e.g., as part of one or more cameras). Device 100 optionally includes one or more intensity sensors 165 that detect the intensity of a contact on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 163 that generate tactile output on device 100 (e.g., generate tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0051] As used herein and in the claims, the term “tactile output” refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, that will be detected by a user with the user's sense of touch. For example, in a situation where a device or a component of a device is in contact with a touch-sensitive surface of a user (e.g., the fingers, palm, or other part of the user's hand), the tactile output produced by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a “downclick” or “upclick” of a physical actuator button. In some cases, a user feels a tactile sensation such as a “downclick” or “upclick” even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's action. As another example, movement of a touch-sensitive surface is optionally interpreted or perceived by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. While such user interpretation of touch depends on the user's personal sensory perception, there are many sensory perceptions of touch that are common to the majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "upclick," "downclick," "roughness"), unless otherwise specified, the generated tactile output corresponds to a physical displacement of the device, or a component of the device, that produces the described sensory perception for a typical (or average) user.Using tactile output to provide haptic feedback to the user improves usability of the device, makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the device), and additionally reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0052] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of its components. The various components shown in FIG. 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0053] Memory 102 optionally includes high-speed random access memory, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripherals interface 118, is optionally controlled by memory controller 122.

[0054] A peripheral interface 118 may be used to couple input and output peripherals of the device with the CPU(s) 120 and memory 102. The one or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions and process data for the device 100.

[0055] In some embodiments, peripheral interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0056] RF (radio frequency) circuitry 108 transmits and receives RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to or from electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. RF circuitry 108 optionally communicates wirelessly with networks, such as the Internet, also known as the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and with other devices. Radio options include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), and code division multiple access (CDMA).Wireless technologies include, but are not limited to, standard IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n, voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP)), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), and Instant Messaging and Presence Services (IMP). The present invention may use any of a number of communication standards, protocols, and technologies, including, but not limited to, Intermediate Message Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0057] Audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuit 110 receives audio data from peripherals interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves audible to humans. Audio circuit 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuit 110 converts the electrical signals into audio data and transmits the audio data to peripherals interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to memory 102 and / or RF circuit 108 by peripherals interface 118. In some embodiments, audio circuit 110 also includes a headset jack (e.g., 212 in FIG. 2A ). The headset jack provides an interface between audio circuitry 110 and a detachable audio input / output peripheral, such as an output-only headphone or a headset with both an output (e.g., mono or binaural headphones) and an input (e.g., a microphone).

[0058] I / O subsystem 106 couples input / output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, with peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from or send electrical signals to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 160 are optionally coupled to any (or none) of a keyboard, infrared port, USB port, stylus, and / or pointer device such as a mouse. The one or more buttons (e.g., 208 in FIG. 2A) optionally include up / down buttons for controlling the volume of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., 206 in FIG. 2A).

[0059] Touch-sensitive display system 112 provides an input and output interface between the device and a user. Display controller 156 receives electrical signals from and / or sends electrical signals to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed towards the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, a button, a slider, an icon, a selectable menu item, a switch, a hyperlink, or other user interface control.

[0060] Touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touch-sensitive display system 112 and display controller 156 (along with any associated modules and / or instruction sets in memory 102) detect contacts (and any movement or disruption of contact) on touch-sensitive display system 112 and translate the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touch-sensitive display system 112. In some embodiments, the point of contact between touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0061] Touch-sensitive display system 112 optionally uses liquid crystal display (LCD), light emitting polymer display (LPD), or light emitting diode (LED) technology, although other display technologies are used in other embodiments. Touch-sensitive display system 112 and display controller 156 optionally detect contact and any movement or disruption thereof using any of a number of now-known or later-developed touch sensing technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.

[0062] Touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or higher). A user optionally contacts touch-sensitive display system 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on a touchscreen than that of a stylus. In some embodiments, the device translates coarse finger input into precise pointer / cursor positions or commands to perform actions desired by the user.

[0063] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0064] Device 100 also includes a power system 162 that provides power to the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of electrical power within a portable device.

[0065] Device 100 also optionally includes one or more light sensors 164 (e.g., as part of one or more cameras). FIG. 1A shows a light sensor coupled to light sensor controller 158 in I / O subsystem 106. Light sensor(s) 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. Light sensor(s) 164 receive light from the environment, projected through one or more lenses, and convert the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), light sensor(s) 164 optionally capture still images and / or video. In some embodiments, the light sensor is located on the back of device 100, opposite touch-sensitive display system 112 on the front of the device, so that the touchscreen can be used as a viewfinder for still and / or video image acquisition. In some embodiments, another light sensor is placed on the front of the device so that an image of the user is captured (e.g., for a selfie, for a video conference while the user is viewing other video conference participants on the touchscreen, etc.).

[0066] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I / O subsystem 106. Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to the touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100, opposite touchscreen display system 112, which is located on the front of device 100.

[0067] Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call).

[0068] Device 100 optionally includes one or more tactile output generators 163. FIG. 1A shows tactile output generators coupled to haptic feedback controller 161 in I / O subsystem 106. In some embodiments, tactile output generator(s) 163 include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear movement, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Tactile output generator(s) 163 receive tactile feedback generation instructions from haptic feedback module 133 and generate tactile outputs on device 100 that can be sensed by a user of device 100. In some embodiments, at least one tactile output generator is juxtaposed with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and generates a tactile output, optionally by moving the touch-sensitive surface vertically (e.g., in / out of the surface of device 100) or horizontally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display system 112, which is located on the front of device 100.

[0069] Device 100 also optionally includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (e.g., as part of an inertial measurement unit (IMU)) for obtaining information regarding the device's location (e.g., position and orientation or attitude). FIG. 1A shows sensors 167, 168, and 169 coupled to peripherals interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver for obtaining information regarding device 100's location.

[0070] In some embodiments, software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and applications (or instruction sets) 136. Additionally, in some embodiments, as shown in Figures 1A and 3, memory 102 stores device / global internal state 157. Device / global internal state 157 includes one or more of: active application state, which indicates which applications, if any, are currently active; display state, which indicates which applications, views, or other information occupy various areas of touch-sensitive display system 112; sensor state, which includes information obtained from the device's various sensors and other input or control devices 116; and position and / or location information regarding the device's attitude (e.g., location and / or orientation).

[0071] Operating system 126 (e.g., an embedded operating system such as iOS, Android, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0072] Communications module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by RF circuitry 108 and / or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is adapted to couple to other devices directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector identical to, similar to, and / or compatible with the 30-pin connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a Lightning connector identical to, similar to, and / or compatible with the Lightning connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a USB Type-C connector that is the same as, similar to, and / or compatible with the USB Type-C connector used in some electronic devices from Apple Inc. of Cupertino, California.

[0073] Contact / motion module 130 optionally detects contact with touch-sensitive display system 112 (in cooperation with display controller 156) and with other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes software components for performing various operations related to detecting contact (e.g., by a finger or stylus), such as determining if contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining if there is contact movement and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining if the contact has stopped (e.g., detecting a finger-up event or an interruption of contact). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact, as represented by the series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact. These actions are optionally applied to a single contact (e.g., a single finger contact or a stylus contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contacts on the touchpad.

[0074] Contact / motion module 130 optionally detects gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or strength of the detected contacts). Thus, gestures are optionally detected by detecting particular contact patterns. For example, detecting a finger tap gesture includes detecting a finger down event, followed by detecting a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger down event, followed by detecting one or more finger drag events, followed by detecting a finger up (lift-off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected with respect to a stylus by detecting particular contact patterns with respect to the stylus.

[0075] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger-down event and detecting a finger-up event, but is not related to the strength of the finger contact between detecting the finger-down event and detecting the finger-up event. In some embodiments, a tap gesture is detected according to determining that the length of time between the finger-down event and the finger-up event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact-detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture can satisfy certain input criteria that do not require the characteristic intensity of the contact to meet a given intensity threshold for the particular input criteria to be met. For clarity, finger contacts in a tap gesture generally need to meet a nominal contact-detection intensity threshold below which the contact is not detected in order to detect a finger-down event. A similar analysis applies to detecting a stylus tap gesture or other contact. In cases where the device is capable of detecting contact of a finger or stylus hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0076] In a similar manner, the same concepts apply to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or long press gestures are optionally detected based on meeting criteria that are either unrelated to the intensity of the contacts included in the gesture or that do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts, a pinch gesture is detected based on the movement of two or more contacts toward each other, a de-pinch gesture is detected based on the movement of two or more contacts away from each other, and a long press gesture is detected based on the duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. Thus, a statement that a particular gesture recognition criterion does not require the intensity of a contact(s) to meet a corresponding intensity threshold in order for the particular gesture recognition criterion to be met means that the particular gesture recognition criterion can be met when the contact(s) in the gesture do not reach the corresponding intensity threshold, and can also be met in situations where one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on a determination that a finger-down event and a finger-up event are detected within a predetermined time period, regardless of whether the contacts are above or below the respective intensity thresholds during the predetermined time period, and a swipe gesture is detected based on a determination that a movement of the contact is greater than a predetermined magnitude, even if the contacts exceed the respective intensity thresholds at the end of the movement of the contacts. Even in implementations in which gesture detection is affected by the intensity of the contact performing the gesture (e.g., the device detects long presses more quickly when the intensity of the contact exceeds an intensity threshold, or the device is slow to detect tap inputs when the intensity of the contact is higher), detection of those gestures does not require the contact to reach a particular intensity threshold, as long as the criteria for recognizing the gesture can be met in situations in which the contact does not reach the particular intensity threshold (e.g., even if the amount of time required to recognize the gesture varies).

[0077] The contact intensity threshold, duration threshold, and movement threshold may, in some circumstances, be combined in various different combinations to create heuristics for distinguishing between two or more different gestures directed at the same input element or region, thereby enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. A statement that a particular set of gesture recognition criteria does not require the intensity of the contact(s) to meet a respective intensity threshold for that particular gesture recognition criterion does not preclude simultaneously evaluating other intensity-dependent gesture recognition criteria to identify other gestures with criteria that are met when the gesture includes a contact having an intensity exceeding a respective intensity threshold. For example, in some circumstances, a first gesture recognition criterion for a first gesture that does not require the intensity of the contact(s) to meet a corresponding intensity threshold for the first gesture recognition criterion to be met competes with a second gesture recognition criterion for a second gesture that relies on the contact(s) reaching a corresponding intensity threshold. In such a competition, a gesture is optionally not recognized as satisfying the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are satisfied first. For example, if the contact reaches the corresponding intensity threshold before moving a predetermined amount of movement, a deep press gesture is detected rather than a swipe gesture. Conversely, if the contact moves a predetermined amount of movement before reaching the corresponding intensity threshold, a swipe gesture is detected rather than a deep press gesture. Even in such a situation, the first gesture recognition criteria for the first gesture still do not require that the intensity of the contact(s) meet the corresponding intensity threshold for the first gesture recognition criteria to be satisfied, because if the contact remained below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact that does not increase in intensity above the corresponding intensity threshold), the gesture would have been recognized by the first gesture recognition criteria as a swipe gesture.In this way, certain gesture recognition criteria that do not require the intensity of the contact(s) to meet a corresponding intensity threshold for the particular gesture recognition criterion to be satisfied are still dependent on the intensity of the contact with respect to an intensity threshold, in the sense that (A) in some circumstances, they ignore the intensity of the contact with respect to the intensity threshold (e.g., for a tap gesture), and / or (B) in some circumstances, the particular gesture recognition criterion (e.g., for a long press gesture) will not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes an input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input (e.g., for a long press gesture that competes with a deep press gesture for recognition).

[0078] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, attitude module 131 optionally detects attitude information about the device, such as the device's attitude (e.g., roll, pitch, yaw, and / or position) within a particular coordinate system. Attitude module 131 includes software components for performing various operations related to detecting the device's position and detecting changes in the device's attitude.

[0079] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other display, including components for modifying the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, video, and animation.

[0080] In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, including coordinate data and other graphic characteristic data, as needed, from an application or the like, and then generates screen image data to output to display controller 156.

[0081] The haptic feedback module 133 includes various software components that generate instructions (e.g., instructions used by the haptic feedback controller 161) that use the tactile output generator(s) 163 to create tactile outputs at one or more locations on the device 100 in response to user interaction with the device 100.

[0082] Text input module 134 is optionally a component of graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0083] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based calling, to the camera 143 as photo / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0084] Virtual / Augmented Reality Module 145 provides virtual and / or augmented reality logic to Application 136, which implements augmented reality functionality, and in some embodiments, virtual reality functionality. Virtual / Augmented Reality Module 145 facilitates the overlay of virtual content, such as virtual user interface objects, onto a representation of at least a portion of the field of view of one or more cameras. For example, with assistance from Virtual / Augmented Reality Module 145, the representation of at least a portion of the field of view of one or more cameras may include corresponding physical objects, and the virtual user interface objects may be displayed in a displayed augmented reality environment determined based on the corresponding physical objects in the field of view of the one or more cameras, or at a position within the virtual reality environment determined based on an orientation of at least a portion of the computer system (e.g., the orientation of a display device used to display a user interface to a user of the computer system).

[0085] Application 136 optionally includes the following modules (or sets of instructions), or a subset or superset thereof: • a contacts module 137 (sometimes called an address book or contact list); ●Telephone module 138, ●Videoconferencing module 139, ● an email client module 140; ● Instant messaging (IM) module 141; ●Training support module 142, a camera module 143 for still and / or video images, ● Image management module 144; ● Browser module 147, ● Calendar module 148, a widget module 149, optionally including one or more of a weather widget 149-1, a stock price widget 149-2, a calculator widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; a widget creation module 150 for creating user-created widgets 149-6; ● Search module 151, • a video and music player module 152, optionally consisting of a video player module and a music player module; ● Memo module 153, Map module 154, and / or ●Online video module 155, Annotation and Modeling Module 195, and / or • Time of Flight (“ToF”) sensor module 196.

[0086] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0087] Contacts module 137, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, includes executable instructions (e.g., stored in memory 102 or in application internal state 192 of contacts module 137 in memory 370) for managing an address book or contact list, including adding name(s) to the address book, removing name(s) from the address book, associating phone number(s), email address(es), physical address(es), or other information with names, associating images with names, categorizing and sorting names, providing phone numbers and / or email addresses to initiate and / or facilitate communication by telephone 138, video conference 139, email 140, or IM 141, etc.

[0088] In cooperation with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for entering a series of characters corresponding to a telephone number, accessing one or more telephone numbers in address book 137, modifying an entered telephone number, dialing each telephone number, conducting a conversation, and disconnecting or hanging up when the conversation is completed. As noted above, wireless communication optionally uses any of a number of communication standards, protocols, and technologies.

[0089] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between a user and one or more other participants according to the user's commands.

[0090] In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 contains executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In cooperation with image management module 144, email client module 140 greatly facilitates the creation and sending of emails with still or video images captured by camera module 143.

[0091] In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant message module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0092] In cooperation with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the training support module 142 includes executable instructions to create workouts (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (in the sports device and smartwatch), receive training sensor data, calibrate sensors used to monitor workouts, select and play music for workouts, and display, store, and transmit workout data.

[0093] Camera module 143, along with touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, includes executable instructions to capture still images or video (including video streams) and store them in memory 102, change characteristics of the still images or video, and / or delete the still images or video from memory 102.

[0094] Image management module 144, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and / or video images.

[0095] Browser module 147, along with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contains executable instructions for browsing the Internet according to user commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0096] Calendar module 148, along with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) according to user commands.

[0097] Widget module 149, along with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, is optionally a mini-application downloaded and used by a user (e.g., weather widget 149-1, stock quotes widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or a mini-application created by a user (e.g., user-created widget 149-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).

[0098] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget creation module 150 contains executable instructions for creating widgets (e.g., turning user-specified portions of a web page into widgets).

[0099] In cooperation with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search memory 102 for text, music, sound, images, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's instructions.

[0100] In cooperation with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions to display, present, or otherwise play videos (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0101] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 contains executable instructions for creating and managing notes, to-do lists, and the like according to user commands.

[0102] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 can be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0103] In cooperation with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 contains executable instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), and play (e.g., on touchscreen 112 or on an external display connected wirelessly or via external port 124) online videos in one or more file formats, such as H.264, and send and otherwise manage emails with links to particular online videos. In some embodiments, instant messaging module 141 is used to send links to particular online videos, rather than email client module 140.

[0104] In cooperation with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, camera module 143, image management module 152, video & music player module 152, and virtual / augmented reality module 145, annotation and modeling module 195 includes executable instructions that enable a user to model the physical environment and / or physical objects therein and annotate (e.g., measure it, draw on it, and / or add virtual objects to it, manipulate virtual objects therein) representations (e.g., live or previously captured) of the physical environment and / or physical objects therein in augmented reality and / or virtual reality, as described in more detail herein.

[0105] In cooperation with camera module 143, ToF sensor module 196 includes executable instructions for capturing depth information of a physical environment. In some embodiments, ToF sensor module 196 operates in cooperation with camera module 143 to provide depth information of a physical environment.

[0106] Each of the above-identified modules and applications corresponds to executable instruction sets that perform one or more of the functions described above and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). The modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of the modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.

[0107] In some embodiments, device 100 is a device in which operation of a predetermined set of functions on the device is performed solely via a touchscreen and / or touchpad. Using the touchscreen and / or touchpad as the primary input control device for operation of device 100 optionally reduces the number of physical input control devices (push buttons, dials, etc.) on device 100.

[0108] The set of predefined functions performed only through the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 to a main menu, home menu, or root menu from any user interface displayed on device 100. In such embodiments, a "menu button" is implemented using the touch-sensitive surface. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touch-sensitive surface.

[0109] 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (in FIG. 3A) includes an event sorter 170 (e.g., within operating system 126) and a respective application 136-1 (e.g., any of applications 136, 137-155, 380-390 described above).

[0110] Event sorter 170 receives the event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of application 136-1. Event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) that are displayed on touch-sensitive display system 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which application(s) are currently active, and application internal state 192 is used by event sorter 170 to determine the application view 191 to which the event information is to be delivered.

[0111] In some embodiments, application internal state 192 includes additional information such as one or more of resume information to be used when application 136-1 resumes execution, user interface state information indicating or ready to display information being displayed by application 136-1, state cues that allow the user to return to a previous state or view of application 136-1, and redo / undo cues of previous actions taken by the user.

[0112] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 167, and / or microphone 113 (via audio circuitry 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0113] In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or for longer than a predetermined period of time).

[0114] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173 .

[0115] Hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when touch-sensitive display system 112 displays more than one view. A view consists of the controls and other elements that a user can see on the display.

[0116] Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a programmatic level within the application's programmatic or view hierarchy. For example, the lowest-level view in which the touch is detected is optionally referred to as the hit view, and the set of events that are recognized as suitable inputs is optionally determined based at least in part on the hit view of the initial touch that initiates the touch gesture.

[0117] Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest-level view in which the initiating sub-event occurs (i.e., the first sub-event in a series of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0118] Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive the particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive the particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that contain the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with one particular view, views higher in the hierarchy still remain actively participating views.

[0119] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information retrieved by each event receiver module 182 in an event queue.

[0120] In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In still other embodiments, event sorter 170 is a stand-alone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0121] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each containing instructions for processing touch events that occur within a respective view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes multiple event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface kit or a higher-level object from which application 136-1 inherits methods and other attributes. In some embodiments, each event handler 190 includes one or more of a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Instead, one or more of the application views 191 include one or more respective event handlers 190. Also, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in each application view 191.

[0122] Each event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (optionally including sub-event delivery instructions).

[0123] The event receiver 182 receives event information from the event sorter 170. The event information includes information about a sub-event, e.g., a touch or a movement of a touch. Depending on the sub-event, the event information also includes additional information, such as the position of the sub-event. When the sub-event involves a movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the current attitude (e.g., position and orientation) of the device.

[0124] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines the event or sub-event, or determines or updates the state of the event or sub-event, based on the comparison. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a predefined sequence of sub-events), such as Event 1 (187-1) and Event 2 (187-2). In some embodiments, the sub-events in Event 187 include, for example, a touch start, a touch end, a touch movement, a touch cessation, and multiple touches. In one example, the definition for Event 1 (187-1) is a double tap on a displayed object. The double tap includes, for example, a first touch (touch start) for a predetermined stage on the displayed object, a first lift-off (touch end) for a predetermined stage, a second touch (touch start) for a predetermined stage on the displayed object, and a second lift-off (touch end) for a predetermined stage. In another example, a definition of event 2 (187-2) is a drag on a displayed object. Drag includes, for example, a touch (or contact) of a predetermined magnitude on a displayed object, a movement of the touch across the touch-sensitive display system 112, and a lift-off of the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event handlers 190.

[0125] In some embodiments, event definition 187 includes a definition of the event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, event comparator 184 uses the results of the hit test to determine which event handler 190 to activate. For example, event comparator 184 selects the event handler associated with the sub-event and object that triggers the hit test.

[0126] In some embodiments, each event 187 definition also includes a delay action that delays delivery of the event information until it is determined whether a set of sub-events corresponds to the event recognizer's event type.

[0127] If the respective event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the respective event recognizer 180 enters an event-disabled, event-failed, or event-ended state and thereafter ignores the next sub-event of the touch-based gesture. In this situation, any other event recognizers that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0128] In some embodiments, each event recognizer 180 includes metadata 183 with configurable properties, flags, and / or lists that indicate to actively participating event recognizers how the event delivery system should perform sub-event delivery. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact with each other or how event recognizers are allowed to interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are delivered to various levels in the view or programmatic hierarchy.

[0129] In some embodiments, each event recognizer 180 activates an event handler 190 associated with an event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and postponing sending) sub-events to the respective hit view. In some embodiments, the event recognizer 180 pops a flag associated with the recognized event, and the event handler 190 associated with the flag captures the flag and performs a predetermined process.

[0130] In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with a set of sub-events or to an actively participating view. The event handler associated with the set of sub-events or the actively participating view receives the event information and performs predetermined processing.

[0131] In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, data updater 176 updates phone numbers used by contacts module 137 or stores video files used by video and music player module 152. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.

[0132] In some embodiments, event handler(s) 190 include or have access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0133] It should be understood that the foregoing description of event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 using input devices, not all of which are initiated on the touchscreen. For example, mouse movements and mouse button presses, contact movements such as tapping, dragging, scrolling on a touchpad, optionally coordinated with single or multiple keyboard presses or holds, pen stylus input, input based on real-time analysis of video images captured by one or more cameras, device movement, verbal commands, detected eye movement, biometric input, and / or any combination thereof, optionally utilize as inputs corresponding to sub-events that define the recognized event.

[0134] FIG. 2A illustrates portable multifunction device 100 (e.g., a view of the front of device 100) with a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A ), according to some embodiments. The touchscreen optionally displays one or more graphics in user interface (UI) 200. In these embodiments, as well as embodiments described below, a user is enabled to select one or more of the graphics by making gestures on the graphics, for example, with one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of the one or more graphics is performed when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with device 100 (right to left, left to right, upward and / or downward). In some implementations or situations, accidental contact with a graphic does not select the graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon optionally does not select the corresponding application.

[0135] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As mentioned above, menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on a touchscreen display.

[0136] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and locking the device, volume control button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. Push button 206 is optionally used to power the device on / off by pressing and holding the button down for a predetermined period of time, to lock the device by pressing and releasing the button before the predetermined time has elapsed, and / or to unlock the device or initiate the unlocking process. In some embodiments, device 100 also accepts verbal input through microphone 113 to activate or deactivate some features. Device 100 also optionally includes one or more contact intensity sensors 165 that detect the intensity of a contact on touch-sensitive display system 112 and / or one or more tactile output generators 163 that generate a tactile output for a user of device 100.

[0137] 2B shows portable multifunction device 100 (e.g., a view of the back of device 100) optionally including optical sensors 164-1 and 164-2 and time-of-flight (“ToF”) sensor 220. When optical sensors (e.g., cameras) 164-1 and 164-2 simultaneously capture representations (e.g., images or video) of the physical environment, portable multifunction device can determine depth information from discrepancies between information simultaneously captured by the optical sensors (e.g., discrepancies between captured images). Depth information (e.g., images) provided by discrepancies determined using optical sensors 164-1 and 164-2 may lack accuracy, but typically provides high resolution. To improve the accuracy of the depth information provided by discrepancies between images, time-of-flight sensor 220 is optionally used in conjunction with optical sensors 164-1 and 164-2. The ToF sensor 220 emits a waveform (e.g., light from a light emitting diode (LED) or laser) and measures the time it takes for the reflection(s) of the waveform (e.g., light) to return to the ToF sensor 220. Depth information is determined from the measured time it takes for the light to return to the ToF sensor 220. ToF sensors typically provide high accuracy (e.g., accuracy of 1 cm or better for measured distance or depth), but may lack high resolution (e.g., the ToF sensor 220 optionally has a resolution that is one-quarter the resolution of the optical sensor 164, or less than one-quarter the resolution of the optical sensor 164, or one-sixteenth the resolution of the optical sensor 164, or less than one-sixteenth the resolution of the optical sensor 164). Therefore, combining the depth information from the ToF sensor with depth information (e.g., images) provided by mismatches determined using an optical sensor (e.g., a camera) provides a depth map that is both accurate and has high resolution.

[0138] FIG. 3A is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes input / output (I / O) interfaces 330, including a display 340, which is optionally a touchscreen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a haptic output generator 357 for generating haptic outputs on device 300 (e.g., similar to haptic output generator(s) 163 described above in connection with FIG. 1A ), sensors 359 (e.g., light sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to those described above in connection with FIG. 1A , and optionally time-of-flight sensor 220 described above in connection with FIG. 2B ). Memory 370 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310.In some embodiments, the memory 370 stores programs, modules, and data structures similar to, or subsets of, the programs, modules, and data structures stored in the memory 102 of the portable multifunctional device 100 (FIG. 1A). Further, the memory 370 optionally stores additional programs, modules, and data structures that do not exist in the memory 102 of the portable multifunctional device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while the memory 102 of the portable multifunctional device 100 (FIG. 1A) optionally does not store those modules.

[0139] Each of the elements in FIG. 3A, specified above, is optionally stored in one or more of the previously mentioned memory devices. Each of the modules specified above corresponds to a set of instructions that perform the functions described above. The modules or programs (i.e., sets of instructions) specified above need not be implemented as separate software programs, procedures, or modules, and thus various subsets of those modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, the memory 370 optionally stores subsets of the modules and data structures specified above. Further, the memory 370 optionally stores additional modules and data structures not described above.

[0140] FIGS. 3B-3C are block diagrams of an exemplary computer system 301, according to some embodiments.

[0141] In some embodiments, the computer system 301 includes and / or communicates with the following. input device(s) (302 and / or 307, e.g., a touch-sensitive surface such as a touch-sensitive remote control or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of the user, such as the user's hand); Virtual / Augmented Reality Logic 303 (e.g., Virtual / Augmented Reality Module 145); display generation component(s) (304 and / or 308, e.g., display, projector, head-mounted display, head-up display, etc.) for displaying virtual user interface elements to the user; ● camera(s) (e.g., 305 and / or 311) for capturing images of the device's field of view, e.g., images used to determine placement of virtual user interface elements, determine the device's pose, and / or display a portion of the physical environment in which the camera(s) are located; • Orientation sensor(s) (e.g., 306 and / or 311) for determining the device's orientation and / or changes in the device's orientation relative to the physical environment.

[0142] In some embodiments, computer system 301 (e.g., camera(s) 305 and / or 311) includes and / or is in communication with a time-of-flight sensor (e.g., time-of-flight sensor 220, FIG. 2B) for capturing depth information as described above with reference to FIG. 2B.

[0143] In some computer systems (e.g., 301-a in FIG. 3B), input device(s) 302, virtual / augmented reality logic 303, display generation component(s) 304, camera(s) 305, and orientation sensor(s) 306 are all integrated into the computer system (e.g., portable multifunction device 100 of FIGS. 1A-1B or device 300 of FIG. 3, such as a smartphone or tablet).

[0144] In some computer systems (e.g., 301-b), in addition to integrated input device(s) 302, virtual / augmented reality logic 303, display generation component(s) 304, camera(s) 305, and orientation sensor(s) 306, the computer system also communicates with additional devices separate from the computer system, such as separate input device(s) 307, such as a touch-sensitive surface, wand, or remote control, and / or separate display generation component(s) 308, such as a virtual reality headset or augmented reality glasses that overlay virtual objects on the physical environment.

[0145] In some computer systems (e.g., 301-c in FIG. 3C), the input device(s) 307, the display generation component(s) 309, the camera(s) 311, and / or the orientation sensor(s) 312 are separate from and in communication with the computer system. In some embodiments, other combinations of components within the computer system 301 and in communication with the computer system are used. For example, in some embodiments, the display generation component(s) 309, the camera(s) 311, and the orientation sensor(s) 312 are incorporated into a headset that is either integrated with or in communication with the computer system.

[0146] In some embodiments, all of the operations described below with reference to FIGS. 5A-5CN are performed on a single computing device having virtual / augmented reality logic 303 (e.g., computer system 301-a described below with reference to FIG. 3B). However, it should be understood that often, different computing devices are linked together to perform the operations described below with reference to FIGS. 5A-5CN, 6A-6Y, and 7A-7AT (e.g., a computing device having virtual / augmented reality logic 303 communicates with a separate computing device having display 450 and / or a separate computing device having touch-sensitive surface 451). In any of these embodiments, the computing device described below with reference to FIGS. 5A-5CN, 6A-6Y, and 7A-7AT is the computing device(s) that includes virtual / augmented reality logic 303. Additionally, it should be understood that in various embodiments, virtual / augmented reality logic 303 may be divided among separate modules or computing devices. However, for purposes of description herein, virtual / augmented reality logic 303 will primarily be referred to as residing within a single computing device so as not to unnecessarily obscure other aspects of the embodiments.

[0147] In some embodiments, the virtual / augmented reality logic 303 includes one or more modules (e.g., one or more event handlers 190 including one or more object updaters 177 and one or more GUI updaters 178, as described in more detail above with reference to FIG. 1B ) that receive the interpreted inputs and, in response to these interpreted inputs, generate instructions for updating the graphical user interface according to the interpreted inputs, which are then used to update the graphical user interface on the display. In some embodiments, the interpreted inputs relative to inputs detected (e.g., by the contact movement module 130 of FIGS. 1A and 3 ), recognized (e.g., by the event recognizer 180 of FIG. 1B ), and / or delivered (e.g., by the event sorter 170 of FIG. 1B ) are used to update the graphical user interface on the display. In some embodiments, the interpreted inputs are generated by a module in the computing device (e.g., the computing device receives raw contact input data so as to identify gestures from the raw contact input data). In some embodiments, some or all of the interpreted input is received by a computing device as the interpreted input (e.g., a computing device including touch-sensitive surface 451 processes the raw contact input data to identify a gesture from the raw contact input data and sends information indicative of the gesture to a computing device including virtual / augmented reality logic 303).

[0148] In some embodiments, both the display and the touch-sensitive surface are integrated with a computer system (e.g., 301-a in FIG. 3B) that includes virtual / augmented reality logic 303. For example, the computer system may be a desktop or laptop computer with an integrated display (e.g., 340 in FIG. 3) and touchpad (e.g., 355 in FIG. 3). As another example, the computing device may be portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2A).

[0149] In some embodiments, the touch-sensitive surface is integrated with a computer system, and the display is not integrated with the computer system that includes virtual / augmented reality logic 303. For example, the computer system may be device 300 (e.g., a desktop or laptop computer) with an integrated touchpad (e.g., 355 in FIG. 3 ) connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.). As another example, the computer system may be portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2A ) connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.).

[0150] In some embodiments, the display is integrated with the computer system, and the touch-sensitive surface is not integrated with the computer system that includes virtual / augmented reality logic 303. For example, the computer system may be device 300 (e.g., a desktop computer, a laptop computer, a television with an integrated set-top box) with an integrated touch display (e.g., 340 in FIG. 3 ) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.). As another example, the computer system may be portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2A ) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, another portable multifunction device with a touchscreen that functions as a remote touchpad, etc.).

[0151] In some embodiments, neither the display nor the touch-sensitive surface is integrated with the computer system (e.g., 301-c in FIG. 3C ) that includes virtual / augmented reality logic 303. For example, the computer system may be a stand-alone computing device 300 (e.g., a set-top box, a game console, etc.) that is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.) and a separate display (e.g., a computer monitor, a television, etc.).

[0152] In some embodiments, the computer system has an integrated audio system (e.g., audio circuitry 110 and speakers 111 in portable multifunction device 100). In some embodiments, the computing device is in communication with an audio system that is separate from the computing device. In some embodiments, the audio system (e.g., an audio system integrated in a television unit) is integrated with a separate display. In some embodiments, the audio system (e.g., a stereo system) is a stand-alone system that is separate from the computer system and the display.

[0153] Attention is now directed to embodiments of a user interface (“UI”) that is optionally implemented on portable multifunction device 100.

[0154] 4A shows an exemplary user interface for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof: signal strength indicator(s) for wireless communication(s), such as cellular and Wi-Fi signals; ●Time, ●Bluetooth (registered trademark) indicator, ● Battery status indicator, Tray 408 with icons of frequently used applications, such as: An icon 416 for the phone module 138, labeled "Phone," optionally including an indicator 414 of the number of missed calls or voicemail messages; icon 418 of the email client module 140, labeled "Mail," optionally including an indicator 410 of the number of unread emails; ○ An icon 420 for the browser module 147, labeled "Browser"; ○ An icon 422 for the video and music player module 152 labeled "Music", and ● Icons of other applications, such as: ○ Icon 424 of IM module 141, labeled "Messages"; icon 426 of the calendar module 148, labeled "Calendar"; ○ Icon 428 of the image management module 144, labeled "Photos" ○ An icon 430 of the camera module 143, labeled "camera"; ○ Icon 432 of the online video module 155, labeled "Online Video"; Icon 434 of Stock Price Widget 149-2, labeled "Stock Price" ○ Icon 436 of the map module 154, labeled "Map" Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of Training Support Module 142, labeled "Training Support"; icon 444 of the Notes module 153, labeled "Notes"; and A settings application or module icon 446 labeled "Settings" that provides access to settings for the device 100 and its various applications 136.

[0155] 4A are merely examples. For example, other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0156] FIG. 4B shows an example user interface on a device (e.g., device 300, FIG. 3A) that has touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3A) that is separate from display 450. While many of the following examples are given with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in FIG. 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) that corresponds to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B) at locations that correspond to respective locations on the display (e.g., in FIG. 4B, 460 corresponds to 468, and 462 corresponds to 470). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462, and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B ) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B ) of the multifunction device. It should be understood that similar methods are optionally used for the other user interfaces described herein.

[0157] Additionally, while the following examples are given primarily with reference to finger input (e.g., finger touches, finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of those finger inputs are replaced with input from another input device (e.g., mouse-based input or stylus input, movement of the device or one or more cameras of the device relative to the surrounding physical environment), and / or user movement relative to the device tracked using one or more cameras. For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch) followed by movement of a cursor along the path of the swipe (e.g., instead of movement of the touch), or by a hand gesture involving moving the user's own hand in a particular direction. As another example, a tap gesture is optionally replaced by a mouse click while the cursor is located over the tap gesture location (e.g., instead of detecting a touch and then ceasing contact detection), or by a corresponding hand gesture representing the tap gesture. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple input devices of a particular type are, optionally, used simultaneously, or multiple input devices of different types are, optionally, used simultaneously.

[0158] As used herein, the term “focus selector” refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other location marker, the cursor functions as a “focus selector” such that when input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3A or touch-sensitive surface 451 in FIG. 4B ) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations that include a touchscreen display (e.g., touch-sensitive display system 112 in FIG. 1A or touchscreen in FIG. 4A ) that enables direct interaction with user interface elements on the touchscreen display, a contact detected on the touchscreen functions as a “focus selector” such that when input (e.g., a press input by contact) is detected on the touchscreen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one region of the user interface to another region of the user interface (e.g., by using the tab key or arrow keys to move focus from one button to another) without a corresponding cursor movement or contact movement on the touchscreen display. In these implementations, the focus selector moves to follow the movement of focus between various regions of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user intends to interact with).For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button indicates that the user intends to activate that corresponding button (rather than other user interface elements shown on the device's display). In some embodiments, a focus indicator (e.g., a cursor or selection indicator) is displayed via a display device to indicate the current portion of the user interface that will be affected by input received from one or more input devices. User Interface and Related Processes

[0159] Attention is now directed to embodiments of user interfaces (“UIs”) and associated processes that may be implemented on a computer system (e.g., portable multifunction device 100 ( FIG. 1A ), device 300 ( FIG. 3A ), or computer system 301 ( FIG. 3B )) that includes (and / or is in communication with) a display generation component (e.g., a display device such as a display, projector, head-mounted display, head-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view, and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hand), optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators.

[0160] 5A-5CN, 6A-6Y, and 7A-7AT show exemplary user interfaces for annotating, measuring, and modeling an environment, such as a physical environment, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including those of FIGS. 8A-8F, 9A-9C, 10A-10C, 11A-11E, 12A-12C, 13A-13D, 14A-14D, 15A-15D, and 16A-16E. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., the centroid of each contact or a point associated with each contact), or the centroid of two or more contacts detected on touch-sensitive display system 112. However, a similar operation is optionally performed on a device having display 450 and separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying the user interface shown in the figure on display 450, along with a focus selector.

[0161] 5A-5CN illustrate exemplary user interfaces for annotating and measuring a physical environment using augmented reality, according to some embodiments.

[0162] FIG. 5A shows a home screen user interface with measurement application 448. FIG. 5B shows input 81 above icon 448. FIG. 5C shows a response to input 581 above icon 448. FIG. 5C shows portable multifunction device 100 displaying augmented reality view 500. Augmented reality view 500 is created based on three-dimensional physical environment 501 that is within the field of view of one or more light sensors 164 of portable multifunction device 100 (e.g., within the field of view of one or more light sensors 164, or of a currently selected light sensor of any one of one or more light sensors 164, or optionally within the combined field of view of two or more light sensors 164). This augmented reality view 500 is used to overlay annotations such as measurements, edge markings, surface markings, bounding boxes, etc. on top of representations of objects found within three-dimensional physical environment 501.

[0163] Three-dimensional physical environment 501 includes a plurality of items, such as television stand 502-1, television (and / or any other type of display) 502-2, sofa 502-3, pillows 502-4, table 502-5, lamp 502-6, and magazine 502-7. Augmented reality view 500 includes corresponding representations of the plurality of items. These corresponding representations of the plurality of items are television stand 503-1 (obscured), television 503-2 (obscured), sofa 503-3, pillows 503-4, table 503-5, lamp 503-6, and magazine 503-7. These corresponding representations of the plurality of items have wavy outlines 569 surrounding them, indicating that measurements (e.g., metrics) of these objects have been determined. Augmented reality view 500 shown on portable multifunction device 100 includes a plurality of user interface objects overlaid on augmented reality view 500. Each of these user interface objects is used to control what is being displayed in augmented reality and also to help the user capture and overlay measurements of objects shown in the three-dimensional physical environment 501. When used with respect to measurements, the terms "capture," "capturing," and "captured" should be understood to mean or include "generate," "generating," and "generated," respectively. When used as a verb, the terms "overlap" and "overlapping" should be understood to mean or include "to display on top of" and "displaying on top of," respectively, or alternatively, to mean being rendered so that it appears to be overlaid (e.g., an object, region, etc.) in a displayed augmented reality or virtual reality user interface.

[0164] In some embodiments, another user interface element displayed in (or overlapping with) the augmented reality view 500 is a mode indication user interface 505. In some embodiments, icons 505 for multiple different measurement modes are displayed within the mode indication user interface 505, allowing a user to switch between five such measurement modes shown in the example of FIG. 5C via icons 505 (e.g., 505-1 through 505-5). The measurement modes are sometimes referred to herein as annotation modes. Each measurement mode is selected by selecting the corresponding measurement mode icon 505, for example, by selecting the automatic measurement mode icon 505-1, the three-dimensional measurement mode icon 505-2, the two-dimensional surface measurement mode icon 505-3, the two-point measurement mode icon 505-4, and the floor plan measurement mode icon 505-5. When selected, automatic measurement mode icon 505-1 can cause portable multifunction device 100 to automatically detect which surface or object a reticle (e.g., reticle 504, shown in FIG. 5E and many subsequent figures) is overlaying (e.g., a three-dimensional object, a two-dimensional surface, etc.), and based on that determination, the device can automatically indicate that the surface or object is measurable. Reticle 504 is an example of an annotation placement user interface element, sometimes simply referred to as a “placement user interface element.” For example, when the automatic measurement mode icon is selected, the device can switch measurement modes depending on what is being measured (e.g., when the reticle is over a table surface, two-dimensional measurements are displayed, and when the reticle is moved over a couch, three-dimensional measurements are shown). In some embodiments, the reticle changes appearance based on the object reticle 504 is overlaying. In some embodiments, there can be instances where a user wants to override automatic mode and manually specify what type of measurement to perform.When this occurs, the user can switch measurement modes by tapping the respective measurement mode icon 505, such as the three-dimensional measurement mode icon 505-2, the two-dimensional surface measurement mode icon 505-3, or the two-point measurement mode icon 505-4. In some embodiments, the user can also swipe in any direction to change modes. The three-dimensional measurement mode icon 505-2, the two-dimensional surface measurement mode icon 505-3, and the two-point measurement mode icon 505-4, when selected, allow the user to instruct the device 100 to generate one or more measurements in either three dimensions (e.g., length, width, and height), two dimensions (e.g., two dimensions selected from length, width, or height), or one dimension from one point to another (e.g., the length of the edge of a table), respectively. In some embodiments, the floor plan measurement mode icon 505-5, when selected, allows the user to use the device 100 to scan the three-dimensional physical environment 501 and create a floor plan (e.g., a bird's-eye view). In some embodiments, reticle 504 (see, e.g., FIG. 5E) visually changes depending on which measurement mode the user selects, thereby indicating to the user which measurement mode is currently in use. In some embodiments, a selection indicator 537 is displayed around the selected icon to indicate which of these mode icons has been selected.

[0165] In some embodiments, another user interface element displayed in (or overlapping with) the augmented reality view 500 is a measurement list button 506, which, when selected, causes the device 100 to display a list of all measurements (e.g., metrics) taken during the current annotation session. In some embodiments, another user interface element displayed in (or overlapping with) the augmented reality view 500 is a measurement anchor user interface object 508, sometimes referred to herein as an “annotation creation button 508,” which, when selected, drops a measurement point at a predetermined location, such as the location of the reticle 504 or the location of the focal point 504-1 of the reticle 504 (see, e.g., FIG. 5BU). In some embodiments, another user interface element displayed in (or overlapping with) the augmented reality view 500 is an undo user interface object 507, which, when selected, removes the most recently dropped measurement point. Finally, in some embodiments, another user interface element displayed on (or overlaid on) the augmented reality view 500 is a record user interface object 550 that, when selected, records the augmented reality view 500, including (e.g., any) displayed measurements, in a video format, an image format, or other suitable format. This recording can then be shared with a user on another device via a messaging application, email, or other suitable method of transferring data to another user.

[0166] FIG. 5D shows the same three-dimensional physical environment 501 in a different orientation (e.g., a real-time representation of a camera's current field of view, as determined by the orientation of portable multifunction device 100). The transition from FIG. 5C to FIG. 5D indicates that the user is holding portable multifunction device 100 in a different orientation, which allows one or more optical sensors 164 to see different portions of three-dimensional physical environment 501. As shown, in this other orientation, front door 502-8, window 502-9 (partially obscured), and dog 502-10 in three-dimensional physical environment 501 are now within the field of view of one or more optical sensors 164, and corresponding representations 503-8 (front door), 502-9 (window), and 502-10 (dog) are shown in augmented reality view 500. These corresponding representations of multiple items have dashed outlines 569 surrounding them, indicating that measurements (e.g., metrics) of these items / objects are being determined.

[0167] 5E shows portable multifunction device 100 with automatic measurement mode icon 505-1 selected, and shows augmented reality view 500 after device 100 and its one or more optical sensors 164 have been moved closer to table 503-5. One of the user interface objects is reticle 504, which shows the user where the measurement will begin if inserted (e.g., dropped).

[0168] 5F shows the augmented reality view 500 after device 100 moves closer to the edge of table 509. As one or more optical sensors 164 of device 100 move closer to the edge of table 509, device 100 recognizes that the entire table is no longer visible and instead provides an edge indication 570 for selection. Although not shown in FIG. 5F , in some embodiments, when an edge region is fully visible and other portions of the physical object adjacent to table 509 are not fully visible, a representation of an edge region (e.g., a first edge) of the physical object (e.g., table 509) includes a visually de-emphasized (e.g., shaded) portion of the edge region adjacent to the fully invisible region. In some embodiments, points within the edge region (e.g., representing a first edge) are progressively de-emphasized as they approach a second portion of the physical object (e.g., a first point in the edge region that is a first distance from the boundary between the first and second portions of the physical object is visually de-emphasized relative to a second point in the edge region that is a second distance from the boundary that is greater than the first distance).

[0169] FIG. 5G shows the augmented reality view 500 with an input 571 on the fix measurement user interface object 508 for fixing the measurement at the edge of the table 509 identified by the indication 570 .

[0170] 5H shows the augmented reality view 500 in response to input 571 onto the measurement fixed user interface object 508 to select the edge of the table 509 identified by the indication 570. As shown in FIG. 5H, the indication 570 is no longer displayed in the augmented reality view 500; instead, the augmented reality view 500 now includes a highlighted line 572 that identifies the edge of the table 509 and an annotation representing the measured length of the table edge (e.g., 4 feet) as measured by the device 100.

[0171] 5I-5K and 5L-5N are alternative embodiments of the augmented reality view 500 of FIG. 5H, in which additional indications of potential corresponding measurements are shown after the initial measurement is made. In some embodiments, as one or more optical sensors 164 move relative to the three-dimensional physical environment 501, device 100 automatically detects whether there are other relevant surfaces that were not previously entirely present in the field of view prior to the movement of one or more optical sensors and that are (now) measurable. When the device detects a measurable relevant surface, additional indications may be shown in the augmented reality view 500. For example, in FIG. 5I, device 100 is moved and one or more optical sensors 164 of device 100 have table surface 560 in view. FIGS. 5I-5K show reticle 504 with an animated transition covering table surface 560. FIG. 5K shows the animation finished and a resulting indication 561 around table surface 560, informing the user that the table measurement can now be fixed. In some embodiments, the indication that other relevant surfaces are measurable includes a representation of one or more additional measurements, such as an additional measurement representing the length of another edge of table surface 560, as shown in FIG. 5K. FIGS. 5L-5N show the entire table 503-3, now within the view of one or more optical sensors 164 or device 100. FIGS. 5L-5N show the appearance of reticle 504 changing (e.g., animated changes in appearance) to cover the entire table. FIG. 5N shows that the change in appearance of reticle 504 is complete and additional indications 562 are obtained around the entire table 503-3, informing the user that the table measurements can be fixed. FIG. 5M shows intermediate stages in the animated change in appearance of reticle 504.

[0172] With respect to the indication 562 shown around the entire table 503-3, when a user of either FIG. 5K or FIG. 5N selects the measurement fixed user interface object 508, the indicated surface or object will be measured and an annotation representing the generated measurement(s) will be added to the augmented reality view 500.

[0173] FIG. 5O shows that the highlighted line 572 in the augmented reality view 500 includes additional detail as one or more optical sensors 164 move closer to the edge of the table 509. In some embodiments, these additional details are checkmarks 510. These checkmarks 510 represent measurement increments (e.g., 1 inch, 2 inches, and 3 inches). In some embodiments, displaying these checkmarks at any time would clutter the user interface, so the checkmarks are displayed when a threshold distance between the device 100 and the object or object feature (e.g., the edge of the table 509) is met. Furthermore, in some embodiments, the granularity of the displayed checkmarks also depends on the proximity of the one or more optical sensors 164 to the object feature (e.g., in some embodiments, as the device 100 moves closer to the object, more checkmarks are displayed because the distance between the optical sensors of the device 100 and the object meets one or more additional (e.g., shorter) threshold distances).

[0174] 5P shows an additional checkmark 563 displayed by device 100 in augmented reality view 500 when device 100 determines that one or more optical sensors 164 have moved closer to an object or object feature, such as the edge of table 509 (e.g., at or within a first distance from the physical feature). In some embodiments, additional checkmark 563 can be used to obtain additional measurements of the object (e.g., table 509). FIG. 5P also shows reticle 504 snapping to the 1.8 inch mark, which was not previously displayed when device 100 was positioned farther away from table 509.

[0175] 5Q shows that when a checkmark, or additional checkmarks, appear in the augmented reality view 500, the user can more precisely position the reticle 504 over the desired measurement point. For example, as the device 100 moves closer to the edge of the table 509, the reticle can snap smaller measurement increments. Here, the reticle has snapped the 2-inch mark.

[0176] FIG. 5R shows an input 511 in a measurement anchored user interface object 508 while the reticle 504 is snapped to a particular checkmark 510-1 (eg, 2 inches).

[0177] 5S illustrates a response to input 511 (e.g., adding an annotation to the augmented reality view 500) in the measurement anchored user interface object 508 while the reticle 504 is snapped to a particular checkmark 510-1. In particular, in this example, a point 564 is displayed in the augmented reality view 500 when the measurement anchored user interface object 508 is selected. This point is located at the location where the reticle 504 was snapped.

[0178] 5T shows the augmented reality view 500 after one or more light sensors 164 of device 100 have moved away from the edge of table 509, with the edge of table 584 still within the view of one or more light sensors 164. Additionally, in some embodiments, or in certain situations when predetermined criteria are met, guide lines 585 are displayed in the augmented reality view 500 to assist in the placement of annotations (e.g., measurements). In this example, when one or more light sensors have moved away from the edge of table 509, checkmark 510 and additional checkmark 563 cease to display. However, previously added annotations (e.g., dot 564, previously placed on checkmark 510-1) continue to display, even though checkmark 510 and additional checkmark 563 are no longer displayed.

[0179] Figure 5U shows portable multifunction device 100 with automatic measurement mode icon 505-1 still selected, and also shows device 100 and its one or more light sensors 164 positioned closer to different objects (in this example, television stand 502-1 and television 502-2) than the previously measured objects. Figure 5U shows reticle 504 over television representation 503-2 in augmented reality view 500. In some embodiments, even though reticle 504 is over television representation 503-2, device 100 automatically detects the television and determines that the measurements of the television (made by device 100) are ready to be fixed.

[0180] 5U-5W show an animated change in the appearance of reticle 504 indicating that the television measurements are ready to be fixed. FIG. 5V shows the animated change in the appearance of reticle 504 in progress. In some embodiments, once the animation is complete, an indication (e.g., bounding box 565) is displayed around the television representation 503-2 in the augmented reality view 500. In some embodiments, the indication (e.g., bounding box 565) may have a shaded fill, signifying that the surface of the object (e.g., the television) is ready to be fixed. In some embodiments, device 100 determines that an object in one or more cameras' views is a television, and in response to that determination, the diagonal length of the television is displayed as an annotation in (e.g., added to) the augmented reality view 500. In some embodiments, other commonly used measurements associated with a particular detected object (e.g., person-specific measurements) may also be displayed in addition to the measurements (e.g., length, width, height, and volume) that are normally displayed. FIG. 5W also shows an input 566 on the measurement lock user interface object 508 to lock to the measurement of television 503-2.

[0181] 5X shows measurements of a fixed television. In some embodiments, to indicate that the measurement has been fixed, device 100 replaces dashed indication (e.g., bounding box) 565 with solid indication (e.g., bounding box) 565-1. Furthermore, in some embodiments, reticle 504 reappears after indication (e.g., bounding box) 565 has been fixed. When device 100 subsequently moves and a different object (e.g., a TV stand) is within the view of one or more of the device's cameras, reticle 504 splits to display a new box around the different object (e.g., the TV stand), and the previously measured object (e.g., television 503-2) reappears in augmented reality view 500 with dotted lines and / or is surrounded by a bounding box with dotted lines. In some embodiments, once the user fixates on the new object (e.g., the TV stand), the representation of the previously measured object in augmented reality view 500 ceases to be surrounded by the corresponding indication (e.g., bounding box) 565. In some embodiments, the representation of the previously measured object in the augmented reality view 500 has a different dotted bounding box than the new object. In some embodiments, the representation of the previously measured object has the same dotted bounding box as the new object. In some embodiments, when the user moves the device 100 so that the reticle 504 is no longer over the representation of the previously measured object without fixing the measurements of the previously measured object, the representation of the previously measured object is displayed with a solid line at least around the periphery of the object to the extent that it is still within the field of view of one or more cameras of the device and / or is displayed within a solid box until the reticle is moved over a new object that can be measured by the device.In some embodiments, when the augmented reality view 500 includes representations of both a previously measured object and a new object, the previously measured object is displayed in the augmented reality view 500 with a bounding box having a different pattern of points than the bounding box displayed around the new object. In some embodiments, the previously measured object is displayed in the augmented reality view 500 with a bounding box having the same pattern of points as the bounding box displayed around the new object.

[0182] FIG. 5Y shows portable multifunction device 100 in another orientation, with front door 502-8 within the view of one or more light sensors 164 of the device, and representation 503-8 of front door is shown in augmented reality view 500. In this orientation, window 502-9 and dog 502-10 are within the view of one or more light sensors 164 of device 100. FIG. 5Y shows portable multifunction device 100 with automatic measurement mode icon 505-1 still selected. FIG. 5Y also shows representation 512 of a person standing in the front door. After the device detects that the person is within the view of one or more light sensors 164, the person is automatically measured. In some embodiments, a reticle (not shown in FIG. 5Y) is placed over representation 512 of person to position portable multifunction device 100 in a bounding line around representation 512 of person to indicate that the person can be measured.

[0183] In some embodiments, in response to automatically detecting that the person 512 is in view, a measurement animation showing the height of the person 512 is shown. FIGS. 5Z-5AB show an animation of how a representation of the height of the person 512 is added to the augmented reality view 500. FIG. 5Z shows a measurement marker 513 in the augmented reality view 500 on the floor where the person 512 is standing. In some embodiments, as the measurement marker 513 moves above the head of the person 512, a corresponding measured distance is displayed. FIG. 5AA shows the measurement marker 513 moving to approximately the waistline of a representation of the person 512, with the corresponding, partially measured height displayed. FIG. 5AB shows the animation completing as the measurement marker 513 moves above the head of the person 512. Once the animation is complete, the representation in the augmented reality view 500 of the corresponding measured distance stops changing. In some embodiments, as the measurement marker 513 moves closer to above the head of the person 512, the measurement animation moves at a slower pace.

[0184] 5AC-5AG depict an embodiment in which a metric other than (or in addition to) a person's height is measured. In FIG. 5AC, person 512 holds hands 514 apart, as if stating how big something is (e.g., "the fish is this big"). Portable multifunction device 100 automatically detects that person 512 is expressing a measurement while holding hands 514 apart. In response, portable multifunction device 100 causes augmented reality view 500 to display a first measurement (e.g., 3 inches) between hands 514 of person 512.

[0185] 5AD shows person 512 keeping hands 514 a different distance apart. Consequently, portable multifunction device 100 causes augmented reality view 500 to display a second measurement (e.g., 6 inches) between hands 514 of person 512. This change in measurement can occur in real time.

[0186] 5AE shows portable multifunction device 100 in another orientation, with person's head 515 in the view of one or more optical sensors 164. In some embodiments, interpupillary distance (“IPD”) 516 is automatically displayed in augmented reality view 500 when predetermined criteria (e.g., regarding the distance of one or more cameras from the user's face, or regarding the portion of augmented reality view 500 occupied by a representation of the person's face, or regarding the placement of reticle 504 on the person's face) are met.

[0187] 5AF shows portable multifunction device 100 in another orientation, with person's feet 517 in the view of one or more optical sensors 164. In some embodiments, foot size (or shoe size) 518 is displayed in augmented reality view 500 when predetermined criteria (e.g., relating to the distance of one or more cameras from user's feet 517, or relating to the portion of augmented reality view 500 occupied by a representation of the person's feet, or relating to the placement of reticle 504 on the person's feet) are met.

[0188] 5AG shows portable multifunction device 100 in another orientation, with dog 502-10 in the view of one or more optical sensors 164. In some embodiments, portable multifunction device 100 indicates that measurements of dog 503-10 can be fixed when predetermined criteria are met (e.g., relating to one or more camera distances from the dog, or relating to the portion of augmented reality view 500 occupied by a representation of the dog, or relating to the placement of reticle 504 over representation 503-10). In some embodiments, the measurements automatically indicated are chest circumference 518 of dog 503-10 and / or length 519 from chest to base of tail of dog 503-10. Optionally, additional or fewer measurements (e.g., only neck circumference) can be indicated.

[0189] While Figures 5AH-5BO depict a sofa and representations of the sofa, their various portions, and various annotations displayed with respect to the sofa, it should be understood that the sofa is merely one example of a three-dimensional physical object, and the various features displayed and depicted with respect to these figures are equally applicable to augmented reality views of a wide range of physical objects. Figure 5AH shows portable multifunction device 100 in another orientation, with sofa 502-3 within the view of one or more optical sensors 164. Figures 5AH, 5AI, and 5AJ show the animation that occurs when the device detects that reticle 504 is placed over a three-dimensional object. Figure 5AJ shows reticle 504 of Figures 5AH and 5AI replaced by (or modified in appearance to) bounding box 589. Figure 5AK shows tap input 520 on two-point measurement mode icon 505-4. Figure 5AL shows the response to tap input 520 on two-point measurement mode icon 505-4. FIG. 5AL shows that two-point measurement mode icon 505-4 is selected and reticle 504 visually changes to two-point measurement mode reticle 504-2 to reflect the changed mode.

[0190] FIG. 5AM shows portable multifunction device 100 with a sofa within the view of one or more optical sensors 164 of device 100. In some embodiments, the transition from FIG. 5AL to FIG. 5AM is caused by moving device 100 and its one or more cameras closer to the sofa and diagonally moving it to the side of the sofa so that two-point measurement mode reticle 504-2 appears near, but not on, an outer edge 521 of representation 503-3 of the sofa in augmented reality view 500. FIG. 5AN shows two-point measurement mode reticle 504-2 snapped to outer edge 521 of the sofa. This snap occurs when reticle 504-2 reaches a threshold distance to the detected edge in augmented reality view 500. FIG. 5AN shows that when reticle 504-2 is over outer edge 521, guide line(s) 522, also referred to herein as guides, are shown along outer edge 521.

[0191] FIG. 5AO shows portable multifunction device 100 with a sofa within the view of one or more optical sensors 164 of device 100. Additionally, two-point measurement mode reticle 504-2 is now displayed near, but not within, a threshold distance of, an outer corner 523 of representation 503-3 of the sofa. FIG. 5AP shows two-point measurement mode reticle 504-2 snapped to outer corner 523, an example anchor point. This snap occurs when reticle 504-2 is positioned within a threshold distance of a detected corner of the sofa in augmented reality view 500 (e.g., by moving device 100 and / or changing the orientation of device 100). FIG. 5AP shows that when reticle 504-2 is over an anchor point, such as outer corner 523 of the displayed representation of the sofa, guides or guide lines 524 are shown along the edge of the object corresponding to the anchor point, e.g., all edges of the sofa corresponding to outer corner 523. Note that corner 523 also corresponds to a feature where multiple surfaces of the sofa meet, and guide 524 shown in FIG. 5AP includes guides extending from corner 523, which is an anchor point, in a direction perpendicular to each of these surfaces.

[0192] Figure 5AQ shows an input 525 on the measurement anchor user interface object 508 that, when selected, drops a point onto the location of the two-point measurement mode reticle 504-2. In response to dropping a point onto the location of the two-point measurement mode reticle 504-2, additional guides (e.g., vertical guide 526-1, horizontal guide 526-2, and horizontal guide 526-3) that were not previously displayed are displayed in the augmented reality view 500 in front of the point being dropped, as shown in Figure 5AR. Note that in the example shown in Figure 5AR, the anchor point for guide 526 is located at the edge of the sofa where two surfaces of the sofa intersect (e.g., the top surface of the sofa's arm and the obscured vertical side of the sofa), and the guide 526 shown in Figure 5AR includes guides extending from the anchor point in a direction perpendicular to each of these surfaces.

[0193] Figure 5AS shows reticle 504-2 moving along horizontal guide 526-3, but when reticle 504-2 reaches another geometric feature 566 (e.g., the end of the arm of a sofa), additional guides are displayed. In Figure 5AS, additional vertical guide 567-1 and horizontal guide 567-2 are displayed.

[0194] 5AT shows reticle 504-2 moving along horizontal guide 526-3 (e.g., in response to movement of device 100), and in response to movement of reticle 504-2 past geometric feature 566 (FIG. 5AS), the display of other guides (e.g., vertical guide 567-1, horizontal guide 567-2-2) is stopped. Furthermore, as reticle 504-2 continues to move along horizontal guide 526-3, in response to corresponding movement of device 100, horizontal guide 526-3 is superimposed with measurement lines 527 that become progressively longer.

[0195] FIG. 5AU shows measurement line 527 and reticle 504-2 snapping to another outer edge 528 of sofa 503-3. In some embodiments, a new guide appears as reticle 504-2 snaps to another outer edge 528 of sofa 503-3. FIG. 5AU also shows input 529 in measurement anchoring user interface object 508 while reticle 504-2 is on another outer edge 528 of sofa 503-3. The display of measurement anchoring user interface object 508 in augmented reality view 500 signifies that measurement line 527 is no longer getting longer and the measurement is complete and can be anchored. FIG. 5AU also shows guide 568 to assist with potential future placement of reticle 504-2. FIG. 5AV shows that after the measurement is completed, guide 568 remains displayed to assist the user in placing the next measurement.

[0196] Figure 5AW shows displayed measurement 530. Measurement 530 corresponds to the final length of measurement line 527. Figure 5AW also shows reticle 504-2, which now does not overlay a representation of a measurable physical object in the physical environment, and as a result, no guide is displayed.

[0197] FIGURE 5AX shows portable multifunction device 100 including representation 503-3 of a couch within the view of one or more optical sensors 164 of device 100. Additionally, two-point measurement mode reticle 504-2 is now displayed on any edge of the couch, or couch cushion 531, away from couch cushion 531. While two-point measurement mode reticle 504-2 is selected, vertical guide 533 is displayed. FIGURE 5AY shows input 534 above measurement list button 506.

[0198] FIG. 5AZ shows an example of a response to input 534 on measurement list user interface object 506. For simplicity and to show a larger view of portable multifunction device 100, FIG. 5AZ removes physical environment 501 in the background of portable multifunction device 100. FIG. 5AZ displays a list of measurements taken during an annotation session (e.g., a session is a period of time during which an application is running in the foreground) in user interface 591. Within measurement user interface 590 is a scrollable list of measurements taken during the current session. When the scrollable list contains more measurements than can be simultaneously displayed, only a subset of the measurements taken are displayed at a time. In the example shown in FIG. 5AZ, the measurements currently displayed are: "Linear Measurements" user interface object 591-2; "Couch Measurements" user interface object 591-3; "Pet Measurements" user interface object 591-4; and "Hand Separation" user interface object 591-5. In some embodiments, names for measurements are automatically generated and included in user interface 591 based on the detected object being measured. For example, the device can communicate when a pet, sofa, shoe, or other object is being measured and name the measurement appropriately (e.g., "Dog Measurement," "Sofa Measurement," "Shoe Measurement," or "Pet Measurement"). In some embodiments, each measurement in user interface 591 also includes an icon indicating what type of measurement was taken (e.g., a 3D measurement, a 2D measurement, a 2D bending measurement, etc.).

[0199] FIG. 5BA shows an input 592 on a “Pet Measurements” user interface object 591-4 located within a list of measurements in user interface 591. FIG. 5BB shows a detailed object measurements (e.g., “Pet Measurements”) user interface 593 displayed in response to input 592 on a respective measurement list user interface object (e.g., “Pet Measurements” user interface object 591-4). The “Pet Measurements” user interface 593 includes an editable measurement name section 593-1. To edit the name of a measurement, input can be made into edit user interface icon 593-2 to edit the measurement name. In some embodiments, measurements can be editable without first clicking an icon. FIG. 5BB also shows a list of measurements (e.g., pet measurements) 593-3. In some embodiments, additional measurements not previously displayed in the list of measurements user interface 591 are displayed in the detailed object measurements user interface 593. In this example, the measurements displayed in the detailed object measurements user interface 593 include chest diameter; chest to tail, head height, and collar size. Additionally, a "Back to List" user interface object 593-4 is displayed which, when selected, displays the previous user interface, as shown in Figure 5BA. There is also a share user interface object 593-5 for sharing this and other measurements with others (e.g., via email or text message). Finally, there is also an exit icon 593-6 for closing the list of measurements user interface 591 or the "Pet Measurements" user interface 593.

[0200] Figure 5BC shows user interface icon 593-2 (obscured by input 594) input 594 for editing the measurement name. Figure 5BD shows that user interface icon 593-2 for editing the measurement name has changed to a "Done" user interface object 593-7, indicating that the measurement name can now be edited. Figures 5BE and 5BF show editable measurement name section 593-1 being re-edited from "Pet's Measurements" to "Lola's Size." In some embodiments, the re-editing occurs in response to input received on a virtual keyboard (not shown in these figures) displayed over at least a portion of the displayed user interface or displayed simultaneously with the re-scaled user interface 593. In some embodiments, an external keyboard can be used.

[0201] Figure 5BF shows input 594 being received at "Done" user interface object 593-7. In response to receiving the input at "Done" user interface object 593-7, edit user interface icon 593-2 replaces "Done" user interface object 593-7, as shown in Figure 5BG.

[0202] 5BH and 5BI show swipe gestures 595-1 and 595-2 (in this example, an upward swipe gesture). In response to the swipe gesture, e.g., swipe gestures 595-1 and 595-2, a user interface element displaying additional options 593-8 is revealed, as shown in FIG. 5BJ. The resulting user interface (an example is shown in FIG. 5BJ) includes additional options (e.g., user interface objects) such as a “Save to Photo” user interface object 593-9 for saving the measurement as a photo for viewing within a photo viewing / editing application; a “Save to File” user interface object 593-10 for saving the measurement to a file system, either locally or remotely; and / or a “Share Measurement” user interface object 593-11 for sharing the measurement (e.g., via email or text message).

[0203] Figure 5BK shows input 596 occurring on "Return to List" user interface object 593-4 (which is partially obscured by input 596). Figure 5BL shows an example of a response to selection of "Return to List" user interface object 593-4. Figure 5BL shows the list of measurements user interface 591, where "Pet Measurements" user interface object 591-4 has now been re-edited to "Lola's Size."

[0204] FIG. 5BM illustrates a swipe gesture, leftward in this example, over a respective user interface element (e.g., over the “hand separation measurements” user interface object 591-5) in the list of measurements user interface 591. While a leftward swipe 597-1 is shown, it should be understood that any other suitable swipe direction can also be used. In some embodiments, a swipe in a predetermined direction, such as leftward, can be used to detect the respective measurements. FIG. 5BN illustrates a leftward swipe 597-2 followed by a leftward swipe. As the respective user interface object 591-5, on which a swipe gesture is being performed, follows the swipe and moves in a leftward direction 597-2, a removal notification 598 begins to display. In some embodiments, the removal notification 598 replaces the entire respective user interface object (e.g., the “hand separation” user interface object 591-5) once it has been swiped completely out of view.

[0205] FIG. 5B0 shows that after a respective user interface object (e.g., "Hand Separation" user interface object 591-5) is swiped off the display (e.g., removed or deleted), and therefore removed from the list of measurements in user interface 591, the measurements that were below it in the list of measurements now appear in the position previously occupied by the removed (respective) user interface object. For example, FIG. 5B0 shows that "Shoe Size" user interface element 599 is now displayed in place of "Hand Separation Measurement" user interface object 591-5.

[0206] FIG. 5BP illustrates a redisplay of the home screen user interface in response to editing the measurement application 448.

[0207] 5BQ shows another input, input 535, directed to the measurement application icon 448 to restart the measurement application. After restarting the measurement application, the user interface 536 of the measurement application 448 is displayed, for example, as shown in FIG. 5BR. In some embodiments, closing the measurement application 448 marks the end of the annotation session, sometimes referred to as a measurement session, which in turn causes the list of annotations (e.g., measurements) captured during that session to be cleared. In some embodiments, restarting the measurement application starts a new annotation session.

[0208] 5BR illustrates a user interface 536 of the measurement application 448, including an augmented reality view 500 that includes a view of a portion of a three-dimensional physical environment 501 that is within the field of view of one or more cameras of the device 100. As shown in FIG. 5BR, the measurement application is operating in automatic measurement mode, as indicated by the selection indicator 537 above the automatic measurement mode icon 505-1.

[0209] FIG. 5BS shows input 538 directed to 3D measurement mode icon 505-2, which corresponds to the 3D measurement mode of the measurement application.

[0210] Figure 5BT shows a user interface 536 of a measurement application operating in a 3D measurement mode in response to input 538. In some embodiments, as shown in Figure 5BT, operation in the 3D measurement mode is indicated by a selection indicator 537 above the 3D measurement mode icon 505-2. As also shown in Figure 5BT, in some embodiments, operation in the 3D measurement mode is indicated by a reticle 504 having an appearance associated with 3D measurement, such as the six-section appearance shown in Figure 5BT.

[0211] FIG. 5BU shows device 100 (e.g., one or more cameras of device 100) in a different position relative to physical environment 501 in FIG. 5BT, e.g., in response to movement of the device from the position shown in FIG. 5BT to the position shown in FIG. 5BU. Accordingly, the live view of one or more cameras displayed in user interface 536 has been updated to reflect the portion of physical environment 501 that is now within the field of view of the one or more cameras. In FIG. 5BU, annotation placement user interface elements (e.g., reticle 504, and more specifically, focal point 504-1 within reticle 504) are over a portion of representation 503-5 of table 502-5. Table 502-5 is entirely within the view of one or more cameras of device 100.

[0212] 5BV-5BX show device 100 initiating a three-dimensional measurement of table 502-5 according to one or more cameras of device 100 positioned such that reticle 504 is displayed over a portion of a representation of a three-dimensional object, such as table 502-5, that is measurable by device 100 (and optionally according to a determination that user interface 536 includes a representation of all or substantially all of table 502-5). For example, FIGS. 5BV-5BX show an animation progression in which reticle 504 is transformed into indication 565-2 (e.g., a first indication including a mark at a corner of the three-dimensional object, as shown in FIG. 5BW, and / or a second indication, such as bounding box 565-3, as shown in FIG. 5BX) superimposed on representation 503-5 of table 502-5, indicating that one or more measurements of table 502-5 can be made by device 100.

[0213] Figure 5BY shows device 100 in a different position relative to physical environment 501 than Figure 5BT, e.g., in response to device 100 being moved from the position shown in Figure 5BT to the position shown in Figure 5BX and then to the position shown in Figure 5BY. For example, device 100 is closer to table 502-5 (which is obscured by device 100) in Figure 5BY than in Figure 5BX. Accordingly, the live view of one or more cameras displayed in user interface 536 has been updated to reflect the portions of physical environment 501 that are now within the field of view of the one or more cameras. In Figure 5BY, less of table 502-5 is within the field of view of the one or more cameras than when device 100 is in the position shown in Figure 5BX. In some embodiments, the measurement application automatically transitions to a two-dimensional measurement mode in accordance with a reticle 504 displayed over a portion of a sub-representation of table 503-5, where sub-representation 503-5 includes a view of at least a threshold amount of the two-dimensional surface (e.g., the tabletop) of table 502-5 without including a view of at least a threshold amount of a third dimension of the table.

[0214] 5BZ and 5CA show device 100 at a similar distance to table 502-5 as in FIG. 5BY, and moved slightly to the right so that reticle 504 (e.g., more specifically, focal point 504-1 of reticle 504, as shown in FIG. 5CA) is over a portion of representation 503-7 (obscured by device 100) of magazine 502-7 located on table 502-5. Magazine 502-7 is entirely within the view of one or more cameras of device 100.

[0215] 5CA illustrates that, upon following one or more cameras of device 100 positioned such that reticle 504, including focal point 504-1, is displayed over a portion of a representation of a two-dimensional feature measurable by device 100, e.g., magazine 502-7 (and optionally following a determination that user interface 536 includes a representation of all or substantially all of the two-dimensional feature), the measurement application transitions to a two-dimensional measurement mode (e.g., from a three-dimensional measurement mode, as shown in FIG. 5BZ ), as indicated, for example, by selection indicator 537 over two-dimensional measurement mode icon 505-3, and as another example, by reticle 504 having an appearance associated with two-dimensional measurement, such as the four-segmented appearance shown in FIG. 5CA . Note that while magazine 502-7 is a three-dimensional object in physical environment 501, magazine 502-7 is sufficiently thin that a two-dimensional region of a page of magazine 502-7 is more likely to be of interest to a user of device 100 than a three-dimensional collection of measurements of magazine 502-7. Thus, in some embodiments, a thin three-dimensional object such as magazine 502-7 is considered by device 100 to be a two-dimensional feature rather than a three-dimensional object.

[0216] 5CA-5CD show device 100 beginning to make two-dimensional measurements of magazine 502-7 according to one or more cameras of device 100, with reticle 504 positioned so that it appears over a portion of magazine representation 503-7. For example, FIGS. 5CB-5CD show a progression of animations in which reticle 504 transforms into an indication (e.g., as shown by indication 565-4 in FIG. 5CC and by indication 565-5 in FIG. 5CD) that is superimposed on representation 503-7 of magazine 502-7, indicating that one or more measurements of magazine 502-7 can be made by device 100.

[0217] FIG. 5CE shows input 539 directed to two-point (eg, one-dimensional) measurement mode icon 505-4, which corresponds to a one-dimensional measurement mode of the measurement application.

[0218] In response to input 539, the measurement application transitions to the one-dimensional measurement mode. Accordingly, in FIG. 5CF, a selection indicator 537 is displayed over two-point measurement mode icon 505-4. FIG. 5CF also illustrates a transition of the measurement application's user interface 552 in response to input 539. According to some embodiments, an animated transition is displayed while the measurement application transitions from one measurement mode (e.g., the measurement mode in which the measurement application is running when the manual selection is received) to another measurement mode (e.g., the selected measurement mode) in response to manual selection of the measurement mode by selecting the corresponding icon. In some embodiments, animated transitions between measurement modes are not displayed during automatic transitions between measurement modes (e.g., no animated transition was displayed during the automatic transition from the three-dimensional measurement mode of FIG. 5BZ to the two-dimensional measurement mode of FIG. 5CA). In some embodiments, the displayed transition includes blurring one or more portions of the user interface 536. 5CF, the displayed transition includes blurring and / or blurring of representations of the fields of view of one or more cameras of device 100 (e.g., optionally without the blurring and / or blurring of the measurement application controls). Additionally, in some embodiments, as in the exemplary transition in FIG. 5CB, reticle 504 ceases to be displayed during the transition to a manually selected measurement mode.

[0219] 5CG shows a user interface 536 of a measurement application operating in one-dimensional measurement mode. Thus, the reticle 504 is displayed with an appearance associated with one-dimensional (e.g., two-point) measurement, such as the two-segmented appearance shown in FIG.

[0220] Figure 5CH shows device 100 in a different position relative to physical environment 501 than in Figure 5CG, e.g., due to movement of device 100 from the location shown in Figure 5CG to the position shown in Figure 5CH. Accordingly, the live view of one or more cameras displayed in user interface 536 has been updated to reflect the portion of physical environment 501 that is now within the field of view of the one or more cameras. In Figure 5CH, device 100 is positioned such that reticle 504 (e.g., more specifically, focal point 504-1 of reticle 504) is over a portion of representation 503-6 of lamp 502-6.

[0221] 5CI shows input 540 directed to annotation creation button 508 to initiate the creation of a measurement annotation, with the current position of focal point 504-1 of reticle 504 indicating the starting point for the one-dimensional measurement. In the example shown in FIG. 5CH, the starting point is the position in physical environment 501 at the foot of lamp 541.

[0222] 5CJ illustrates adding an annotation (e.g., a measurement annotation) within user interface 536, as described with reference to FIG. 5CI, that is a representation of a measurement 546 extending from a starting point and extending to a current point in physical environment 501 corresponding to focal point 504-1 of reticle 504. Measurement representation 546 includes measurement segment 547 and label 548. Measurement segment 547 is displayed with an appearance (e.g., a dashed line) that indicates that the measurement is still in progress (e.g., the endpoint of a one-dimensional measurement has not yet been identified). Label 548 indicates the current length of the measurement (e.g., the distance between the starting point and the current point in physical environment 501).

[0223] Figure 5CK shows input 540 directed to create annotation button 508 to finish or complete the creation of a measurement annotation, with focal point 501-4 of reticle 504 identifying the endpoint of the one-dimensional measurement that began in Figure 5CH and continued in Figure 5CJ. In the example shown in Figure 5CK, the endpoint is a location within physical environment 501 at the top of ramp 541.

[0224] FIG. 5CL shows a completed representation 542 of the measurement in response to input 540 while the focal point 501-4 of the reticle 504 identifies the endpoint for the measurement, as the measurement segment 543 is displayed with an appearance (e.g., a filled appearance) indicating that the measurement has been completed.

[0225] FIG. 5CM shows input 544 directed to the measurement list button 506 .

[0226] In FIG. 5CN, the background physical environment 501 of device 100 has been removed for simplicity and to show a larger view of device 100. In FIG. 5CN, in response to input 544, device 100 displays a measurement list user interface 545. In the example shown in FIG. 5CN, measurement list user interface 545 is displayed over a portion of the measurement application's user interface 536, simultaneously with one or more other portions of the user interface 536, such as controls for the measurement application and / or live view of one or more cameras (e.g., mode indication user interface 505, measurement list button 506, undo user interface object 507, measurement freeze user interface object 508, and record user interface object 550). To resume the measurement application described herein with reference to FIG. 5BQ, the measurement list user interface 545 in FIG. 5CN lists only measurements made during the current measurement session. Thus, the measurement list user interface 545 in FIG. 5CN includes line measurements 543 (described with reference to 5CJ-5CL). The measurement list user interface 536 of FIG. 5CN does not include any of the measurements described with reference to FIGS. 5A-5BO that were made during a previous measurement session that was previously completed (e.g., as described with reference to FIGS. 5BP and 5BQ).

[0227] 6A-6Y show exemplary user interfaces for measuring non-linear surfaces (eg, curved surfaces) and retrieving measurement information from previously captured media items according to some embodiments.

[0228] FIG. 6A illustrates selection of automatic measurement mode icon 605-1 via input 650. When selected, automatic measurement mode icon 605-1 can cause portable multifunction device 100 to automatically detect a surface or object (e.g., a three-dimensional object, a two-dimensional surface, etc.) over which a reticle is located, and based on that determination, the device can automatically indicate that the surface or object is measurable. Three-dimensional measurement mode icon 605-2, two-dimensional surface measurement mode icon 605-3, and two-point measurement mode icon 605-4, when selected, allow a user to measure in three dimensions (e.g., length, width, and height), two dimensions (e.g., two dimensions selected from length, width, or height), or from one point to another (e.g., the length of a table edge), respectively. In some embodiments, floor plan measurement mode icon 605-5 is also present, which, when selected, allows a user to scan three-dimensional physical environment 601 to create a floor plan (e.g., a bird's-eye view). In some embodiments, any combination of these mode icons can be displayed. For example, in some embodiments, only the automatic mode and floorplan mode may be displayed, in some embodiments, additional mode icons may be added, such as a curved surface mode.

[0229] 6B illustrates some aspects of device operation after automatic measurement mode icon 605-1 is detected. FIG. 6B shows portable multifunction device 100 displaying augmented reality view 600. Augmented reality view 600 is created based on three-dimensional physical environment 601 that is within the field of view of one or more optical sensors 164. This augmented reality view 600 is used to display annotations (e.g., measurements, bounding boxes, etc.) on or near objects found in three-dimensional physical environment 601.

[0230] 6B also shows that three-dimensional physical environment 601 includes a plurality of items, such as television stand 602-1, television (and / or any other type of display) 602-2, sofa 602-3, pillows 602-4, table 602-5, lamp 602-6, and magazine 602-7. Augmented reality view 600 includes corresponding representations of the plurality of items displayed in three-dimensional physical environment 601. In this example, the corresponding representations of the plurality of items include representations of television stand 603-1, television 603-2, sofa 603-3, pillows 603-4, table 603-5, lamp 603-6, and magazine 603-7. Augmented reality view 600 shown on portable multifunction device 100 has a plurality of user interface objects overlaid on augmented reality view 600. These user interface objects are used to control what is being displayed in augmented reality and also to help the user capture and overlay measurements (e.g., metrics) of objects shown in the three-dimensional physical environment 601. In some embodiments, one of the user interface objects is an annotation placement user interface (e.g., reticle 604) that shows the user where a measurement will start if inserted (e.g., dropped).

[0231] Specifically, Figure 6B shows that the two-point measurement mode icon 605-4 is now not selected, and the selection indicator 635 is now over the automatic measurement mode icon 605-1. In some embodiments, the reticle 604 visually changes depending on which mode the user selects, indicating to the user which mode is currently being used. Figure 6B also shows a partially curved guide 636 that matches the surface 638 of a sofa cushion, with the surface continuing in three-dimensional space.

[0232] FIG. 6C shows input 640 on measurement anchor user interface object 608 while automatic measurement mode icon 605-1 is selected.

[0233] 6D shows a response to input 640 on measurement anchored user interface object 608. In response to input 640 on measurement anchored user interface object 608, the partially curved guide 636, which conformed to the surface 638 of the sofa cushion, is replaced with a fully curved guide 642. This curved guide 642 is suitable for measuring along the curve of the sofa, as it covers a surface that conforms to the curves of the sofa's surface, as opposed to a straight (e.g., linear) guide line for measuring distance from one point to another (e.g., a curved measurement that takes into account the irregular shape of a physical surface will be longer than a straight, two-point measurement).

[0234] Figure 6E shows reticle 604 moving along a fully curved guide 642, moving from the vertical surface or plane of Figure 6D to the horizontal surface or plane of Figure 6E. Additionally, checkmarks 644-1 are also optionally displayed along at least a portion of curved guide 642. Optionally, a live measurement readout 646 can also be displayed to indicate the distance from the beginning of guide 642 to the current of reticle 604.

[0235] 6F shows reticle 604 continuing to move along a complete curved guide 642, with checkmarks 644-2 displayed along a portion of curved guide 642 between the beginning of the guide and the current position of reticle focal point 504-1. Optionally, a live measurement readout 647 may also be displayed to indicate the distance from the beginning of guide 642 to the current position of reticle 604.

[0236] Figure 6G shows an input 640 on the measurement anchored user interface object 608. Such an input indicates that the measurement is complete. Figure 6H shows a response to the input 640 on the measurement anchored user interface object 608. Specifically, Figure 6H shows a anchored measurement 648, sometimes referred to as a "completed measurement."

[0237] 6I-6Y show how measurements can be viewed among stored media items (e.g., photos, videos, etc.) FIG. 6I shows device 100 returning to the home screen, including photo application icon 428.

[0238] FIG. 6J illustrates input 651 on media application icon 428. FIG. 6K illustrates a photo application user interface 653 being displayed on device 100 in response to input 651 on the media application icon. Within media application user interface 653, a first media item 655-1 is selected and displayed in an expanded view. Additionally, a thumbnail scrubber bar 657 is displayed below first media item 655-1. Scrubber bar 657 is used to display thumbnail images of the media items (e.g., first media item thumbnail 656-1, second media item thumbnail 656-2, third media item thumbnail 656-3, and fourth media item thumbnail 656-4), which correspond to first media item 655-1, second media item 655-2 (see FIG. 6Q), third media item 655-3 (see FIG. 6T), and fourth media item 655-4 (see FIG. 6X), respectively. Each one of these thumbnail images, when selected, causes device 100 to display the selected media item in an expanded view.

[0239] 6K also shows several user interface objects for interacting with user interface 653. First, a share button 658 is displayed to share a media item. Next, a display measurements button 660 is displayed, which, when selected, displays all measurements available for the expanded media item. Next, a like button 660 is displayed to allow the user to select a media item they like. Next, a delete icon 662 is displayed, which, when selected, causes the media in view (e.g., the expanded media item) to be deleted.

[0240] FIG. 6K also shows that within the first media item 655-1, a partial view of TV stand 603-1 is shown, sofa 603-3 is fully visible, table 603-5 is partially visible, lamp 603-6 is fully visible, and magazine 603-7 is also fully visible.

[0241] FIG. 6L shows an input 666 on the measure button 660. FIG. 6M shows that, in response to receiving the input 666, (1) the measure button 660 is displayed in a selected (activated) state, and (2) the first media item 655-1 is overlaid with all available measurements, e.g., measurements based on the first media item, including information such as depth information for physical features in the portion of the physical environment represented by the first media item. In some embodiments, the measure button 660 can be toggled to either display or hide the available measurements. Specifically, the measure button 660 is now surrounded by a selection indicator 670. The available measurements for the first media item include a sofa measurement 672.

[0242] Figure 6N shows an input 674 on table 603-5 to provide additional measurements. Figure 6O shows that in response to input 674, no additional measurements are shown in media application user interface 653. In some embodiments, no additional measurements are shown in response to input 674 because all measurements are displayed when measurement button 660 is selected.

[0243] FIG. 6P shows an input 676 on the second media item thumbnail 656-2 to bring the second media item 655-2 in an expanded state. FIG. 6Q shows the second media item 655-2 replacing the display of the first media item 655-1. Furthermore, the measurement button 660 is deactivated when another media item is displayed. In some embodiments, optionally, the measurement button 660 can be persistently activated when changing through media items (e.g., available measurements are automatically displayed as each media item is selected). FIG. 6Q also shows an input 678 on the measurement button 660 to cause measurements to be displayed on the second media item 655-2. The second media item 655-2 is a media item that corresponds to a different part of the same physical environment as the first media item 655-1. A second media item 655-2 includes a partial view of sofa 603-3 and a full view of table 603-5, lamp 603-6, and magazine 603-7. Figure 6Q also shows that scrubber bar 657 has been scrolled from its position in Figures 6K-6P to indicate that second media item thumbnail 656-2 has been selected.

[0244] Figure 6R shows that in response to input 678, available measurements are displayed in a second media item 655-2, where the media item includes information, such as depth information, sufficient to determine measurements 680 for only the table 603-5 and lamp 603-6. Figure 6R also shows that the measurement button 660 is now surrounded by a selection indicator 670.

[0245] FIG. 6S shows an input 682 on the third media item thumbnail 656-3 to bring the third media item 655-3 in an expanded state. FIG. 6T shows the third media item 655-3 replacing the display of the second media item 655-2. Furthermore, the measurement button 660 is deactivated when another media item is displayed. In some embodiments, optionally, the measurement button 660 can be persistently activated when changing through media items (e.g., available measurements are automatically displayed as each media item is selected). The third media item 655-3 is a media item in a different part of the same physical environment as the first media item 655-1 and the second media item 655-2. The third media item 655-3 includes a partial display of the table 603-5 and does not include the other objects shown in the first media item 655-1 and the second media item 655-2. FIG. 6T also shows that the scrubber bar 657 has been scrolled to a new position to indicate that the third media item thumbnail 656-3 has been selected.

[0246] Figure 6U also shows an input 684 on the measure button 660 to cause measurements to be displayed on the third media item 655-2. Figure 6V shows that available measurements are displayed in response to the input 684. The third media item 655-3 includes information, such as depth information, sufficient to determine measurements for only a portion of table 603-5, allowing measurements 686 in table 603-5 to be determined and displayed in response to the input 684. Figure 6V also shows that the measure button 660 is now surrounded by a selection indicator 670.

[0247] Figure 6W shows an input 688 on the fourth media item thumbnail 656-4 to bring the fourth media item 655-4 in an expanded state. Figure 6X shows the fourth media item 655-4 replacing the display of the third media item 655-3. The fourth media item 655-4 is a media item in a different physical environment than the physical environments of the first media item 655-1, the second media item 655-2, and the third media item 655-3. The fourth media item 655-4 includes a representation of a building 692.

[0248] Figure 6Y shows that available measurements are displayed in response to input 690. In this fourth media item 655-4, information about the physical features of building 692, such as depth information, is recorded to enable the determination and display of measurements 694. Figure 6Y also shows that measurement button 660 is now surrounded by selection indicator 670.

[0249] 7A-7AT show exemplary user interfaces for scanning and modeling an environment and interacting with the generated schematic representation, according to some embodiments.

[0250] Figure 7A illustrates an exemplary transition from Figure 5CG. In particular, Figure 7A illustrates a user interface 700 of a measurement application while the measurement application is running in one-dimensional measurement mode, as indicated by a selection indicator 537 displayed (e.g., outlined) over two-point measurement mode icon 505-4. Figure 7A also illustrates an input 702 (e.g., a swipe gesture) at a position on user interface 700 that corresponds to a live view of one or more cameras of device 100 (e.g., and does not correspond to any control of the measurement application).

[0251] In response to input 702 (FIG. 7A), the measurement application transitions to the next measurement mode in mode indication user interface 705. In the example shown in FIG. 7A, the next measurement mode after the two-point measurement mode is floor plan modeling mode (also referred to herein as "floor plan measurement mode"), represented by floor plan modeling mode icon 505-5. Accordingly, FIG. 7B shows selection indicator 537 displayed above floor plan modeling mode icon 505-5. Additionally, in some embodiments, the floor plan measurement mode includes a start / stop scan button 704 that, when activated, toggles an environmental scan (e.g., while a scan is not in progress, start / stop scan button 704 appears as a start scan button to start a scan, and while a scan is in progress, start / stop scan button 704 appears as a stop scan button to end the scan).

[0252] 7B also illustrates an exemplary transition of user interface 700 (e.g., in response to user input rather than automatically based on objects detected in the field of view of one or more cameras) according to some embodiments in which an animated transition is displayed to transition the measurement application from one measurement mode to another (e.g., during the transition). In the example of FIG. 7B, the displayed transition includes temporarily (e.g., for a time period of less than 1 second, or less than 2 seconds) blurring and / or obscuring the live view of one or more cameras of device 100, as described herein with reference to FIG. 5CF.

[0253] 7C-7D show the transition of reticle 706 from an appearance associated with one-dimensional measurement (FIG. 7A) to an appearance associated with floorplan modeling as the metrology application transitions to floorplan modeling mode. In the example shown in FIG. 7C, reticle 706 changes to a four-segmented appearance with the segments moving outward from the center of 706, and in the example shown in FIG. 7D, the four segments of reticle 706 move outward to the corners of a rectangle or rectangle.

[0254] 7D-7G show exemplary reticle animations illustrating the process of scanning a physical environment and progressively mapping out a floor plan 708 of the physical environment while the user moves around the physical environment. In some embodiments, the reticle animation is displayed when transitioning from the metrology application to floor plan modeling mode before scanning the physical environment. In some embodiments, the reticle animation loops repeatedly (e.g., after all reticle animations have played, as shown in FIGS. 7D, 7E, and 7F, the reticle animation restarts from the beginning, as shown in FIG. 7G). In some embodiments, the reticle animation plays while the metrology application is in floor plan modeling mode and whenever a scan of the physical environment is not in progress (e.g., the reticle animation is replayed after a scan of the physical environment has finished and before a subsequent scan of the physical environment begins). In some embodiments, the user is presented with instructions to move around the physical environment (e.g., an animation of the user's representation moving around a representation of the physical environment).

[0255] FIG. 7G shows an input 710 commanding the start scan button 704-1 to initiate a scan of the physical environment 701.

[0256] In response to input 710 ( FIG. 7G ), scanning of physical environment 701 begins to capture information indicative of physical environment 701 (e.g., depth or other topological information) that can be used to generate a floor plan. In FIG. 7H , the portion of physical environment 701 that was scanned (e.g., from which information was captured) is shown by an overlay 712 displayed on a representation of the field of view of one or more cameras in user interface 700. Overlay 712 provides a first-person perspective view of the portion(s) of scanned physical environment 701. Additionally, inset 714 includes map 716 that provides a different view (e.g., an overhead view) of the portion(s) of scanned physical environment 701. Map 716 optionally indicates current camera position 718 and / or current camera field of view 720 of one or more cameras of device 100 relative to map 716, and thus relative to physical environment 701. Additionally, as shown in FIG. 7H, in some embodiments, large objects in the physical environment 701, such as television 502-2 and television stand 502-1 (e.g., the edges of these objects), are visually emphasized (e.g., highlighted) in the overlay 712 of the user interface 700.

[0257] 7I illustrates a transition from FIG. 7H as device 100 (e.g., one or more cameras of device 100) moves relative to physical environment 701 (e.g., from facing television 502-2 to changing orientation to face sofa 502-3). As device 100 moves, device 100 captures information indicative of additional portions of physical environment 701, as indicated by the expansion of overlay 712 of user interface 700 and, in some embodiments, as reflected by updating map 722 in inset 714 to include a representation of the additional captured portions of physical environment 701. For example, sofa 502-3 (e.g., an edge of sofa 502-3) is visually emphasized within overlay 712 of user interface 700. The edges of television 502-2 and television stand 502-1 are also visually emphasized, but in this example, to a lesser extent than sofa 502-3 and fade over time because these physical features were captured first by the device. The visual emphasis of sofa 502-3, which initially appears in Figure 7H with the same degree of visual emphasis as television 502-2 and television stand 502-1, also fades over time, as shown in Figures 7K-7L. Furthermore, in the example shown in Figure 7I, map 722 translates and rotates within inset 714 as device 100 moves and rotates within physical environment 701, while camera location 718 and camera field of view 720 remain stationary relative to inset 714.

[0258] Figure 7J shows an alternative transition from Figure 7H. In contrast to Figure 71, the map 716 in Figure 7J does not rotate within the inset 714. Instead, the map 716 is displayed in the inset 714 of Figure 7J in the same orientation as in Figure 7H, and the camera orientation 718 and camera field of view 720 change relative to the map 716, and therefore relative to the inset 714, to reflect the horizontal translation and rotation of the device 100 within the physical environment 701.

[0259] FIG. 7K shows a change in the field of view of one or more cameras from the field of view of one or more cameras in FIG. 7H or 7J due to ball 724 moving through physical environment 701 and physical objects (e.g., ball 724) that enter the field of view of one or more cameras, as reflected in the live view of one or more cameras displayed on user interface 700 of device 100.

[0260] 7L shows that in response to detecting movement of a physical object (e.g., ball 724) in physical environment 701, device 100 displays an alert 726 (e.g., "Do not move things in the room") instructing the user not to move objects in physical environment 701 while scanning. Objects that move in the physical environment while the device is scanning the physical environment prevent accurate capture of information about the physical environment, and additionally, the objects themselves are difficult to scan, especially when the objects are moving quickly, thus preventing accurate floor plan generation.

[0261] Figure 7M shows the result of continued movement of device 100 in physical environment 701, e.g., the transition in Figure 7L, and the capture of information about further portions of physical environment 701, including further expansion of overlay 712 in user interface 700 and further updating of map 716 in inset 714. Figure 7M also shows the edges of table 502-5 visually highlighted in overlay 712 of user interface 700. The visual highlighting decreases over time, as shown in Figure 7N.

[0262] In Figure 7M, the portion of physical environment 701 within the field of view of one or more cameras of device 100 includes a mirror 728. In Figure 7N, following a determination that the field of view of one or more cameras includes a mirror (e.g., following a determination that a representation of the field of view of one or more cameras includes a representation of a mirror), device 100 displays alert 730 (e.g., "Do not point at mirrors") instructing the user not to point one or more cameras (e.g., focal points of one or more cameras corresponding to a center point or area of ​​the display of device 100) toward mirrors in the physical environment during scanning.

[0263] FIG. 7O shows the result of continued movement of device 100 in physical environment 701, for example, in the transition in FIG. 7M, and the capture of information about further portions of physical environment 701, including further expansion of overlay 712 in user interface 700 and further updating of map 716 in inset 714.

[0264] 7P illustrates the results of continued movement of device 100 in physical environment 701, e.g., the transition in FIG. 7O, and the capture of information about further portions of physical environment 701, including further expansion of overlay 712 in user interface 700 and further updating of map 716 in inset 714. Following a determination that device 100 (e.g., one or more cameras of device 100) is moving too quickly to allow the device to capture accurate information about physical environment 701, device 100 displays alert 732 (e.g., "Slow down") instructing the user to slow the movement of device 100 and allow device 100 time to capture information about physical environment 701.

[0265] FIG. 7Q shows the result of continued and slowed movement of device 100 in physical environment 701, for example, at the transition in FIG. 7P, and the capture of information about further portions of physical environment 701, including further expansion of overlay 712 in user interface 700 and further updating of map 716 in inset 714.

[0266] In FIG. 7R, device 100 displays alert 734 (e.g., "Please go back and scan the missing spots") following a determination that one or more cameras have moved past a portion of physical environment 701 where information was not fully captured due to one or more cameras moving too quickly, as indicated by missing portion 736 of map 716 between FIG. 7O and FIG. 7P.

[0267] FIG. 7S shows that, in response to alert 734 (FIG. 7R), device 100 is moved by the user back to a position in physical environment 701 between the position of device 100 in FIG. 7O and the position of device 100 in FIG. 7P, as indicated by camera orientation 718, to again attempt to capture information about the missing portion of physical environment 701. Note that in FIGS. 7P-7R, even though information about physical environment 701 was at least partially missing, overlay 712 appears as a continuous region. That is, in some embodiments, an alert such as alert 734 is provided to instruct the user to rescan the missing portion of physical environment 701, rather than displaying holes or gaps in overlay 712 to indicate the area of ​​missing information.

[0268] In FIG. 7S, the portion of physical environment 701 that is within the field of one or more cameras of device 100 includes an exit to the current room in which device 100 (e.g., its one or more cameras) is currently located. In some embodiments, device 100 allows the user to mark an entrance between the room in which device 100 is currently located (e.g., the room in which the user is holding device 100) and an adjacent room. An exemplary mechanism by which a user can mark an entrance between adjacent rooms is shown in FIG. 7T. Here, device 100 displays a prompt 738 requesting the user to indicate whether the user intends to move from the current room in physical environment 701 to another room through doorway 740. FIG. 7U shows input 742 directed to a "No" button 744, declining the user's move to the adjacent room.

[0269] Figure 7V shows the results of continued movement of device 100 through physical environment 701. In Figure 7V, one or more cameras of device 100 are located near a wall 746 of physical environment 701, as indicated by a large visible window 748 in the live view of one or more cameras of user interface 700 and by a camera orientation 718 in map 716 near a representation 750 of wall 746, with the camera's field of view 720 facing toward the representation 750 of wall 746. In Figure 7W, following a determination that an object in the field of view of one or more cameras is too close to one or more cameras, device 100 displays an alert 749 (e.g., "Too Close") prompting the user to move one or more cameras away from the object in the field of view, as shown in Figure 7X.

[0270] FIG. 7Y illustrates the result of continued movement of device 100 through the transition in physical environment 701, e.g., FIG. 7S. In FIG. 7Y, the portion of physical environment 701 within the field of view of one or more cameras of device 100 includes stairwell 747, which is an exit for the current room in which device 100 is currently located. In some embodiments, following a determination that the field of view of one or more cameras includes an exit for the current room (e.g., an exterior exit or stairs to a floor of a different building) other than an entrance to an adjacent (e.g., interior) room, e.g., stairwell 747, device 100 displays an alert instructing the user not to leave the current room through that exit during scanning. In the example shown in FIG. 7Z, device 100 displays alert 745 (e.g., "Do not go up") instructing the user not to go to an upper floor of the building through stairwell 747 during scanning. In another example, device 100 displays an alert (e.g., "Do not go down") instructing the user not to go to a lower floor of the building through the stairs during scanning. In yet another example, device 100 displays an alert to the user (e.g., "DO NOT GO OUTSIDE") instructing the user not to go outdoors while scanning.

[0271] FIG. 7AA shows the result of continued movement of device 100 in physical environment 701, for example, in the transition in FIG. 7Y, and the capture of information about further portions of physical environment 701, including further expansion of overlay 712 in user interface 700 and further updating of map 716 in inset 714.

[0272] FIG. 7AB illustrates the transition from FIG. 7AA, including, for example, expanding the overlay 712 in the user interface 700 so that the overlay 712 is displayed over each portion of the representation of the physical environment 701 of the current room, and further updating the map 716 in the inset 714 so that the map 716 provides a complete view of the current room of the physical environment 701, and the result of continued movement of the device 100 through the physical environment 701 in capturing information for a final portion of the physical environment 701. FIG. 7AB also illustrates an input 743 for commanding the stop scan button 704-2 to terminate the scan of the physical environment 701. In response to the input 743, the scan of the physical environment 701 is terminated, and in some embodiments, a floor plan is generated and displayed using the information captured about the physical environment 701 during the scanning process. Examples of such floor plans are described in further detail herein with reference to FIGS. 7AC-7AT.

[0273] FIG. 7AC shows portable multifunction device 100 displaying floor plan user interface 751 using information obtained while scanning rooms (as described above with reference to FIGS. 7A-7AB). Floor plan user interface 751 includes a floor plan for each room, e.g., “family room” 752. Each room's floor plan includes multiple measurements of bounding exterior walls 753-1 through 753-5 (e.g., their lengths and their angles relative to each other), doors 754, fixtures 755 (e.g., mirrors), and windows 756. Additionally, each room's floor plan includes measurements of physical objects within each room, such as table 757, sofa 758, television 759, and television stand 760. In this example, stairs 761 are also shown in each room's floor plan. In addition to the floor plan measurements, floor plan user interface 751 includes options user interface objects 762 that, when selected, display controls for changing the visual attributes of the displayed floor plan. An information box 763 is also displayed, which includes information regarding the scale of the floor plan relative to the physical environment 763-1 (e.g., in FIG. 7AC the scale is that 1 inch in the physical environment equals 10 feet on the visual floor plan), and the orientation of the room relative to baseline direction 763-2 (shown, for example, as a representation of a compass or compass pointer). Finally, there is an "Exit" button 765, which when selected closes the floor plan user interface 751 and also prompts the user to either save the floor plan to a photo application, save the floor plan locally or remotely to a file repository, or delete the floor plan.

[0274] FIG. 7AD shows de-pick gestures 766-1 and 766-2 occurring on the floor plan user interface 751, and in particular on the floor plans of the respective rooms. FIG. 7AE shows the response to de-pick gestures 766-1 and 766-2. In response to de-pick gestures 766-1 and 766-2, the rooms again scale. However, in some embodiments, the text labels and line thickness do not again scale at the same rate as the floor plan. FIG. 7AE also shows the de-pick gestures 766-1 and 766-2 continuing. FIG. 7AF shows the response to the continuation of de-pick gestures 766-1 and 766-2. In response to the continuation of de-pick gestures 766-1 and 766-2, the rooms again scale. However, the text labels and line thickness again do not again scale at the same rate as the floor plan. Figure 7AF illustrates the cessation or termination of rescaling of the respective rooms in response to the cessation of movement of the unpinch gestures 766-1 and 766-2. Figure 7AF also illustrates the floor plan scale for the physical environment 763-1 being updated.

[0275] FIG. 7AG shows that the drawing scale of the rescaled floor plan for each room is automatically snapped to a new rescale threshold, or to a respective drawing scale within a predetermined set of drawing scales. In the example shown in FIG. 7AG, the new rescale threshold is defined as 6 feet in the physical environment equals 1 inch in the virtual floor plan. In the rescaled environment, the new scale setting is reflected by the floor plan scale for physical environment 763-1 being updated.

[0276] In some embodiments, as shown in Figures 7AC-7AG, one or more of the scanned objects (e.g., a table and a sofa) are automatically identified by the device, and based on that identification and without human intervention, the displayed representations of these objects in the floor plan are assigned titles or names. Consequently, there may be instances where a user wishes to change the title or name of a designated object to a title or name specified by the user. Figures 7AH-7AJ illustrate such an interaction.

[0277] FIG. 7AH shows input 767 over the displayed name of the identified object, which in this example is labeled as “Table” 768-1. FIG. 7AI shows the resulting user interface, allowing the user to rename the identified object. In some embodiments, the name change (e.g., re-titling) occurs in response to input received at a virtual keyboard that is overlaid on the underlying floor plan user interface 751 or displayed simultaneously with a rescaled version of the underlying floor plan user interface 751. In some embodiments, an external keyboard can be used that does not cause the underlying floor plan user interface 751 to overlay or rescale. FIG. 7AJ shows the identified object, which was labeled “Table” 768-1, now labeled “Coffee Table” 768-2.

[0278] FIG. 7AK shows a panning input 769 (e.g., a single-finger swipe input moving rightward in this example). FIG. 7AL shows a response to the panning input 769. FIG. 7AL shows the floor plan of each room (e.g., "Family Room" 752 in this example) being moved in accordance with the panning input 769. FIG. 7AL also shows the panning input 769 continuing over the floor plan of each room. FIG. 7AM shows a response to the panning input 769 continuing. The floor plan of "Family Room" 752 is no longer completely displayed on the display. In some embodiments, as shown in FIG. 7AM, following a determination that the panning input 769 causes portions of the floor plan of each room to disappear, measurements are moved to continue displaying them even though their original location is no longer displayed. Additionally, in some embodiments, the title or name of each room (e.g., "Family Room" title 752-1) is moved, and the title or name remains on the display even though its original location is no longer displayed.

[0279] Figure 7AN shows a tap input 770 on a door 754. The door in Figure 7AN shows the door 754 opening inwards relative to the "family room" 752. In response to the tap input 770, Figure 7AO shows the door 754 opening outwards relative to the "family room" 752. In other words, a tap input on a door can change the opening or closing direction of that door.

[0280] FIG. 7AP shows an input 771 on an options user interface object 762. FIG. 7AQ shows a response to the input 771 on the options user interface object 762. In FIG. 7AQ, the options user interface object 762 is replaced by or overlapped with an expanded options user interface 772. The expanded options user interface 772 includes a hand-drawing toggle 772-1, an additional measurements toggle 772-2, and a display furniture toggle 772-4. When the hand-drawing toggle 772-1 is toggled to its activated position, it changes the appearance of the floor plan by making the floor plan appear as if it had been drawn by hand (e.g., a sketch). When the additional measurements toggle 772-2 is toggled to its activated position, it displays additional linear measurements (e.g., metrics) that were not originally displayed. In some embodiments, when the additional measurements toggle 772-2 is toggled to its activated position, it displays additional angles (e.g., angles other than 90-degree angles) that were not originally displayed. The Show Furniture toggle 772-4, when selected, causes the furniture in each room identified by the device to be displayed on the floor plan for each room.

[0281] FIG. 7AR shows an input 773 above the additional measurement toggle 772-3. FIG. 7AS shows the response to the input 773. As noted above, the additional measurement toggle 772-3, when switched to its activated position, displays (on the floor plan) additional linear measurements (e.g., metrics) that were not originally displayed. FIG. 7AS shows the additional measurements displayed on the floor plan. For example, the new measurements included include the diagonal length 776 of the table 757, the width 777 of the fixture 755 (e.g., a mirror), and the width 778 of the window 756. While specific additional measurements are shown in this example, many other measurements can be displayed in addition to or instead of the measurements shown.

[0282] 7AT shows an alternative embodiment in which an input 775 on additional measurement toggle 772-2 causes angles other than 90 degrees to be displayed. For example, FIG. 7AT shows additional angles (774-1 and 774-2) being displayed in response to activation of additional measurement toggle 772-2. In some embodiments, these additional angles are angles determined within the room to be greater than or less than 90 degrees (e.g., 45 degrees or 135 degrees).

[0283] 8A-8F are flow diagrams illustrating a method 800 for displaying automatically determined measurements of a physical environment using augmented reality, according to some embodiments. Method 800 is implemented in a computer system (e.g., portable multifunction device 100 (FIG. 1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B)) that includes a display device (e.g., a display (optionally touch-sensitive), a projector, a head-mounted display, a head-up display, etc., such as touchscreen 112 (FIG. 1A), display 340 (FIG. 3A), or display generation component(s) 304 (FIG. 3B)), one or more cameras (e.g., light sensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)), and, optionally, one or more depth-sensing devices, e.g., depth sensors (e.g., one or more depth sensors, such as time-of-flight sensor 220 (FIG. 2B)). Some operations in method 800 are optionally combined and / or the order of some operations is optionally changed.

[0284] As described herein, method 800 indicates when a reticle in an annotation placement user interface (e.g., reticle 504 as described with reference to FIGS. 5I-5N) is over a representation of a measurable physical feature, and in response to a request to perform one or more measurements of the physical feature, automatically determines the type of measurements that constitute the physical feature based on the feature type for which the physical feature was determined, without requiring a user to provide input to specify the type of measurement (e.g., without selecting a particular annotation mode in the annotation placement user interface). Performing an operation (e.g., automatically) when a set of conditions is met, without requiring further user input, improves system usability and provides a more efficient user-device interface (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently, reducing device power usage and improving battery life.

[0285] The computer system displays 802 an annotation placement user interface (e.g., a user interface of an augmented reality measurement application) via the display device. The annotation placement user interface includes a representation of one or more camera fields of view (e.g., as shown in FIGS. 5C and 5D ), including a representation of a portion of the three-dimensional physical environment within the field of view of the one or more cameras, where the representation of the field of view is updated over time based on changes within the field of view of the one or more cameras (e.g., the representation of the field of view is a live view updated based on changes in the physical environment within the field of view and / or based on movement of one or more cameras that changes which portion of the physical environment is within the field of view), and a placement user interface element (e.g., reticle 504) that indicates (e.g., via a dot or other marker on a reticle) where a virtual annotation (e.g., a representation of a measurement value) will be placed on the representation of the field of view in response to receiving an annotation placement input (e.g., as described with reference to FIGS. 5R-5S ).

[0286] In some embodiments, while displaying the annotation placement user interface, the computer system detects (804) a first movement of one or more cameras relative to the physical environment.

[0287] In response to detecting a first movement of the one or more cameras relative to the physical environment, the computer system updates (806) the representation of the field of view based on the first movement of the one or more cameras (e.g., as described with reference to Figures 5C and 5D).

[0288] Following a determination that a positioning user interface element (e.g., as described with reference to reticle 504 being positioned based on movement of device 100) is over at least a portion of a representation of a physical feature in the measurable physical environment, the computer system changes (808) the appearance of the positioning user interface element according to one or more aspects of the representation of the physical feature (e.g., to match or adapt to one or more aspects of the representation of the physical feature, e.g., to highlight or mark one or more edges, vertices, boundaries, etc. of the physical feature to visually accentuate the representation of the physical feature relative to the representation of the field of view, and / or to indicate certain measurements that may be performed on the physical feature, such as length, height, area, and / or volume measurements) (e.g., as described with reference to the changing reticle 504 in Figures 5I-5N, 5T-5X, and 5AI-5AJ).

[0289] While displaying the annotation placement user interface, the computer system receives (810) an annotation placement input including a request to perform one or more measurements of a physical feature (e.g., as described with reference to Figures 5G-5H, 5R-5T, and 5W-5X).

[0290] In some embodiments, the one or more measurements are performed by the system regardless of (e.g., before or without receiving) user input requesting the performance of the one or more measurements. In some embodiments, the received annotation placement input includes a request to display one or more measurements of a physical feature (e.g., instead of a request to perform one or more measurements of the physical feature). As otherwise stated, in some embodiments, the user input that is annotation placement input includes or corresponds to a request to add or display one or more representations of one or more measurements already performed or determined by the system.

[0291] In response to receiving 812 an input corresponding to a request to perform one or more measurements of the physical feature, and following a determination 814 that the physical feature is a first type of physical feature (e.g., an edge of a physical object), the computer system displays a representation of a first set of one or more measurements of a first measurement type (e.g., physical feature) (e.g., a one-dimensional measurement type that measures the distance between two points, e.g., the length of an edge of an object such as a table) over the representation of the physical feature, indicating that the physical feature is a first type of physical feature (e.g., an edge of a physical object). Following a determination (816) that the first type of feature is a second type of physical feature (e.g., a two-dimensional surface of a physical object, such as a table, wall, window, etc.) different from the first type of physical feature, the computer system displays a representation of a second set of one or more measurements (e.g., of the physical feature) of a second measurement type different from the first measurement type (e.g., as described with reference to FIGS. 5I-5K) over the representation of the physical feature (e.g., a two-dimensional measurement type measuring the area of ​​the two-dimensional surface, etc., and optionally, the distance between one or more points on the length of an edge of the surface within the field of view of one or more cameras, etc.). In some embodiments, the first type of feature can be any type of feature, such as one of the following types of features: a piece of furniture (822), a person (824), or part of a person (e.g., a person's face, eyes (828), one or both hands (832), or one or both feet (830)), an animal (826), etc. Examples of such features and their measurements are described with reference to FIGS. 5F-5AL. In some embodiments, the second type of feature can be any one of the same set of feature types, provided that the second type of feature is different from the first type of feature.

[0292] In some embodiments, the physical features are of a first respective type of physical features (818), where the physical features are a first portion (e.g., corresponding to a first dimension) of a physical object in the physical environment that is within the field of view of one or more cameras, and a second portion (e.g., corresponding to a second dimension) of the physical object is at most partially within the field of view of the one or more cameras (e.g., the second portion is not entirely within the field of view), examples of which are shown in Figures 5F-5N. In some embodiments, after displaying one or more representations of a first respective set of measurements of the first respective measurement type (e.g., according to physical features that are of the first respective type of physical features) over the representations of the physical features, the computer system detects a second movement of the one or more cameras relative to the physical environment such that the second portion of the physical object is (e.g., entirely) within the field of view of the one or more cameras. In some embodiments, in response to detecting a second movement of the one or more cameras, the computer system updates the representation of the field of view, including displaying a representation of the physical object that includes a representation of a second portion of the physical object (e.g., and optionally, a representation of one or more other portions of the physical object, such as the first portion, that are within the field of view of the one or more cameras) based on the second movement of the one or more cameras.

[0293] In some embodiments, pursuant to a determination that a placement user interface element (e.g., reticle 504) is over at least a portion of the representation of the physical object (e.g., and pursuant to a determination that the physical object is measurable), the computer system modifies the appearance of the placement user interface element according to one or more aspects of the representation of the physical object, including a second portion of the physical object (e.g., taking into account one or more aspects of the representation of the second portion by aligning or matching it with, in addition to, the appearance of the displayed representation of any other portion of the physical object, such as the first portion; e.g., highlighting or marking one or more edges, vertices, boundaries, etc. of the portion(s) of the physical object to visually highlight the representation of the physical object relative to the representation in the field of view; and / or indicating certain measurements that may be performed on the physical object, such as length, height, area, and / or volume measurements), examples of which are described above with reference to FIGS. 5F-5T.

[0294] In some embodiments, while displaying the annotation placement user interface including the representation of the physical object, the computer system receives a second annotation placement input including a request to perform one or more measurements of the physical object. In some embodiments, in response to receiving the input corresponding to the request to perform the one or more measurements of the physical object, the computer system displays a representation of a second respective set of one or more measurements of a second respective measurement type based on a second portion of the physical object over the representation of the physical object, examples of which are described above with reference to Figures 5I-5N.

[0295] In some embodiments, the second respective measurement type is based on the second portion of the physical object and any other portion, such as the first portion, that is within the field of view of one or more cameras. In some embodiments, if the first portion of the physical object is not within the field of view of one or more cameras, the second respective measurement type is based on the second portion of the physical object rather than the first portion of the physical object. In some embodiments, the second respective measurement type is based on the second portion and the first portion of the physical object, regardless of whether the first portion is within (e.g., remains within) the field of view of one or more cameras, using previously obtained information about the first portion. For example, the first portion of the physical object is a first edge of the physical object, and the first respective measurement type measures the distance between two points (e.g., the length of the first edge). In some scenarios of this example, the second portion of the physical object is a second edge of the physical object adjacent to the first edge, and the second respective measurement type measures area (e.g., a surface of the physical object defined by at least the first edge and the second edge) and, optionally, a length of the first edge and a length of the second edge. In a second example, the first portion of the physical object is a first surface of the physical object, and the first respective measurement type measures area of ​​the first surface (and, optionally, length(s) of the edge(s) of the first surface) (e.g., as described above with reference to FIGS. 5I-5N). In some scenarios of this second example, the second portion of the physical object is an edge of the physical object associated with the first surface, and the second respective measurement type measures volume of the physical object and, optionally, area of ​​the first surface, length(s) of the edge(s) of the first surface, and / or length(s) of the associated edge(s).

[0296] As movement of one or more cameras brings additional portions of the physical object into the field of view of one or more cameras, the appearance of the configuration user interface elements changes to take into account the additional portions of the physical object, providing the user with visual feedback indicating that different types of measurements of the physical object are now available, and different types of measurements can be created automatically, allowing the user to access additional control options without having to navigate through complex menu hierarchies. Providing the user with improved visual feedback and providing additional control options (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring additional user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0297] In some embodiments, a first portion of a physical object in the physical environment is within the field of view of one or more cameras (820), and the first portion of the physical object includes an edge region adjacent to a second portion of the physical object that is not within the field of view of one or more cameras or that is only partially within the field of view of one or more cameras (e.g., the field of view of one or more cameras includes a partial view of the physical object). In some embodiments, the computer system displays a representation of the first portion of the physical object in the representation within the field of view of the one or more cameras, including visually de-emphasizing (e.g., fading) the representation of the edge region adjacent to the second portion of the physical object (e.g., as described with reference to FIGS. 5F-5G). In some embodiments, points within the edge region are progressively de-emphasized as they approach the second portion of the physical object (e.g., a first point in the edge region that is a first distance from a boundary between the first and second portions of the physical object is visually de-emphasized relative to a second point in the edge region that is a second distance from the boundary that is greater than the first distance).

[0298] By visually de-emphasizing (e.g., fading) the representation of the visible portion of the physical object adjacent to the portion of the physical object that is not in view, visual feedback is provided that informs the user that the physical object is not entirely in view and prompts the user to move one or more cameras if the user wishes to view and / or measure the physical object more fully. Providing improved visual feedback to the user improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and improves battery life.

[0299] In some embodiments, determining that the physical feature is a first type of physical feature includes determining by the system that the physical feature is a piece of furniture (822), and the measurements of the first measurement type include one or more of a height, a width, a depth, and a volume of the physical feature.

[0300] Performing measurements of the height, width, depth, and / or volume of a physical feature (e.g., a physical object) determined to be a piece of furniture provides superior measurement functionality for measuring dimensions and aspects of physical features of a type that the user is likely to be interested in, without the user having to provide input to specify which type(s) of measurement the user wants to make of the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met, without cluttering the user interface with additional displayed controls and without requiring further user input, improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently, and reduces device power usage and improves battery life.

[0301] In some embodiments, determining that the physical feature is a first type of physical feature includes the system determining 824 that the physical feature is a person, and the measurements of the first measurement type include a distance between the person's feet (e.g., the soles of the person's feet or a surface on which the person's feet are placed while standing or sitting) and the top of the person's head (e.g., the top of the person's head). In some embodiments, the representation of the distance from the person's feet to the top of the person's head includes a measurement segment from the person's feet to the top of the person's head (e.g., as described with reference to FIGS. 5Z-5AB) and / or a text label accounting for the distance. In some embodiments, if the person is standing, the distance from the person's feet to the top of the person's head is the person's height. In some embodiments, if the person is sitting, the distance from the person's feet to the top of the person's head is the person's height while seated. In some embodiments, the representation of the set of measurements for the person includes an indication of the person's feet (e.g., an annotation indicating the soles of the person's feet or the surface on which the person's feet are placed). In some embodiments, the person's feet are automatically identified by the system without requiring user input to mark the location of the person's feet.

[0302] Performing height measurements of physical features determined by the system to be human provides superior measurement functionality, measuring dimensions and aspects of physical features that are likely to interest a user, without requiring the user to provide input to specify which type(s) of measurement the user wants to make from the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met, without cluttering the user interface with additional displayed controls and without requiring further user input, improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve their intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and improves battery life.

[0303] In some embodiments, determining that the physical feature is a first type of physical feature includes the system determining (826) that the physical feature is an animal (e.g., a pet such as a dog or cat), and the measurements of the first measurement type include one or more of the circumference of the animal's neck (e.g., as described with reference to FIG. 5AG), the circumference of the animal's chest, and the length of the animal (e.g., the length of the animal's back from the base of the animal's neck to the base of the animal's tail).

[0304] By performing measurements of neck circumference, chest circumference, girth, and length of physical features determined by the system to be animal (e.g., pet), superior measurement functionality is provided to measure dimensions and aspects of physical features that are likely to be of interest to the user (e.g., in selecting apparel and accessories for the pet) without the user having to provide input to specify which type(s) of measurement the user wants to make from the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring further user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0305] In some embodiments, determining that the physical feature is a first type of physical feature includes the system determining (828) that the physical feature is a human face, and the first measurement type measurements include a distance between a first pupil of a first eye of the human and a second pupil of a second eye of the human (e.g., the distance between the centers of the pupils, also known as the “pupillary distance (PD)” or “interpupillary distance (IPD)”) (e.g., as described with reference to FIG. 5AE).

[0306] Performing interpupillary distance measurements of physical features determined by the system to be a person's face and eyes provides superior measurement functionality for measuring dimensions and aspects of physical features that are likely to be of interest to a user (e.g., in selecting eyeglasses) without requiring the user to provide input to specify which type(s) of measurement the user wants to make from the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring further user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and improves battery life.

[0307] In some embodiments, determining that the physical feature is a first type of physical feature includes determining by the system (830) that the physical feature is a person's foot, and the measurement of the first measurement type includes a foot length (e.g., as described with reference to FIG. 5AF). In some embodiments, the representation of the foot length is displayed as a shoe size corresponding to the foot length (e.g., the shoe size is determined based on a size chart for the person's current geographic location or a previously defined geographic location or region, e.g., according to system-defined or, optionally, user-defined settings).

[0308] Performing a length or shoe size measurement of a physical feature determined by the system to be a person's foot (e.g., optionally taking into account whether the person is barefoot or wearing shoes) provides superior measurement functionality for measuring dimensions and aspects of a physical feature that is likely to be of interest to the user (e.g., in selecting a pair of shoes) without requiring the user to provide input to specify which type(s) of measurement the user wants to make from the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring further user input improves system usability and provides a more efficient user-device interface (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0309] In some embodiments, determining that the physical feature is a first type physical feature includes determining (832) by the system that the physical feature includes a first hand and a second hand that are separated in space (e.g., from the same person), and the measurements of the first measurement type include a distance between the first hand and the second hand (e.g., the distance between corresponding points on the first and second hands, such as the distance between the center of the palm of the first hand and the center of the palm of the second hand) (e.g., as described with reference to Figures 5AC-5AD).

[0310] By performing distance measurements between two hands for physical features determined to comprise a pair of hands (e.g., palms facing each other), the system provides superior measurement functionality, measuring dimensions and aspects of physical features that are likely to be of interest to the user (e.g., where a subject in one or more cameras' fields of view is holding their hands apart, indicating the size of an object), without the user having to provide input to specify which type(s) of measurement the user wants to make from the physical feature. Performing an operation (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring further user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0311] In some embodiments, while displaying a respective set of one or more representations of measurements over the representation of the physical feature (834), where each set includes a representation of a first measurement, and the first representations include (e.g., optionally) first measurement labels and first measurement segments displayed using a first level of detail (e.g., a first granularity) while the one or more cameras are positioned at a first distance from the physical feature (e.g., pursuant to a determination that the first distance is within a first range of distances corresponding to a first level of detail), the computer system detects movement of the one or more cameras that positions the one or more cameras at a second distance from the physical feature that is less than the first distance (e.g., the one or more cameras are moving closer to a portion of the physical feature that corresponds to the first representation of the measurement). In some embodiments, the computer system ceases displaying the first measurement labels while the one or more cameras are positioned at a second distance from the physical feature (e.g., pursuant to a determination that the second distance is within a second range of distances different from the first range of distances corresponding to a second level of detail). Alternatively, in some embodiments, the display of the first measurement labels is maintained. In some embodiments, while the one or more cameras are positioned at a second distance from the physical feature, the computer system displays the first measurement segment using a second level of detail that is different from (e.g., greater than) the first level of detail (e.g., the second level of detail is associated with a second granularity that is finer than the first granularity) (e.g., as described above with reference to Figures 5O-5T).

[0312] Displaying measurements of a physical feature as distances between one or more cameras and changes in the physical feature using different levels of detail (e.g., with multiple sets of scale markers of different granularities) superiorly provides information about measurements at a level of detail (e.g., scale markers at each level of granularity) that are relevant to the current distance and likely to be of interest to the user, without requiring the user to provide input to specify the level of detail at which the user wants to see measurements. Furthermore, turning off the display of measurement levels for measurement segments when displaying scale markers for measurement segments reduces clutter in the user interface and avoids measurements that obscure the scale markers, or vice versa. Performing an operation (e.g., automatically) when a set of conditions is met without cluttering the user interface with additional displayed controls and without requiring further user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0313] In some embodiments, displaying the first measurement segment using a second level of detail includes displaying a set of scale markers (e.g., check marks) at intervals along the first measurement segment (e.g., as described with reference to FIGS. 5P-5S). In some embodiments, while displaying the first measurement segment using the second level of detail, pursuant to a determination by the system that a placement user interface element (e.g., a placement user interface element, or a selection indicator for a placement user interface element, is within a threshold distance of the respective scale marker or within the anchor point of the respective scale marker) is over a respective scale marker (e.g., pursuant to a determination that the placement user interface element is over a single scale marker, or pursuant to a determination that the distance between the center of the placement user interface element and the respective scale marker is less than the distance between the center of the placement user interface element and any other scale marker in the set of scale markers), (e.g., level In response to receiving annotation placement input by snapping a reticle to an anchor point (where snapping includes, for example, moving the reticle from a predetermined position within the annotation placement user interface to the anchor point, or moving a selection indicator (e.g., a dot or other marker) of the reticle from a predetermined position within the reticle (e.g., the center of the reticle) to the anchor point (e.g., a selection indicator that is being moved independently of other portions of the reticle) and / or expanding the selection indicator relative to the outline of the reticle), the computer system changes the appearance of the placement user interface element to indicate that the annotation will be placed at the anchor point on the respective scale marker (e.g., the midpoint or end point of the scale marker). Examples are shown in FIGS. 5P-5T.

[0314] Snapping a placement user interface element to an anchor point on a respective scale marker displayed for a measurement segment provides visual feedback to the user indicating that the annotation will be added at the identified anchor point, facilitating the user to add annotations at locations identified by the scale markers, which are typically useful reference points along a measurement (e.g., feet or inches, meters or centimeters, or simple fractions thereof), without the user having to carefully position the placement user interface element. By providing the user with improved visual feedback and performing an operation (e.g., automatically) when a set of conditions is met, reducing the number and / or extent of inputs required to perform an operation improves system operability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device power usage and battery life.

[0315] In some embodiments, while displaying the first measurement segment using the second level of detail (838), the computer system receives input (e.g., annotation placement input) corresponding to a request to add annotations at respective locations in the representation of the field of view indicated by the second level of detail and corresponding to respective physical locations in the physical environment (e.g., displaying the first measurement segment using the second level of detail includes displaying a set of scale markers (e.g., check marks) at intervals along the first measurement segment, and the input corresponds to the request to add annotations at each scale marker, e.g., by a user moving a reticle over each scale marker and providing annotation placement input while the reticle is snapped to each scale marker). In some embodiments, in response to the input corresponding to the request to add annotations, the computer system adds annotations at each location indicated by the second level of detail.

[0316] In some embodiments, after adding the annotation, the computer system detects movement of one or more cameras that positions one or more cameras at (or alternatively within) a first distance from the physical feature (e.g., as described with reference to FIGS. 5O-5P). In some embodiments, in response to detecting movement of one or more cameras that positions one or more cameras at a first distance from the physical feature, the computer system updates the representation of the field of view based on the movement of one or more cameras (e.g., when a checkmark disappears due to moving partially away, as described with reference to FIGS. 5S-5T). In some embodiments, while the one or more cameras are located at (or alternatively within) a first distance from the physical feature (e.g., following a determination that the third distance is within a first range of distances corresponding to the first level of detail), the computer system displays the first measurement segment using the first level of detail, the first measurement label, and annotations at each position in the updated representation of the field of view corresponding to each physical location in the physical environment (e.g., previously represented by the second level of detail), regardless of whether the respective locations are represented by the first level of detail (e.g., displaying the first measurement segment using the first level of detail includes ceasing to display any scale markers, or a different set of scale markers, at intervals along the first measurement segment that are longer than the intervals of the second level of detail, and the annotations are maintained at the same positions in the representation of the field of view that correspond to their respective physical locations before moving the one or more cameras back to the first distance, even if the first level of detail does not include scale markers corresponding to the annotation positions).

[0317] By maintaining annotations added to anchor points on each scale marker along the measurements so that the respective scale markers are not displayed even when one or more cameras are moved (e.g., away from the measured physical feature), improved visual feedback is provided to the user that the annotations are located at useful and previously marked reference points (e.g., feet or inches, meters or centimeters, or simple fractions thereof) along the measurements, without cluttering the user interface with overly detailed measurements. Providing improved visual feedback to the user improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user failures when operating / interacting with the system), which further enables the user to use the device more quickly and efficiently, reducing device power usage and improving battery life.

[0318] It should be understood that the particular order described of the operations in Figures 8A-8F is merely an example, and that the described order is not intended to indicate the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. Additionally, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 900, 1000, 1100, 1200, 1300, 1400, 1500, and 1600) are also applicable in a similar manner to method 800 described above with respect to Figures 8A-8F. For example, the user interfaces, user interface elements, physical environments and features and objects therein, feature types, annotations, measurement representations, measurement types, and scale markers described above with reference to method 800, optionally have one or more of the characteristics of the user interfaces, user interface elements, physical environments and features and objects therein, feature types, annotations, measurement representations, measurement types, and scale markers described herein with reference to other methods described herein (e.g., methods 900, 1000, 1100, 1200, 1300, 1400, 1500, and 1600), the details of which will not be repeated here for the sake of brevity.

[0319] 9A-9C are flow diagrams illustrating a method 900 for providing positioning guidance based on automatically determined anchor points within an augmented reality environment, according to some embodiments. Method 900 is implemented in a computer system (e.g., portable multifunction device 100 ( FIG. 1A ), device 300 ( FIG. 3A ), or computer system 301 ( FIG. 3B )) that includes a display device (e.g., a display (optionally touch-sensitive), a projector, a head-mounted display, a head-up display, etc., such as touchscreen 112 ( FIG. 1A ), display 340 ( FIG. 3A ), or display generation component(s) 304 ( FIG. 3B )), one or more cameras (e.g., light sensor(s) 164 ( FIG. 1A ) or camera(s) 305 ( FIG. 3B )), and, optionally, one or more depth-sensing devices, e.g., depth sensors (e.g., one or more depth sensors, such as time-of-flight sensor 220 ( FIG. 2B )). Some operations in method 900 are optionally combined and / or the order of some operations is optionally changed.

[0320] As described herein, when a reticle in an annotation placement user interface snaps to an anchor point on a representation of a physical feature, method 900 displays a set of guides extending in one or more directions from the anchor point to provide the user with visual feedback identifying orientations at which one or more measurements of the physical feature can be made (e.g., as described with reference to FIGS. 5AN and 5AP), thus assisting the user in making measurements that may be of interest to the user. Furthermore, method 900 advantageously displays different guides based on the determined type of feature the physical feature is, without requiring the user to provide input specifying which type of guide to display. Providing improved visual feedback to the user and performing operations (e.g., automatically) without requiring further user input improves system usability and makes the user-device interface more efficient (e.g., by helping the user achieve intended results and by reducing user errors when operating / interacting with the system), which in turn allows the user to use the device more quickly and efficiently and reduces device ...

Claims

1. 1. A computer system having a display device and one or more cameras, displaying within a first region of a user interface a representation of a field of view of the one or more cameras, the one or more cameras being within a three-dimensional physical environment, the representation of the field of view including a representation of a first respective portion of the physical environment that is within the field of view of the one or more cameras; capturing depth information indicative of a first subset of the first respective portions of the physical environment; displaying, on the representation of the field of view, a first indication indicating the extent of the first respective portion of the physical environment for which the depth information was captured, including displaying the first indication overlaid on at least a first portion of the representation of the field of view that includes a representation of the first subset, and displaying at least a second portion of the representation of the field of view without the first indication overlaid; detecting movement of the one or more cameras that moves the field of view to include a second respective portion of the physical environment; In response to detecting movement of the one or more cameras, updating the representation of the field of view of the one or more cameras to include a representation of the second respective portion of the physical environment; capturing additional depth information indicative of a second subset of the second respective portions of the physical environment; updating the first indication displayed on the updated representation of the field of view to indicate the extent of the second portion of the physical environment for which depth information was captured, including displaying the updated first indication overlaid on the second portion of the representation of the field of view; the second portion of the representation of the field of view includes a representation of the second subset of the second respective portion of the physical environment; updating the updated first indication to indicate additional portions of the physical environment for which depth information was captured in response to detecting movement of the one or more cameras; A method comprising:

2. The method of claim 1 , comprising displaying, within the representation of the field of view, one or more representations of one or more portions of the physical environment for which no depth information has been captured.

3. The method of claim 1 , comprising smoothing one or more edges of the first indication.

4. The method of claim 1 , wherein the first indication is a contiguous area.

5. The method of claim 1 , comprising displaying instructions to a user prompting the user to move the one or more cameras around the physical environment prior to detecting the movement of the one or more cameras.

6. displaying a first user interface element that, when activated, initiates capture of depth information indicative of a physical environment in which the one or more cameras are located; receiving an input activating the first user interface element; capturing depth information indicative of the physical environment after receiving the input activating the first user interface element, the capturing of depth information including capturing the depth information indicative of the first subset and capturing the depth information indicative of the second subset according to movement of the one or more cameras; The method of claim 1 , comprising:

7. displaying a second user interface element that, when activated, initiates a display of an orthographic view of the portion of the physical environment for which depth information was captured; receiving an input activating the second user interface element; displaying an orthographic view of the physical environment after receiving the input activating the second user interface element; and The method of claim 1 , comprising:

8. 2. The method of claim 1, wherein displaying the first indication comprises visually highlighting a portion of the first indication that is overlaid on a representation of a set of physical features of a particular feature type detected within the representation of the field of view.

9. 9. The method of claim 8, wherein visually enhancing the portion of the first indication overlaid on the representation of the physical characteristic comprises highlighting the portion of the first indication for a predetermined period of time.

10. The method of claim 8 , comprising gradually discontinuing visual emphasis of the portion of the first indication over time.

11. the user interface is a user interface for a respective application; 2. The method of claim 1, wherein displaying the first indication, detecting the movement of the one or more cameras and updating the representation of the field of view in response to detecting the movement of the one or more cameras, capturing the depth information indicative of the second subset, and updating the first indication are performed while the respective application is operating in a first of a plurality of modes of the respective application.

12. 2. The method of claim 1, comprising displaying a representation of a second view of the physical environment within a second region of the user interface, wherein, in accordance with capturing the depth information indicative of the first subset of the first respective portion of the physical environment, the representation of the second view indicates the range of the physical environment for which depth information was captured and includes an indication that the depth information indicative of the first subset was captured.

13. 13. The method of claim 12, wherein the movement of the one or more cameras comprises one or more camera rotations about respective axes by respective amounts, the method comprising, in response to detecting the movement of the one or more cameras, rotating the representation of the second view of the physical environment by the respective amounts.

14. 13. The method of claim 12, wherein the movement of the one or more cameras comprises one or more camera rotations about respective axes by respective amounts, the method comprising maintaining an orientation of the second view of the physical environment of the user interface in response to detecting the movement of the one or more cameras.

15. displaying one or more alerts with information about the location or movement of the one or more cameras, the one or more alerts comprising: an alert indicating that a distance from the one or more cameras to an object in the field of view of the one or more cameras is outside a threshold distance; an alert indicating that the speed of movement of the one or more cameras is outside a threshold speed; an alert indicating that the one or more cameras should remain in a predetermined portion of the physical environment; and an alert indicating that the field of view of the one or more cameras needs to move away from one or more reflective surfaces in the physical environment; an alert indicating that an object within the field of view of the one or more cameras should remain substantially stationary; and an alert indicating that the one or more cameras should be moved to a previous position where at least some depth information was not captured for each portion of the physical environment that was in the field of view of the one or more cameras at the previous position of the one or more cameras; an alert indicating that the one or more cameras should move to an initial position where the one or more cameras were located when capture of depth information indicative of the physical environment began; and The method of claim 1 , comprising:

16. receiving a user input indicating a boundary between a first space in the physical environment and a second space in the physical environment that is different from and adjacent to the first space; 2. The method of claim 1, wherein the orthographic view of the physical environment displayed after receiving the user input indicating the boundary between the first space and the second space includes an indication of the boundary between the first space and the second space.

17. A display device; one or more cameras; one or more processors; a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs comprising: displaying within a first region of a user interface a representation of a field of view of the one or more cameras, the one or more cameras being within a three-dimensional physical environment, the representation of the field of view including a representation of a first respective portion of the physical environment that is within the field of view of the one or more cameras; capturing depth information indicative of a first subset of the first respective portions of the physical environment; displaying, on the representation of the field of view, a first indication indicating the extent of the first respective portion of the physical environment for which the depth information was captured, including displaying the first indication overlaid on at least a first portion of the representation of the field of view that includes a representation of the first subset, and displaying at least a second portion of the representation of the field of view without the first indication overlaid; detecting movement of the one or more cameras that moves the field of view to include a second respective portion of the physical environment; In response to detecting movement of the one or more cameras, updating the representation of the field of view of the one or more cameras to include a representation of the second respective portion of the physical environment; capturing additional depth information indicative of a second subset of the second respective portions of the physical environment; updating the first indication displayed on the updated representation of the field of view to indicate the extent of the second portion of the physical environment for which depth information was captured, including displaying the updated first indication overlaid on the second portion of the representation of the field of view; the second portion of the representation of the field of view includes a representation of the second subset of the second respective portion of the physical environment; updating the updated first indication to indicate additional portions of the physical environment for which depth information was captured in response to detecting movement of the one or more cameras; a memory containing instructions for A computer system comprising:

18. 18. A computer system according to claim 17, wherein the one or more programs comprise instructions for carrying out the method of any one of claims 2 to 16.

19. When executed by a computer system including a display device and one or more cameras, the computer system: displaying within a first region of a user interface a representation of a field of view of the one or more cameras, the one or more cameras being within a three-dimensional physical environment, the representation of the field of view including a representation of a first respective portion of the physical environment that is within the field of view of the one or more cameras; capturing depth information indicative of a first subset of the first respective portions of the physical environment; displaying, on the representation of the field of view, a first indication indicating the extent of the first respective portion of the physical environment for which the depth information was captured, including displaying the first indication overlaid on at least a first portion of the representation of the field of view that includes a representation of the first subset, and displaying at least a second portion of the representation of the field of view without the first indication overlaid; detecting movement of the one or more cameras that moves the field of view to include a second respective portion of the physical environment; In response to detecting movement of the one or more cameras, updating the representation of the field of view of the one or more cameras to include a representation of the second respective portion of the physical environment; capturing additional depth information indicative of a second subset of the second respective portions of the physical environment; updating the first indication displayed on the updated representation of the field of view to indicate the extent of the second portion of the physical environment for which depth information was captured, including displaying the updated first indication overlaid on the second portion of the representation of the field of view; the second portion of the representation of the field of view includes a representation of the second subset of the second respective portion of the physical environment; updating the updated first indication to indicate additional portions of the physical environment for which depth information was captured in response to detecting movement of the one or more cameras; A computer program containing instructions to cause a program to be executed.

20. 20. A computer program according to claim 19, wherein the program comprises instructions which, when executed by the computer system, cause the computer system to carry out the method of any one of claims 2 to 16.

Images