Enhancing simulated vision

Abstract

Some examples of the present disclosure relate to systems and methods for selectively displaying an image of a portion of a physical environment of an electronic device with enhanced visibility. In some examples, the display of the image overlies a portion of a view of the physical environment. In some examples, the electronic device displays the image according to one or more first criteria that are satisfied. In some examples, the electronic device abandons display of the image according to the one or more criteria that are not satisfied.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 585,921 filed on September 27, 2023, U.S. Provisional Application No. 63 / 691,976 filed on September 6, 2024, and U.S. Patent Application No. 18 / 892,082 filed on September 20, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention generally relates to systems and methods for presenting an extended reality environment, and more particularly to displaying one or more images of a physical environment while presenting an extended reality environment. Background Art

[0004] Some computer graphics environments provide two-dimensional environments and / or three-dimensional environments in which at least some of the objects presented for viewing by a user are virtual and generated by a computer. In some examples, a computer graphics environment may be based on one or more images of a physical environment of the computer. Summary of the invention

[0005] The present invention generally relates to systems and methods for presenting an extended reality environment, and more specifically to displaying one or more images of a physical environment when presenting an extended reality environment. In some examples, presenting an extended reality environment using an electronic device includes presenting a transparent video of the physical environment of the electronic device. As described herein, for example, presenting a transparent video may include displaying a virtual or video transparent, wherein the electronic device uses a display to display an image of the physical environment. In some examples, presenting an extended reality environment using an electronic device includes presenting a real or true optical perspective, wherein portions of the physical environment are visible to a user through transparent portions of the display. In some cases, the light level (e.g., brightness) in the physical environment of the electronic device may be too low for the user, so that the user cannot see the physical environment clearly. In some examples, in response to detecting that one or more criteria are met, the electronic device can display one or more (e.g., static and / or video) images of a portion of the physical environment at an increased brightness level, so that the user can more easily see the portion of the physical environment. Example criteria include detecting that the brightness level of the physical environment is below a predefined threshold, receiving user input, and criteria based on contextual data about the physical environment.

[0006] A comprehensive description of these examples is provided in the drawings and detailed description, and it should be understood that this summary does not limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various examples described herein, reference should be made to the following detailed description and the following drawings. Throughout the drawings, like reference numerals generally refer to corresponding parts.

[0008] Figure 1 An electronic device presenting an extended reality environment according to some examples of the present disclosure is shown.

[0009] Figure 2 A block diagram illustrating an example architecture of a device according to some examples of the present disclosure.

[0010] FIG. 3A to FIG. 3B An electronic device displaying an image of a portion of a physical environment of the electronic device is shown according to some examples of the present disclosure.

[0011] Figure 4 An electronic device displaying an image of a portion of a physical environment of the electronic device is shown according to some examples of the present disclosure.

[0012] Figure 5 An electronic device is shown that forgoes displaying an image of a portion of a physical environment with enhanced visibility based on the context of the physical environment of the electronic device according to some examples of the present disclosure.

[0013] Figure 6 An example of an electronic device that forgoes displaying an image of a portion of an environment with enhanced visibility based on determining that an amount of ambient light is above a predefined threshold in accordance with some examples of the present disclosure is shown.

[0014] Figure 7 An example of an electronic device presenting an image with enhanced visibility including a color filter is shown according to some examples of the present disclosure.

[0015] Figure 8 An example of an electronic device presenting an image with enhanced visibility including edge enhancement is shown in accordance with some examples of the present disclosure.

[0016] Fig. 9 An example method of selectively displaying an image of a portion of a physical environment of an electronic device according to some examples of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The present invention generally relates to systems and methods for presenting an extended reality environment, and more specifically to displaying one or more images of a physical environment when presenting an extended reality environment. In some examples, presenting an extended reality environment using an electronic device includes presenting a transparent video of the physical environment of the electronic device. As described herein, for example, presenting a transparent video may include displaying a virtual or video transparent, wherein the electronic device uses a display to display an image of the physical environment. In some examples, presenting an extended reality environment using an electronic device includes presenting a real or true optical perspective, wherein portions of the physical environment are visible to a user through transparent portions of the display. In some cases, the light level (e.g., brightness) in the physical environment of the electronic device may be too low for the user, so that the user cannot see the physical environment clearly. In some examples, in response to detecting that one or more criteria are met, the electronic device can display one or more (e.g., static and / or video) images of a portion of the physical environment at an increased brightness level, so that the user can more easily see the portion of the physical environment. Example criteria include detecting that the brightness level of the physical environment is below a predefined threshold, receiving user input, and criteria based on contextual data about the physical environment.

[0018] In some examples, a three-dimensional object is displayed in a computer-generated three-dimensional environment at a particular orientation that controls one or more behaviors of the three-dimensional object (e.g., when the three-dimensional object moves within the three-dimensional environment). In some examples, the orientation in which the three-dimensional object is displayed in the three-dimensional environment is selected by a user of the electronic device or automatically selected by the electronic device. For example, when initiating presentation of the three-dimensional object in the three-dimensional environment, the user may select a particular orientation of the three-dimensional object, or the electronic device may automatically select the orientation of the three-dimensional object (e.g., based on the type of the three-dimensional object).

[0019] In some examples, a three-dimensional object may be displayed in a three-dimensional environment in a world-locked orientation, a body-locked orientation, a tilt-locked orientation, or a head-locked orientation, as described below. As used herein, an object displayed in a three-dimensional environment in a body-locked orientation has a distance and orientation offset relative to a portion of the user's body (e.g., the user's torso). Alternatively, in some examples, the body-locked object has a fixed distance from the user, and the orientation of the content does not reference any part of the user's body (e.g., may be displayed in the same basic direction relative to the user regardless of head and / or body movement). Additionally or alternatively, in some examples, the body-locked object may be configured to always maintain gravity or horizon (e.g., perpendicular to gravity) alignment, so that changes in the head and / or body in the rolling direction will not cause the body-locked object to move within the three-dimensional environment. Instead, translational movement in either configuration will cause the body-locked object to reposition within the three-dimensional environment to maintain the distance offset.

[0020] As used herein, an object displayed in a head-locked orientation in a three-dimensional environment has a distance and orientation offset relative to the user's head. In some examples, the head-locked object moves within the three-dimensional environment when the user's head moves (when the user's viewpoint changes).

[0021] As used herein, an object displayed in a world-locked orientation in a three-dimensional environment has no distance or orientation offset relative to the user.

[0022] As used herein, an object displayed in a three-dimensional environment with a tilt-locked orientation (referred to herein as a tilt-locked object) has a distance offset relative to a user, such as a portion of a user's body (e.g., a user's torso) or a user's head. In some examples, the tilt-locked object is displayed with a fixed orientation relative to the three-dimensional environment. In some examples, the tilt-locked object moves according to a polar (e.g., spherical) coordinate system centered on a pole passing through the user (e.g., the user's head). For example, the tilt-locked object is moved in the three-dimensional environment based on the movement of the user's head in a spherical space around the user's head (e.g., centered on the user's head). Thus, if the user tilts his head relative to gravity (e.g., up or down in a pitch direction), the tilt-locked object will follow the head tilt and move radially along the sphere, so that the tilt-locked object is repositioned within the three-dimensional environment to the same distance offset relative to the user as before the head tilt, while optionally maintaining the same orientation relative to the three-dimensional environment. In some examples, if the user moves his head in a rolling direction (e.g., clockwise or counterclockwise) relative to gravity, the tilt-locked object is not repositioned within the three-dimensional environment.

[0023] Figure 1 An electronic device 101 is shown presenting an extended reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples of the present disclosure. In some examples, such as Figure 1 As shown, the electronic device 101 is a head mounted display or other head mounted device configured to be worn on the head of the user of the electronic device 101. Figure 2 An example of an electronic device 101 is described by using an architectural block diagram of FIG. Figure 1 As shown, electronic device 101 and table 106 are located in a physical environment. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lights, etc.). In some examples, electronic device 101 may be configured to detect and / or capture an image of the physical environment including table 106 (shown in the field of view of electronic device 101).

[0024] In some examples, such as Figure 1As shown, the electronic device 101 includes one or more internal image sensors 114a (eg, as described below with reference to FIG. 1 ) oriented toward the user's face. Figure 2 101 ). In some examples, internal image sensor 114a is used for eye tracking (e.g., detecting the user's gaze). Internal image sensor 114a is optionally arranged on the left and right portions of display 120 to enable eye tracking of the user's left and right eyes. In some examples, electronic device 101 also includes external image sensors 114b and 114c facing outward from the user to detect and / or capture the physical environment of electronic device 101 and / or movement of the user's hands or other body parts.

[0025] In some examples, the display 120 has a field of view visible to the user (e.g., it may or may not correspond to the field of view captured by the external image sensors 114b and 114c). Because the display 120 is optionally part of the head-mounted device, the field of view of the display 120 is optionally the same as or similar to the field of view of the user's eyes. In other examples, the field of view of the display 120 may be smaller than the field of view of the user's eyes. In some examples, the electronic device 101 may be an optical see-through device, in which the display 120 is a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, the display 120 may be included in a transparent lens and may overlap with all or only a portion of the transparent lens. In other examples, the electronic device may be a video pass-through device, in which the display 120 is an opaque display configured to display an image of the physical environment captured by the external image sensors 114b and 114c.

[0026] In some examples, in response to a trigger, the electronic device 101 may be configured to display in the XR environment Figure 1 , which does not exist in the physical environment but is displayed in the XR environment positioned on top of the real-world table 106 (or its representation). Optionally, in response to detecting the plane of the table 106 in the physical environment 100, the virtual object 104 can be displayed on the surface of the table 106 in the XR environment displayed via the display 120 of the electronic device 101.

[0027] It should be understood that virtual object 104 is a representative virtual object and may include and render one or more different virtual objects (e.g., virtual objects with various dimensions, such as two-dimensional or other three-dimensional virtual objects) in a three-dimensional XR environment. For example, a virtual object may represent an application or user interface displayed in an XR environment. In some examples, a virtual object may represent content corresponding to an application and / or displayed via a user interface in an XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures, such as air pinch gestures, air tap gestures, and / or air touch gestures), so that a user can virtually touch, tap, move, rotate, or otherwise interact with virtual object 104.

[0028] In some examples, displaying an object in a three-dimensional environment may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiation of display of an object in a three-dimensional environment may include interaction with one or more virtual options / indicators displayed in the three-dimensional environment. In some examples, when display of an object is initiated in a three-dimensional environment, an electronic device may track the user's line of sight as an input for identifying one or more virtual options / indicators as targets for selection. For example, line of sight may be used to identify one or more virtual options / indicators as targets for selection using another selection input. In some examples, a virtual option / indicator may be selected using hand tracking input detected via an input device that communicates with the electronic device. In some examples, an object displayed in a three-dimensional environment may be moved and / or reoriented in the three-dimensional environment based on movement input detected via an input device.

[0029] In the following discussion, an electronic device that communicates with a display generation component and one or more input devices is described. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. In addition, as described above, it should be understood that the described electronic device, display, and touch-sensitive surface are optionally distributed between two or more devices. Therefore, as used in the present disclosure, information displayed on or by an electronic device is optionally used to describe information output by the electronic device to be displayed on an independent display device (touch-sensitive or non-touch-sensitive). Similarly, as used in the present disclosure, input received on an electronic device (e.g., touch input received on a touch-sensitive surface of an electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received on an independent input device, and the electronic device receives input information from the independent input device.

[0030] The device typically supports a variety of applications, such as one or more of the following applications: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, a television channel browsing application, and / or a digital video player application.

[0031] Figure 2 A block diagram of an example architecture of an electronic device 201 according to some examples of the present disclosure is shown. In some examples, the electronic device 201 includes one or more electronic devices. For example, the electronic device 201 can be a portable device, an auxiliary device that communicates with another device, a head-mounted display, etc. In some examples, the electronic device 201 corresponds to the above reference Figure 1 An electronic device 101 is described.

[0032] like Figure 2 As shown, the electronic device 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally corresponding to Figure 1 114a and / or external image sensors 114b and 114c in the embodiment of the present invention), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors), one or more display generation components 214 (optionally corresponding to Figure 1 The display 120 in the electronic device 201, one or more speakers 216, one or more processors 218, one or more memories 220 and / or communication circuits 222. One or more communication buses 208 are optionally used for communication between the above components of the electronic device 201.

[0033] The communication circuit 222 optionally includes circuits for communicating with electronic devices, networks such as the Internet, intranets, wired networks and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communication circuit 222 optionally includes circuits for using near field communication (NFC) and / or short-range communication such as A circuit for communicating.

[0034] Processor 218 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage device) storing computer-readable instructions, which are configured to be executed by processor 218 to perform the techniques, processes, and / or methods described below. In some examples, memory 220 may include more than one non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may be any medium (e.g., excluding signals) that can tangibly contain or store computer-executable instructions for use by or in combination with an instruction execution system, device, and apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, a magnetic storage device, an optical storage device, and / or a semiconductor storage device. Examples of such storage devices include magnetic disks, optical disks based on compact disc (CD), digital versatile disc (DVD) or Blu-ray technology, and persistent solid-state memory (such as flash memory, solid-state drives, etc.).

[0035] In some examples, display generation component 214 includes a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generation component 214 includes multiple displays. In some examples, display generation component 214 may include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, electronic device 201 includes touch-sensitive surfaces 209 for receiving user inputs (such as tap inputs and swipe inputs or other gestures). In some examples, display generation component 214 and touch-sensitive surface 209 form a touch-sensitive display (e.g., a touch screen integrated with electronic device 201 or a touch screen that communicates with electronic device 201 outside electronic device 201).

[0036] The electronic device 201 optionally includes an image sensor 206. The image sensor 206 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real-world environment. The image sensor 206 also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor 206 also optionally includes one or more cameras configured to capture the movement of physical objects in the real-world environment. The image sensor 206 also optionally includes one or more depth sensors configured to detect the distance of the physical object from the electronic device 201. In some examples, information from the one or more depth sensors may allow the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may allow the device to determine the texture and / or topography of an object in the real-world environment.

[0037] In some examples, the electronic device 201 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the electronic device 201. In some examples, the image sensor 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor work in tandem and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the electronic device 201 uses the image sensor 206 to detect the positioning and orientation of the electronic device 201 and / or the display generation component 214 in the real-world environment. For example, the electronic device 201 uses the image sensor 206 to track the positioning and orientation of the display generation component 214 relative to one or more fixed objects in the real-world environment.

[0038] In some examples, the electronic device 201 includes a microphone 213 or other audio sensor. The electronic device 201 optionally uses the microphone 213 to detect sounds from the user and / or the user's real-world environment. In some examples, the microphone 213 includes a microphone array (multiple microphones) that optionally operate in conjunction to identify ambient noise or locate a sound source in the space of the real-world environment.

[0039] The electronic device 201 includes a location sensor 204 for detecting the location of the electronic device 201 and / or the display generating component 214. For example, the location sensor 204 may include a global positioning system (GPS) receiver that receives data from one or more satellites and allows the electronic device 201 to determine the absolute location of the device in the physical world.

[0040] The electronic device 201 includes an orientation sensor 210 for detecting the orientation and / or movement of the electronic device 201 and / or the display generation component 214. For example, the electronic device 201 uses the orientation sensor 210 to track changes in the position and / or orientation of the electronic device 201 and / or the display generation component 214, such as relative to physical objects in the real world environment. The orientation sensor 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0041] In some examples, the electronic device 201 includes a hand tracking sensor 202 and / or an eye tracking sensor 212 and / or other body tracking sensors, such as a leg tracking sensor, a torso tracking sensor, and / or a head tracking sensor. The hand tracking sensor 202 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to the extended reality environment, relative to the display generation component 214, and / or relative to another defined coordinate system. The eye tracking sensor 212 is configured to track the position and movement of the user's line of sight (more generally, eyes, face, or head) relative to the real world or extended reality environment and / or relative to the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented together with the display generation component 214. In some examples, the hand tracking sensor 202 and / or the eye tracking sensor 212 are implemented separately from the display generation component 214.

[0042] In some examples, the hand tracking sensor 202 (and / or other body tracking sensors, such as leg tracking sensors, torso tracking sensors, and / or head tracking sensors) may use an image sensor 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that captures three-dimensional information from the real world including one or more body parts (e.g., the legs, torso, head, or hands of a human user). In some examples, the hands may be resolved with sufficient resolution to distinguish fingers and their corresponding positioning. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view and an interaction space of the image sensor 206, in which the finger / hand position, orientation, and / or movement captured by the image sensor is used as input (e.g., to distinguish from the user's resting hand or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) may be advantageous because it does not require the user to touch, hold, or wear any type of beacon, sensor, or other marker.

[0043] In some examples, eye tracking sensor 212 includes at least one eye tracking camera (e.g., an infrared (IR) camera) and / or an illumination source (e.g., an IR light source such as an LED) that emits light toward the user's eyes. The eye tracking camera may be pointed toward the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by corresponding eye tracking cameras and illumination sources, and focus / line of sight may be determined by tracking both eyes. In some examples, one eye (e.g., the dominant eye) is tracked by one or more corresponding eye tracking cameras / illumination sources.

[0044] The electronic device 201 is not limited to Figure 2 Rather than components and configurations of the electronic device 201, it may include fewer components, other components, or additional components in a variety of configurations. In some examples, the electronic device 201 may be implemented between two electronic devices (e.g., as a system). In some such examples, each (or more) electronic device may each include one or more of the same components discussed above, such as various sensors, one or more display generating components, one or more speakers, one or more processors, one or more memories, and / or communication circuits. One or more people using the electronic device 201 are optionally referred to herein as one or more users of the device.

[0045] FIG. 3A to FIG. 3B An electronic device 101 is shown displaying an image 308 of a portion of the physical environment of the electronic device according to some examples of the present disclosure. For example, the electronic device 101 optionally has the same physical environment as described above. Figure 1 and / or Figure 2The electronic device 101 and / or the electronic device 201 described have the same architecture.

[0046] exist Figure 3A In the example shown, the electronic device 101 is operating in a physical environment with a relatively low ambient light level. The ambient light level 312a in this example is indicated by the scale 310. For example, the electronic device 101 is operating outdoors at night without artificial lighting or with relatively low artificial lighting (and / or insufficient moonlight). The current ambient light level 312a may be below a predefined threshold. The electronic device 101 optionally presents a three-dimensional environment 300 including a view and / or representation of the physical environment. For example, in Figure 3A , the electronic device 101 presents a view of a set of stairs in the physical environment of the electronic device 101. Rendering the three-dimensional environment 300 optionally includes displaying virtual elements, such as the image 308 described in more detail below, and / or using video or virtual see-through to display representations of real objects in the physical environment. Rendering the three-dimensional environment 300 optionally includes presenting a view of the physical environment through a transparent portion of the display generation component 120 using real or true optical see-through.

[0047] In situations where the ambient light level falls below a predefined threshold, such as Figure 3A In some embodiments, electronic device 101 optionally displays user interface element 302 including image 308 of a portion of a physical environment of electronic device 101. In some examples, image 308 has enhanced visibility compared to the visibility of the portion of the physical environment presented using display generation component 120 without displaying the image. In some examples, image 308 has enhanced visibility compared to the visibility of the portion of the physical environment viewed without electronic device 101. Enhanced visibility optionally includes one or more of increased brightness, contrast, color saturation, sharpness, and / or clarity. Additionally or alternatively, enhanced visibility optionally includes, for example, Figure 7 The color filters shown and / or Figure 8 Application of object edge enhancement technique shown.

[0048] In some examples, electronic device 101 may be used in addition to, for example, Figure 3AThe electronic device 101 may be configured to display an image of a physical environment in one or more contexts other than a context in which enhanced visibility is provided in an environment with low ambient light levels of an environment. For example, the electronic device 101 captures an image (e.g., a still image or a video) of the physical environment using a camera application. In some examples, the electronic device 101 applies image processing to image 308 (and other visibility-enhanced images according to the present disclosure) to enhance visibility of image 308, which the electronic device 101 does not apply to images of the physical environment captured in other contexts, such as when capturing photos using a camera application. In some examples, the electronic device 101 applies image processing to increase brightness, color saturation, color contrast, and / or sharpness of image 308 to a greater extent than the electronic device 101 applies one or more of these techniques to images of the physical environment in other contexts, such as when capturing and / or editing photos for a photo application using a camera application. For example, the electronic device 101 applies fewer of these techniques, applies these techniques to a lesser extent, or does not apply these techniques to images of the physical environment, such as images displayed in a camera application and / or a photo application. In some examples, images displayed in a camera application are processed to the extent that they reproduce or simulate the appearance of a physical environment, wherein images displayed for enhanced visibility are processed to have an appearance that is different from (e.g., easier to see) the appearance of the physical environment.

[0049] In some examples, the electronic device 101 displays an image 308 of a portion of the physical environment based on determining that one or more first criteria are met. The one or more first criteria optionally include criteria that are met when the electronic device 101 receives a user input requesting display of the image 308. For example, the electronic device 101 displays the image 308 on demand in response to receiving the input. Example inputs include input received through a hardware input device; line of sight detection; detecting air gestures or other movements of a user's fingers, hands, and / or arms; and / or voice input. For another example, the electronic device 101 displays the image 308 in response to a setting activated on the electronic device 101, the setting including displaying the image 308 in response to detecting one or more of the other first criteria described herein. The electronic device 101 optionally activates the setting in response to receiving one or more user inputs.

[0050] The one or more first criteria optionally include a criterion that is satisfied when the current ambient light level 312a is below a predefined threshold level. For example, the predefined threshold level corresponds to the amount of ambient light when visibility is reduced. In some examples, the predefined threshold level is a system setting of the electronic device 101 that the user cannot change. In some examples, the predefined threshold level is a user setting of the electronic device 101 that the user can change. In some examples, the predefined threshold level is static, optionally unless and until the user adjusts the threshold by changing the setting. In some examples, the predefined threshold level is dynamic, depending on the time window average amount of ambient light, wherein the predefined threshold changes according to a predefined function based on the current ambient light level 312a and / or historical ambient light levels (e.g., the last 5 seconds, the last 30 seconds, the last 60 seconds, etc.). For example, if the electronic device 101 moves from a well-lit room to a dark environment, the predefined threshold level may start from an initial value and decrease as the electronic device 101 stays in the dark environment because the user's eyes may gradually adjust to the darkness. In some examples, the electronic device 101 adjusts the predefined threshold level in response to detecting that the user's eyes are adapted to a change in the amount of ambient light. For example, in response to a user moving from a well-lit room to a dark environment, the electronic device 101 detects the adaptation of the user's eyes to the darkness (e.g., by measuring the diameter of the user's pupils) and adjusts (e.g., reduces) the predefined threshold in response to detecting that the user's eyes have adjusted to the darkness.

[0051] The one or more first criteria optionally include a criterion based on the context of the physical environment of the electronic device 101. For example, in Figure 3A , a user is wearing an electronic device 101 while walking near stairs in a dark environment. In this context (uneven or blocked path), the user may want to view an image 308 of the physical environment with enhanced visibility, such as to avoid falling. The electronic device 101 optionally uses movement data, gaze tracking data, semantic understanding of one or more captured images of the physical environment, movement speed, movement acceleration, and / or position data to determine the context of the physical environment of the electronic device 101. For example, the electronic device uses one or more of these factors to determine the user's direction of travel and whether there are obstacles in the user's path of motion. The electronic device 101 optionally uses algorithms and / or machine learning techniques to determine whether the context corresponds to the displayed image 308.

[0052] In some examples, the user interface element 302 also includes a zoom option 306 overlaid on the image 308. For example, in response to detecting an input selecting the zoom option 306, the electronic device 101 displays a plurality of options corresponding to different zoom levels. In response to detecting a selection of one of the options corresponding to the different zoom levels, the electronic device 101, for example, zooms in or out of the image 308 corresponding to the selected zoom level. Available example zoom levels include 0.5x, 1x, 1.5x, 2x, 3x, and / or 5x. In some examples, the user interface element 302 also includes a low light level indication 304 overlaid on the image 308. In some examples, the electronic device 101 displays the low light indication 304 in response to detecting that the current ambient light level 312a is less than the predefined threshold level described above.

[0053] In some examples, image 308 includes a video corresponding to a portion of the physical environment in which electronic device 101 overlays user interface element 302. For example, the video is a live video feed or a near live video feed (e.g., having a latency less than a threshold, such as 1 ms, 50 ms, etc.). Figure 3B As shown, in response to detecting that the position of electronic device 101 has changed relative to the physical environment, electronic device 101 updates image 308 included in user interface element 302 to correspond to the updated view of the physical environment. FIG. 3A to FIG. 3B As shown, electronic device 101 displays user interface element 302 overlaid on a portion of the physical environment shown in image 308. In some examples, electronic device 101 displays the user interface element 302 in a manner that is greater than that used when the amount of ambient light is greater than FIG. 3A to FIG. 3B The image 308 is displayed at a low brightness in an environment with an amount of ambient light 312a, and / or other user interface elements, as described below with reference to Figure 6 Described in more detail.

[0054] In some examples, electronic device 101 modifies the frame rate and fidelity metrics of the video of image 308 based on the context of the physical environment of electronic device 101. Electronic device 101 optionally makes a tradeoff between the frame rate indicated by scale 314 and video / image fidelity based on the context of the physical environment. Figures 3A to 3B In the example of , the electronic device 101 displays a video of the image 308 at a relatively high frame rate and relatively low fidelity (indicated by the value 316a of the scale 314). FIG. 3A to FIG. 3BAs shown, in the scenario of walking down the stairs in the dark, the frame rate may be more important to assist the user in moving in the physical environment. In some examples, the electronic device 101 uses one or more second criteria corresponding to a preference for frame rate rather than picture fidelity to determine that the scenario corresponds to a high frame rate and low fidelity. The second criterion is optionally based on motion data, line of sight data, semantic understanding of an image of a physical environment, and / or position data of the electronic device 101. Additional or alternative scenarios that meet one or more second criteria are possible. It should be understood that the scale 314 and other scales used herein are shown for schematic illustration and are not necessarily displayed by the electronic device 101.

[0055] For example, in FIG. 3A to FIG. 3B , motion data captured by electronic device 101 indicates that the amount of movement of the user is greater than a threshold amount. Electronic device 101 optionally captures motion data using an IMU, one or more cameras, and / or other motion sensors. In some examples, detecting motion above the threshold amount corresponds to increasing the frame rate of the video of image 308. Increasing the frame rate based on detecting motion above the threshold amount can be advantageous in maintaining the correspondence of image 308 with a portion of the physical environment in which user interface element 302 is located.

[0056] For example, in FIG. 3A to FIG. 3B , the line of sight data captured by the electronic device 101 indicates that the user is not reading or attempting to understand fine details of a portion of the physical environment. For example, the line of sight data corresponding to reading may have specific characteristics that the electronic device 101 can detect algorithmically, and when these characteristics are not detected, the electronic device 101 optionally determines that the user is not reading. For another example, the line of sight data may correspond to a user attempting to use their vision to walk steadily in the physical environment. The electronic device 101 optionally uses one or more cameras, other eye sensors, and / or other line of sight detection sensors to capture the line of sight data. In some examples, detecting line of sight data that does not correspond to reading or viewing fine details corresponds to reducing the fidelity of the picture quality, which optionally enables the electronic device 101 to increase the frame rate of the video of the image 308.

[0057] For example, in FIG. 3A to FIG. 3B In the example, the electronic device 101 uses the position data to make a trade-off between image fidelity and frame rate. The electronic device 101 optionally uses a global positioning satellite (e.g., GPS) sensor and / or other position sensors and / or a connection to other electronic devices via a wireless network (e.g., Bluetooth and / or WiFi) to determine the current location of the electronic device 101. In some examples, the electronic device 101 determines whether the current location of the electronic device 101 corresponds to a location where the user may prefer high fidelity or high frame rate for the video content of the image 308.

[0058] For example, in FIG. 3A to FIG. 3B In some examples, the electronic device 101 captures one or more images of the physical environment of the electronic device 101, and uses semantic understanding of the environment to determine that a relatively high frame rate is more important than preserving the fidelity of the picture quality. For example, the electronic device 101 uses algorithms and / or machine learning techniques to identify features of the physical environment, such as stairs. Additionally or alternatively, in some examples, the electronic device 101 uses semantic understanding to analyze other data related to the trade-off between image fidelity and frame rate, such as movement, line of sight, and position data.

[0059] FIG. 3A to FIG. 3B The electronic device 101 is shown displaying an image 308 within the user interface element 302. Additionally or alternatively, in some examples, the electronic device 101 can display one or more images that outline the edges of objects in the physical environment or otherwise annotate portions of the physical environment for improved situational awareness in low-light environments. For example, the electronic device 101 can display FIG. 3A to FIG. 3B In some examples, electronic device 101 displays an image outlining the edges of objects in the physical environment having one or more of the characteristics described above with reference to image 308, such as in response to detecting ambient light (in FIG. 3A to FIG. 3B The image is displayed when the frame rate is less than the threshold amount and / or the image is displayed with the frame rate and image fidelity trade-off selected according to one or more of the above criteria. Figure 8 Additional examples are described for augmenting the presentation of edges of objects in a real environment of an electronic device.

[0060] Figure 4 An electronic device 101 is shown displaying an image 408 of a portion of the physical environment of the electronic device 101 according to some examples of the present disclosure. For example, the electronic device 101 optionally has the same physical environment as described above. Figure 1 and / or Figure 2 The electronic device 101 and / or the electronic device 201 described have the same architecture.

[0061] In some examples, the electronic device 101 displays FIG. 3A to FIG. 3B For example, electronic device 101 displays image 408 based on determining that one or more first criteria similar to those described above are satisfied, the one or more first criteria including criteria based on the amount of ambient light 412a indicated by scale 410, criteria based on receiving one or more inputs, and / or criteria based on Figure 4 The physical environment of the electronic device 101 is described in detail below. Figure 4As shown, electronic device 101 displays image 408 in user interface element 402 similar to user interface element 302, which includes indication 404 similar to indication 304 and option 406 similar to option 306. In some examples, when electronic device 101 is in an environment with relatively low ambient light, the image 408 is displayed in a user interface element 402 similar to user interface element 302. Figure 4 When the electronic device 101 is in an environment with an amount of ambient light 412a (represented on scale 410) greater than the amount of ambient light 412a in the environment, the electronic device 101 displays the image 408 at a brightness lower than the amount of brightness used by the electronic device 101 to display images and / or other user interface elements. In some examples, the electronic device 101 displays the user interface element 402 overlaid on a view of a portion of the physical environment represented by the image 408.

[0062] Figure 4 The context and physical environment of FIG. 3A to FIG. 3B The physical environment in the context is different. For example, Figure 4 In some examples, the user of electronic device 101 optionally reads a menu in a dimly lit restaurant (or generally reads text in an environment with insufficient lighting). In some examples, electronic device 101 identifies a menu item based on satisfying one or more third criteria. Figure 4 The scenario of corresponds to a display video of image 408 having a higher fidelity and a lower frame rate (as indicated by value 416b of scale 414). For example, because a user may try to Figure 4 The third criterion is optionally based on motion data, line of sight data, semantic understanding of the image of the physical environment, and / or position data of the electronic device. Additional or alternative contexts that meet one or more third criteria are possible.

[0063] For example, in Figure 4 In some examples, the motion data captured by the electronic device 101 indicates that the user's movement is less than a threshold amount. In some examples, the motion data captured by the electronic device 101 indicates that the stability of the user's head / line of sight / overall position is at least a threshold amount. For example, detecting that the user is moving their hands can still include the user's overall movement below the threshold amount. The electronic device 101 optionally uses the above reference FIG. 3A to FIG. 3B One or more of the listed sensors captures motion data. In some examples, detecting motion below a threshold amount corresponds to increasing image fidelity of video of image 408. Increasing image fidelity based on detecting motion below a threshold amount can be advantageous in enabling a user to read objects in the physical environment of electronic device 101, such as a menu in a restaurant.

[0064] For example, in Figure 4In the embodiment of the present invention, the gaze data captured by the electronic device 101 indicates that the user is reading an object in a portion of the physical environment. For example, the user's gaze moves in a pattern associated with reading. For another example, the gaze data may not correspond to the user attempting to use their vision to walk steadily in the physical environment. The electronic device 101 optionally uses the above reference FIG. 3A to FIG. 3B One or more of the listed sensors captures gaze data. In some examples, detecting gaze data corresponding to reading or viewing fine details corresponds to improving the fidelity of picture quality, which optionally causes the electronic device 101 to reduce the frame rate of the video of the image 408.

[0065] For example, in Figure 4 In the embodiment, the electronic device 101 uses the position data to make a trade-off between image fidelity and frame rate. The electronic device 101 optionally uses the above reference FIG. 3A to FIG. 3B One or more of the listed sensors determines the current location of the electronic device 101. In some examples, the electronic device 101 determines whether the current location of the electronic device 101 corresponds to a location where the user may prefer high fidelity or high frame rate for the video content of the image 408. For example, the electronic device 101 optionally determines that the user is in a restaurant where the user may wish to read a menu using a higher fidelity image 408 generated by the electronic device 101.

[0066] For example, in Figure 4 In some examples, the electronic device 101 captures one or more images of the physical environment of the electronic device 101 and uses semantic understanding of the environment to determine that preserving the fidelity of the picture quality is more important than the frame rate. For example, the electronic device 101 uses algorithms and / or machine learning techniques to recognize features of the physical environment, such as writing on a menu. Additionally or alternatively, in some examples, the electronic device 101 uses semantic understanding to analyze other data related to the trade-off between image fidelity and frame rate, such as movement, line of sight, and position data.

[0067] Figure 5 An electronic device 101 is shown forgoing displaying an image of a portion of a physical environment with enhanced visibility based on the context of the physical environment of the electronic device 101 according to some examples of the present disclosure. Figure 5 The electronic device 101 optionally has a reference Figure 1 The electronic device 101 described and / or the above reference Figure 2 The electronic device 201 is described with the same architecture.

[0068] exist Figure 5 In some examples, the electronic device 101 operates in the physical environment of a movie theater. In some examples, the context is similar to abandoning the display of an image of a portion of the environment with enhanced visibility (e.g., similar to FIG. 3A to FIG. 3BThe image 308 and / or Figure 4 For example, in the context of a movie theater, a user may not want to view an image of a portion of the environment with enhanced visibility because their vision may adapt to the darkness in order to watch the movie.

[0069] In some examples, the electronic device 101 follows the above reference FIG. 3A to FIG. 4 One or more of the described first criteria identify a context. Figure 5 The first criterion that is not met in the example of is optionally based on one or more of motion data, position data, receiving or not receiving one or more user inputs, and / or semantic understanding of one or more images of the physical environment. For example, the position data may indicate that the electronic device 101 is being used in a movie theater, and the electronic device 101 may automatically abandon the display of an image of a portion of the environment with enhanced visibility in response to detecting the location of the electronic device 101 in or near the movie theater. For another example, the electronic device 101 may recognize that the electronic device 101 is in a movie theater based on the semantic understanding of one or more captured images of the physical environment of the electronic device 101. For another example, the electronic device 101 may receive one or more inputs requesting the electronic device 101 to abandon the display of an image of a portion of the environment with enhanced visibility. For example, the electronic device 101 presents a prompt confirming that the feature should be disabled in response to the semantic understanding of the position data and / or the image of the physical environment, and receives a user input to turn off the feature when the prompt is displayed. For another example, the electronic device 101 does not receive an input requesting to turn on the feature, so the electronic device 101 does not turn on the feature of displaying an image of a portion of the environment with enhanced visibility. Additional or alternative scenarios are possible where the ambient light (represented by the scale) is less than a predefined threshold but the electronic device 101 abstains from displaying an image of a portion of the environment with enhanced visibility.

[0070] Figure 6 An example of an electronic device 101 forgoing displaying an image of a portion of an environment with enhanced visibility based on determining that the amount of ambient light indicated by 612b (represented on scale 610) is above a predefined threshold in accordance with some examples of the present disclosure. Figure 6 The electronic device 101 optionally has a reference Figure 1 The electronic device 101 described and / or the above reference Figure 2 The electronic device 201 is described with the same architecture.

[0071] exist Figure 6 In the example of FIG. 1 , the electronic device 101 is operating outdoors on a sunny day. In this example, the amount of ambient light 612 b is greater than a predefined threshold. In some examples, based on determining that the amount of ambient light 612 b is greater than the threshold, the electronic device 101 abandons displaying an image of a portion of the environment with enhanced visibility (e.g., similar to FIG. 1 ). FIG. 3A to FIG. 3B The image 308 and / or Figure 4 Additionally or alternatively, in some examples, in response to receiving user input requesting electronic device 101 to abandon displaying the image of the portion of the environment with enhanced visibility, electronic device 101 abandons displaying the image of the portion of the environment with enhanced visibility. Additionally or alternatively, in some examples, electronic device 101 abandons displaying the image of the portion of the environment with enhanced visibility in response to not receiving input requesting display of the image.

[0072] like Figure 6 As shown, electronic device 101 displays a visual indication 602 of an incoming message received at electronic device 101 overlaid on a view of environment 600. In some examples, electronic device 101 displays a visual indication 602 of an incoming message received at electronic device 101 overlaid on a view of environment 600. FIG. 3A to FIG. 3B The image 308 and / or Figure 4 Visual indication 602 is displayed at a brightness higher than the brightness of image 408 in the image. In some examples, the brightness of indication 602 is greater than the brightness of image 308 or image 408 because Figure 6 The amount 612b of the ambient light 610 is greater than FIG. 3A to FIG. 3B or Figure 4 In some examples, even if the amount of ambient light is the same, electronic device 101 displays image 308 and image 408 at a lower brightness than electronic device 101 would display other elements such as indication 610. Additionally or alternatively, in some examples, electronic device 101 displays image 308 and image 408 at a lower brightness than electronic device 101 would display other elements such as indication 610. FIG. 3A to FIG. 3B The image 308 and / or Figure 4 The high brightness in the image 408 shows different virtual elements.

[0073] Thus, as described above, in some examples, the electronic device 101 provides enhanced low-light vision, including presenting an image of the physical environment of the electronic device 101 with enhanced visibility compared to the visibility of the physical environment without the enhancement. In some examples, the electronic device 101 activates the enhanced low-light vision feature in response to detecting that the ambient light level is below a threshold amount and / or in response to receiving an input requesting activation of the enhanced low-light vision feature. Additionally or alternatively, as described above with reference to Figures 3A to 5 As described, the electronic device 101 controls the tradeoff between the frame rate at which the electronic device 101 updates the enhanced image and the resolution of the enhanced image and / or whether to automatically activate the enhanced low-light vision feature (e.g., without receiving input requesting to activate the feature).

[0074] As described above, presenting an enhanced image of the physical environment of the electronic device 101 optionally includes adjusting the brightness, contrast, color saturation, sharpness, and / or clarity of the enhanced image. Optionally, the enhanced image is an image of a portion of the physical environment of the electronic device 101 visible through the display generation component 120, wherein other portions of the physical environment are presented without enhancement and / or with a lesser degree of enhancement. Additionally or alternatively, in some examples, the electronic device 101 applies a colored light filter (e.g., a red light filter) and / or edge enhancement to the enhanced image, as described below with reference to Figures 7 and 8 In some examples, the electronic device 101 will refer to Figures 7 and 8 The techniques described are similar to those in the above references Figures 3A to 6 The techniques described are combined together.

[0075] Figure 7 An example of an electronic device 101 presenting an image 708 with enhanced visibility including a color filter according to some examples of the present disclosure is shown. In some examples, the electronic device 101 presents the image 708 to the user interface element 702 using an enhanced visibility technique, which includes the same techniques as described above with reference to FIG. Figures 3A to 6 For example, the electronic device 101 is described above with reference to Figures 3A to 6 In one of the described cases, an image 708 including a color filter is presented, and / or the image 708 includes the above-referenced Figures 3A to 6 In some examples, user interface element 702 also includes a low light indication 704 similar to indication 304 and / or indication 404 described above and a zoom option 706 similar to zoom option 306 and / or zoom option 406 described above.

[0076] In some examples, the color filter applied to image 708 presents a specific color and removes or reduces a specific color from image 708. For example, image 708 includes a red light filter that presents image 708 that includes red and / or removes or reduces other colors (e.g., green and / or blue). Applying a red light filter can be advantageous because viewing red light does not affect a person's night vision to the same extent that viewing other colors of light affects a person's night vision. However, in some cases, electronic device 101 applies other color filters to image 708.

[0077] As described above, the electronic device 101 is used in a manner such as the one described above. Figures 3A to 6The image 708 is presented using a color filter in situations such as those described herein. For example, the electronic device 101 presents the image 708 based on a determination that a current ambient light level 712a in the physical environment of the electronic device 101 is less than a predefined threshold level (as indicated by the scale 710) and / or based on other determinations related to the physical environment of the electronic device 101 (such as the context of the physical environment described above). In some examples, the electronic device 101 applies the color filter based on a user-defined setting, such as a setting for selecting a visual effect for the electronic device 101 to be applied to an enhanced visibility image of the physical environment of the electronic device. Optionally, the setting includes an available configuration that combines one or more visual effects described herein for the electronic device 101 to apply to the enhanced visibility image of the physical environment.

[0078] Figure 8 An example of an electronic device 101 presenting an image 808 with enhanced visibility including edge enhancement according to some examples of the present disclosure is shown. In some examples, the electronic device 101 presents the image 808 to the user interface element 802 using an enhanced visibility technique, which includes the same techniques as described above with reference to FIG. FIG. 3A to FIG. 7 For example, the electronic device 101 in the above reference Figures 3A to 6 In one of the described cases, an image 808 including edge enhancement is presented, and / or the image 808 includes the above reference FIG. 3A to FIG. 7 In some examples, user interface element 802 also includes a low light indication 804 similar to indication 304, indication 404, and / or indication 704 described above and a zoom option 806 similar to zoom option 306, zoom option 406, and / or zoom option 706 described above.

[0079] In some examples, the edge enhancement applied by the electronic device 101 to the image 808 includes increasing the visibility of the edges of objects in the physical environment 800 of the electronic device 101. For example, the electronic device 101 displays the edges of the objects in the image 808 using highlights, outlines, and / or other added visual enhancements that are not visible or not easily visible in the physical environment 800 without the use of the electronic device 101. For example, the electronic device 101 adds an image of the boundaries between the objects in the environment 800 overlaid on the image 808 at locations corresponding to the boundaries between the objects in the image 808. To facilitate edge enhancement, the electronic device 101 identifies the edges of the objects in the image 808 using an image processing algorithm.

[0080] In some examples, electronic device 101 enhances the edges of all objects included in image 808. In some examples, electronic device 101 enhances the edges of some of the objects included in image 808 according to one or more criteria, but does not enhance the edges of other objects. For example, electronic device 101 enhances the edges of objects to which the user's attention (e.g., including line of sight) is directed, and does not enhance the edges of other objects in image 808 or enhances the edges of other objects in image 808 to a lesser extent. For another example, electronic device 101 enhances the edges of objects adjacent to the user's hands (e.g., objects that the user is touching and / or pointing to with his or her hands), and does not enhance the edges of other objects in image 808 or enhances the edges of other objects in image 808 to a lesser extent. In some examples, if image 808 includes the user's hands, such as Figure 8 In the case of the hand, the electronic device 101 does not use edge enhancement to display a portion of the image 808 including the hand, such as Figure 8 In some examples, electronic device 101 displays a portion of image 808 including a user's hand using edge enhancement.

[0081] As described above, the electronic device 101 is used in a manner such as the one described above. FIG. 3A to FIG. 7 The image 808 is presented using edge enhancement techniques in situations such as those described. For example, the electronic device 101 presents the image 808 based on a determination that a current ambient light level 812a in the physical environment of the electronic device 101 is less than a predefined threshold level (as indicated by the scale 810) and / or based on other determinations related to the physical environment of the electronic device 101 (such as the context of the physical environment described above). In some examples, the electronic device 101 applies edge enhancement techniques based on user-defined settings, such as settings for selecting visual effects for the electronic device 101 to be applied to an enhanced visibility image of the physical environment of the electronic device. In some examples, where the electronic device 101 applies edge enhancement to some objects in the image 808 but not to other objects in the image 808, the electronic device 101 applies other techniques, such as adjusting hue, saturation, contrast, and / or sharpness, to portions of the image that include the objects to which the electronic device 101 applies edge enhancement, and to portions of the image that do not include the objects to which the electronic device 101 applies edge enhancement.

[0082] Fig. 9 An example method 900 of selectively displaying an image of a portion of the physical environment of an electronic device 101 according to some examples of the present disclosure is shown. In some examples, the electronic device 101 is configured to display an image of a portion of the physical environment of the electronic device 101 according to the above reference. FIG. 3A to FIG. 8 One or more techniques described may be used to implement method 900 .

[0083] In some examples, method 900 includes presenting (902) a view of a physical environment of an electronic device using one or more displays. For example, FIG. 3A to FIG. 3B , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 Included are views of physical environments 300, 400, 500, 600, 700, and 800, respectively, as described in more detail above.

[0084] In some examples, method 900 includes, while presenting the view of the physical environment (906), displaying (908) using one or more displays one or more images of a portion of the physical environment that has a greater visibility than the physical environment based on determining that one or more first criteria are satisfied, as described above. FIG. 3A to FIG. 3B , Figure 4 , Figure 7 and Figure 8 , the electronic device 101 displays image 308, image 408, image 708 and image 808 respectively.

[0085] In some examples, method 900 includes, when presenting the view of the physical environment (906), based on determining that one or more first criteria are not met, abandoning (910) displaying one or more images of a portion of the physical environment, as described above. Figure 6 , the electronic device 101 abandons displaying an image similar to image 308 or image 408.

[0086] Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied when the electronic device receives input corresponding to a request to display one or more images of a portion of the physical environment using one or more input devices. Additionally or alternatively, in some examples, the one or more first criteria include criteria that are satisfied in response to sensing data indicating a first context of the physical environment using one or more sensors, wherein the criteria are not satisfied in response to sensing data indicating a second context of the physical environment different from the first context using one or more sensors. Additionally or alternatively, in some examples, displaying one or more images of a portion of the physical environment includes: displaying one or more images having a first fidelity metric and a first frame rate based on determining that one or more second criteria different from the one or more first criteria are satisfied; and displaying one or more images having a second fidelity metric less than the first fidelity metric and a second frame rate greater than the first frame rate based on determining that one or more third criteria different from the one or more first criteria and different from the one or more second criteria are satisfied. Additionally or alternatively, in some examples, the one or more second criteria include criteria that are satisfied when the electronic device senses first motion data indicating a movement less than a threshold amount of movement using an inertial measurement unit (IMU), and the one or more third criteria include criteria that are satisfied when the electronic device senses second motion data indicating a movement not less than the threshold amount of movement using the IMU. Additionally or alternatively, in some examples, the one or more second criteria include criteria satisfied when the electronic device senses first gaze data using a gaze tracking device, and the one or more third criteria include criteria satisfied when the electronic device senses second gaze data different from the first gaze data using a gaze tracking device. Additionally or alternatively, in some examples, the one or more second criteria include criteria satisfied when the electronic device senses data indicating a first context of the physical environment using one or more sensors, and the one or more third criteria include criteria satisfied when the electronic device senses data indicating a second context of the physical environment different from the first context of the physical environment using one or more sensors. Additionally or alternatively, in some examples, displaying one or more images of a portion of the physical environment includes displaying a selectable option using one or more displays, which, when selected, causes the electronic device to adjust a zoom level of one or more images of a portion of the physical environment. Additionally or alternatively, in some examples, displaying one or more images of a portion of the physical environment includes displaying one or more images of a portion of the physical environment with a first brightness, and the method further includes: when presenting a view of the physical environment, based on determining that one or more first criteria are not satisfied, including criteria satisfied when the ambient light level of the physical environment is less than a threshold amount: using one or more displays to display a user interface element different from the one or more images of a portion of the physical environment at a second brightness greater than the first brightness.Additionally or alternatively, in some examples, method 600 includes, while presenting the physical environment, displaying, using one or more displays, a user interface of a camera application, the user interface including a second image of a portion of the physical environment, the second image being different from one or more images of a portion of the environment, wherein: the one or more images of the portion of the environment have greater visibility than the second images. Additionally or alternatively, in some examples, displaying the one or more images includes applying one or more image processing techniques to the one or more images to a greater extent than the extent to which the one or more image processing techniques are applied to the second image when the second image is displayed. Additionally or alternatively, in some examples, the one or more images are displayed outside of the user interface of the camera application. Additionally or alternatively, in some examples, method 600 includes, while presenting the view of the physical environment: based on determining that one or more fourth criteria are satisfied, displaying, using the one or more displays, a visual indication of an edge of one or more physical objects in the physical environment overlaid on the view of the physical environment at a corresponding location of the edge of the one or more physical objects. Additionally or alternatively, in some examples, the one or more images include a video. Additionally or alternatively, in some examples, presenting the view of the physical environment includes presenting an extended reality environment based on the physical environment. Additionally or alternatively, in some examples, displaying one or more images of a portion of the physical environment includes displaying the one or more images overlaid on the view of the physical environment at corresponding locations of the portion of the physical environment.

[0087] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Based on the above teachings, many modifications and variations are possible. The examples are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so as to enable other technical personnel in the field to best use the present disclosure and the various described examples with various modifications suitable for the specific purpose contemplated.

Claims

1. A method, comprising: At an electronic device in communication with one or more displays: presenting a view of a physical environment of the electronic device using the one or more displays; While presenting the view of the physical environment: displaying one or more images of a portion of the physical environment using the one or more displays based on determining that one or more first criteria are satisfied, wherein the one or more first criteria include a criterion satisfied when an ambient light level of the physical environment is below a threshold amount of ambient light; as well as Based on determining that the one or more first criteria are not satisfied, displaying the one or more images of the portion of the physical environment is foregone.

2. The method of claim 1, wherein the one or more first criteria comprise criteria that are satisfied when the electronic device receives input corresponding to a request to display the one or more images of the portion of the physical environment using one or more input devices.

3. A method according to claim 1, wherein the one or more first criteria include a criterion that is satisfied in response to sensing data indicating a first context of the physical environment using one or more sensors, and wherein the criterion is not satisfied in response to sensing data indicating a second context of the physical environment that is different from the first context using the one or more sensors.

4. The method of claim 1 , wherein displaying the one or more images of the portion of the physical environment comprises: displaying the one or more images having a first fidelity metric and a first frame rate based on determining that one or more second criteria different than the one or more first criteria are satisfied; as well as Based on determining that one or more third criteria different from the one or more first criteria and different from the one or more second criteria are satisfied, the one or more images are displayed having a second fidelity metric less than the first fidelity metric and having a second frame rate greater than the first frame rate.

5. According to the method of claim 4, the one or more second criteria include criteria that are satisfied when the electronic device uses an inertial measurement unit (IMU) to sense first motion data indicating a movement less than a movement threshold amount, and the one or more third criteria include criteria that are satisfied when the electronic device uses the IMU to sense second motion data indicating a movement not less than the movement threshold amount.

6. A method according to claim 4, wherein the one or more second criteria include criteria that are satisfied when the electronic device uses a gaze tracking device to sense first gaze data, and the one or more third criteria include criteria that are satisfied when the electronic device uses the gaze tracking device to sense second gaze data different from the first gaze data.

7. A method according to claim 4, wherein the one or more second criteria include criteria that are satisfied when the electronic device uses one or more sensors to sense data indicating a first context of the physical environment, and the one or more third criteria include criteria that are satisfied when the electronic device uses one or more sensors to sense data indicating a second context of the physical environment that is different from the first context of the physical environment.

8. A method according to claim 1, wherein displaying the one or more images of the portion of the physical environment includes displaying a selectable option using the one or more displays, which, when selected, causes the electronic device to adjust the zoom level of the one or more images of the portion of the physical environment.

9. The method of claim 1 , wherein displaying the one or more images of the portion of the physical environment comprises displaying the one or more images of the portion of the physical environment having a first brightness, and the method further comprises: In presenting the view of the physical environment, in accordance with determining that the one or more first criteria are not satisfied, including a criterion that is satisfied when an ambient light level of the physical environment is less than a threshold amount: A user interface element of the one or more images other than the portion of the physical environment is displayed using the one or more displays at a second brightness greater than the first brightness.

10. The method according to claim 1, further comprising: When presenting the physical environment, displaying, using the one or more displays, a user interface for a camera application, the user interface comprising a second image of the portion of the physical environment, the second image being different from the one or more images of the portion of the environment, wherein: The one or more images of the portion of the environment have greater visibility than the second image.

11. The method according to claim 10, wherein: Displaying the one or more images includes applying one or more image processing techniques to the one or more images to a greater extent than the one or more image processing techniques are applied to the second image when displaying the second image. 12 . The method of claim 10 , wherein the one or more images are displayed outside of the user interface of the camera application.

13. The method according to claim 1, further comprising: While presenting the view of the physical environment: Based on determining that one or more fourth criteria are satisfied, displaying, using the one or more displays, visual indications of edges of one or more physical objects in the physical environment overlaid on the view of the physical environment at corresponding locations of the edges of the one or more physical objects.

14. The method of claim 1, wherein the one or more images comprise a video.

15. The method of claim 1, wherein presenting the view of the physical environment comprises presenting an extended reality environment based on the physical environment.

16. The method of claim 1, wherein displaying the one or more images of the portion of the physical environment comprises displaying the one or more images overlaid on the view of the physical environment at respective locations of the portion of the physical environment.

17. A method according to claim 1, wherein displaying the one or more images of the portion of the physical environment includes displaying the one or more images using a color filter that enhances one or more first colors in the one or more images relative to one or more second colors in the one or more images.

18. An electronic device, comprising: Memory; as well as One or more processors coupled to the memory and configured to perform the method according to any one of claims 1 to 17.

19. A non-transitory computer-readable storage medium storing instructions, which, when executed by an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 17.

Images