Managing devices with additional displays

Abstract

To provide one or more techniques for managing a virtual object between one or more displays.SOLUTION: According to one embodiment, an exemplary technique for displaying a virtual object is described. According to one embodiment, an exemplary technique for causing an object to transition between displays on the basis of direction information is described. According to one embodiment, an exemplary technique for displaying a corrected expression of the object is described. According to one embodiment, an exemplary technique for managing one or more displays on the basis of data related to one or more processes is described.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present disclosure relates generally to computer-generated reality environments, and more specifically to methods and techniques for managing devices with additional displays for use in computer-generated reality environments.

[0002] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 067461, entitled "MANAGING DEVICES HAVING ADDITIVE DISPLAYS," filed August 19, 2020, and U.S. Provisional Patent Application No. 62 / 907207, entitled "TECHNIQUES FOR MANAGING DEVICES HAVING ADDITIVE DISPLAYS FOR USE IN A COMPUTER-GENERATED REALITY ENVIRONMENT," filed September 27, 2019. The contents of the foregoing applications are incorporated herein by reference in their entireties. [Background technology]

[0003] Users often interact with computer-generated reality environments using wearable devices, such as head-mounted display (HMD) devices. The use of wearable devices is often limited by the battery life of the wearable device. Thus, one or more techniques are required to manage power usage to conserve battery life of a wearable device. Summary of the Invention

[0004] According to some embodiments, a method is performed in a system having one or more processors, memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The method includes displaying a first portion of a virtual object via the secondary display and displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0005] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display spanning the field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for displaying a first portion of a virtual object via the secondary display and displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0006] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs include instructions for displaying a first portion of a virtual object via the secondary display and displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0007] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs include instructions for displaying a first portion of a virtual object via the secondary display and displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0008] According to some embodiments, the system includes one or more processors. The system also includes means for displaying a first portion of the virtual object via the secondary display and means for displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object in accordance with a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0009] According to some embodiments, a method is performed in a system having one or more processors, memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The method includes receiving directional information corresponding to a position of an object outside of a field of view of the primary display of the HMD device and the secondary display of the HMD device. The method also includes, in response to receiving the directional information corresponding to a position of an object outside of a field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display in accordance with a determination that a first criterion is met that includes a criterion that is met when the directional information of the object is permitted to be displayed via the secondary display, and displaying a second representation of the directional information via the tertiary display in accordance with a determination that a second criterion is met that includes a criterion that is met when the directional information of the object is permitted to be displayed via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0010] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. The one or more programs also include instructions for, in response to receiving directional information corresponding to a position of an object outside the field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display in accordance with a determination that a first criterion is met including a criterion that is met when the directional information of the object is authorized to be displayed via the secondary display, and displaying a second representation of the directional information via the tertiary display in accordance with a determination that a second criterion is met including a criterion that is met when the directional information of the object is authorized to be displayed via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0011] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. The one or more programs also include instructions for, in response to receiving the directional information corresponding to a position of an object outside the field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display in accordance with a determination that a first criterion is met, the first criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the secondary display, and displaying a second representation of the directional information via the tertiary display in accordance with a determination that a second criterion is met, the second criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0012] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. The one or more programs also include instructions for, in response to receiving the directional information corresponding to a position of an object outside the field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display in accordance with a determination that a first criterion is met, the first criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the secondary display, and displaying a second representation of the directional information via the tertiary display in accordance with a determination that a second criterion is met, the second criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0013] According to some embodiments, the system includes one or more processors. The system also includes means for receiving directional information corresponding to a position of an object outside of a field of view of a primary display of a head mounted display (HMD) device and a secondary display of the HMD device. The system further includes means for, in response to receiving the directional information corresponding to a position of the object outside of a field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display in accordance with a determination that a first criterion is met, the criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the secondary display, and displaying a second representation of the directional information via the tertiary display in accordance with a determination that a second criterion is met, the criterion including a criterion that is met when the directional information of the object is permitted to be displayed via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0014] According to some embodiments, a method is performed in a system having one or more processors, memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The method includes detecting an object in a computer generated reality (CGR) environment at a first location, and displaying, in response to detecting the object in the CGR environment at the first location, a first modified representation of the object via the secondary display of the HMD device, the first modified representation being visually distinct from a first actual representation of the object in accordance with a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0015] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for detecting an object in a computer generated reality (CGR) environment at a first location, and displaying, in response to detecting the object in the CGR environment at the first location, a first modified representation of the object via the secondary display of the HMD device, the first modified representation being visually distinct from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0016] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs include instructions for detecting an object in a computer generated reality (CGR) environment at a first location, and in response to detecting the object in the CGR environment at the first location, displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinct from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0017] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs include instructions for detecting an object in a computer generated reality (CGR) environment at a first location, and in response to detecting the object in the CGR environment at the first location, displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinct from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0018] According to some embodiments, the system includes one or more processors. The system also includes means for detecting an object in a computer generated reality (CGR) environment at a first location. The system further includes means for displaying, in response to detecting the object in the CGR environment at the first location, a first modified representation of the object via a secondary display of the HMD device according to a determination that the first location is within a first predetermined distance outside the field of view of a primary display of the HMD device, the first modified representation of the object being visually distinct from a first actual representation of the object.

[0019] According to some embodiments, the method is performed in a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The method includes receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. In response to receiving the information corresponding to a change status of the process, the method also includes displaying, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that the first criterion is met, and displaying, via the tertiary display of the HMD device, a second representation corresponding to the status of the process, where the second representation is different from the first representation, according to a determination that the second criterion is met.

[0020] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display spanning the field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving information corresponding to a change in a status of the process. In response to receiving the information corresponding to the changed status of the process, the one or more programs also include displaying, via the secondary display of the HMD device, a first representation corresponding to the status of the process in accordance with a determination that a first criterion is met, and displaying, via the tertiary display of the HMD device, a second representation corresponding to the status of the process, where the second representation is different from the first representation in accordance with a determination that a second criterion is met.

[0021] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in a status of the process. In response to receiving the information corresponding to the changed status of the process, the one or more programs also include displaying, in accordance with a determination that a first criterion is met, a first representation corresponding to the status of the process via the secondary display of the HMD device, and in accordance with a determination that a second criterion is met, a second representation corresponding to the status of the process via the tertiary display of the HMD device, where the second representation is different from the first representation.

[0022] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having one or more sensors, a primary display spanning a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in a status of the process. In response to receiving the information corresponding to the changed status of the process, the one or more programs also include displaying, in accordance with a determination that a first criterion is met, a first representation corresponding to the status of the process via the secondary display of the HMD device, and in accordance with a determination that a second criterion is met, a second representation corresponding to the status of the process via the tertiary display of the HMD device, where the second representation is different from the first representation.

[0023] According to some embodiments, the system includes one or more processors. The system also includes means for receiving directional information corresponding to a position of an object outside the field of view of a primary display of a head mounted display (HMD) device and a secondary display of the HMD device. The system further includes means for, in response to receiving the information corresponding to a change status of the process, displaying, via the secondary display of the HMD device, a first representation corresponding to the status of the process in accordance with a determination that a first criterion is met, and displaying, via a tertiary display of the HMD device, a second representation corresponding to the status of the process in accordance with a determination that a second criterion is met, where the second representation is different from the first representation.

[0024] Executable instructions for performing these functions are optionally contained within a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. Executable instructions for performing these functions are optionally contained within a transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. In the following description, reference is made to the accompanying drawings, which form a part hereof and which illustrate several examples of the present disclosure. It is understood that other examples may be utilized and structural and operational changes may be made without departing from the scope of the present disclosure. The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0025] [Figure 1A] FIG. 1 illustrates an exemplary system for use in various computer-generated reality technologies. [Figure 1B] FIG. 1 illustrates an exemplary system for use in various computer-generated reality technologies. [Figure 2A]FIG. 1 illustrates an exemplary system including a device having an additional display for use in various computer-generated reality environments. [Figure 2B] FIG. 1 illustrates an exemplary system including a device having an additional display for use in various computer-generated reality environments. [Diagram 3] FIG. 1 illustrates an example virtual object according to some embodiments. [Figure 4] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Diagram 5] FIG. 1 illustrates an exemplary area on a device having an additional display for displaying virtual objects according to some embodiments. [Figure 6A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 6B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 6C] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 7] 1 is a flow diagram illustrating a method for displaying a virtual object according to some embodiments. [Figure 8A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 8B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 9A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 9B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 10A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 10B]FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 11A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 11B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 12A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 12B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 13A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 13B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 14A] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 14B] FIG. 1 illustrates an exemplary technique for displaying a virtual object, according to some embodiments. [Figure 15] 1 is a flow diagram illustrating a method for transitioning objects between displays based on orientation information according to some embodiments. [Figure 16] 1 is a flow diagram illustrating a method for displaying a modified representation of an object according to some embodiments. [Figure 17A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 17B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 18] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 19A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 19B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 20A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 20B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 20C] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 20D] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 21A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 21B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 21C] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 21D] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 22A]FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 22B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 23A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 23B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 24A] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 24B] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 24C] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Figure 24D] FIG. 1 illustrates an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. [Diagram 25] 1 is a flow diagram illustrating a method for managing one or more displays based on data related to one or more processes, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Various examples of electronic systems and techniques for using such systems in connection with various computer-generated reality technologies are described.

[0027] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses of sight, touch, hearing, taste, smell, etc.

[0028] In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's physical movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to conform to at least one law of physics. For example, a CGR system may detect the rotation of a person's head and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to properties of virtual objects within the CGR environment may be made in response to representations of physical movements (e.g., voice commands).

[0029] A person can sense and / or interact with a CGR object using any of the senses, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. In another example, an audio object can enable audio transparency that selectively incorporates ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or interact with only the audio object.

[0030] Examples of CGR include virtual reality and mixed reality.

[0031] A virtual reality (VR) environment (or virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with the virtual objects in the VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0032] In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to incorporate sensory input from, or representations of, a physical environment in addition to including computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, an MR environment is anywhere between, but not including, a fully physical environment at one end and a VR environment at the other end.

[0033] In some MR environments, computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Also, some electronic systems for presenting MR environments can track position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system can account for movement so that a virtual tree appears stationary relative to the physical ground.

[0034] Examples of MR include augmented reality and augmented virtuality.

[0035] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present the virtual objects on the transparent or semi-transparent display, whereby the person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment that are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment to perceive the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment shown on an opaque display is referred to as a "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses these images to present the AR environment on the opaque display. Further alternatively, the system may have a projection system that projects virtual objects into the physical environment, for example as holograms or onto physical surfaces, so that a person using the system perceives the virtual objects as superimposed on the physical environment.

[0036] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may transform one or more sensor images to apply a selected viewpoint (e.g., a viewpoint) that differs from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of it, such that the modified portion may be a representative version rather than a photorealistic version of the originally captured image. As a further example, the representation of the physical environment may be transformed by graphically removing or obscuring a portion of it.

[0037] An augmented virtual (AV) environment refers to a simulated environment in which the virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images captured of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0038] There are various types of electronic systems that allow a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated viewing capabilities, windows with integrated viewing capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to the person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In one example, a transparent or semi-transparent display may be configured to be selectively opaque. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems may also be configured to project virtual objects into the physical environment, for example as holograms or onto physical surfaces.

[0039] 1A and 1B illustrate an exemplary system 100 for use in various computer-generated reality technologies.

[0040] 1A, system 100 includes device 100a. Device 100a includes various components, such as a processor 102, RF circuitry 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of device 100a.

[0041] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop) and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, and the HMD device communicates with the base station device. In some examples, device 100a is implemented in the base station device or the HMD device.

[0042] 1B, in some examples, the system 100 includes two (or more) devices that communicate via wired or wireless connections, etc. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate via a communication bus 150 of the device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate via a communication bus 150 of the device 100c.

[0043] In some examples, the system 100 is a mobile device. In some examples, the system 100 is an HMD device. In some examples, the system 100 is a wearable HUD device.

[0044] System 100 includes a processor 102 and a memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) that store computer-readable instructions configured to be executed by processor 102 to perform the techniques described below.

[0045] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks such as the Internet, an intranet, and / or wireless networks such as cellular networks and wireless local area networks (LANs). RF circuitry 104 optionally includes circuitry for communicating using near-field and / or short-range communications such as Bluetooth®.

[0046] System 100 includes display 120. In some examples, display 120 includes a first display (e.g., a left-eye display panel) and a second display (e.g., a right-eye display panel), each display displaying an image to a corresponding eye of the user. The corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, resulting in a parallax effect that provides the user with an illusion of depth of the object on the display. In some examples, the first display can include multiple other displays (e.g., sub-displays), such as a primary display and a secondary display. In some embodiments, the primary display has a different resolution than the secondary display when system 100 is operating. In some examples, display 120 includes a single display (e.g., a first display or a second display). In some examples, the single display includes multiple other displays (e.g., sub-displays), such as a primary display and a secondary display. The corresponding images are displayed simultaneously on a first region and a second region of the single display for each eye of the user. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, resulting in a parallax effect that provides a user with the illusion of depth of the object on a single display.

[0047] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user input, such as tap and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0048] The system 100 includes an image sensor 108. The image sensor 108 optionally includes one or more visible light image sensors, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide-semiconductor (CMOS) sensor operable to capture images of physical objects from the real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR dot emitter, for emitting infrared light into the real environment. The image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. The image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of the physical objects from the system 100. In some examples, the system 100 detects the physical environment around the system 100 using a combination of a CCD sensor, an event camera, and a depth sensor. In some examples, the image sensor 108 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of a physical object in the real environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as a hand gesture. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0049] In some examples, system 100 includes a microphone 112. System 100 uses microphone 112 to detect sounds from a user and / or the user's real environment. In some examples, microphone 112 includes an array of microphones (including multiple microphones), optionally operating in tandem, such as to identify ambient noise or to locate a sound source within the space of the real environment.

[0050] System 100 includes an orientation sensor 110 for detecting an orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in a real environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0051] 2A illustrates a device 202. In some embodiments, the device 202 may include one or more components of the system 100, such as one or more processors (e.g., processor 102), memory (e.g., memory 106), a camera sensor (e.g., image sensor 108), or a motion sensor (e.g., orientation sensor 110). For example, the device 202 may be an embodiment of the device 100a of the system 100 illustrated in FIG. 1A. Alternatively, in some embodiments, the device 202 may be an embodiment of the second device 100c illustrated in FIG. 1B.

[0052] As shown in FIG. 2A , device 202 is a wearable device that includes a first display (e.g., a left-eye display panel of device 202) and a second display (e.g., a right-eye display panel of device 202). The first display and second display each include a primary display 204 that is surrounded by a secondary display 206. As shown in FIG. 2A , secondary display 206 surrounds at least a portion of primary display 204. In some embodiments, secondary display 206 extends across the entire first or second display. In some embodiments, primary display 204 overlaps only a portion (e.g., a center) of secondary display 206.

[0053] In some embodiments, the primary display 204 and the secondary display 206 are two different types of displays. In some embodiments, the primary display 204 is a waveguide display. In some embodiments, the secondary display 206 is an organic light emitting diode display.

[0054] 2A , primary display 204 is not only physically coupled (e.g., connected, attached) to secondary display 206, but is also electronically coupled (e.g., connected, attached) to secondary display 206. In some embodiments, one or more regions or additional displays may be positioned between primary display 204 and secondary display 206.

[0055] 2A, the primary display 204 and the secondary display 206 are shown as oval-shaped displays. However, it should be appreciated that the primary display 204 and the secondary display 206 can have a variety of shapes, such as a rectangular shape (e.g., a rectangular prism). Also, in some embodiments, the shape of the primary display 204 may differ from the shape of the secondary display 206. In some embodiments, if the primary display 204 and / or the secondary display 206 are rectangular in shape, the device 202 can apply visual effects to the primary display 204 and / or the secondary display 206 (e.g., blurring or fading corners of the primary display and / or the secondary display) such that the primary display 204 and / or the secondary display 206 appears round.

[0056] 2A , an edge 208 is defined between the primary display 204 and the secondary display 206. In some embodiments, the edge 208 is transparent and / or translucent. In some embodiments, the edge 208 is an edge of the primary display 204 or an edge of the secondary display 206. In some embodiments, the edge 208 is a physical boundary between the primary display 204 and the secondary display 206.

[0057] In some embodiments, the primary display 204 can operate at a higher maximum resolution than the maximum potential resolution of the secondary display 206. Because displays capable of operating at higher resolutions typically require more energy, the device 202 can use more energy when displaying a virtual object (or a CGR object) via the primary display 204 than when displaying the same object via the secondary display 206. Thus, the device 202 can preserve battery life based on a decision to display one or more virtual objects on the secondary display 206 instead of displaying the one or more virtual objects on the primary display 204.

[0058] 2A, the device 202 includes a frame 210 disposed around the secondary display 206. The frame 210 includes a tertiary display, such as a light emitting diode (LED). Alternatively, in some embodiments, the frame 210 does not include a tertiary display.

[0059] As described below, in some embodiments, the device 202 emits one or more LEDs. In some embodiments, the LEDs do not surround the interior of the secondary 206 located near the user's nose.

[0060] 2B, device 202 is shown being worn by user 250 such that the nose of user 250 is located under bridge 214 of device 202. As described below, device 202 enables user 250 to view the CGR environment.

[0061] Figure 2B shows a user 250 standing in a physical environment with the eyes of the user 250 in a position such that the pupils 216 of the user 250 are aligned with a portion of the primary display 204. As a result, in Figure 2B the central field of view or line of sight of the user 250 is aligned with a portion of the field of view of the primary display 204, which indicates that objects displayed on the primary display 204 are displayed in the user's central field of view.

[0062] 2B further illustrates the peripheral field of view of user 250 aligned with a portion of the field of view of secondary display 206, indicating that objects displayed on secondary display 206 are displayed in the user's peripheral field of view. As described below, in some embodiments, when one or more LEDs are illuminated, light is displayed outside or at the edge of the peripheral field of view of user 250. In some embodiments, one or more sensors of device 202 may track the movement of the user's eyes, including pupil 216. In some embodiments, device 202 uses the tracking of movement to adjust or modify one or more techniques described below.

[0063] Figure 3 illustrates an example virtual object according to some embodiments. In particular, Figure 3 illustrates the virtual object as a pencil 302 shaded in a single color (e.g., black, blue). The pencil 302 has different portions 302a-302c.

[0064] Figure 4 illustrates device 202 displaying a portion of pencil 302. In particular, in Figure 4, pencil 302 is shown falling from a position that is outside the field of view of primary display 204 and secondary display 206. Specifically, in Figure 4, pencil 302 is shown in a position where portions 302a and 302b of pencil 302 are within the field of view of primary display 204 and portion 302c of pencil 302 is within the field of view of secondary display 206, but where a portion of pencil 302, shown in dashed lines in Figure 4, is outside the field of view of primary display 204 and secondary display 206.

[0065] As shown in Figure 4, portion 302c is blurred in comparison to portions 302a and 302b, where the blurring of portion 302c is indicated by portion 302c being shaded less strongly than was shown in Figure 3. In contrast, portions 302a and 302b include little or no blurring, which is indicated by portions 302a-302b being shaded the same colors as were shown in Figure 3. Portion 302c has more blurring than portions 302a and 302b because portion 302c is displayed via secondary display 206, which may have a lower maximum resolution than the maximum resolution of primary display 204 displaying portions 302a-302b.

[0066] When device 202 is worn by a user, the edge between the user's central and peripheral vision is typically near region 404. As shown in Figure 4, region 404 corresponds to the area of ​​primary display 204 and secondary display 206 that display sections of portions 302b and 302c.

[0067] Because portion 302c is blurred compared to portion 302b, hard clipping may occur at region 404 and / or at the edge between the user's central and peripheral vision. In some embodiments, hard clipping occurs when there is a visual distortion between a portion of an object displayed on one display and a portion of an object displayed on another display. As described further below, hard clipping may occur when an object transitions from one display to another.

[0068] In some embodiments, hard clipping in region 404 is an unintended visual effect. In particular, hard clipping can cause the user interface displayed by device 202 to be distracting to a user wearing device 202. For example, the user interface can be distracting to a user wearing device 202 as objects move in and out of the user's central visual field.

[0069] FIG. 5 illustrates exemplary regions on a device having an additional display, according to some embodiments. As will be further described below, hard clipping may occur between one or more exemplary regions on the device. Because the field of view of the primary display 204 is typically in the user's central field of view and the field of view of the secondary display 206 is in the user's peripheral field of view, reducing hard clipping of objects transitioning between the primary display 204 and the secondary display 206 reduces high-frequency noise (e.g., snow blindness, random and / or sudden fluctuations of objects, irregular movements or spatial relationships between objects) in the user's central and / or peripheral fields of view. Reducing high-frequency noise in the user's central and / or peripheral fields of view is advantageous because high-frequency noise may cause the user to unnecessarily see objects in the user's peripheral field of view. Thus, high-frequency noise may affect the user's ability to focus on objects in the user's central field of view, which may affect the device 202's ability to track and / or make accurate decisions based on the user's eye movements.

[0070] 5 illustrates a device 202 having an exemplary first region 504a and an exemplary second region 504b of the primary display 204. The first region 504a extends across the area of ​​the primary display 204 from an edge 208 of the primary display 204 to a distance 510a from the edge 208 of the primary display 204. In some embodiments, at least a portion of the region 404 illustrated in FIG. 4 overlaps with a portion of the first region 504a illustrated in FIG. 5. In some embodiments, when a user is wearing the device 202, at least a portion of the edges of the user's central and peripheral vision are in a portion of the first region 504a.

[0071] The second region 504b extends across the area of ​​the primary display 204 from the center of the primary display 204 to a distance 510b toward the edge 208 of the primary display 204. In some embodiments, no portion of the region 404 shown in Figure 4 overlaps with any portion of the first region 504b shown in Figure 5. In some embodiments, when the user is wearing the device 202, no portion of the edge of the user's central and peripheral vision is present within any portion of the first region 504a.

[0072] In some embodiments, the first region 504a and / or the second region 504b extend across other regions of the primary display 204 not shown by Figure 5, or do not extend across the regions shown by Figure 5. In some embodiments, the first region 504a and / or the second region 504b extend across the entire primary display 204.

[0073] 6A-6C and 7 illustrate an exemplary technique for displaying virtual objects, according to some embodiments. In particular, the exemplary techniques and processes described below in FIGS. 6A-6C and 7 can reduce hard clipping of objects (e.g., abrupt changes when an object transitions from one display to another) when the object transitions between displays having different resolutions (e.g., between the primary display 204 and the secondary display 206), for example, as shown in FIG. 4 above. As described above, reducing hard clipping of objects transitioning between the primary display 204 and the secondary display 206 reduces high frequency noise in the user's central and / or peripheral vision. Reducing high frequency noise in the user's central and / or peripheral vision is advantageous because high frequency noise can affect the device 202's ability to track and / or make accurate decisions based on the user's eye movements.

[0074] Figure 6A shows an environment including device 202 and pencil 302. In Figure 6A, user 250 is wearing device 202, as described above in connection with Figure 2B. However, for ease of illustration, user 250 is not shown in Figure 6A.

[0075] 6A shows device 202 displaying pencil 302 falling from a position outside the field of view of primary display 204 and secondary display 206. In Figures 6A-6C, the portion of pencil 302 represented by dashed lines corresponds to the portion of 302 that is not visible when viewed through primary display 204 and / or secondary display 206 while a user is wearing device 202.

[0076] As shown in Figure 6A, the pencil 302 has been dropped from a position outside the field of view of the primary display 204 and the secondary display 206 to a position within the field of view of the secondary display 206 and outside the field of view of the primary display 204. In Figure 6A, portion 302a is presented via the secondary display 206. In some embodiments, the device 202 generates a virtual representation of portion 302a and displays the virtual representation via the secondary display 206.

[0077] In Figure 6A, device 202 displays a virtual representation of portion 302a presented at a resolution corresponding to the maximum resolution of secondary display 206. As shown in Figure 6A, portion 302a is blurred (e.g., indicated by its lighter shading) as compared to portion 302a in Figure 3. In some embodiments, the virtual representation of portion 302a displayed in Figure 6A does not have visual effects applied to create a blur change.

[0078] As the pencil 302 continues to fall for a period of time following that shown in Figure 6A, Figure 6B shows the portion 302a that has fallen within a particular region of the primary display 204 (e.g., region 504a described in Figure 5 below). While viewing the pencil 302 via the secondary display 206 (e.g., within the field of view of the secondary display 206), it is determined that the pencil 302 has fallen to a position within a distance 510a of the edge 208 of the primary display 204 (e.g., moving into the primary display 204), as shown in Figure 6B. Based on this determination, the device 202 generates a virtual representation by applying a visual effect to the portion 302a of the pencil 302, and displays the virtual representation on the primary display 204, as shown in Figure 6B.

[0079] Here, the virtual representation corresponding to portion 302a is blurred by applying a visual effect to portion 302a. As shown in FIG. 6B, applying the visual effect causes portion 302a to appear as a gradient (e.g., from light to dark as shown) to provide a smoother transition at the edges of the user's central and peripheral vision (and / or region 404 described in FIG. 4). In some embodiments, after applying the visual effect to portion 302a, the virtual representation corresponding to portion 302a is faded. In some embodiments, the virtual representation is faded inversely proportional to the detected amount of blur of secondary display 206 (or the amount of blur of at least one region of secondary display 206) after applying the visual effect to portion 302a. In some embodiments, the virtual representation does not include a gradient after applying the visual effect to portion 302a. In other embodiments, applying the visual effect includes applying a blur function to portion 302a. In some embodiments, the amount of blur applied to portion 302a at edge 208 can be selected to match the blur of secondary display 206. In doing so, the appearance of virtual object 302 on either side of edge 208 can be the same or similar, thereby reducing the hard clipping effect described above. The amount of blur applied to portion 302a (or other virtual content) within portion 504a can be decreased based on distance from edge 208, with no blur applied at distance D1 from edge 208. The blur can be decreased linearly or non-linearly with distance from edge 208.

[0080] As shown in Figure 6B, portion 302b has also moved into the field of view of secondary display 206. Thus, in addition to displaying a virtual representation corresponding to portion 302a on primary display 204, device 202 also displays a virtual representation corresponding to portion 302b of pencil 302. In some embodiments, device 202 displays a virtual representation corresponding to portion 302b using the techniques described above with respect to displaying a virtual representation corresponding to portion 302a in Figure 6A.

[0081] In particular, as shown in Figure 6B, the virtual representations corresponding to portions 302a and 302b are blurred, and this blurring is represented by differences in shading of the respective virtual representations corresponding to the respective portions of pencil 302. In Figure 6B, the virtual representation corresponding to portion 302b has more blurring than the virtual representation corresponding to portion 302a.

[0082] 6B, the virtual representation corresponding to portion 302b is blurred because it is displayed via secondary display 206, which is a lower resolution display that is blurrier than primary display 204. Meanwhile, the virtual representation corresponding to portion 302a is blurred because device 202 applied a visual effect to portion 302a before displaying the virtual representation corresponding to portion 302a via primary display 204. Thus, the virtual representation corresponding to portion 302a with the visual effect applied is more similar (e.g., has a closer blur value) to the virtual representation corresponding to portion 302b than the virtual representation corresponding to portion 302a without the visual effect applied.

[0083] In some embodiments, device 202 applies visual effects to portion 302a such that the virtual representation of portion 302a appears to visually match the virtual representation corresponding to portion 302b displayed on secondary display 206 (e.g., if no visual effects are applied to portion 302b). Because secondary display 206 has a lower resolution than primary display 204, device 202 applies visual effects to portion 302a to cause the visual appearance of the virtual representation corresponding to portion 302a to match the visual appearance of the virtual representation corresponding to portion 302b.

[0084] In some embodiments, device 202 applies visual effects to portion 302 to smooth the transition from displaying a virtual representation corresponding to portion 302a of FIG. 6A via a display having a lower resolution (e.g., a secondary display) to displaying a virtual representation corresponding to portion 302a of FIG. 6B via a display having a higher resolution (e.g., the primary display 204).

[0085] As shown in FIG. 6C, the pencil 302 has fallen to a new position that is within the field of view of the primary display 204 and the secondary display 206. As shown in FIG. 6C, the portion 302a is a distance from the edge 208 of the primary display 204 that is outside of the distance 510a (e.g., a distance that is the sum of the distance 510a and the distance 510b). Because the portion 302a is outside of the distance 510a from the edge 208 of the primary display 204, the portion 302a has entered a second region of the primary display 204 (e.g., region 504b in FIG. 5) that is outside of the first region of the primary display 204. Thus, no visual effects are applied to the virtual representation of the portion 302a that is in the second region 504b (shown in FIG. 5). In some embodiments, no visual effects (or visual effects different from the visual effects applied to the first region) are applied to the representation of the portion of the virtual object in the second region 504b of the primary display 204. In some embodiments, no visual effects are applied because portion 302a is within the user's central visual field and device 202 determines that the user would likely be distracted by a virtual object (e.g., or a particular visual object) without the visual effects applied. As shown in FIG. 6C, device 202 displays a virtual representation of portion 302c, a new portion of pencil 302, within the field of view of secondary display 206, using the techniques described above with respect to displaying the virtual representation of portion 302a in FIG. 6A. Additionally, in FIG. 6C, device 202 displays a virtual representation of portion 302b with the visual effects applied because portion 302b is within distance 510a from edge 208 of primary display 204 (e.g., using techniques similar to those described above with respect to portion 302a in FIG. 6C).

[0086] In some embodiments, if an object (e.g., pencil 302) is entirely within the field of view of the primary display (e.g., no part of the object is within the field of view of the secondary display), no visual effects are applied to any portion of the representation of the virtual object, including the portion of the representation of the virtual object displayed in second region 504b described above with respect to FIG. 5.

[0087] In some embodiments, the device 202 may determine to display different virtual objects differently. In some embodiments, the device 202 may apply a visual effect to one virtual object while not applying the visual effect to another virtual object. In some embodiments, the device 202 may apply different visual effects to virtual objects. In some embodiments, the device 202 uses different determinations to apply visual effects to different sets of virtual objects. In some embodiments, the device 202 may apply a visual effect to one type of virtual object based on a determination that the virtual object has entered an area of ​​the display, but may forgo applying the visual effect to another type of virtual object based on a determination that another type of virtual object has entered the area.

[0088] 6A-6C may be reversed (e.g., FIGS. 6C-6A). For example, a virtual object may transition from being within the field of view of primary display 204 and / or secondary display 206 to being outside the field of view of primary display 204 and secondary display 206 using techniques similar to those described above.

[0089] 7 is a flow diagram illustrating a method 700 for displaying a virtual object, according to some embodiments. In some embodiments, the method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, the method is performed by device 202 (FIG. 2A). In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0090] At block 702, a first portion of the virtual object is displayed via the secondary display. For example, as described above with reference to FIG. 6C, the first portion 302c of the virtual object 302 is displayed via the secondary display 206. In some embodiments, at a first time, the virtual object is presented on the secondary display (e.g., secondary display 206) without displaying the virtual object on the primary display (e.g., primary display 204). In some embodiments, when the virtual object is presented on the secondary display but the virtual object is not presented on the primary display, the virtual object is within the field of view of the secondary display and the virtual object is not within the field of view of the primary display.

[0091] 7, at block 704, a second portion of the virtual object is displayed via the primary display. For example, as described above with reference to FIG. 6C, the second portion 302b of the virtual object 302 is displayed via the primary display 204. In some embodiments, the second portion of the virtual object is displayed via the primary display when the device moves within a predetermined distance from the virtual object. In some embodiments, the second portion of the virtual object is displayed via the primary display when the virtual object moves within a predetermined distance from an edge of the primary display or within a predetermined distance from the field of view of the primary display (e.g., an edge of the field of view).

[0092] 7, as part of displaying the second portion of the virtual object (block 704), visual effects are applied to the second portion of the virtual object in accordance with a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display in block 706. For example, as part of displaying the second portion 302b of the virtual object 302, as described above with reference to FIG. 6C, visual effects are applied to the second portion 302b of the virtual object 302 in accordance with a determination that the second portion 302b is within a predetermined distance 510a from an edge of the primary display 204.

[0093] In some embodiments, visual effects are applied to portions of the one or more virtual objects while the portions of the one or more virtual objects are in a first region of the primary display. For example, as described above in connection with FIGURE 6C, visual effects are applied while second portion 302b is displayed in region 504a shown in FIGURE 5. In some embodiments, region 504a is within a predetermined distance from an edge of the primary display that is adjacent to an edge of the secondary display.

[0094] In some embodiments, the amount of visual effects applied may be the same for two or more virtual objects in a particular region. Alternatively, in some embodiments, the amount of visual effects applied to a particular virtual object is dynamic. For example, the amount of visual effects applied may be based on one or more detected characteristics associated with a user wearing the device or based on one or more user settings. In some embodiments, the user characteristics may include the user's pupil size, pupil direction, pupil position, ocular dominance, blink rate, etc. at different times.

[0095] In some embodiments, the amount of visual effects applied depends on one or more characteristics of the virtual object. For example, in some embodiments, the amount of visual effects applied depends on the size, movement (e.g., speed, acceleration), color, etc. of the virtual object. In some embodiments, the amount of visual effects applied is less for virtual objects moving at slower velocities and more for virtual objects moving at faster velocities.

[0096] In some embodiments, the amount of visual effects applied depends on one or more characteristics of the device, such as the speed or acceleration of the device, hi some embodiments, the amount of visual effects applied is less when the device is moving at a slower speed and more when the device is moving at a faster speed.

[0097] In some embodiments, visual effects are applied to a portion of a virtual object when it is displayed in one area of ​​the primary display, but visual effects are not applied to another portion of the virtual object that is displayed in another area of ​​the primary display. For example, as described above in connection with FIG. 6C, portion 302a does not have visual effects applied to it while it is displayed in area 504b shown in FIG.

[0098] In some embodiments, applying the visual effect to the second portion of the virtual object includes blurring the second portion of the virtual object. In some embodiments, prior to applying the visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first blurred appearance. In some embodiments, after applying the visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second blurred appearance that is different (e.g., more blurred) than the first blurred appearance.

[0099] In some embodiments, a second portion of the virtual object is blurred based on characteristics of a pupil aligned with a portion of the primary and / or secondary display detected via one or more image sensors. In some embodiments, a portion of the virtual object is blurred or faded in response to one or more characteristics of the detected pupil, while another portion of the virtual object is not. In some embodiments, the blurring or fading of a portion of the virtual object increases as the detected pupil is further away from an edge of the primary display.

[0100] In some embodiments, after blurring the second portion of the virtual object, the second portion of the virtual object includes an amount of blur based on the amount of blur of the first portion of the virtual object. In some embodiments, the amount of blur (e.g., of the portion of the virtual object) matches the blur of the first portion of the virtual object. In some embodiments, as part of blurring the second portion of the virtual object, the amount of blur of the second portion of the virtual object changes based on the corresponding distance between the virtual object and the secondary display (e.g., continuously as the distance between the object and the device 202 changes).

[0101] In some embodiments, as part of applying the visual effect to the second portion of the virtual object, the second portion of the virtual object is faded (e.g., faded in or faded out). In some embodiments, fading the second portion of the virtual object includes feathering edges of the primary display and / or the virtual object adjacent to the second area.

[0102] In some embodiments, prior to applying the visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first faded appearance. In some embodiments, after applying the visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second faded appearance that is different (e.g., more faded) than the first faded appearance.

[0103] In some embodiments, the second portion of the virtual object is faded in proportion to the blur of the secondary display (e.g., display 206) (e.g., at least one region of the secondary display). In some embodiments, the second portion of the virtual object is faded out to a level inversely proportional to the blur of the secondary display.

[0104] In some embodiments, the second portion of the virtual object is faded from the first color to the second color. In some embodiments, the second portion of the virtual object is faded such that a gradient is applied to the second portion of the virtual object. In some embodiments, if a gradient is applied to the second portion of the virtual object, the second portion of the virtual object includes a gradient (e.g., a gradient that transitions from the first color to the second color).

[0105] In some embodiments, as part of applying the visual effect to the second portion of the virtual object, the corners of the second portion of the virtual object are rounded. In some embodiments, the edges between the primary display and the secondary display (and / or the first region) are rounded or modified to appear rounded. In some embodiments, the corners are faded or have a particular visual effect applied to them (e.g., a visual effect that fades the corners is applied to the corners, while other portions of the second portion of the virtual object are not faded or do not have the particular visual effect applied). In some embodiments, when the corners are faded, a gradient is applied to the corners.

[0106] 7, as part of displaying the second portion (block 704), applying visual effects to the second portion of the virtual object is abandoned pursuant to a determination that the second portion of the virtual object is not within a predetermined distance from an edge of the primary display. For example, as described above with reference to FIG. 6C for displaying the second portion 302c of the virtual object 302 via the primary display 204, applying visual effects to the second portion 302c of the virtual object 302 is abandoned pursuant to a determination that the second portion 302c of the virtual object 302 is not within a predetermined distance 510a from an edge of the primary display 204.

[0107] 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, and 14A-14B illustrate example techniques for transitioning objects between displays, according to some embodiments. Among other advantages, the techniques and processes described below help conserve battery life of the device 202 while continuing to maximize the device's ability to present information (e.g., present information to the user 250 of FIG. 2B). The device 202 conserves battery life by intelligently shifting the presentation of information from the primary display 204 to one or more of the secondary display 206 and the tertiary display (e.g., LEDs around the frame 210). By shifting the information, battery life is preserved because the secondary display 206 and the tertiary display can have a lower maximum resolution than the resolution of the primary display 204. Thus, the secondary display 206 and the tertiary display can use less energy to display information than the primary display 204.

[0108] 8A, 9A, 10A, 11A, 12A, 13A, and 14A illustrate an example scenario in which a user 250 interacts with an object 806 while wearing a device 202. However, this particular scenario is provided for illustrative purposes only. The techniques described below, or similar techniques, may be effectively applied to a variety of other scenarios.

[0109] 8A illustrates a scenario in which a user 250 is wearing a device 202. In this scenario, the device 202 is positioned a distance 804a away from an object 806, such as a set of keys. In some embodiments, the object 806 is a physical item or object located within the physical environment in which the user 250 is located. In some embodiments, the object 806 is a virtual object located within the CGR environment that is displayed to the user 250 via the device 202.

[0110] In some embodiments, the object 806 moves while the user 250 and / or the device 202 are stationary, or alternatively, non-stationary.

[0111] In some embodiments, the environment includes one or more objects other than object 806. In some embodiments, device 202 detects that object 806 is in the environment shown in FIG. 8A. In some embodiments, device 202 determines that the object is a useful or important object (e.g., more useful or important than other objects in the environment). In some embodiments, device 202 can use one or more algorithms and / or one or more user preferences to determine whether an object is useful or important to a user or an application corresponding to a particular object (e.g., an application communicating with device 202). In some embodiments, device 202 receives a request or notification regarding the object and makes this determination based on whether the object is useful or important based on the request or notification.

[0112] Figure 8B illustrates the device 202 at a time corresponding to the scenario illustrated in Figure 8A. The device 202 includes (in addition to one or more components described in Figure 2A above) a primary display 204 and a secondary display 206. In Figure 8B, the primary display 204 and the secondary display 206 are not displaying (or presenting) visual information related to directional information corresponding to the position of the object 806 relative to the device 202 (directional information). In some embodiments, at least some visual information is not displayed via the primary display 204 and / or the secondary display 206 because the device 202 is too far from the object 806 (or the distance 804a exceeds a predetermined threshold distance).

[0113] As shown in Figure 8B, the device 202 further includes LEDs 810a-810f and 812a-812f (LEDs). In Figure 8B, the LEDs are not emitting light (e.g., emitting light to indicate directional information). In some embodiments, one or more of the LEDs are not emitting light because the device 202 is too far from the object 806 (or the distance 804a exceeds a predetermined threshold distance).

[0114] FIG. 9A shows the object 806 and / or device 202 moved from the position of FIG. 8A to the position of FIG. 9A.

[0115] 9B illustrates the device 202 at a time corresponding to the scenario depicted in FIG. 9A. With reference to FIGS. 9A and 9B, assume that the device 202 receives directional information when the device 202 is a distance 804b away from the object 806. In response to receiving the directional information, the device 202 determines that the object 806 is a type of object for which directional information may be displayed (e.g., configured and determined to be so) via one or more LEDs. In some embodiments, the device 202 makes this determination because the device 202 is a distance 804b away from the object 806 (or the distance 804b is less than a predetermined threshold distance (e.g., a threshold distance corresponding to a maximum distance for displaying directional information).

[0116] As shown in Fig. 9B, because the device 202 is a distance 804b away from the object 806 (or the object 806 is a type of object for which directional information can be displayed via one or more LEDs), the device 202 illuminates the LEDs 812a-812f in a pattern to indicate directional information. As shown in Fig. 9B, the LEDs 812a-812f are on the right side of the device 202 in an upward rather than downward orientation as positioned on the frame 210 of the device 202. Thus, the LEDs 812a-812f indicate that the object 806 is in front of and to the right of the device 202, which coincides with the general direction of the path between the device 202 and the object 806 in Fig. 9A.

[0117] 9B, LEDs 812a and 812f are hatched while LEDs 812b-812e are solid to indicate a gradient field of light, where the gradient field of light can indicate points of weak directional pull (e.g., hatched LEDs 812a, 812f) to points of strong directional pull of directional information between object 806 and device 202 (e.g., solid LEDs 812b-812e). In particular, the gradient field can provide more directional information to the user, which can help the user navigate more efficiently towards object 806.

[0118] In some embodiments, at distances equal to or greater than a predetermined threshold distance (e.g., distance 804b), the directional information is displayed via LEDs rather than the primary display 204 and / or secondary display 206. In some embodiments, the directional information is displayed via LEDs rather than the primary display 204 and / or secondary display 206 because displaying the directional information via LEDs instead of the primary display 204 and / or secondary display 206 conserves battery power.

[0119] In some embodiments, directional information is displayed via LEDs rather than the primary display 204 and / or secondary display 206 based on the type of information the device 202 is permitted to display when it is a particular distance away from the object 806. In some embodiments, the device 202 displays directional information that is not based on the object's appearance (e.g., a blinking LED) when the device 202 is further away from the object 806 (or within a first set of distances from the object). In some embodiments, the device 202 displays directional information that is based on the object's appearance (e.g., a visual representation of the object or a proxy object displayed via the primary display 204 and / or secondary display 206) when the device 202 is closer to the object (or within a second set of distances from the object that is different from the first set of distances). In some embodiments, the device 202 can display directional information based on the appearance of the object and directional information not based on the appearance of the object (e.g., when the device 202 is within a third set of distances from the object that is between the first set of distances and the second set of distances from the object. In some embodiments, the directional information is displayed via LEDs rather than on the primary display 204 and / or secondary display 206 based on whether the object is within the FOV of the primary display 204 and / or secondary display 206. In some embodiments, the directional information is displayed via LEDs rather than on the primary display 204 and / or secondary display 206 when the object is a certain distance outside the FOV of the primary display 204 and / or secondary display 206.

[0120] Figure 10A shows object 806 and / or device 202 having been moved from the position in Figure 9A to the position in Figure 10A. In particular, in Figure 10A, device 202 is positioned at a distance 804c from object 806, where distance 804c is different from distance 804b in Figure 9A.

[0121] FIG. 10B illustrates device 202 at a time corresponding to the scenario illustrated in FIG. 10A. With reference to FIGS. 10A and 10B, assume that device 202 receives directional information. In response to receiving the directional information, using techniques similar to those described above with respect to FIGS. 9A and 9B, device 202 determines that object 806 is a type of object for which directional information may be displayed via LEDs (e.g., because device 202 is a distance 804c away from object 806). Then, based on this determination, device 202 causes LEDs 812d-812f to emit light corresponding to the directional information and causes LEDs 812a-812c to cease emitting light. Now, the updated LEDs of device 202 indicate the newly received directional information (e.g., indicating the location of object 806 below and directional information of a relative path to the left of the location of device 202).

[0122] As shown in Figure 10B, the device 202 reduces the number of illuminated LEDs from the number of LEDs that were illuminated in Figure 9B. In some embodiments, when the device 202 is closer to the object 806 than in Figure 9B (e.g., in some embodiments, fewer LEDs can provide better directional information when the user 250 is closer to the object). In some embodiments, when there are physical and / or virtual structures or objects between the device 202 and the object 806, the device 202 continuously updates the LEDs (and / or the primary display 204 and / or the secondary display 206) of the device 202 (e.g., LEDs illuminating in a pattern) to enable the user 250 to navigate around the physical and / or virtual structures or objects between the device 202 and the object 806.

[0123] Figure 11A shows object 806 and / or device 202 having been moved from the position in Figure 10A to the position in Figure 11A. In particular, in Figure 11A, device 202 is positioned at a distance 804d from object 806, where distance 804d is different from distance 804c in Figure 10A.

[0124] Figure 11B illustrates device 202 at a time corresponding to the scenario illustrated in Figure 11A. With reference to Figures 11A and 11B, assume that device 202 updates secondary display 206 in addition to the LED because device 202 is closer (e.g., within a set of predetermined threshold distances) at distance 804d from object 806 (as opposed to device 202 at distance 804c from object 806 in Figure 10B). In some embodiments, device 202 updates secondary display 206 in addition to the LED because object 806 is within the FOV of secondary display 206.

[0125] Also assume that device 202 receives directional information. In response to receiving the directional information, using techniques similar to those described above with respect to Figures 9A and 9B, device 202 determines that object 806 is a type of object for which directional information can be displayed via LED 812e (e.g., because device 202 is a distance 804d away from object 806). Based on this determination, device 202 causes LED 812e to emit light corresponding to the directional information and causes LEDs 812d and 812f to cease emitting light, using one or more techniques described above with respect to Figures 9B and 10B.

[0126] If the device 202 determines that the object 806 is a type of object for which directional information can be displayed via the secondary display 206, then the device 202 displays the directional information via the secondary display 206. The device 202 makes this determination because it is now a distance 804d away from the object 806.

[0127] In some embodiments, because the device 202 is a distance 804d away from the object 806 and / or the object 806 is within a predetermined threshold distance outside the field of view of the primary display 204, the device 202 causes the secondary display 206 to display a modified representation 826a of the object 806, where the modified representation 826a is a representation of a proxy object or a symbolic representation of the object 806 that is visually distinguishable from the object 806. As shown in FIG. 11B, the modified representation 826a is a blob of pixels at the edge of the secondary display 206.

[0128] In some embodiments, when the device 202 is a distance 804d away from the object 806, the object 806 is in the field of view of the secondary display 206. In some embodiments, the object 806 is in a position in the field of view of the secondary display 206 that corresponds to the position on the secondary display 206 where the modified representation 826a is displayed.

[0129] In some embodiments, when the device 202 is a distance 804d away from the object 806, the object 806 is outside the field of view of the secondary display 206. Thus, in some embodiments, the device 202 displays the modified representation 826a via the secondary display 206 when the object 806 is outside the field of view of the secondary display 206. In some embodiments, displaying the modified representation 826a in a corresponding position on the secondary display 206 when the object 806 is outside the field of view of the secondary display 206 provides directional information for locating the object that is outside the field of view of the display of the device 202.

[0130] 11B, the modified representation 826a does not resemble the actual representation of the object 806. In some embodiments, the modified representation 826a does not resemble the object 806 because it does not have two or more characteristics of the object 806, such as the size, shape, color, texture, blur, etc.

[0131] The modified representation 826a indicates that the object (e.g., an object that is useful or important as described above in connection with FIG. 8A ) is within or outside of the user's peripheral vision (e.g., the peripheral vision of a user receiving visual information presented or displayed via the secondary display 206). The modified representation 826a may also indicate a general direction of the object's location relative to the location of the device 202. In some embodiments, the general direction is indicated by placing the modified object representation 826a in a particular location on the secondary display 206.

[0132] In some embodiments, the modified representation 826a is static (e.g., does not move, pulsate, or animate), but the light emanating from one or more LEDs is not static (e.g., moves, pulsates, animates), or vice versa. In some embodiments, the modified representation 826a and the LEDs 812a-812l are both static (e.g., or not static).

[0133] Figure 12A shows object 806 and / or device 202 moved from the position in Figure 11A to the position in Figure 12A. In particular, in Figure 12A, device 202 is positioned at a distance 804e from object 806, where distance 804e is different from distance 804d in Figure 11A.

[0134] Figure 12B illustrates device 202 at a point in time corresponding to the scenario depicted in Figure 12B. With reference to Figures 12A and 12B, assume that device 202 receives new directional information. Now, based on the new directional information, device 202 updates primary display 204 and secondary display 206 to display a representation of the directional information and abandons displaying the directional information via the LEDs.

[0135] In some embodiments, because the device 202 is a distance 804e away from the object 806, the device 202 determines that the object 806 is not a type of object for which directional information can be displayed via the LEDs 812a-812l. For example, the device 202 may determine that the distance 804e is a distance at which the device 202 is too close to the object 806. In some embodiments, the device 202 stops displaying the directional information via the LEDs once the directional information or one or more types of directional information (e.g., representation 828) is displayed via the primary display 204 and / or secondary display 206.

[0136] In some embodiments, in response to receiving the directional information, the device 202 determines, using techniques similar to those described above in FIGURE 11B, that the object 806 is a type of object for which directional information of the object may be displayed via the secondary display 512 (e.g., because the device 202 is a distance 804e away from the object 806 and / or the object 806 is within a predetermined threshold distance outside the field of view of the primary display 204). To make this determination, the device 202 causes the secondary display 206 to display a modified representation 826b.

[0137] In Figure 12B, modified representation 826b is larger than modified representation 826a because device 202 is closer to object 806 in Figure 12B than in Figure 11B. As shown in Figure 12B, modified representation 826b is displayed in a new position on secondary display 206 that is closer to primary display 204 than the position of modified representation 826a as shown in Figure 11A. Modified representation 826b is displayed in the new position because object 806 is closer to the field of view of primary display 204.

[0138] In some embodiments, the modified representation 826b occupies a portion of the secondary display 206 that was occupied by the modified representation 826a. In some embodiments, the modified representation 826b grows (e.g., via animation) from the modified representation 826a. In some embodiments, the device 202 displays the modified representation 826b and / or the modified representation 826a at a brightness based on the size of the object 806. For example, the device 202 displays the modified representation of a large object at a lower or dimmer brightness than the brightness of the modified representation of a small object.

[0139] In some embodiments, in response to receiving the directional information, the device 202 determines that the object 806 is a type of object for which directional information of the object may be displayed via the primary display 204. In Figure 12B, the device 202 makes this determination based on a determination that the distance 804e is sufficiently close to the object 806. Based on this determination, the device 202 causes the representation 828 to be displayed via the primary display 204.

[0140] 12B , representation 828 is an arrow pointing to the location of object 806. In some embodiments, device 202 displays representation 828 to indicate that user 250 can adjust the field of view of primary display 204 (e.g., by moving user 250's head while standing still) so that object 806 can be positioned within the field of view of primary display 204. In some embodiments, device 202 animates representation 828. In some embodiments, device 202 causes representation 828 (or a representation corresponding to representation 828) to be displayed via secondary display 206.

[0141] Figure 13A shows object 806 and / or device 202 moved from the positions of Figure 12A to the positions of Figure 13A. In particular, in Figure 13A, device 202 is positioned at a distance 804f from object 806, where distance 804f is different than distance 804e of Figure 12A. Additionally, as shown in Figure 13A, object 806 is within the field of view of primary display 204 of device 202.

[0142] FIG. 13B illustrates the device 202 at a time corresponding to the scenario illustrated in FIG. 13A. With reference to FIGS. 13A and 13B, assume that the device 202 receives new direction information. Because the device 202 is a distance 804f away from the object 806 and / or at least a portion of the object 806 is within the field of view of the primary display and within the field of view of the secondary display, the device 202 displays an actual representation of the portion of the object 806 via the primary display 204 and a modified representation 826c of the object 806 via the secondary display 206. In some embodiments, the actual representation of the portion of the object resembles the object 806 (e.g., has two or more characteristics that are similar to the object 806), but the modified representation does not resemble it. In some embodiments, the actual representation of the portion of the object is displayed using a first model of the object, but the modified representation is displayed using a second model of the object. In some embodiments, the first model of the object is different from the second model of the object because the second model of the object is simpler than the first model of the object. In some embodiments, the second model of the object is simpler than the first model of the object because the second model has less detail (eg, a three-dimensional model with fewer polygons) than the first model.

[0143] In some embodiments, the portion of the modified representation 826b moves into the primary display 204 by applying a visual effect to the virtual representation of the portion of the object 806 using techniques similar to those described above in Figure 7 (e.g., to reduce clipping of the object 806 transitioning between the primary display 204 and the secondary display 206). In some embodiments, using techniques similar to those described above in Figure 7, the device 202 replaces the modified representation 826b or 826c with a representation of the actual portion of the object 806 on the secondary display 206 when transitioning the actual representation of the portion of the object 806 displayed via the primary display 204.

[0144] Figure 14A shows object 806 and / or device 202 moved from the position in Figure 13A to the position in Figure 14A. In particular, in Figure 14A, device 202 is positioned at a distance 804g from object 806, where distance 804g is different than distance 804f in Figure 13A. Additionally, as shown in Figure 14A, the entirety of object 806 is within the field of view of primary display 204 of device 202.

[0145] Figure 14B illustrates the device 202 at a point in time corresponding to the scenario illustrated in Figure 14A. With reference to Figures 14A and 14B, the device 202 displays the actual representation of the object 806 (or, in some embodiments, a portion of the object 806) displayed via the primary display 204 and stops displaying the modified representation of the object 806 via the secondary display 206. In some embodiments, the device 202 produces the actual representation of the object 806 via the primary display 204 and stops displaying the modified representation of the object 806 via the secondary display 206 because the device 202 is within the field of view of the primary display 204 and is not within the field of view of the secondary display 206.

[0146] 13B and 14B, instead of displaying an actual representation of the object 806 via the primary display 204, the device 202 may display a proxy representation of the object 806 via the secondary display 206. In some embodiments, the secondary display 206 may overlap the primary display 204 such that the secondary display 206 covers most (or all) of the primary display 204. Thus, in some of these embodiments, the proxy representation of the object 806 may be shown via the secondary display 206 over a portion of the primary display 204 until the device 202 receives a request to interact with the object 806. For example, the request to interact with the object 806 may be received when an input (e.g., audio input, physical input) is determined to be directed at the object 806 and / or when a process is initiated to perform an operation involving one or more components of the object 806 and / or an object associated with the object 806 (e.g., indirectly or directly). In some embodiments, in response to receiving a request to interact with the object 806, the device 202 stops displaying the proxy representation of the object 806 via the secondary display 206 and displays the actual representation of the object 806 via the primary display 204. Thus, in embodiments in which the device 202 uses more battery power to display the object via the primary display 204 than the secondary display 206, battery power of the device 202 is conserved because the primary display 204 is not used to display a representation of the object 806 until a determination is made that the object 806 can be interacted with.

[0147] 13B and 14B, instead of displaying an actual representation of the object 806 via the primary display 204, the device 202 may display a proxy representation of the object 806. In some embodiments, the proxy representation of the object 806 has less visual detail and / or requires less power to display than the actual representation of the object 806. In some embodiments, while displaying the proxy representation of the object 806 via the primary display 204, the device 202 receives a request to interact with the object 806. For example, the request to interact with the object 806 may be received when an input (e.g., voice input, physical input) is determined to be directed to the object 806 and / or when a process is initiated to perform an operation involving one or more components of the object 806 and / or an object associated with the object 806. In some embodiments, in response to receiving a request to interact with the object 806, the device 202 replaces the display of the proxy representation of the object 806 with the actual representation of the object 806. Thus, in some embodiments, the actual representation of the object 806 is displayed only after a determination has been made that the object 806 may be interacted with, thereby conserving battery power and / or reducing the number of processes performed by the device 202. In some embodiments, the LED is inactive (or is not included as part of the device 202) while the proxy representation of the object 806 and / or the actual representation of the object 806 are displayed.

[0148] 15 is a flow diagram illustrating a method for transitioning objects between displays based on orientation information, according to some embodiments. In some embodiments, the method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, the method is performed by device 202 (FIG. 2A). In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0149] At block 1502, directional information, e.g., 804a-804d), corresponding to a position (e.g., north, south, east, west, right, left, back, front, other cardinal directions, or any combination thereof) of an object that is outside the field of view of a primary display (e.g., 204) of the device and a secondary display of the device is received. In some embodiments, the object is a physical object or item in a physical environment, and in some embodiments, the object is a virtual object in a virtual environment.

[0150] In some embodiments, the primary display, secondary display, and / or tertiary display are different types of displays. In some embodiments, the primary display is a waveguide display. In some embodiments, the secondary display is an organic light emitting diode display. In some embodiments, the tertiary display is a plurality of light emitting diodes.

[0151] At blocks 1504 and 1506, in response to receiving directional information corresponding to a position of an object outside the field of view of the primary display and the secondary display, and following a determination that a first criterion is met, a first representation of the directional information is displayed via the secondary display.

[0152] In some embodiments, the first criterion includes a criterion that is met if the object is of a type for which directional information of the object can be displayed on a secondary display. In some embodiments, the directional information of the object is permitted to be displayed via the secondary display if the object is a predetermined distance away from or next to a portion (e.g., the primary display, the secondary display, the frame of the device). In some embodiments, the directional information of the object is permitted to be displayed via the secondary display if directional information is received from an application or an object corresponding to the application, and the object is configured by the application (or other process) to be displayed via the secondary display if the object has one or more characteristics or features (e.g., size, shape, distance) that allow it to be displayed via the secondary display). In some embodiments, the directional information of the object is permitted to be displayed via the secondary display if the object is not in a particular position or orientation, or the object causes a representation of the object to be displayed adjacent to a portion of the device (e.g., inside the frame of the device, near the bridge of the device, near the user's nose).

[0153] In blocks 1504 and 1508, in response to receiving directional information corresponding to the position of an object outside the field of view of the primary display and the secondary display, and in accordance with a determination that a second criterion is met, a second representation (e.g., 812a-812g) of the directional information (e.g., 804b-804d) is displayed via the tertiary display.

[0154] In some embodiments, the second criterion includes a criterion that is met if the object is a type of object for which directional information of the object may be displayed on a tertiary display. In some embodiments, when directional information is received from an application or an object corresponding to the application, the object is configured by the application (or other process) to be displayed via the tertiary display if the object is a predetermined distance away from or next to a portion of the device (e.g., primary display, secondary display, frame), if the object has one or more properties or characteristics (e.g., size, shape, distance) that allow it to be displayed via the tertiary display, if the object is not in a particular position or orientation, or if the object causes a representation of the object to be displayed adjacent to a portion of the device (e.g., inside the frame of the device, near the bridge of the device, near the user's nose) or on a portion of the tertiary display.

[0155] In some embodiments, the first representation is a virtual object (e.g., the first representation is a modified representation of the object) and the second representation is not a virtual object in the CGR environment. In some embodiments, the first modified representation of the virtual object can be visually distinguishable from an actual representation of the object. For example, the first modified representation can have a visual appearance that is dissimilar or dissimilar to the actual object, while the actual representation of the object can have a visual appearance that is similar or resembling the actual object. In some embodiments, the device cannot use image recognition techniques to identify the actual object when processing the modified representation of the object, but can identify the actual object when processing the actual representation of the object. In some embodiments, the first modified representation can be a representation that is dissimilar to the actual object (e.g., does not have two or more of the same characteristics, such as color, shape, texture, size, brightness, thickness, fade, etc.), while the actual representation is a representation that resembles the actual object.

[0156] In some embodiments, the first representation is a static representation. For example, a static representation is one that does not move or is not animated over a period of time (e.g., does not change, does not pulsate, is not in a predefined sequence or pattern, etc.). In some embodiments, the second representation is a dynamic representation. For example, a dynamic representation is one that moves or is animated over a period of time (e.g., pulsates, is displayed in a predefined sequence or pattern, changes color over a period of time). In some embodiments, the first representation is a different color than the second representation. In some embodiments, the first representation is a different size than the second representation. In some embodiments, the first representation is not displayed while the second representation is displayed, or vice versa. In some embodiments, at least one of the first representation and the second representation includes a gradient (e.g., a gradient field that indicates a direction (e.g., solid lines and hatched LEDs)).

[0157] In some embodiments, the primary display and the secondary display may display different representations of the object. In some embodiments, the first representation of the directional information is a first representation of the object displayed via the secondary display. In some embodiments, while the first representation of the directional information is displayed via the secondary display, the object is detected within a field of view of the primary display of the HMD device. In some embodiments, in response to detecting that the object is within a field of view of the primary display, a second representation of the object is displayed via the primary display. In some embodiments, the first representation of the object displayed via the secondary display is displayed based on a first model of the object. In some embodiments, the second representation of the object displayed via the primary display is displayed based on a second model of the object. In some embodiments, the second model of the object is simpler (e.g., has more visual features, more visual details) than the first model of the object.

[0158] In some embodiments, the tertiary display surrounds a first portion of the secondary display. In some embodiments, the tertiary display surrounds a first portion of the secondary display but does not surround a second portion of the secondary display. In some embodiments, the primary display overlaps the secondary display.

[0159] In some embodiments, a representation of the object distinct from the first and second representations of directional information is displayed via the primary display and the first representation is displayed via the secondary display. In some embodiments, the representation of the directional information is associated with a position of the object. For example, the representation of the directional information may indicate a position of the device relative to the position of the object.

[0160] In some embodiments, a representation of the object may not be displayed via the primary display until the object is interacted with. In some embodiments, directional information corresponding to a position of the object within a field of view of the primary display of the HMD device and the secondary display of the HMD device is received. In some embodiments, in response to receiving the directional information corresponding to a position of the object within a field of view of the primary display of the HMD device and the secondary display of the HMD device, a fourth representation of the object is displayed via the secondary display without displaying a fifth representation of the object via the primary display. In some embodiments, a request to interact with the object (e.g., select the object, initiate a process involving the object) is received. In some embodiments, in response to receiving the request to interact with the object, a fifth representation of the object is displayed via the primary display and a fourth representation of the object ceases to be displayed via the secondary display.

[0161] In some embodiments, a request to interact with the object is received, hi some embodiments, a representation of the object is displayed via the primary display in response to receiving the request to interact with the object.

[0162] In some embodiments, while the first representation and the second representation are displayed, the first representation is displayed via the secondary display at a first position in a corresponding orientation relative to the primary display, and the second representation is not displayed via the tertiary display at a second position in a corresponding orientation relative to the primary display.

[0163] 16 is a flow diagram illustrating a method for displaying a modified representation of an object, according to some embodiments. In some embodiments, the method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, the method is performed by device 202 (FIG. 2A). In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0164] At block 1602, an object (e.g., a virtual object, a virtual object that is a representation of a physical object, a physical object in a physical environment) is detected in a computer generated reality (CGR) environment at a first location (or a location). In some embodiments, the object is determined to be important or useful. For example, the object may be determined to be important or useful by receiving a user input associated with the object, by receiving a user interaction associated with the object, by receiving data identifying the object as important or useful. In some embodiments, the data identifying the object as important or useful is received from one or more applications in communication with or running by device 202.

[0165] At blocks 1604 and 1606, in response to detecting an object in the CGR environment at a first location and according to a determination that the first location is within a first predetermined distance outside the field of view of the device's primary display, a first modified representation of the virtual object is displayed.

[0166] The first modified representation of the virtual object may be visually distinguishable from the actual representation of the object. For example, the first modified representation may have a visual appearance that is dissimilar or dissimilar to the actual object, while the actual representation of the object may have a visual appearance that is similar or resembling the actual object. In some embodiments, the device may not be able to identify the actual object when processing the modified representation of the object using image recognition techniques, but may be able to identify the actual object when processing the actual representation of the object. In some embodiments, the first modified representation may be a representation that is dissimilar to the actual object (e.g., does not have two or more of the same characteristics, such as color, shape, texture, size, brightness, thickness, fade, etc.), while the actual representation is a representation that resembles the actual object.

[0167] In some embodiments, the first modified representation has different visual content (e.g., data representing the displayed representation) than the visual content of the first actual representation. In some embodiments, the modified representation of the object has a different structure than the structure of the actual object, and the actual representation of the object has the same structure as the object. For example, if the actual object is a tree, the actual representation of the object visually looks like a tree, but the first modified representation looks like a blob of pixels that does not have the shape of a tree.

[0168] In some embodiments, the first modified representation has a size based on a distance between the first location and a field of view of the device's primary display. In some embodiments, when the object's location is a first distance outside the primary display, the first modified representation is a first size. In some embodiments, when the object's location is a second distance outside the primary display, the first modified representation is a second size. In some embodiments, when the first distance is closer to the primary display than the second distance, the first modified representation displayed at the first size is larger than the first modified representation displayed at the second size. In some embodiments, when the object is detected at a location closer to the field of view of the device's primary display than the first location, the size of the first modified representation is increased.

[0169] In some embodiments, the first modified representation has a brightness based on the size of the object (e.g., larger objects are made darker). In some embodiments, if the object is a first size, the first modified representation has a first brightness. In some embodiments, if the object is a second size, the first modified representation has a second brightness. In some embodiments, if the object's first size is smaller than the object's second size, the first modified representation displayed at the first brightness is brighter than the first modified representation displayed at the second brightness.

[0170] In some embodiments, the first modified representation has a brightness based on the size of the object relative to another object displayed via the device. For example, the brightness of the first modified representation may increase as the size of the object becomes larger than the size of another object, and vice versa. In some embodiments, if the size of the object is substantially equal to the size of another object, the brightness of the modified representation of the object may be equal to the brightness of the representation of the other object. Thus, in some embodiments, as the user moves throughout the environment, the brightness of the modified representation of the object may change based on other objects in the environment.

[0171] In some embodiments, the first modified representation has a higher blur than the first actual representation. In some embodiments, the first modified representation is more faded than the first actual representation.

[0172] In some embodiments, an animation may be displayed as part of displaying the first modified representation, for example, the animation may be a pulsating animation, an animation that shifts or moves the first modified representation, an animation that transforms the first actual representation of the object into the first modified representation, an animation that fades in or out the first modified representation, etc.

[0173] In some embodiments, a second actual representation of the first portion of the object is displayed via the primary display in response to detecting the object in the CGR environment at the first location and according to a determination that the first location is within the field of view of the primary display and the field of view of the secondary display. For example, as shown in FIG. 13B, a second actual representation (e.g., 806) of the first portion of the object may be displayed via the primary display if the device is within a threshold distance within the field of view of the secondary display.

[0174] While displaying a second actual representation of the first portion of the object via the primary display, a second modified representation of a second portion of the object is displayed via the secondary display. For example, as shown in FIG. 13B, a second modified representation (e.g., 826c) is displayed via the secondary display.

[0175] In some embodiments, the second portion of the object is a different portion of the object than the first portion of the object. In some embodiments, the first portion of the object is within the field of view of the primary display but not within the field of view of the secondary display. In some embodiments, the second portion of the object is within the field of view of the secondary display but not within the field of view of the primary display. In some embodiments, the second actual representation is displayed at a higher resolution than the second modified representation. In some embodiments, the second actual representation is displayed adjacent an edge of the primary display adjacent to the secondary display. In some embodiments, the second modified representation is displayed adjacent an edge of the secondary display adjacent to the primary display.

[0176] In some embodiments, as part of displaying the second actual representation of the object via the primary display, a visual effect is applied to the second actual representation of the object. In some embodiments, the second actual representation of the object after the visual effect is applied has a greater blur than the blur of the second actual representation of the object before the visual effect is applied. In some embodiments, one or more of the second actual representation of the first portion of the object displayed via the primary display and the second modified representation of the second portion of the object displayed via the secondary display are displayed using one or more of the techniques described above in connection with FIG.

[0177] In some embodiments, a representation corresponding to directional information for locating an object is displayed via the primary display in response to detecting an object in the CGR environment at a first location and following a determination that the first location is within a second predetermined distance outside the field of view of the device's primary display. For example, as shown in Figures 12A and 12B, a representation (e.g., 828) corresponding to directional information for locating an object is displayed via the primary display when the distance between user 250 and object 806 is distance 804e away from each other, unlike in Figure 11A where user 250 and object 806 are distance 804d away from each other.

[0178] In some embodiments, the representation corresponding to the directional information for locating an object differs from the actual representation and may provide an indication of the location of the object, the proximity of the object to the device, and whether the device is moving away from or towards the object.

[0179] In some embodiments, a modified representation of the object is displayed via a secondary display while displaying a representation corresponding to the object's position via the primary display.

[0180] In some embodiments, in response to detecting the object at the first location, and following a determination that the first location is within the field of view of the primary display and not within the field of view of the secondary display, the first actual representation (e.g., 806) is displayed via the primary display, but the first modified representation ceases to be displayed via the secondary display. In some embodiments, a portion of the first actual representation of the object is displayed. In some embodiments, the entire object is displayed.

[0181] Figures 17A-17B, 18, and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments. In Figure 17A, device 202 is shown to have one or more components, as described above in Figures 2A-2B and 8A.

[0182] 17A illustrates device 202 displaying a number of application icons, including a messaging application icon 1702a, a health application icon 1702b, a phone application icon 1702c, and an email application icon 1702d.

[0183] 17A, assume that the device 202 receives a new message via a messaging application corresponding to the messaging icon 1702a. In response to receiving the new message, the device 202 can determine whether information related to the new message is displayed via a particular display (e.g., the primary display 204, the secondary display 206, an LED, etc.).

[0184] In some embodiments, in response to receiving a new message, the device 202 determines that information related to the new message is permitted (or configured) to be displayed via the secondary display 206. As shown in FIG. 17B, the device 202 updates the messaging application icon 1702a to include a message status indicator 1702a1 to make this determination. As shown in FIG. 17B, the message status indicator 1702a1 is a "1" adjacent to the messaging icon 1702a, indicating that one additional new or unread message has been received. Unread messages may include messages that have not received user interaction, such as voicemail messages that have not been listened to, text or email messages that have not been read, notifications or alerts from applications that have not yet been opened, notifications or alerts that have not been deleted, etc.

[0185] In some embodiments, in response to receiving a new message, device 202 determines that information related to the new message is permitted (or configured) to be displayed via a tertiary display, such as an LED. As shown in FIG. 17B, device 202 illuminates an LED to make this determination. In some embodiments, device 202 illuminates only a portion of the LED.

[0186] One or more other types of alerts, notifications, sensor data, requests or data from an application, an incoming phone call, etc. may be received. In some embodiments, device 202 performs similar actions in response to receiving them as described above with respect to new messages received in FIGS. 17A-17B. For example, device 202 may illuminate an LED if it determines that health data detected by a heart rate sensor (or one or more other sensors detecting personal or health data) is above or below a threshold level and information related to the data is permitted to be displayed via the LED.

[0187] The device 202 may also perform other actions in response to receiving one or more messages, alerts, notifications, sensor data, requests or data from applications, incoming calls, etc. For example, as shown in Figure 18, the device 202 displays a color across at least a portion of the secondary display 206 based on a determination that information related to a new message is permitted to be displayed via the secondary display 206. In some embodiments, a portion of the primary display 204 is updated due to this determination.

[0188] In some embodiments, the device 202 may display an animation via one or more displays. In some embodiments, the animation may include one or more pulsating colors, one or more moving objects, one or more objects that change shape or transition into one or more other objects, etc. For example, the device 202 may display a pulsating color animation on the primary display 204 and / or the secondary display 206 based on a determination that the user's health data is above / below a threshold level and information related to the health data is permitted to be displayed on the corresponding display.

[0189] 19A-19B and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0190] 19A illustrates a device 202 displaying a CGR environment including representations of multiple objects 1910a-1910d. Objects 1910a-1910d are automobiles within the field of view of primary display 204 and secondary display 206. Here, objects 1910a-1910d represent physical automobiles within the physical environment as viewed through the primary 204 and secondary 206 displays of device 202. In some embodiments, objects 1910a-1910d represent virtual objects displayed by the primary 204 and secondary 206 displays of device 202.

[0191] 19A, assume that the device 202 receives data from an application indicating that the object 1910a is associated with a user of the device 202. For example, in FIG. 19A, the application may be a car sharing application, a taxi application, a car location application, etc. In some embodiments, the device 202 determines that the object 1910a is associated with a user of the device 202 without receiving data from a dedicated application using one or more user settings, data stored in memory, one or more machine learning algorithms, etc.

[0192] In response to receiving the data from the application, the device 202 may determine whether information associated with the data from the application is displayed via a particular display. In some embodiments, in response to receiving the data from the application, the device 202 determines that information associated with the data is permitted to be displayed via the secondary display 206. Based on this determination, the device 202 displays an indicator 1912 around the object 1910a on the secondary display 206, as shown in FIG.

[0193] Indicator 1912 is displayed around object 1910a to visually distinguish object 1910a from objects 1910b-1910d. In some embodiments, device 202 visually distinguishes object 1910a from other objects in other ways. For example, device 202 can stop displaying objects 1910b-1910d, minimize objects 1910b-1910d, enlarge object 1910a, highlight object 1910a, etc.

[0194] In some embodiments, in response to receiving data from an application, the device 202 determines that information related to the data is permitted to be displayed via LEDs, and based on this determination, the device 202 illuminates one or more LEDs, as shown in FIG.

[0195] 20A-20D and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0196] 20A illustrates a device 202 displaying a CGR environment including a representation of time 2002. The representation of time 2002 is displayed via the primary display 204 to indicate that a process has 60 seconds remaining. In some embodiments, the representation of time 2002 represents a process associated with a timer application, a broadcast application, a workout application, a meeting application, a productivity application, etc. In some embodiments, a determination has been made that the representation of time 2002 is permitted to be displayed via the primary display 204, so the representation of time 2002 is displayed via the primary display 204.

[0197] Simultaneously with the representation of time 2002, a representation of time 2004 is displayed via secondary display 2006. The representation of time 2004 includes a colored overlay covering the entire secondary display 206 to indicate the 60 seconds remaining. In some embodiments, the representation of time 2004 indicates the time represented by the representation of time 2002. In other words, in some embodiments, a representation displayed via one display may correspond to or be representative of a representation displayed via another display.

[0198] In some embodiments, a determination has been made that the representation of time 2004 is permitted to be displayed via secondary display 206, so the representation of time 2002 is displayed via secondary display 206.

[0199] 20A, the representation of time 2004 shows the percentage of time remaining compared to the start time of a timer or process. For example, assuming a process is started and will finish in 60 seconds, the representation 2004 shows that the percentage of time remaining is 100%.

[0200] Simultaneously with the representation of time 2002 and the representation of time 2004, the device 202 illuminates the LEDs as shown in Figure 20A. As with the representation of time 2004, the LEDs indicate the time represented by the representation of time 2002 and therefore correspond to the representation of time 2002. Now, assuming the process starts and ends in 60 seconds, all the LEDs are illuminated since the percentage of time remaining is 100%.

[0201] Assume that device 202 receives an indication that the process has 30 seconds remaining, as shown in FIG. 20B. In response to receiving the indication, device 202 updates the representation of time 2002 and the representation of time 2004. In FIG. 20B, the representation of time 2004 includes a colored overlay that covers approximately half of the area of ​​secondary display 206. In some embodiments, representation 2004 in FIG. 20B represents that the time remaining (e.g., 30 seconds) is 50% of the original time that the process was set for (e.g., 60 seconds in FIG. 20A). Additionally, device 202 also stops illuminating half of the LEDs (e.g., 810a-810l and 812a-812l) in response to the time remaining being at least 50% of the original time.

[0202] As shown in Figures 20C-20D, the time remaining is 15 seconds and 5 seconds, respectively. As shown in Figures 20C-20D, device 202 continues to update primary display 204, secondary display 206, and LEDs (e.g., 810a-810l and 812a-812l) based on the time remaining in the process (e.g., 15 seconds in Figure 20C and 5 seconds in Figure 20D).

[0203] However, in contrast to Figures 20A-20C, device 202 illuminates more than an expected portion of the LEDs (e.g., more than 1 / 12) in Figure 20D, where device 202 illuminates all of the LEDs to provide an indication that the time remaining is less than a predetermined minimum time.

[0204] In some embodiments, the device 202 animates (e.g., pulsates, brightens) one or more LEDs and / or secondary displays 206 as the remaining time approaches zero. In some embodiments, the secondary displays 206 and / or LEDs change to a different color and / or shade when a predetermined time remains. For example, the secondary displays 206 may change from translucent to red when a predetermined time remains. In some embodiments, one or more LEDs and / or secondary displays 206 animate while one or more other displays do not animate. In some embodiments, the secondary displays 206 may overlap the primary display 204 such that animations shown in the overlapping portions of the secondary displays 206 are shown in the center of the display of the device 202 (e.g., in an area corresponding to around the user's pupil).

[0205] 21A-21D and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0206] 21A shows device 202 displaying a CGR environment including device 2102a and device 2102b. As shown in FIG. 21A, device 2102a and device 2102b are within the field of view of primary display 204 and secondary display 206.

[0207] 21A, a user interface (UI) element 2110 is displayed on the device 2102a. For example, the UI element 2110 may be one or more icons, files, application shortcuts, text, etc.

[0208] 21B, the device 202 detects a drag-and-drop input 2150 on the UI element 2110. In response to detecting the drag-and-drop input 2150 on the UI element 2110, the device 202 determines whether information related to a status of the drag-and-drop input can be displayed via an LED. As shown in FIG. 21B, the device 202 illuminates an LED for this determination.

[0209] 21B, assume that device 202 detects movement of a drag-and-drop input 2150. In response to detecting movement of the drag-and-drop input 2150, device 202 displays a UI element 2110 that moves from a position on device 2102a to a position on device 2102b.

[0210] Figure 21C shows the UI element 2110 in a position on device 2102b. As shown in Figure 21C, the LED continues to illuminate because the UI element 2110 remains selected.

[0211] With reference to FIG. 21C, assume that the device 202 detects that the UI element 2110 is not selected. In response to detecting that the UI element 2110 is not selected, the device 202 abandons illuminating the LEDs, as shown in FIG. 21D. In some embodiments, in response to detecting that the UI element 2110 is not selected, the device 202 illuminates one or more LEDs differently. For example, the device 202 may cause one or more LEDs to emit a different color of light while the UI element 2110 is selected than the color they previously emitted in FIG. 21C.

[0212] In some embodiments, lighting one or more LEDs differently indicates the completion or effectiveness of a user action. For example, device 202 may light LEDs differently according to the rhythm of a media file being played after a user drags a media file from device 2102a to device 2102b.

[0213] In some embodiments, device 202 performs the techniques described above with respect to drag-and-drop input 2150 (e.g., in FIGS. 21A-21D) to indicate other user interactions and / or the state of an application. For example, in some embodiments, device 202 performs the techniques described above when a drag, release, copy, edit, clip, or move operation is detected. In some embodiments, device 202 performs the techniques described above when device 202 receives data regarding the state of an application or process, such as a state associated with downloading a file, a state associated with playing media, a state associated with completing an exercise routine, a state associated with completing a periodic goal (e.g., health goal, productivity goal), etc.

[0214] 22A-22B and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0215] 22A shows a device 202 displaying a CGR environment including a person 2202. The person 2202 is within the field of view of and can be seen through the primary display 204 and secondary display 206.

[0216] As shown in Figure 22A, the person 2202 is frowning. With reference to Figure 22A, assume that the device 202 receives data representing the mood of the person 2202, which is processed by one or more algorithms, such as one or more facial recognition algorithms. In some embodiments, the one or more algorithms may determine that the person 2202 is frowning and transmit data representing this determination.

[0217] In response to receiving data representing the person 2202 frowning, the device 202 determines that information related to the data is permitted to be displayed via the LEDs. In some embodiments, in accordance with this determination, the device 202 causes the LEDs to illuminate in a first state. For example, FIG. 22A shows the LEDs in a first state (e.g., the LEDs are filled). In some embodiments, the device 202 updates only the portion of the LEDs shown in solid lines in FIG. 22A.

[0218] In some embodiments, the device 202 updates the secondary display 206 with a colored overlay suggestive of the mood of the person 22A (e.g., pursuant to a determination that the data is permitted to be displayed via the secondary display 206).

[0219] Figure 22B shows device 202 displaying a CGR environment including person 2204. As shown in Figure 22B, person 2204 is smiling, in contrast to person 2202 of Figure 22A. With reference to Figure 22B, assume that device 202 receives data representing the mood of person 2204, which is processed by one or more algorithms, such as one or more facial recognition algorithms.

[0220] In response to receiving the data representing the mood of the person 2204, the device 202 determines that information related to the data is permitted to be displayed via the LED. In some embodiments, in accordance with this determination, the device 202 illuminates the LED in a second state as opposed to a first state. For example, FIG. 22B indicates that the LED is in the second state by showing the LED as hatched. Here, the device 202 illuminates the LED differently in the second state than in the first state because the data represents a different mood of the user.

[0221] In some embodiments, data representing other moods (e.g., sad, happy, etc.) or other types of data may be displayed using techniques similar to those described above in connection with Figures 22A-B. For example, other types of data may include financial market conditions, weather forecasts, ratings (e.g., restaurant ratings, movie ratings, etc.).

[0222] 23A-23B and 25 illustrate example techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0223] 23A illustrates a device 202 displaying a CGR environment including a representation 2302 of a progress indicator displayed via a secondary display 206. In some embodiments, the representation 2302 illustrates one or more metrics being tracked. For example, the metric may be a number of steps taken (or remaining to achieve a goal), a download speed, a number of workouts over a period of time, and / or any other metric. In some embodiments, the representation 2302 corresponds to data received from one or more wellness applications, such as a walking application, a workout application, a strength training application, etc.

[0224] In some embodiments, device 202 causes LEDs to emit light representative of the progress indicators based on a determination that information related to the progress indicators is permitted to be displayed via the LEDs. For example, as shown in FIG 23A, LEDs 810a and 810h-810l are illuminated and correspond to the length and position of representation 2302. LEDs 810a and 810h-810l indicate the level of the progress indicator.

[0225] In some embodiments, LEDs 812a, 812h, 812i, 812l are illuminated, as shown in Figure 23A. LEDs 812a, 812h, 812i, 812l are on the right side of device 202 along the length of representation 2302. LEDs 812a, 812h, 812i, 812l also indicate the level of a progress indicator. Notably, LEDs 812j and 812k of device 202 are not illuminated because they are within a certain distance from the nose of a user wearing device 202.

[0226] 23B, device 202 updates representation 2302 via secondary display 206 and LEDs to indicate progress has been made in the tracked metric. In particular, LEDs 812j and 812k remain unilluminated because they are within a particular distance from the nose of the user wearing device 202.

[0227] 24A-24D illustrate an example technique for managing one or more displays based on data related to one or more processes, according to some embodiments. 24A-24D illustrate a scenario in which device 202 sequentially illuminates one or more of LEDs 810a-210l and 812a-812l in a clockwise pattern around the eyes of a user wearing device 202. In some embodiments, the pattern may indicate a direction to the user and may be used in addition to, instead of, or as an alternative to the examples discussed above in connection with FIGS. 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, 14A-14B, 15, and 16.

[0228] For example, the LEDs may emit a pattern, where the LEDs are sequentially turned on and off, similar to a dot moving around a circle. The circle may be animated in a clockwise or counterclockwise direction around one or more eyes of a user wearing device 202. In some embodiments, when tracing a pattern around both eyes of a user, device 202 transitions between LEDs on the right and left sides of device 202, as shown in Figures 24B-C, where device 202 skips some of the LEDs in the sequence (e.g., 810g and 810h).

[0229] 25 is a flow diagram illustrating a method for managing one or more displays based on data associated with one or more processes, according to some embodiments. In some embodiments, the method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, the method is performed by device 202 (FIG. 2A). In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0230] At block 2502, information corresponding to a change in status of a process is received. In some embodiments, the information may be one or more notifications, alerts, and / or one or more outputs from one or more functions, processes, applications, sensors, etc. In some embodiments, the sensors may include one or more health sensors, heart rate sensors, pedometers, heat sensors, etc.

[0231] In some embodiments, the information is calculated based on one or more environmental factors or data from one or more applications. For example, in Figures 20A-20D, device 202 may calculate the start or end of a process as described above. In some embodiments, environmental factors may include data corresponding to weather, characteristics of one or more users or devices in an environment, etc.

[0232] 25, in response to receiving information corresponding to a change status of the process at blocks 2504 and 2506, and following a determination that a first criterion is met, a first representation corresponding to the status of the process is displayed via the secondary display. For example, the representation may include one or more representations such as representation status indicator 1702a1 of FIG. 17B, a colored overlay displayed across secondary display 206 of FIG. 18, indicator 1912 of FIG. 19B, representation 2004 of FIG. 20A-D, progress indicator representation 2302 of FIG. 23A-B, etc.

[0233] In some embodiments, the first criterion includes a criterion that is met when a determination is made that displaying object directional information via the secondary display is permitted, as described above in connection with Figures 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, 14A-14B, 15 and 16.

[0234] 25, at blocks 2504 and 2506, in response to receiving information corresponding to a change in the status of the process and following a determination that a second criterion is met, a second representation corresponding to the status of the process is displayed, for example, 810a-810l and 812a-812l in Figures 17A-17B, 18, 19A-19B, 20A-20D, 21A-21D, 22A-22B, 23A-23B, and 24A-24D.

[0235] In some embodiments, the second criterion includes a criterion that is met when a determination is made that the object is permitted to be displayed via a tertiary display, as described above in connection with Figures 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, 14A-14B, 15 and 16.

[0236] In some embodiments, the second representation is different from the first representation, hi some embodiments, the first representation is a virtual object and the second representation is not a virtual object.

[0237] In some embodiments, one or more of the first and second representations are associated with a third representation presented via the primary display (e.g., corresponding to one or more processes of the same application). For example, in Figures 20A-20D, a representation 2004 displayed via a secondary display and an LED (e.g., a tertiary display) is both associated with a representation of time 2002 displayed via the primary display 204.

[0238] In some embodiments, while presenting the first representation, a second virtual object is displayed simultaneously with the first representation. In some embodiments, the first representation highlights at least a portion of the second virtual or physical object displayed via the secondary display. For example, in FIG. 19B, indicator 1912 highlights (or brackets) object 1910a displayed via secondary display 206.

[0239] In some embodiments, one or more of the first and second representations include a progress indicator, which is updated to indicate a change in a metric associated with the device, as described above with respect to representation 2302 of FIGS.

[0240] In some embodiments, the first representation includes an application icon. In some embodiments, the process is associated with an application that includes a status indicator that is updated based on the process. For example, in Figures 17A-17B, status indicator 1702a1 is displayed in response to receiving data related to a messaging application.

[0241] In some embodiments, one or more of the first representation and the second representation are displayed based on one or more sensors. In some embodiments, the one or more sensors include a heart rate sensor. In some embodiments, the received information corresponds to data detected via the heart rate sensor. In some embodiments, as part of displaying one or more of the first representation (e.g., 206 of FIG. 18) and the second representation based on at least one of the one or more sensors, an animation is displayed based on data detected via the heart rate sensor.

[0242] In some embodiments, the received information corresponds to an alert corresponding to the application. In some embodiments, one or more of the first representation and the second representation are presented based on the alert corresponding to the application, for example, as described above in connection with FIG.

[0243] In some embodiments, the received information corresponds to an alert corresponding to a calendar event. In some embodiments, one or more of the first representation and the second representation are presented based on the alert corresponding to the calendar event, for example, as described above in connection with FIG.

[0244] In some embodiments, the received information corresponds to data obtained via a facial recognition algorithm. In some embodiments, one or more of the first representation and the second representation are presented based on the data obtained via a facial recognition algorithm, for example as described above with respect to the LEDs of Figures 22A and 22B.

[0245] In some embodiments, the received information corresponds to a state of the application. In some embodiments, one or more of the first representation and the second representation are presented based on the state of the application. For example, in Figures 21A-21D, an LED is updated based on whether a drag-and-drop input 2150 is detected.

[0246] In some embodiments, the received information corresponds to a detected posture of the user. In some embodiments, one or more of the first representation and the second representation are presented based on a detected posture of a user wearing the device. In some embodiments, the first representation is displayed based on a detected eye position of the user. For example, the representation may be a line representing a current posture or eye position of the user. In some embodiments, in response to detecting a change in the posture or eye position of the user, the line is updated to indicate the new posture (or, alternatively, the change in posture) or eye position. In some embodiments, the line is displayed only via the secondary display. In some embodiments, the LEDs may be illuminated in various combinations to indicate a new posture (or, alternatively, the change in posture) or eye position.

[0247] As noted above, one aspect of the present technology is to collect and use data available from various sources to provide specialized resource management for devices with additional displays (e.g., devices with additional displays) to conserve battery life for users and provide specialized content to users of the devices. The present disclosure contemplates that in some examples, this collected data may include personal information data that uniquely identifies or can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0248] The present disclosure recognizes that the use of such personal information data in the present technology can be used to the benefit of the user. For example, the personal information data can be used to conserve the battery life of the user's device. Thus, for example, the use of such personal information data can help the system properly manage resources to conserve the battery life of the device. Additionally, other uses of the personal information data that benefit the user are also contemplated by the present disclosure. For example, health and fitness data can be used to provide insight into the overall health of the user, or can be used as positive feedback to individuals using the technology to pursue health goals.

[0249] This disclosure contemplates that entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will follow established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized to meet or exceed industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily available to users and should be updated if data collection and / or use changes. Personal information from users should be collected for the entity's lawful and reasonable use and should not be shared or sold outside of those lawful uses. Furthermore, such collection / sharing should occur after receiving the user's informed consent. Furthermore, such entities should consider taking all necessary measures to protect and secure access to such personal information data and ensure that others with access to the personal information data comply with the privacy policies and procedures. Furthermore, such entities may be subject to a third-party assessment to attest that they adhere to widely accepted privacy policies and practices. Additionally, policies and practices should be tailored to the specific types of personal data collected and / or accessed, and to applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, collection of or access to certain health data may be regulated by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and must be addressed accordingly. Thus, different privacy practices may need to be maintained for different personal data types in each country.

[0250] Notwithstanding the above, the present disclosure also contemplates instances where a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, when managing resources of a low-power device, the present technology may be configured to allow a user to choose to "opt-in" or "opt-out" from participating in the collection of personal information data during registration for the service or at any time thereafter. In another example, a user may choose not to provide eye tracking data, such as pupil position, pupil dilation, and / or blink rate, for specialized resource management. In yet another example, a user may choose to limit the amount of time eye tracking data is maintained or to completely prohibit the development of a baseline eye tracking profile. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications regarding access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an app, and then again just before the personal information data is accessed by the app.

[0251] Further, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintended or unauthorized access or use. Limiting data collection and deleting data that is no longer needed can minimize the risk. Additionally, where applicable, including in certain health-related applications, anonymization of data can be used to protect user privacy. Anonymization can be facilitated as needed by removing certain identifiers (e.g., date of birth, etc.), controlling the amount or specificity of data stored (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0252] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, the present disclosure also contemplates that the various examples may be implemented without the need for access to such personal information data. That is, the various examples of the present technology are not inoperable due to the absence of all or a portion of such personal information data. For example, resources of a low-power device may be managed and content (e.g., status updates and / or objects) may be selected and delivered to a user by inferring preferences based on non-personal information data or minimal personal information, such as content requested by a device associated with the user, other non-personal information available to the system controlling the device, or publicly available information.

Claims

1. 1. A method of controlling a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device including a primary display spanning a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, comprising: Detecting an object in a computer generated reality (CGR) environment at a first location; in response to detecting the object in the CGR environment at the first location, displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinct from a first actual representation of the object in accordance with a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device; and displaying, via the primary display, a representation corresponding to directional information for locating the object in accordance with a determination that the first position is within a second predetermined distance outside the field of view of the primary display of the HMD device, the second predetermined distance being different from the first predetermined distance, wherein the representation corresponding to the directional information for locating the object is different from the first actual representation.

2. The method of claim 1 , wherein a first revised representation has visual content that differs from visual content of the first actual representation.

3. The method of claim 1 or 2, wherein the first modified representation has a size based on the distance between the first position and the field of view of the primary display of the HMD device.

4. The method of claim 1 , wherein the first modified representation has a brightness that is based on a size of the object.

5. The method of claim 1 , wherein the first modified representation has a higher blur than a blur of the first actual representation.

6. The method of claim 1 , wherein displaying the first modified representation comprises displaying an animation.

7. in response to detecting the object in the CGR environment at the first location, in response to determining that the first location is within the field of view of the primary display and the field of view of the secondary display, displaying a second actual representation of a first portion of the object via the primary display; and 7. The method of claim 1, further comprising: displaying a second modified representation of a second portion of the object via the secondary display while displaying the second actual representation of the first portion of the object via the primary display.

8. The method of claim 7 , wherein the second actual representation is displayed at a higher resolution than the second revised representation.

9. 9. The method of claim 7 or 8, wherein the second actual representation is displayed adjacent an edge of the primary display.

10. displaying the second actual representation via the primary display includes applying a visual effect to the second actual representation; 10. The method of claim 7, wherein the second actual representation of the object after applying the visual effect has a greater blur than a blur of the second actual representation of the object before applying the visual effect.

11. in response to detecting the object in the CGR environment at the first location, in response to determining that the first location is within the field of view of the primary display and not within the field of view of the secondary display, displaying the first actual representation via the primary display; and The method of claim 1 , further comprising: abandoning displaying the first modified representation via the secondary display.

12. A computer program product causing a computer to carry out the method according to any one of claims 1 to 11.

13. A memory for storing a computer program according to claim 12; one or more processors capable of executing the computer programs stored in the memory, The head mounted display (HMD) device comprises: one or more image sensors; a primary display spanning the entire field of view and having a first resolution; A head mounted display (HMD) device physically and electronically coupled to the primary display and configured to communicate with a secondary display having a second resolution lower than the first resolution.

14. one or more image sensors; a primary display spanning the entire field of view and having a first resolution; a head mounted display (HMD) device physically and electronically coupled to the primary display and configured to communicate with a secondary display having a second resolution lower than the first resolution, A head mounted display (HMD) device comprising means for carrying out the method according to any one of claims 1 to 11.

Images