Managing devices with additive displays

Abstract

The invention relates to managing a device with an additive display. One or more techniques are described for managing virtual objects between one or more displays. According to some embodiments, exemplary techniques for displaying virtual objects are described. According to some embodiments, exemplary techniques are described for transitioning an object between displays based on directional information. According to some embodiments, exemplary techniques are described for displaying a modified representation of an object. According to some embodiments, exemplary techniques are described for managing one or more displays based on data associated with one or more processes.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 048842, international application date August 31, 2020, date of entry into the Chinese national phase September 30, 2021, Chinese national application number 202080026845.1, and invention name “Managing devices with additive displays”.

[0003] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 067,461, entitled “MANAGING DEVICES HAVING ADDITIVE DISPLAYS,” filed on August 19, 2020, and U.S. Provisional Patent Application Serial No. 62 / 907,207, entitled “TECHNIQUES FOR MANAGING DEVICES HAVING ADDITIVE DISPLAYS FOR USE IN A COMPUTER-GENERATED REALITY ENVIRONMENT,” filed on September 27, 2019. The contents of the above applications are hereby incorporated by reference in their entirety. Technical Field

[0004] The present disclosure relates generally to computer-generated reality environments, and more particularly to methods and techniques for managing devices with additive displays for use in computer-generated reality environments. Background Art

[0005] Users often use wearable devices, such as head mounted display (HMD) devices, to interact with computer generated reality environments. The use of wearable devices is often limited by the amount of battery life of the wearable devices. Therefore, one or more techniques are needed to manage power usage to extend battery life in wearable devices. Summary of the invention

[0006] According to some embodiments, a method is performed at a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device comprising a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The method comprises: displaying a first portion of a virtual object via the secondary display; and displaying a second portion of the virtual object via the primary display, wherein displaying the second portion of the virtual object via the primary display comprises: applying a visual effect to the second portion of the virtual object based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display.

[0007] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The HMD device also includes one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: displaying a first portion of a virtual object via the secondary display; and displaying a second portion of the virtual object via the primary display, wherein displaying the second portion of the virtual object via the primary display includes: applying a visual effect to the second portion of the virtual object based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display.

[0008] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The one or more programs include instructions for: displaying a first portion of a virtual object via the secondary display; and displaying a second portion of the virtual object via the primary display, wherein displaying the second portion of the virtual object via the primary display includes: applying a visual effect to the second portion of the virtual object based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display.

[0009] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The one or more programs include instructions for: displaying a first portion of a virtual object via the secondary display; and displaying a second portion of the virtual object via the primary display, wherein displaying the second portion of the virtual object via the primary display includes: applying a visual effect to the second portion of the virtual object based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display.

[0010] According to some embodiments, the system includes one or more processors. The system also includes means for displaying a first portion of the virtual object via the secondary display; and means for displaying a second portion of the virtual object via the primary display, wherein displaying the second portion of the virtual object via the primary display includes: applying a visual effect to the second portion of the virtual object based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display.

[0011] According to some embodiments, a method is performed at a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The method includes receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. The method also includes, in response to receiving the directional information corresponding to the position of the object outside the field of view of the primary display and the secondary display, displaying a first representation of the directional information via the secondary display based on determining that a first criterion is satisfied, wherein the first criterion includes a criterion that is satisfied when displaying the directional information of the object via the secondary display; and displaying a second representation of the directional information via the tertiary display based on determining that a second criterion is satisfied, wherein the second criterion includes a criterion that is satisfied when displaying the directional information of the object via the tertiary display, wherein the first representation is a virtual object within a CGR environment, and wherein the second representation is not a virtual object within the CGR environment.

[0012] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a main display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the main display and having a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of the main display of the HMD device and the secondary display of the HMD device. The one or more programs also include instructions for the following operations: in response to receiving direction information corresponding to the position of an object outside the field of view of the main display and the secondary display, displaying a first representation of the direction information via the secondary display based on determining that a first criterion is met, wherein the first criterion includes a criterion that is met when the direction information of the object is allowed to be displayed via the secondary display; and displaying a second representation of the direction information via the tertiary display based on determining that a second criterion is met, wherein the second criterion includes a criterion that is met when the direction information of the object is allowed to be displayed via the tertiary display, wherein the first representation is a virtual object within a CGR environment, and wherein the second representation is not a virtual object within the CGR environment.

[0013] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device including one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of a primary display of an HMD device and a secondary display of the HMD device. The one or more programs also include instructions for the following operations: in response to receiving direction information corresponding to the position of an object outside the field of view of the primary display and the secondary display, displaying a first representation of the direction information via the secondary display based on determining that a first criterion is satisfied, wherein the first criterion includes a criterion that is satisfied when the direction information of the object is allowed to be displayed via the secondary display; and displaying a second representation of the direction information via the tertiary display based on determining that a second criterion is satisfied, wherein the second criterion includes a criterion that is satisfied when the direction information of the object is allowed to be displayed via the tertiary display, wherein the first representation is a virtual object within a CGR environment, and wherein the second representation is not a virtual object within the CGR environment.

[0014] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device including one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of a primary display of an HMD device and a secondary display of the HMD device. The one or more programs also include instructions for the following operations: in response to receiving direction information corresponding to the position of an object outside the field of view of the primary display and the secondary display, displaying a first representation of the direction information via the secondary display based on determining that a first criterion is satisfied, wherein the first criterion includes a criterion that is satisfied when the direction information of the object is allowed to be displayed via the secondary display; and displaying a second representation of the direction information via the tertiary display based on determining that a second criterion is satisfied, wherein the second criterion includes a criterion that is satisfied when the direction information of the object is allowed to be displayed via the tertiary display, wherein the first representation is a virtual object within a CGR environment, and wherein the second representation is not a virtual object within the CGR environment.

[0015] According to some embodiments, the system includes one or more processors. The system also includes a device for receiving directional information corresponding to the position of an object outside the field of view of a primary display of a head-mounted display (HMD) device and a secondary display of the HMD device. The system also includes a device for performing the following operations in response to receiving directional information corresponding to the position of an object outside the field of view of the primary display and the secondary display: displaying a first representation of the directional information via the secondary display based on determining that a first criterion is met, wherein the first criterion includes a criterion that is met when the directional information of the object is allowed to be displayed via the secondary display; and displaying a second representation of the directional information via a tertiary display based on determining that a second criterion is met, wherein the second criterion includes a criterion that is met when the directional information of the object is allowed to be displayed via the tertiary display, wherein the first representation is a virtual object within a CGR environment, and wherein the second representation is not a virtual object within the CGR environment.

[0016] According to some embodiments, a method is performed at a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The method includes: detecting an object at a first location in a computer generated reality (CGR) environment; and in response to detecting the object at the first location in the CGR environment, displaying a first modified representation of the object that is visually distinguishable from a first actual representation of the object via the secondary display of the HMD device based on determining that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0017] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The HMD device also includes one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for the following operations: detecting an object at a first location in a computer generated reality (CGR) environment; and in response to detecting the object at the first location in the CGR environment, displaying a first modified representation of the object that is visually distinguished from a first actual representation of the object via the secondary display of the HMD device based on determining that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0018] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The one or more programs include instructions for the following operations: detecting an object at a first location in a computer-generated reality (CGR) environment; and in response to detecting the object at the first location in the CGR environment, displaying a first modified representation of the object that is visually distinguishable from a first actual representation of the object via the secondary display of the HMD device based on determining that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0019] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution. The one or more programs include instructions for: detecting an object at a first location in a computer-generated reality (CGR) environment; and in response to detecting the object at the first location in the CGR environment, displaying a first modified representation of the object that is visually distinguishable from a first actual representation of the object via the secondary display of the HMD device based on determining that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0020] According to some embodiments, the system includes one or more processors. The system also includes means for detecting an object at a first location in a computer generated reality (CGR) environment. The system also includes means for displaying a first modified representation of the object visually distinguishable from a first actual representation of the object via a secondary display of the HMD device in response to detecting the object at the first location in the CGR environment based on determining that the first location is within a first predetermined distance outside of a field of view of a primary display of the HMD device.

[0021] According to some embodiments, a method is performed at a system having one or more processors, a memory, one or more image sensors, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The method includes receiving directional information corresponding to a position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. In response to receiving information corresponding to a changed state of the process, the method also includes: based on determining that a first criterion is satisfied, displaying a first representation corresponding to the state of the process via the secondary display of the HMD device; and based on determining that a second criterion is satisfied, displaying a second representation corresponding to the state of the process via the tertiary display of the HMD device, wherein the second representation is different from the first representation.

[0022] According to some embodiments, a head mounted display (HMD) device includes one or more image sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving information corresponding to a change in the state of a process. In response to receiving information corresponding to a changed state of the process, the one or more programs also include: based on determining that a first criterion is met, displaying a first representation corresponding to the state of the process via the secondary display of the HMD device; and based on determining that a second criterion is met, displaying a second representation corresponding to the state of the process via the tertiary display of the HMD device, wherein the second representation is different from the first representation.

[0023] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in a state of a process. In response to receiving information corresponding to a changed state of the process, the one or more programs also include: based on determining that a first criterion is met, displaying a first representation corresponding to the state of the process via the secondary display of the HMD device; and based on determining that a second criterion is met, displaying a second representation corresponding to the state of the process via the tertiary display of the HMD device, wherein the second representation is different from the first representation.

[0024] According to some embodiments, a transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device comprising one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in a state of a process. In response to receiving information corresponding to a changed state of the process, the one or more programs also include: based on determining that a first criterion is met, displaying a first representation corresponding to the state of the process via the secondary display of the HMD device; and based on determining that a second criterion is met, displaying a second representation corresponding to the state of the process via the tertiary display of the HMD device, wherein the second representation is different from the first representation.

[0025] According to some embodiments, the system includes one or more processors. The system also includes a device for receiving directional information corresponding to the position of an object outside the field of view of a primary display of a head-mounted display (HMD) device and a secondary display of the HMD device. The system also includes a device for performing the following operations in response to receiving information corresponding to a changed state of the process: displaying a first representation corresponding to the state of the process via the secondary display of the HMD device based on determining that a first criterion is met; and displaying a second representation corresponding to the state of the process via the tertiary display of the HMD device based on determining that a second criterion is met, wherein the second representation is different from the first representation.

[0026] Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transient computer-readable storage medium or other computer program product configured for execution by one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the following description, reference is made to the accompanying drawings which form a part thereof and which illustrate several examples of the present disclosure. It should be understood that other examples may be utilized and structural and operational changes may be made without departing from the scope of the present disclosure. The same reference symbols are used in different drawings to represent similar or identical items.

[0028] Figure 1A to Figure 1B Depicted are exemplary systems for use in various computer-generated reality techniques.

[0029] FIG. 2A to FIG. 2B An exemplary system for use in various computer-generated reality environments is depicted, the system including a device having an additive display.

[0030] Figure 3 Depicted are exemplary virtual objects according to some embodiments.

[0031] Figure 4 Depicted are example techniques for displaying virtual objects according to some embodiments.

[0032] Figure 5 Depicted are example areas on a device with an additive display for displaying virtual objects according to some embodiments.

[0033] FIG. 6A to FIG. 6C Depicted are example techniques for displaying virtual objects according to some embodiments.

[0034] Figure 7is a flowchart illustrating a method for displaying a virtual object according to some embodiments.

[0035] FIG. 8A to FIG. 8B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0036] FIG. 9A to FIG. 9B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0037] FIG. 10A to FIG. 10B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0038] FIG. 11A to FIG. 11B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0039] FIG. 12A to FIG. 12B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0040] FIG. 13A to FIG. 13B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0041] FIG. 14A to FIG. 14B Depicted are example techniques for displaying virtual objects according to some embodiments.

[0042] Fig.15 is a flowchart illustrating a method for transitioning an object between displays based on directional information according to some embodiments.

[0043] Fig.16 is a flowchart illustrating a method for displaying a modified representation of an object according to some embodiments.

[0044] FIG. 17A to FIG. 17B Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0045] Fig.18 Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0046] FIG. 19A to FIG. 19B Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0047] FIG. 20A to FIG. 20D Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0048] FIG. 21A to FIG. 21D Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0049] FIG. 22A to FIG. 22B Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0050] FIG. 23A to FIG. 23B Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0051] FIG. 24A to FIG. 24D Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0052] Fig.25 is a flowchart illustrating a method for managing one or more displays based on data associated with one or more processes according to some embodiments. DETAILED DESCRIPTION

[0053] Various examples of electronic systems and techniques for using such systems in connection with various computer-generated reality techniques are described.

[0054] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0055] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0056] A person may sense and / or interact with CGR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object, which creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0057] Examples of CGR include virtual reality and mixed reality.

[0058] A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0059] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, an MR environment is anything between a fully physical environment at one end and a VR environment at the other end, but not including both ends.

[0060] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0061] Examples of MR include augmented reality and augmented virtuality.

[0062] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0063] An AR environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not true version of the original captured image. For another example, a representation of a physical environment may be transformed by graphically eliminating or blurring a portion thereof.

[0064] An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0065] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one example, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0066] Figure 1A and Figure 1B An exemplary system 100 for use in various computer-generated reality techniques is shown.

[0067] In some examples, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0068] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some examples, device 100a is implemented in either the base station device or the HMD device.

[0069] like Figure 1B As shown, in some examples, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate through a communication bus 150 of device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate through a communication bus 150 of device 100c.

[0070] In some examples, system 100 is a mobile device. In some examples, system 100 is a HDM device. In some examples, device 100 is a wearable HUD device.

[0071] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the following techniques.

[0072] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0073] System 100 includes a display 120. In some examples, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display images to the corresponding eyes of the user. The corresponding images are displayed on the first display and the second display simultaneously. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of the object on the display. In some examples, the first display may include a plurality of other displays (e.g., sub-displays), such as a main display and a secondary display. In some embodiments, when system 100 is operating, the main display has a different resolution from the secondary display. In some examples, display 120 includes a single display (e.g., a first display or a second display). In some examples, the single display includes a plurality of other displays (e.g., sub-displays), such as a main display and a secondary display. For each eye of the user, the corresponding image is displayed simultaneously on the first area and the second area of ​​the single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides a user with a stereoscopic effect of the objects on a single display.

[0074] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user inputs, such as tap inputs and swipe inputs. In some examples, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0075] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the real environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of physical objects from system 100. In some examples, system 100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment around system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some examples, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some examples, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0076] In some examples, the system 100 includes a microphone 112. The system 100 uses the microphone 112 to detect sounds from the user and / or the user's real environment. In some examples, the microphone 112 includes a microphone array (including multiple microphones), which are optionally operated in series to identify ambient noise or locate a sound source in the space of the real environment.

[0077] System 100 includes an orientation sensor 110 for detecting orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in a real environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0078] Figure 2ADevice 202 is shown. In some embodiments, device 202 may include one or more components of system 100, such as one or more processors (e.g., processor 102), memory (e.g., memory 106), camera sensors (e.g., image sensor 108), or motion sensors (e.g., orientation sensor 110). For example, device 202 may be Figure 1A Alternatively, in some embodiments, device 202 may be Figure 1B An implementation of the second device 100c depicted in .

[0079] like Figure 2A As shown, device 202 is a wearable device including a first display (e.g., a left-eye display panel of device 202) and a second display (e.g., a right-eye display panel of device 202). The first display and the second display each include a primary display 204 surrounded by a secondary display 206. Figure 2A As shown, the secondary display 206 surrounds at least a portion of the primary display 204. In some embodiments, the secondary display 206 extends across the entire first display or the second display. In some embodiments, the primary display 204 overlaps only a portion (e.g., the center) of the secondary display 206.

[0080] In some embodiments, the primary display 204 and the secondary display 206 are two different types of displays. In some embodiments, the primary display 204 is a waveguide display. In some embodiments, the secondary display 206 is an organic light emitting diode display.

[0081] exist Figure 2A In the embodiment of the present invention, the primary display 204 is physically coupled (e.g., connected, attached) to the secondary display 206 and electronically coupled (e.g., connected, attached) to the secondary display 206. In some embodiments, one or more areas or additional displays may be provided between the primary display 204 and the secondary display 206.

[0082] exist Figure 2A, the primary display 204 and the secondary display 206 are depicted as oval displays. However, it should be appreciated that the primary display 204 and the secondary display 206 may have various shapes, such as a rectangular shape (e.g., a rectangular prism). Additionally, in some embodiments, the shape of the primary display 204 may be different from the shape of the secondary display 206. In some embodiments, when the primary display 204 and / or the secondary display 206 are rectangular in shape, the device 202 may apply a visual effect (e.g., blurring, fading the corners of the primary display and / or the secondary display) to the primary display 204 and / or the secondary display 206 so that the primary display 204 and / or the secondary display 206 appear to be round.

[0083] like Figure 2A As shown, an edge 208 is defined between the primary display 204 and the secondary display 206. In some embodiments, the edge 208 is transparent and / or translucent. In some embodiments, the edge 208 is an edge of the primary display 204 or an edge of the secondary display 206. In some embodiments, the edge 208 is a physical boundary between the primary display 204 and the secondary display 206.

[0084] In some embodiments, the primary display 204 may operate at a maximum resolution that is higher than the maximum potential resolution of the secondary display 206. Since displays that may operate at a higher resolution generally require more energy, the device 202 may use more energy when displaying a virtual object (or CGR object) via the primary display 204 than when displaying the same object via the secondary display 206. Therefore, the device 202 may extend battery life based on determining to display one or more virtual objects in the secondary display 206 instead of displaying the one or more virtual objects in the primary display 204.

[0085] like Figure 2A As shown, the device 202 includes a frame 210 positioned around the secondary display 206. The frame 210 includes a tertiary display, such as a light emitting diode (LED). Alternatively, in some embodiments, the frame 210 does not include a tertiary display.

[0086] As will be described below, in some embodiments, device 202 causes one or more of these LEDs to emit light. In some embodiments, these LEDs do not surround an inner portion of secondary display 206 located near the user's nose.

[0087] exist Figure 2B , the device 202 is depicted as being worn by a user 250 such that the nose of the user 250 is positioned below the bridge 214 of the device 202. As will be described below, the device 202 enables the user 250 to view a CGR environment.

[0088] Figure 2B A user 250 is depicted standing in a physical environment, with both eyes of the user 250 positioned such that the pupils 216 of the user 250 are aligned with a portion of the primary display 204. Figure 2B At , the central vision or line of sight of the user 250 is aligned with a portion of the field of view of the primary display 204, indicating that the object displayed in the primary display 204 is displayed in the central vision of the user.

[0089] also, Figure 2B The peripheral vision of user 250 is depicted aligned with a portion of the field of view of secondary display 206, indicating that an object displayed in secondary display 206 is displayed in the user's peripheral vision. As will be described below, in some embodiments, when one or more LEDs are emitting light, the light is displayed in the outer portion or edge of the peripheral vision of user 250. In some embodiments, one or more sensors of device 202 can track the movement of the user's eyes, including pupil 216. In some embodiments, device 202 uses this motion tracking to adjust or modify one or more of the techniques discussed below.

[0090] Figure 3 Depicted are exemplary virtual objects according to some embodiments. Specifically, Figure 3 The virtual object is depicted as a pencil 302 that is colored in a solid color (eg, black, blue). The pencil 302 has different portions 302a-302c.

[0091] Figure 4 A device 202 is depicted showing a portion of a pencil 302. Specifically, in Figure 4 , pencil 302 is depicted as falling from a location outside the field of view of primary display 204 and secondary display 206. Specifically, in Figure 4 , pencil 302 is depicted in a position where portions 302a and 302b of pencil 302 are within the field of view of primary display 204 and portion 302c of pencil 302 is within the field of view of secondary display 206, but Figure 4 The portion of the pencil 302 depicted in dashed lines in FIG. 3 is outside the field of view of the primary display 204 and the secondary display 206 .

[0092] like Figure 4 As shown, portion 302c is blurred when compared to portions 302a and 302b. Figure 3 The blurriness of portion 302c is shown by the lighter shade depicted in FIG. In contrast, portions 302a and 302b contain little to no blurriness, which is indicated by coloring portions 302a-302b as being the same as Figure 3Portion 302c has more blur than portions 302a and 302b because portion 302c is displayed via secondary display 206, which may have a lower maximum resolution than primary display 204 that is displaying portions 302a-302b.

[0093] When device 202 is worn by a user, the edge between the user's central vision and peripheral vision is typically near region 404. Figure 4 As depicted in FIG. 4 , region 404 corresponds to the area of ​​primary display 204 and secondary display 206 that display segments of portions 302 b and 302 c .

[0094] Because portion 302c is blurry compared to portion 302b, hard clipping can occur in region 404 and / or the edge between the user's central vision and peripheral vision. In some embodiments, hard clipping occurs when there is visual distortion between a portion of an object displayed on one display and a portion of an object displayed on another display. As described further below, hard clipping can occur when an object transitions from one display to another display.

[0095] In some embodiments, hard clipping in region 404 is an unintended visual effect. Specifically, hard clipping can cause the user interface displayed by device 202 to be disruptive to the user wearing device 202. For example, the user interface can be disruptive to the user wearing device 202 when an object enters or leaves the user's central vision.

[0096] Figure 5 Depicted are exemplary regions on a device with an additive display according to some embodiments. As further explained below, hard clipping may occur between one or more exemplary regions on a device. Since the field of view of the primary display 204 is typically in the user's central vision and the field of view of the secondary display 206 is in the user's peripheral vision, reducing the hard clipping of objects that transition between the primary display 204 and the secondary display 206 can reduce high-frequency noise (e.g., visual snow, random fluctuations and / or sudden fluctuations of objects, irregular motion or spatial relationships between objects) in the user's central vision and / or peripheral vision. Reducing high-frequency noise in the user's central vision and / or peripheral vision is advantageous because high-frequency noise may cause the user to unnecessarily view objects in the user's peripheral vision. Therefore, high-frequency noise may affect the user's ability to focus on objects in the user's central vision, which may affect the ability of the device 202 to track the user's eye movements and / or make accurate determinations based on the user's eye movements.

[0097] Figure 52 shows a device 202 having an exemplary first region 504a and an exemplary second region 504b of a primary display 204. The first region 504a extends across the area of ​​the primary display 204 from an edge 208 of the primary display 204 to a distance 510a from the edge 208 of the primary display 204. In some embodiments, Figure 4 At least a portion of the region 404 depicted in Figure 5 In some embodiments, when the user wears the device 202, at least a portion of the edge of the user's central vision and peripheral vision is located in a portion of the first area 504a.

[0098] The second region 504b extends across the area of ​​the primary display 204 from the center of the primary display 204 to a distance 510b toward the edge 208 of the primary display 204. In some embodiments, Figure 4 No portion of the region 404 depicted in Figure 5 In some embodiments, when the user is wearing the device 202, no portion of the edge of the user's central vision and peripheral vision is located in a portion of the first area 504a.

[0099] In some embodiments, the first region 504a and / or the second region 504b extends across Figure 5 Other areas of the primary display 204 that are not shown may not extend across Figure 5 In some embodiments, the first region 504a and / or the second region 504b extend across the entire primary display 204.

[0100] FIG. 6A to FIG. 6C and Figure 7 Describes exemplary techniques for displaying virtual objects according to some embodiments. Notably, in FIG. 6A to FIG. 6C and Figure 7 In the case of an object such as Figure 4 As shown, the following exemplary techniques and processes can reduce hard clipping of objects (e.g., abrupt changes when objects transition from one display to another) when transitioning between displays having different resolutions (e.g., between the primary display 204 and the secondary display 206). As described above, reducing hard clipping of objects transitioning between the primary display 204 and the secondary display 206 can reduce high-frequency noise in the user's central vision and / or peripheral vision. Reducing high-frequency noise in the user's central vision and / or peripheral vision is advantageous because high-frequency noise can affect the ability of the device 202 to track the user's eye movements and / or make accurate determinations based on the user's eye movements.

[0101] Fig. 6A An environment including device 202 and pencil 302 is shown. Fig. 6A At , user 250 wears device 202, as described above with respect to Figure 2B However, for ease of discussion, Fig. 6A User 250 is not depicted.

[0102] Fig. 6A The device 202 is depicted showing the pencil 302 falling from a location outside the field of view of the primary display 204 and the secondary display 206. FIG. 6A to FIG. 6C , the portion of pencil 302 represented by the dashed line corresponds to the portion of 302 that is not visible to a user viewing through primary display 204 and / or secondary display 206 while wearing device 202.

[0103] like Fig. 6A As shown, pencil 302 has fallen from a position outside the field of view of primary display 204 and secondary display 206 to a position within the field of view of secondary display 206 and outside the field of view of primary display 204. Fig. 6A , portion 302a is presented via secondary display 206. In some embodiments, device 202 generates a virtual representation of portion 302a and displays the virtual representation via secondary display 206.

[0104] exist Fig. 6A , device 202 displays a virtual representation of portion 302a, which is rendered at a resolution corresponding to the maximum resolution of secondary display 206. Fig. 6A As shown, when Figure 3 In some embodiments, portion 302a is blurred (e.g., as shown by its lighter color). Fig. 6A The virtual representation of portion 302a shown in FIG. 1 has no visual effects applied to create blurry changes.

[0105] exist Fig. 6A Some time after the time depicted in FIG. 1 , as pencil 302 continues to fall, Figure 6B Depicts that portion 302a has fallen into a particular area of ​​primary display 204 (e.g., as described below in Figure 5 When the pencil 302 is displayed via the secondary display 206 (e.g., within the field of view of the secondary display 206), the pencil 302 is determined to have fallen to a position within a distance 510a from the edge 208 of the primary display 204 (e.g., moved into the primary display 204), as shown in FIG. Figure 6B Based on the determination, the device 202 generates a virtual representation by applying a visual effect to the portion 302a of the pencil 302 and displays the virtual representation in the primary display 204, as shown. Figure 6B shown.

[0106] Here, the virtual representation corresponding to portion 302a is blurred by applying the visual effect to portion 302a. Figure 6B As shown, applying the visual effect causes portion 302a to appear as a gradient (e.g., from a light color to a dark color as shown) to appear at the edges of the user's central vision and peripheral vision (and / or Figure 4 404 discussed in . In some embodiments, after the visual effect is applied to portion 302a, the virtual representation corresponding to portion 302a is faded. In some embodiments, after the visual effect is applied to portion 302a, the virtual representation fades in a manner inversely proportional to the detected amount of blur of the secondary display 206 (or blur of at least one area of ​​the secondary display 206). In some embodiments, after the visual effect is applied to portion 302a, the virtual representation does not include a gradient. In other embodiments, applying the visual effect includes applying a blur function to portion 302a. In some embodiments, the amount of blur applied to portion 302a at edge 208 may be selected to match the blur of the secondary display 206. In this case, the appearance of the virtual object 302 on opposite sides of the edge 208 may be the same or similar, thereby reducing the hard clipping effect described above. The amount of blur applied to portion 302a (or other virtual content) within portion 504a may be reduced based on the distance from edge 208, and no blur is applied at distance D1 from edge 208. Blur may be reduced linearly or non-linearly as a function of distance from edge 208.

[0107] like Figure 6B 2, portion 302b has also been moved into the field of view of secondary display 206. Thus, in addition to displaying a virtual representation corresponding to portion 302a in primary display 204, device 202 also displays a virtual representation corresponding to portion 302b of pencil 302. In some embodiments, device 202 uses the above-described method with respect to displaying and Fig. 6A The techniques described herein are used to display the virtual representation corresponding to portion 302a in FIG.

[0108] It is worth noting that Figure 6B As shown, the virtual representations corresponding to portions 302a and 302b are blurred, where the blur is represented by the difference in coloring of the respective virtual representations corresponding to the respective portions of pencil 302. Figure 6B , the virtual representation corresponding to portion 302b has a greater blur than the virtual representation corresponding to portion 302a.

[0109] exist Figure 6B, the virtual representation corresponding to portion 302b is blurred because it is displayed via the secondary display 206, which is a low-resolution display having greater blur than the primary display 204. On the other hand, the virtual representation corresponding to portion 302a is blurred because the device 202 applies the visual effect to portion 302a before displaying the virtual representation corresponding to portion 302a via the primary display 204. Therefore, the virtual representation corresponding to portion 302a to which the visual effect is applied is more similar to (e.g., has a blur value closer to) the virtual representation corresponding to portion 302b than the virtual representation corresponding to portion 302a to which the visual effect is not applied.

[0110] In some embodiments, the device 202 applies the visual effect to portion 302a so that the virtual representation of portion 302a appears to visually match a virtual representation corresponding to portion 302b displayed in the secondary display 206 (e.g., where the visual effect has not yet been applied to portion 302b). Because the secondary display 206 has a lower resolution than the primary display 204, the device 202 applies the visual effect to portion 302a so that the visual appearance of the virtual representation corresponding to portion 302a matches the visual appearance of the virtual representation corresponding to portion 302b.

[0111] In some embodiments, device 202 applies the visual effect to portion 302 to cause the image to be displayed via a display having a lower resolution (e.g., a secondary display). Fig. 6A The virtual representation corresponding to the portion 302a in the image is displayed via a display having a higher resolution (e.g., the main display 204). Figure 6B The transition of the virtual representation corresponding to the portion 302a in FIG.

[0112] like Figure 6C As shown, pencil 302 has fallen to a new position within the field of view of primary display 204 and secondary display 206. Figure 6C As shown, portion 302a is a distance (e.g., the sum of distance 510a and distance 510b) from edge 208 of primary display 204, which is outside distance 510a. Since portion 302a is outside distance 510a from edge 208 of primary display 204, portion 302a has entered a second area of ​​primary display 204 that is outside the first area of ​​primary display 204 (e.g., Figure 5 And, therefore, no visual effect is applied to the second area 504b ( Figure 5In some embodiments, no visual effect (or a visual effect different from the visual effect applied to the first area) is applied to the representation of the portion of the virtual object in the second area 504b of the primary display 204. In some embodiments, no visual effect is applied because the portion 302a is within the user's central vision and the device 202 has determined that the user may be distracted by the virtual object (e.g., or a particular visual object) without the visual effect applied. Figure 6C As shown, device 202 uses the above relative Fig. 6A The techniques described above for displaying a virtual representation of portion 302a in the secondary display 206 are used to display a virtual representation of portion 302c (i.e., the new portion of pencil 302 in the field of view of secondary display 206). Figure 6C At , device 202 displays a virtual representation of portion 302b to which the visual effect is applied (e.g., using the same method as above with respect to Figure 6C 302a in FIG. 2 ), because portion 302b is within a distance 510a from edge 208 of primary display 204.

[0113] In some embodiments, when the entire object (e.g., pencil 302) is in the field of view of the primary display (e.g., no portion of the object is in the field of view of the secondary display), no visual effects are applied to any portion of the representation of the virtual object, including the above with respect to Figure 5 A portion of a representation of a virtual object displayed in the second region 504b is depicted.

[0114] In some embodiments, the device 202 may determine to display different virtual objects in different ways. In some embodiments, the device 202 may apply a visual effect to one virtual object while not applying the visual effect to another virtual object. In some embodiments, the device 202 may apply different visual effects to virtual objects. In some embodiments, the device 202 uses different determinations to apply visual effects to different groups of virtual objects. In some embodiments, the device 202 may apply a visual effect to one type of virtual object based on determining that the virtual object has entered an area of ​​the display, but abandon applying the visual effect to another type of virtual object based on determining that other types of virtual objects have entered the area.

[0115] In some embodiments, FIG. 6A to FIG. 6C The operations described can be described in reverse (e.g., FIG. 6C to FIG. 6A ). For example, a virtual object may transition from within the field of view of the primary display 204 and / or the secondary display 206 to outside the field of view of the primary display 204 and the secondary display 206 using techniques similar to those described above.

[0116] Figure 7is a flow chart illustrating a method 700 for displaying a virtual object according to some embodiments. In some embodiments, the system 100 ( Figure 1A and Figure 1B ) performs the method. In some embodiments, the device 202 ( Figure 2A ) performs the method. In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0117] At block 702, a first portion of a virtual object is displayed via a secondary display. For example, as described above with reference to Figure 6C 2, a first portion 302c of the virtual object 302 is displayed via the secondary display 206. In some embodiments, at a first time instance, the virtual object is presented on the secondary display (e.g., the secondary display 206) and the virtual object is not displayed on the primary display (e.g., the primary display 204). In some embodiments, when the virtual object is presented on the secondary display and when the virtual object is not presented on the primary display, the virtual object is in the field of view of the secondary display and the virtual object is not in the field of view of the primary display.

[0118] refer to Figure 7 At block 704, a second portion of the virtual object is displayed via the primary display. Figure 6C As described, the second portion 302b of the virtual object 302 is displayed via the main display 204. In some embodiments, when the device has moved within a predefined distance from the virtual object, the second portion of the virtual object is displayed via the main display. In some embodiments, when the virtual object moves within a predefined distance from an edge of the main display or within a predefined distance from a field of view of the main display (e.g., an edge of the field of view), the second portion of the virtual object is displayed via the main display.

[0119] refer to Figure 7 As part of displaying the second portion of the virtual object (block 704), at block 706, based on determining that the second portion of the virtual object is within a predefined distance from an edge of the primary display, a visual effect is applied to the second portion of the virtual object. Figure 6C As described, as part of displaying the second portion 302 b of the virtual object 302 , based on determining that the second portion 302 b is within a predefined distance 510 a from an edge of the primary display 204 , a visual effect is applied to the second portion 302 b of the virtual object 302 .

[0120] In some embodiments, a visual effect is applied to portions of one or more virtual objects when the portions of the one or more virtual objects are in the first area of ​​the primary display. Figure 6C As described, the second portion 302b is Figure 5 The visual effect is applied when displayed in the area 504a shown. In some embodiments, the area 504a is within a predefined distance from the edge of the primary display adjacent to the edge of the secondary display.

[0121] In some embodiments, the amount of visual effect applied may be the same for two or more virtual objects in a particular area. Alternatively, in some embodiments, the amount of visual effect applied to a particular virtual object is dynamic. For example, the amount of visual effect applied may be based on one or more detected features associated with the user wearing the device or based on one or more user settings. In some embodiments, user features may include pupil size, pupil direction, pupil position, eye deviation, blink rate, etc. of the user at different points in time.

[0122] In some embodiments, the amount of visual effect applied depends on one or more characteristics of the virtual object. For example, in some embodiments, the amount of visual effect applied depends on the size, motion (e.g., speed, acceleration), color, etc. of the virtual object. In some embodiments, the amount of visual effect applied is smaller for virtual objects moving at a slower speed and larger for virtual objects moving at a higher speed.

[0123] In some embodiments, the amount of visual effects applied depends on one or more characteristics of the device, such as the speed or acceleration of the device. In some embodiments, the amount of visual effects applied is smaller when the device moves at a slower speed and larger when the device moves at a greater speed.

[0124] In some embodiments, a visual effect is applied to a portion of a virtual object when it is displayed in one area of ​​the primary display, while the visual effect is not applied to another portion of the virtual object displayed in another area of ​​the primary display. Figure 6C As described, part 302a is Figure 5 Region 504b is shown displayed with no visual effects applied.

[0125] In some embodiments, applying the visual effect to the second portion of the virtual object includes blurring the second portion of the virtual object. In some embodiments, before applying the visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first blurred appearance. In some embodiments, after applying the visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second blurred appearance that is different from (e.g., more blurred than) the first blurred appearance.

[0126] In some embodiments, a second portion of the virtual object is blurred based on a feature of a pupil that is aligned with a portion of the primary display and / or the secondary display detected via the one or more image sensors. In some embodiments, a portion of the virtual object is blurred or faded based on one or more features of the detected pupil while not blurring or fading another portion of the virtual object. In some embodiments, the blurring or fading of a portion of the virtual object increases as the detected pupil is further from an edge of the primary display.

[0127] In some embodiments, after blurring the second portion of the virtual object, the second portion of the virtual object includes an amount of blur based on the amount of blur of the first portion of the virtual object. In some embodiments, the amount of blur (e.g., of the portion of the virtual object) matches the blur of the first portion of the virtual object. In some embodiments, as part of blurring the second portion of the virtual object, the amount of blur of the second portion of the virtual object is changed (e.g., continuously changed as the distance between the object and the device 202 changes) based on a corresponding distance between the virtual object and the secondary display.

[0128] In some embodiments, as part of applying the visual effect to the second portion of the virtual object, the second portion of the virtual object is faded (e.g., faded in or faded out). In some embodiments, fading the second portion of the virtual object includes feathering an edge of the primary display and / or the virtual object adjacent to the second area.

[0129] In some embodiments, before applying the visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first faded appearance. In some embodiments, after applying the visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second faded appearance that is different from (e.g., more faded than) the first faded appearance.

[0130] In some embodiments, the second portion of the virtual object is faded in proportion to the blurriness of the secondary display (e.g., display 206) (e.g., at least one area of ​​the secondary display). In some embodiments, the second portion of the virtual object is faded out to a level inversely proportional to the blurriness of the secondary display.

[0131] In some embodiments, the second portion of the virtual object fades from the first color to the second color. In some embodiments, the second portion of the virtual object is faded so that a gradient is applied to the second portion of the virtual object. In some embodiments, when the gradient is applied to the second portion of the virtual object, the second portion of the virtual object includes a gradient (e.g., a gradient that transitions from the first color to the second color).

[0132] In some embodiments, as part of applying a visual effect to the second portion of the virtual object, the corners of the second portion of the virtual object are rounded. In some embodiments, the edge between the primary display and the secondary display (and / or the first area) is rounded or modified to appear rounded. In some embodiments, the corner is faded or a specific visual effect is applied (e.g., a visual effect is applied to the corner, thereby fading the corner, while other portions of the second portion of the virtual object are not faded or the specific visual effect is not applied). In some embodiments, a gradient is applied to the corner when the corner is faded.

[0133] Reference again Figure 7 As part of displaying the second portion (block 704), based on determining that the second portion of the virtual object is not within a predefined distance from an edge of the primary display, foregoing applying the visual effect to the second portion of the virtual object. Figure 6C As described by displaying the second portion 302c of the virtual object 302 via the primary display 204, based on determining that the second portion 302c of the virtual object 302 is not within a predefined distance 510a from an edge of the primary display 204, applying the visual effect to the second portion 302c of the virtual object 302 is abandoned.

[0134] FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11B , FIG. 12A to FIG. 12B , FIG. 13A to FIG. 13B and FIG. 14A to FIG. 14B Describes exemplary techniques for transitioning an object between displays according to some embodiments. Among other benefits, the following techniques and processes help extend the battery life of the device 202 while continuing to enable the device 202 to present information (e.g., present information to Figure 2BThe device 202 extends battery life by intelligently shifting the presentation of information from the primary display 204 to one or more of the secondary display 206 and the tertiary display (e.g., LEDs around the frame 210). Shifting the information can extend battery life because the secondary display 206 and the tertiary display can have a lower maximum resolution than the resolution of the primary display 204. Therefore, the secondary display 206 and the tertiary display can use less energy than the primary display 204 to display information.

[0135] As discussed below, Fig. 8A , Fig. 9A , Fig. 10A , Fig.11A , Fig. 12A , Fig.13A and Fig.14A An exemplary scenario is depicted in which user 250 interacts with object 806 while wearing device 202. However, this particular scenario is provided for exemplary purposes only. The techniques discussed below or similar techniques may also be effectively applied to a variety of other scenarios.

[0136] As mentioned above, Fig. 8A A scene is shown in which user 250 is wearing device 202. In this scene, device 202 is positioned at a distance 804a from object 806 (such as a bunch of keys). In some embodiments, object 806 is a physical article or physical object located in a physical environment with user 250. In some embodiments, object 806 is a virtual object located in a CGR environment displayed to user 250 by device 202.

[0137] In some embodiments, the object 806 moves while the user 250 and / or device 202 is stationary or alternatively non-stationary.

[0138] In some embodiments, the environment includes one or more objects in addition to object 806. In some embodiments, device 202 detects object 806 in Fig. 8A In some embodiments, the device 202 determines that the object is a useful or important object (e.g., or more useful or important than other objects in the environment). In some embodiments, the device 202 may use one or more algorithms and / or one or more user preferences to determine whether the object is useful or important to the user or an application corresponding to a particular object (e.g., an application communicating with the device 202). In some embodiments, the device 202 receives a request or notification related to the object and makes the determination as to whether the object is useful or important based on the request or notification.

[0139] Figure 8B Shown in Fig. 8AThe device 202 includes a primary display 204 and a secondary display 206 (as described above). Figure 2A in addition to one or more of the components described above). Figure 8B 806 relative to the device 202. In some embodiments, at least some of the visual information is not displayed via the primary display 204 and / or the secondary display 206 because the device 202 is too far from the object 806 (or the distance 804a exceeds a predetermined threshold distance).

[0140] like Figure 8B As shown, device 202 also includes LEDs 810a-810f and 812a-812f (LEDs). Figure 8B In some embodiments, one or more of these LEDs are not illuminated because the device 202 is too far from the object 806 (or the distance 804a exceeds a predetermined threshold distance).

[0141] Fig. 9A The object 806 and / or device 202 is shown as having been removed from its Fig. 8A Move the position in Fig. 9A The position in.

[0142] Fig. 9B Shown in Fig. 9A The depicted scene corresponds to the time of device 202. Fig. 9A and Fig. 9B , assume that device 202 receives direction information when device 202 is separated from object 806 by distance 804b. In response to receiving the direction information, device 202 determines that object 806 is a type of object for which direction information can be (e.g., configured to be, determined to be) displayed via one or more of the LEDs. In some embodiments, device 202 makes this determination because device 202 is separated from object 806 by distance 804b (or distance 804b is less than a predetermined threshold distance (e.g., a threshold distance corresponding to a maximum distance for displaying direction information).

[0143] like Fig. 9B As shown, device 202 causes LEDs 812a-812f to illuminate in a pattern to show directional information because device 202 is a distance 804b from object 806 (or object 806 is a type of object for which directional information may be displayed via one or more of these LEDs). Fig. 9BAs shown, LEDs 812a-812f are located on the right side of device 202 in a direction that is more upward than downward on frame 210 of device 202. Thus, LEDs 812a-812f indicate that object 806 is in front of and to the right of device 202, which matches Fig. 9A The approximate direction of the path between device 202 and object 806 in FIG.

[0144] exist Fig. 9B , LEDs 812a and 812f are shaded, while LEDs 812b-812e are solid to demonstrate a gradient field of light. Here, the gradient field of light can be shown from a weaker point of directional pull of directional information between object 806 and device 202 (e.g., shaded LEDs 812a, 812f) to a stronger point of directional pull (e.g., solid LEDs 812b-812e). Notably, the gradient field can provide more directional information to the user, which can help the user navigate toward object 806 more effectively.

[0145] In some embodiments, at a distance greater than or equal to a predetermined threshold distance (e.g., a distance greater than or equal to distance 804b), directional information is displayed via these LEDs instead of the primary display 204 and / or the secondary display 206. In some embodiments, directional information is displayed via these LEDs instead of the primary display 204 and / or the secondary display 206 because displaying directional information via these LEDs instead of the primary display 204 and / or the secondary display 206 can extend battery power.

[0146] In some embodiments, directional information is displayed via these LEDs instead of the primary display 204 and / or the secondary display 206, based on the type of information that is allowed to be displayed when the device 202 is at a certain distance from the object 806. In some embodiments, when the device 202 is farther away from the object 806 (or within a first set of distances from the object), the device 202 displays directional information that is not based on the appearance of the object (e.g., flashing LEDs). In some embodiments, when the device 202 is closer to the object 806 (or within a second set of distances from the object, where the second set of distances is different from the first set of distances), the device 202 displays directional information based on the appearance of the object (e.g., a visual representation of the object or proxy object displayed via the primary display 204 and / or the secondary display 206). In some embodiments, the device 202 may display directional information based on the appearance of the object and directional information that is not based on the appearance of the object (e.g., when the device 202 is within a third set of distances from the object, the third set of distances being between the first set of distances from the object and the second set of distances). In some embodiments, directional information is displayed via the LEDs instead of the primary display 204 and / or secondary display 206 based on whether the object is within the FOV of the primary display 204 and / or secondary display 206. In some embodiments, directional information is displayed via the LEDs instead of the primary display 204 and / or secondary display 206 when the object is a certain distance outside the FOV of the primary display 204 and / or secondary display 206.

[0147] Fig. 10A The object 806 and / or device 202 is shown as having been removed from its Fig. 9A Move the position in Fig. 10A Specifically, Fig. 10A , device 202 is positioned at a distance 804c from object 806, where distance 804c is different from Fig. 9A The distance 804b in.

[0148] Fig. 10B Shown in Fig. 10A The depicted scene corresponds to the time of device 202. Fig. 10A and Fig. 10B , assuming that device 202 receives direction information. In response to receiving the direction information, using the same Fig. 9A and Fig. 9BUsing techniques similar to those discussed above, device 202 determines that object 806 is a type of object for which directional information can be displayed via these LEDs (e.g., because device 202 is separated from object 806 by distance 804c). Based on this determination, device 202 then causes LEDs 812d-812f to emit light corresponding to the directional information and causes LEDs 812a-812c to stop emitting light. Here, the updated LEDs of device 202 show the newly received directional information (e.g., directional information showing a relative path to the location of object 806 is downward and to the left of the location of device 202).

[0149] like Fig. 10B As shown, device 202 from Fig. 9B In some embodiments, when the device 202 is larger than the LED 202, the number of LEDs that emit light is reduced. Fig. 9B In some embodiments, when there are physical and / or virtual structures or objects between the device 202 and the object 806, the device 202 continuously updates the plurality of LEDs (and / or the primary display 204 and / or the secondary display 206) of the device 202 (e.g., LEDs that illuminate in a certain pattern) so that the user 250 can navigate around the physical and / or virtual structures or objects between the device 202 and the object 806.

[0150] Fig.11A The object 806 and / or device 202 is shown as having been removed from its Fig. 10A Move the position in Fig.11A Specifically, Fig.11A , device 202 is positioned at a distance 804d from object 806, where distance 804d is different from Fig. 10A The distance 804c in.

[0151] Fig. 11B Shown in Fig.11A The depicted scene corresponds to the time of device 202. Fig.11A and Fig. 11B , assuming that device 202 updates not only the LEDs but also the secondary display 206, this is because device 202 is closer (e.g., within a set of predetermined threshold distances), at a distance 804d from object 806 (rather than Fig. 10B In some embodiments, the device 202 updates not only the LEDs, but also the secondary display 206 because the object 806 is within the FOV of the secondary display 206.

[0152] It is also assumed that device 202 receives direction information. In response to receiving the direction information, the device 202 uses the same method as above with respect to Fig. 9A and Fig. 9B Similar to the techniques discussed above, device 202 determines that object 806 is a type of object for which directional information can be displayed via LED 812e (e.g., because device 202 is separated from object 806 by distance 804d). Based on this determination, device 202 uses the above-described method with respect to Fig. 9B and Fig. 10B One or more techniques are described to cause LED 812e to emit light corresponding to the direction information and cause LEDs 812d and 812f to stop emitting light.

[0153] If device 202 determines that object 806 is a type of object for which direction information may be displayed via secondary display 206, device 202 displays the direction information via secondary display 206. Here, device 202 makes this determination because it is separated from object 806 by distance 804d.

[0154] In some embodiments, because the device 202 is separated from the object 806 by a distance 804d and / or the object 806 is within a predetermined threshold distance outside the field of view of the primary display 204, the device 202 causes the secondary display 206 to display a modified representation 826a of the object 806. Here, the modified representation 826a is a representation of a proxy object or a symbolic representation of the object 806 and is visually distinguishable from the object 806. Fig. 11B As shown, the modified representation 826a is a patch of multiple pixels at the edge of the secondary display 206 .

[0155] In some embodiments, when the device 202 is separated from the object 806 by a distance 804d, the object 806 is within the field of view of the secondary display 206. In some embodiments, the object 806 is at a position within the field of view of the secondary display 206 that corresponds to the position on the secondary display 206 where the modified representation 826a is displayed.

[0156] In some embodiments, when the device 202 is separated from the object 806 by a distance 804d, the object 806 is outside the field of view of the secondary display 206. Thus, in some embodiments, when the object 806 is outside the field of view of the secondary display 206, the device 202 displays the modified representation 826a via the secondary display 206. In some embodiments, displaying the modified representation 826a at a corresponding position on the secondary display 206 when the object 806 is outside the field of view of the secondary display 206 provides directional information for locating an object outside the field of view of the display of the device 202.

[0157] exist Fig. 11B, the modified representation 826a does not resemble the actual representation of the object 806. In some embodiments, the modified representation 826a does not resemble the object 806 because it does not have two or more features of the object 806, such as size, shape, color, texture, blur, etc.

[0158] Modified representation 826a shows an object (e.g., as described above with respect to Fig. 8A The modified representation 826a may also show the approximate direction of the location of the object relative to the location of the device 202. In some embodiments, the approximate direction is shown by positioning the modified object representation 826a at a specific location on the secondary display 206.

[0159] In some embodiments, the modified representation 826a is static (e.g., does not move, does not pulse, is not animated), while the light emitted from one or more of the LEDs is not static (e.g., moves, pulses, is animated), or vice versa. In some embodiments, the modified representation 826a and LEDs 812a-812l are both static (e.g., or are not static).

[0160] Fig. 12A The object 806 and / or device 202 is shown as having been removed from its Fig.11A Move the position in Fig. 12A Specifically, Fig. 12A , device 202 is positioned at a distance 804e from object 806, where distance 804e is different from Fig.11A The distance in is 804d.

[0161] Fig. 12B Shown in Fig. 12B The depicted scene corresponds to the time of device 202. Fig. 12A and Fig. 12B , assuming that the device 202 receives new direction information. Here, based on the new direction information, the device 202 updates the primary display 204 and the secondary display 206 to display a representation of the direction information and abandons displaying the direction information via the LEDs.

[0162] In some embodiments, the device 202 determines that the object 806 is not a type of object for which directional information can be displayed via the LEDs 812a-812l because the device 202 is separated from the object 806 by a distance 804e. For example, the device 202 may determine that the distance 804e is a distance at which the device 202 is too close to the object 806. In some embodiments, when the directional information or one or more types of directional information (e.g., representation 828) is displayed via the primary display 204 and / or the secondary display 206, the device 202 ceases to display the directional information via the LEDs.

[0163] In some embodiments, in response to receiving the direction information, device 202 uses the method described above in Fig. 11B 806 is a type of object for which directional information may be displayed via secondary display 512 (e.g., because device 202 is separated from object 806 by distance 804e and / or object 806 is within a predetermined threshold distance outside the field of view of primary display 204). As a result of this determination, device 202 causes secondary display 206 to display modified representation 826b.

[0164] exist Fig. 12B , modified representation 826b is greater than modified representation 826a because device 202 is Fig. 12B In Fig. 11B is closer to the object 806. Fig. 12B As shown, the modified representation 826b is displayed in a new position on the secondary display 206, the new position being, for example, Fig.11A Modified representation 826a is shown positioned closer to primary display 204. Modified representation 826b is displayed in a new position because object 806 is closer to primary display 204's field of view.

[0165] In some embodiments, the modified representation 826b occupies a portion of the secondary display 206 that was occupied by the modified representation 826a. In some embodiments, the modified representation 826b grows from the modified representation 826a (e.g., via an animation). In some embodiments, the device 202 displays the modified representation 826b and / or the modified representation 826a at a brightness based on the size of the object 806. For example, the device 202 displays the modified representation of a large object at a lower or darker brightness than the modified representation of a small object.

[0166] In some embodiments, in response to receiving the directional information, device 202 determines that object 806 is a type of object for which directional information can be displayed via primary display 204. Fig. 12BAt , device 202 makes this determination based on determining that distance 804e is close enough to object 806. Based on this determination, device 202 causes representation 828 to be displayed via primary display 204.

[0167] exist Fig. 12B , representation 828 is an arrow pointing to the location of object 806. In some embodiments, device 202 displays representation 828 to indicate that user 250 can adjust the field of view of primary display 204 (e.g., by moving the head of user 250 while standing still) so that object 806 can be located within the field of view of primary display 204. In some embodiments, device 202 causes representation 828 to be displayed in an animated manner. In some embodiments, device 202 causes representation 828 (or a representation corresponding to representation 828) to be displayed via secondary display 206.

[0168] Fig.13A The object 806 and / or device 202 is shown as having been removed from its Fig. 12A Move the position in Fig.13A Specifically, Fig.13A , device 202 is positioned at a distance 804f from object 806, where distance 804f is different from Fig. 12A In addition, if Fig.13A As depicted, object 806 is within the field of view of primary display 204 of device 202 .

[0169] Fig. 13B Shown in Fig.13A The depicted scene corresponds to the time of device 202. Fig.13A and Fig. 13B , assuming that device 202 receives new direction information. Because device 202 is separated from object 806 by distance 804f and / or because at least a portion of object 806 is in the field of view of the primary display and in the field of view of the secondary display, device 202 displays an actual representation of a portion of object 806 via primary display 204 and displays a modified representation 826c of object 806 via secondary display 206. In some embodiments, the actual representation of the portion of the object is similar to object 806 (e.g., having two or more features similar to object 806), while the modified representation is not similar to object 806. In some embodiments, the actual representation of the portion of the object is displayed using a first model of the object, while the modified representation is displayed using a second model of the object. In some embodiments, the first model of the object is different from the second model of the object because the second model of the object is simpler than the first model of the object. In some embodiments, the second model of the object is simpler than the first model of the object because the second model has fewer details than the first model (e.g., a three-dimensional model with fewer polygons).

[0170] In some embodiments, by using the same Figure 7 204 (e.g., to reduce clipping of object 806 transitioning between primary display 204 and secondary display 206) applies a visual effect to a virtual representation of a portion of object 806 to cause a portion of modified representation 826b to move into primary display 204. In some embodiments, when using the same techniques as described above in Figure 7 When an actual representation of a portion of object 806 is transformed for display via primary display 204 using techniques similar to those described in , device 202 replaces modified representation 826 b or 826 c with a representation of the actual portion of object 806 on secondary display 206 .

[0171] Fig.14A The object 806 and / or device 202 is shown as having been removed from its Fig.13A Move the position in Fig.14A Specifically, Fig.14A , device 202 is positioned at a distance 804g from object 806, where distance 804g is different from Fig.13A In addition, if Fig.14A As depicted, the entire object 806 is now within the field of view of the primary display 204 of the device 202 .

[0172] Fig. 14B Shown in Fig.14A The depicted scene corresponds to the time of device 202. Fig.14A and Fig. 14B , device 202 displays the actual representation of object 806 (or, in some embodiments, a portion of object 806) for display via primary display 204 and ceases to display the modified representation of object 806 via secondary display 206. In some embodiments, device 202 causes the actual representation of object 806 to be displayed via primary display 204 and ceases to display the modified representation of object 806 via secondary display 206 because device 202 is within the field of view of primary display 204 and not within the field of view of secondary display 206.

[0173] In some embodiments, Fig. 13B and Fig. 14BAt , device 202 may display a proxy representation of object 806 via secondary display 206 instead of displaying an actual representation of object 806 via primary display 204. In some embodiments, secondary display 206 may overlap primary display 204 so that secondary display 206 covers most (or all) of primary display 204. Thus, in some of these embodiments, a proxy representation of object 806 may be shown within a portion of primary display 204 via secondary display 206 until device 202 receives a request to interact with object 806. For example, a request to interact with object 806 may be received upon determining that input (e.g., voice input, physical input) is directed to object 806 and / or a process is initiated to perform an operation involving (e.g., indirectly or directly) one or more components of object 806 and / or an object associated with object 806. In some embodiments, in response to receiving a request to interact with object 806, device 202 stops displaying the proxy representation of object 806 via secondary display 206 and displays an actual representation of object 806 via primary display 204. Thus, in embodiments where device 202 would use more battery power by displaying an object via primary display 204 than by displaying the object via secondary display 206, battery power of device 202 is conserved because primary display 204 is not used to display a representation of object 806 until it has been determined that object 806 can be interacted with.

[0174] In some embodiments, Fig. 13B and Fig. 14BAt , device 202 may display a proxy representation of object 806 via main display 204 instead of displaying an actual representation of object 806. In some embodiments, the proxy representation of object 806 has less visual detail and / or requires less power to display than the actual representation of object 806. In some embodiments, while displaying the proxy representation of object 806 via main display 204, device 202 receives a request to interact with object 806. For example, a request to interact with object 806 may be received when determining that input (e.g., voice input, physical input) is directed to object 806 and / or a process for initiating an operation involving (e.g., indirectly or directly) one or more components of object 806 and / or an object associated with object 806 is initiated. In some embodiments, in response to receiving a request to interact with object 806, device 202 replaces the display of the proxy representation of object 806 with the actual representation of object 806. Thus, in some embodiments, battery power is conserved and / or the number of processes performed by device 202 is reduced because the actual representation of object 806 is only displayed after it has been determined that it is possible to interact with object 806. In some embodiments, the LED is not active (or not included as part of device 202) when the proxy representation of object 806 and / or the actual representation of object 806 is displayed.

[0175] Fig.15 is a flow chart illustrating a method for transitioning an object between displays based on directional information according to some embodiments. In some embodiments, the system 100 ( Figure 1A and Figure 1B ) performs the method. In some embodiments, the device 202 ( Figure 2A ) performs the method. In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0176] At block 1502, directional information (e.g., 804a-804d) corresponding to the position (e.g., north, south, east, west, right, left, back, front, other cardinal directions, or any combination thereof) of an object outside the field of view of a primary display (e.g., 204) of a device and a secondary display of the device is received. In some embodiments, the object is a physical object or artifact in a physical environment; in some embodiments, the object is a virtual object in a virtual environment.

[0177] In some embodiments, the primary display, secondary display, and / or tertiary display are different types of displays. In some embodiments, the primary display is a waveguide display. In some embodiments, the secondary display is an organic light emitting diode display. In some embodiments, the tertiary display is a plurality of light emitting diodes.

[0178] At blocks 1504 and 1506 , in response to receiving directional information corresponding to a location of an object outside of the fields of view of the primary and secondary displays and in accordance with determining that a first criterion is satisfied, a first representation of the directional information is displayed via the secondary display.

[0179] In some embodiments, the first criterion includes a criterion that is met when the object is an object type for which the directional information of the object can be displayed in a secondary display. In some embodiments, when the object is at a predetermined distance from or next to a portion (e.g., a primary display, a secondary display, a frame of a device), the directional information of the object is allowed to be displayed via a secondary display. In some embodiments, when directional information is received from an application or an object corresponding to an application, the directional information of the object is allowed to be displayed via a secondary display, and when the object has one or more characteristics or features (e.g., size, shape, distance) that allow it to be displayed via a secondary display, the object is configured by the application (or another process) to be displayed via a secondary display. In some embodiments, when the object is not in a specific position or orientation or the object will cause a representation of the object to be displayed near a portion of the device (e.g., inside the frame of the device, close to a bridge of the device, close to the user's nose), the directional information of the object is allowed to be displayed via a secondary display.

[0180] At boxes 1504 and 1508, in response to receiving directional information corresponding to the position of an object outside the field of view of the primary display and the secondary display and based on determining that the second criterion is met, a second representation (e.g., 812a-812g) of the directional information (e.g., 804b-804d) is displayed via the tertiary display.

[0181] In some embodiments, the second criterion includes a criterion that is met when the object is a type of object for which directional information for the object can be displayed in a tertiary display. In some embodiments, the object is configured by the application (or another process) to be displayed via a tertiary display when the object is a predetermined distance from or next to a portion of the device (e.g., a primary display, a secondary display, a frame) and when directional information is received from an application or an object corresponding to the application, the directional information for the object is allowed to be displayed via the tertiary display, and when the object has one or more characteristics or features (e.g., size, shape, distance) that allow it to be displayed via the tertiary display, when the object is not in a particular position or orientation, or when the object will cause a representation of the object to be displayed near a portion of the device (e.g., inside the frame of the device, close to a bridge of the device, close to the user's nose) or in a portion of the tertiary display.

[0182] In some embodiments, the first representation is a virtual object (e.g., the first representation is a modified representation of the object), and the second representation is not a virtual object within the CGR environment. In some embodiments, the first modified representation of the virtual object can be visually distinguished from the actual representation of the object. For example, the first modified representation may have a visual appearance that is not similar to or does not look like the actual object, while the actual representation of the object may have a visual appearance that is similar to or looks like the actual object. In some embodiments, the device will not be able to use image recognition technology to identify the actual object when processing the modified representation of the object, but will be able to identify the actual object when processing the actual representation of the object. In some embodiments, the first modified representation can be a representation that is not similar to the actual object (e.g., there are no two or more identical features, such as color, shape, texture, size, brightness, coarseness, fade), while the actual representation is a representation that is similar to the actual object.

[0183] In some embodiments, the first representation is a static representation. For example, a static representation is a representation that does not move or animate (e.g., does not change, does not pulsate, does not adopt a predetermined sequence or pattern, etc.) within a certain period of time. In some embodiments, the second representation is a dynamic representation. For example, a dynamic representation is a representation that moves or animate (e.g., pulsates within a certain period of time, is displayed in a predetermined sequence or pattern, changes color) within a certain period of time. In some embodiments, the first representation is a different color from the second representation. In some embodiments, the first representation is a different size from the second representation. In some embodiments, the first representation is not displayed, while the second representation is displayed, or vice versa. In some embodiments, at least one of the first representation and the second representation includes a gradient (e.g., a gradient field showing direction (e.g., solid and shaded LEDs)).

[0184] In some embodiments, the primary display and the secondary display may display different representations of the object. In some embodiments, the first representation of the directional information is the first representation of the object displayed via the secondary display. In some embodiments, when the first representation of the directional information is displayed via the secondary display, the object is detected within the field of view of the primary display of the HMD device. In some embodiments, in response to detecting that the object is within the field of view of the primary display, a second representation of the object is displayed via the primary display. In some embodiments, the first representation of the object displayed via the secondary display is displayed based on a first model of the object. In some embodiments, the second representation of the object displayed via the primary display is displayed based on a second model of the object. In some embodiments, the second model of the object is simpler than the first model of the object (e.g., has more visual features, more visual details).

[0185] In some embodiments, the tertiary display surrounds a first portion of the secondary display. In some embodiments, the tertiary display surrounds a first portion of the secondary display without surrounding a second portion of the secondary display. In some embodiments, the primary display overlaps the secondary display.

[0186] In some embodiments, a representation of an object that is different from the first and second representations of the directional information is displayed via a primary display, while the first representation is displayed via a secondary display. In some embodiments, the representation of the directional information is associated with the location of the object. For example, the representation of the directional information may indicate the location of the device relative to the location of the object.

[0187] In some embodiments, a representation of an object may not be displayed via the primary display until the object is interacted with. In some embodiments, directional information corresponding to the position of the object within the field of view of the primary display of the HMD device and the secondary display of the HMD device is received. In some embodiments, in response to receiving directional information corresponding to the position of the object within the field of view of the primary display of the HMD device and the secondary display of the HMD device, a fourth representation of the object is displayed via the secondary display without displaying a fifth representation of the object via the primary display. In some embodiments, a request to interact with the object (e.g., select the object, start a process involving the object) is received. In some embodiments, in response to receiving a request to interact with the object, the fifth representation of the object is displayed via the primary display, and displaying the fourth representation of the object via the secondary display is stopped.

[0188] In some embodiments, a request to interact with an object is received. In some embodiments, in response to receiving the request to interact with the object, a representation of the object is displayed via a primary display.

[0189] In some embodiments, when displaying the first representation and the second representation, the first representation is displayed via the secondary display in a first position in a corresponding direction relative to the main display, and the second representation is not displayed via the tertiary display in a second position in a corresponding direction relative to the main display.

[0190] Fig.16 is a flow chart illustrating a method for displaying a modified representation of an object according to some embodiments. In some embodiments, the system 100 ( Figure 1A and Figure 1B ) performs the method. In some embodiments, the device 202 ( Figure 2A ) performs the method. In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0191] At block 1602, an object (e.g., a virtual object, a virtual object as a representation of a physical object, a physical object in a physical environment) is detected at a first position (or orientation) in a computer generated reality (CGR) environment. In some embodiments, the object is determined to be important or useful. For example, the object may be determined to be important or useful by receiving user input related to the object, by receiving user interaction related to the object, by receiving data identifying the object as important or useful, etc. In some embodiments, the data identifying the object as important or useful is received from one or more applications in communication with or executed by the device 202.

[0192] At blocks 1604 and 1606 , in response to detecting the object at a first location in the CGR environment and based on determining that the first location is within a first predetermined distance outside of a field of view of a primary display of the device, a first modified representation of the virtual object is displayed.

[0193] The first modified representation of the virtual object may be visually distinguishable from the actual representation of the object. For example, the first modified representation may have a visual appearance that is not similar to or does not look like the actual object, while the actual representation of the object may have a visual appearance that is similar to or looks like the actual object. In some embodiments, the device may not be able to recognize the actual object using image recognition technology when processing the modified representation of the object, but will be able to recognize the actual object when processing the actual representation of the object. In some embodiments, the first modified representation may be a representation that is not similar to the actual object (e.g., does not have two or more identical features, such as color, shape, texture, size, brightness, thickness, fade), while the actual representation is a representation that is similar to the actual object.

[0194] In some embodiments, the first modified representation has visual content (e.g., data representing a displayed representation) that is different from the visual content of the first actual representation. In some embodiments, the modified representation of the object has a structure that is different from the structure of the actual object, and the actual representation of the object has the same structure as the object. For example, if the actual object is a tree, the actual representation of the object may visually look like a tree, while the first modified representation looks like a blob of pixels that is not in the shape of a tree.

[0195] In some embodiments, the first modified representation has a size based on the distance between the first position and the field of view of the main display of the device. In some embodiments, when the position of the object is at a first distance outside the main display, the first modified representation is a first size. In some embodiments, when the position of the object is at a second distance outside the main display, the first modified representation is a second size. In some embodiments, when the first distance is closer to the main display than the second distance, the first modified representation displayed at the first size is larger than the first modified representation displayed at the second size. In some embodiments, when the object is detected at a position closer to the field of view of the main display of the device than the first position, the size of the first modified representation increases in size.

[0196] In some embodiments, the first modified representation has a brightness based on the size of the object (e.g., making larger objects darker). In some embodiments, when the object is a first size, the first modified representation has a first brightness. In some embodiments, when the object is a second size, the first modified representation has a second brightness. In some embodiments, when the first size of the object is smaller than the second size of the object, the first modified representation displayed at the first brightness is brighter than the first modified representation displayed at the second brightness.

[0197] In some embodiments, the first modified representation has a brightness based on the size of the object relative to another object displayed via the device. For example, the brightness of the first modified representation may increase when the size of the object becomes larger than the size of the other object, and vice versa. In some embodiments, when the size of the object is substantially equal to the size of the other object, the brightness of the modified representation of the object may be equal to the brightness of the representation of the other object. Thus, in some embodiments, as the user moves throughout the environment, the brightness of the modified representation of the object may change based on other objects within the environment.

[0198] In some embodiments, the first modified representation has a higher blur than the blur of the first actual representation. In some embodiments, the first modified representation is more faded than the first actual representation.

[0199] In some embodiments, as part of displaying the first modified representation, an animation may be displayed. For example, the animation may be a pulsating animation, an animation that transfers or moves the first modified representation, an animation that transforms the first actual representation of the object into the first modified representation, an animation that fades the first modified representation in or out.

[0200] In some embodiments, in response to detecting an object at a first location in the CGR environment and based on determining that the first location is within the field of view of the primary display and the field of view of the secondary display, a second actual representation of a first portion of the object is displayed via the primary display. Fig. 13B As shown, when the device is within a threshold distance within the field of view of the secondary display, a second actual representation of the first portion of the object can be displayed via the primary display (eg, 806).

[0201] While displaying the second actual representation of the first portion of the object via the primary display, displaying the second modified representation of the second portion of the object via the secondary display. Fig. 13B As shown, a second modified representation (eg, 826c) is displayed via the secondary display.

[0202] In some embodiments, the second portion of the object is a portion of the object that is different from the first portion of the object. In some embodiments, the first portion of the object is within the field of view of the primary display and not in the field of view of the secondary display. In some embodiments, the second portion of the object is within the field of view of the secondary display and not in the field of view of the primary display. In some embodiments, the second actual representation is displayed at a higher resolution than the second modified representation. In some embodiments, the second actual representation is displayed near an edge of the primary display adjacent to the secondary display. In some embodiments, the second modified representation is displayed near an edge of the secondary display adjacent to the primary display.

[0203] In some embodiments, as part of displaying the second actual representation of the object via the primary display, the visual effect is applied to the second actual representation of the object. In some embodiments, the second actual representation of the object after the visual effect is applied has a greater blur than the blur of the second actual representation of the object before the visual effect was applied. In some embodiments, one or more of the second actual representation of the first portion of the object displayed via the primary display and the second modified representation of the second portion of the object displayed via the secondary display uses the above with respect to Figure 7 One or more of the techniques discussed are shown.

[0204] In some embodiments, in response to detecting an object at a first location in the CGR environment and based on determining that the first location is within a second predetermined distance outside the field of view of a primary display of the device, a representation corresponding to the directional information of the located object is displayed via the primary display. Fig. 12A and Fig. 12B As shown, when the distance between the user 250 and the object 806 is a distance 804e from each other (which is different from Fig.11A When the user 250 and the object 806 are separated from each other by a distance 804d), a representation (eg, 828) corresponding to the direction information of the positioning object is displayed via the main display.

[0205] In some embodiments, the representation corresponding to the directional information of the located object is different from the actual representation. The representation corresponding to the directional information of the located object can provide an indication of the location of the object, the proximity of the object to the device, and whether the device is moving away from or close to the object.

[0206] In some embodiments, a modified representation of the object is displayed via the secondary display while a representation corresponding to the location of the object is displayed via the primary display.

[0207] In some embodiments, in response to detecting the object at the first location and based on determining that the first location is within the field of view of the primary display and not within the field of view of the secondary display, the first actual representation (e.g., 806) is displayed via the primary display and the first modified representation ceases to be displayed via the secondary display. In some embodiments, a portion of the first actual representation of the object is displayed. In some embodiments, the entire object is displayed.

[0208] FIG. 17A to FIG. 17B , Fig.18 and Fig.25 Described are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments. Fig.17A In the embodiment, device 202 is depicted as having one or more components, as described above in FIG. 2A to FIG. 2B and Fig. 8A discussed in .

[0209] Fig.17A A device 202 is depicted displaying a plurality of application icons. The plurality of application icons includes a messaging application icon 1702a, a health application icon 1702b, a phone application icon 1702c, and an email application icon 1702d.

[0210] refer to Fig.17A, assuming that device 202 receives a new message via a messaging application corresponding to messaging icon 1702a. In response to receiving the new message, device 202 may determine whether to display information related to the new message via a specific display (e.g., primary display 204, secondary display 206, LED, etc.).

[0211] In some embodiments, in response to receiving a new message, device 202 determines that information related to the new message is allowed (or configured) to be displayed via secondary display 206. Fig. 17B As depicted, as a result of this determination, the device 202 updates the messaging application icon 1702a to include a message status indicator 1702a1. Fig. 17B As shown, the message status indicator 1702a1 is a "1" adjacent to the messaging icon 1702a and indicates that an additional new or unread message has been received. Unread messages may include messages that have not yet received any user interaction, such as voicemail messages that have not yet been listened to, text or email messages that have not yet been read, notifications or prompts from applications that have not yet been opened, notifications or prompts that have not yet been deleted, etc.

[0212] In some embodiments, in response to receiving a new message, device 202 determines that information related to the new message is allowed (or configured) to be displayed via a three-level display (such as an LED). Fig. 17B As depicted, as a result of the determination, device 202 causes the LEDs to emit light. In some implementations, device 202 causes only a portion of the LEDs to emit light.

[0213] One or more other types of prompts, notifications, sensor data, requests or data from applications, incoming phone calls, etc. may be received. In some embodiments, as described above with respect to FIG. 17A to FIG. 17B As described in the new message received in the above, device 202 performs similar actions in response to their receipt. For example, when it is determined that the health data detected by the heart rate sensor (or one or more other sensors that detect personal or health data) is above or below a threshold level and allows information related to the data to be displayed via these LEDs, device 202 can cause these LEDs to illuminate.

[0214] Device 202 may also perform other actions in response to receiving one or more messages, prompts, notifications, sensor data, requests or data from applications, incoming phone calls, etc. For example, Fig.18 As depicted, device 202 displays a color across at least a portion of secondary display 206 based on determining that information related to the new message is allowed to be displayed via secondary display 206. In some embodiments, a portion of primary display 204 is updated as a result of the determination.

[0215] In some embodiments, the device 202 may display an animation via one or more displays. In some embodiments, the animation may include one or more pulsating colors, one or more moving objects, one or more objects changing shape or transforming into one or more other objects, etc. For example, the device 202 may display an animation of pulsating colors on the primary display 204 and / or the secondary display 206 based on determining that the user's health data is above / below a threshold level and allowing information related to the health data to be displayed on the corresponding display.

[0216] FIG. 19A to FIG. 19B and Fig.25 Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0217] Fig.19A A device 202 is depicted displaying a CGR environment that includes representations of a plurality of objects 1910a-1910d. Objects 1910a-1910d are cars in the field of view of a primary display 204 and a secondary display 206. Here, objects 1910a-1910d represent physical cars in the physical environment viewed through the primary display 204 and the secondary display 206 of the device 202. In some embodiments, objects 1910a-1910d represent virtual objects displayed by the primary display 204 and the secondary display 206 of the device 202.

[0218] refer to Fig.19A , assume that device 202 receives data from an application indicating that object 1910a is associated with a user of device 202. For example, Fig.19A In some embodiments, the application can be a car sharing application, a taxi application, a car location application, etc. In some embodiments, the device 202 uses one or more user settings, data stored in memory, one or more machine learning algorithms, etc. to determine that the object 1910a is associated with the user of the device 202 without receiving data from a dedicated application.

[0219] In response to receiving data from an application, the device 202 may determine whether to display information related to the data from the application via a particular display. In some embodiments, in response to receiving data from an application, the device 202 determines to allow information related to the data to be displayed via the secondary display 206. Fig.19B As depicted, based on the determination, the device 202 displays an indicator 1912 around the object 1910a on the secondary display 206 .

[0220] An indicator 1912 is displayed around the object 1910a to visually distinguish the object 1910a from the objects 1910b-1910d. In some embodiments, the device 202 causes the object 1910a to be visually distinguished from the other objects in other ways. For example, the device 202 may stop displaying the objects 1910b-1910d, minimize the objects 1910b-1910d, enlarge the object 1910a, highlight the object 1910a, etc.

[0221] In some embodiments, in response to receiving data from an application, the device 202 determines to allow information related to the data to be displayed via the LED. Fig.19B As depicted, device 202 causes one or more of the LEDs to emit light based on the determination.

[0222] FIG. 20A to FIG. 20D and Fig.25 Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0223] Fig. 20A A device 202 is depicted displaying a CGR environment that includes a time representation 2002. The time representation 2002 is displayed via the primary display 204 and indicates that the process has a remaining time of 60 seconds. In some embodiments, the time representation 2002 represents a process associated with a timer application, a delivery application, a fitness application, a meeting application, a productivity application, etc. In some embodiments, the time representation 2002 is displayed via the primary display 204 because it has been determined that the display of the time representation 2002 via the primary display 204 is allowed.

[0224] Simultaneously with time representation 2002, time representation 2004 is displayed via secondary display 2006. Time representation 2004 includes a colored overlay covering the entire secondary display 206 to indicate the remaining time of 60 seconds. In some embodiments, time representation 2004 indicates the time represented by time representation 2002. In other words, in some embodiments, a representation displayed via one display may correspond to or represent a representation displayed via another display.

[0225] In some implementations, because it has been determined that display of the time representation 2004 via the secondary display 206 is permitted, the time representation 2002 is displayed via the secondary display 206 .

[0226] like Fig. 20AAs depicted, time representation 2004 indicates the percentage of time remaining when compared to the start time of a timer or process. For example, when assuming that a process is started that will end within 60 seconds, representation 2004 indicates that the percentage of time remaining is 100%.

[0227] Simultaneously with time representation 2002 and time representation 2004, device 202 causes LED to emit light, such as Fig. 20A As with the time representation 2004, these LEDs indicate the time represented by the time representation 2002 and thus correspond to the time representation 2002. Here, since the percentage of time remaining when assuming that the process to be completed within 60 seconds is started is 100%, all the LEDs are lit.

[0228] like Fig. 20B As shown, assume that device 202 receives an indication that the process has 30 seconds remaining. In response to receiving the indication, device 202 updates time representation 2002 and time representation 2004. Fig. 20B , the time representation 2004 includes a colored overlay that substantially covers the general area of ​​the secondary display 206. In some embodiments, Fig. 20B The representation 2004 in FIG. 2004 indicates that the remaining time (eg, 30 seconds) is the original time to which the process was set (eg, Fig. 20A In addition, device 202 also causes half of the LEDs (eg, 810a-810l and 812a-812l) to stop emitting light based on the remaining time being at least 50% of the original time.

[0229] like FIG. 20C to FIG. 20D As shown in , the remaining time is 15 seconds and 5 seconds respectively. FIG. 20C to FIG. 20D As shown, the device 202 continues to perform the process based on the remaining time of the process (e.g., Fig. 20C 15 seconds and Fig.20D 5 seconds in total) to update the primary display 204, secondary display 206, and LEDs (e.g., 810a-810l and 812a-812l).

[0230] However, with FIG. 20A to FIG. 20C On the contrary, Fig.20D Here, device 202 causes more than an expected portion (eg, more than 1 / 12) of the LEDs to emit light. Here, device 202 causes all LEDs to emit light to provide an indication that the remaining time is less than a predetermined minimum time.

[0231] In some embodiments, as the remaining time gets closer to zero, the device 202 causes one or more of the LED and / or secondary display 206 to be displayed in an animated manner (e.g., pulsating, brightening). In some embodiments, when a predetermined amount of time is left, the secondary display 206 and / or LED changes to a different color and / or hue. For example, when a predetermined amount of time is left, the secondary display 206 may change from translucent to red. In some embodiments, one or more of the LED and / or secondary display 206 is displayed in an animated manner, while one or more of the other displays are not displayed in an animated manner. In some embodiments, the secondary display 206 may overlap with the primary display 204 so that the animation shown in the overlapping portion of the secondary display 206 is shown in the center of the display of the device 202 (e.g., an area corresponding to the user's pupil).

[0232] FIG. 21A to FIG. 21D and Fig.25 Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0233] Fig.21A A device 202 is depicted displaying a CGR environment, the CGR environment including a device 2102a and a device 2102b. Fig.21A As depicted, device 2102a and device 2102b are in the field of view of primary display 204 and secondary display 206 .

[0234] like Fig.21A As shown, a user interface (UI) element 2110 is displayed on device 2102a. For example, UI element 2110 can be one or more icons, files, application shortcuts, characters, etc.

[0235] refer to Fig. 21B , device 202 detects a drag and drop input 2150 on UI element 2110. In response to detecting the drag and drop input 2150 on UI element 2110, device 202 determines whether information associated with the state of the drag and drop input is displayable via the LED. Fig. 21B As depicted, as a result of this determination, device 202 causes the LEDs to emit light.

[0236] refer to Fig. 21B , assume that device 202 detects movement of drag-and-drop input 2150. In response to detecting movement of drag-and-drop input 2150, device 202 displays UI element 2110 moving from a location on device 2102a to a location on device 2102b.

[0237] Fig. 21C The UI element 2110 at the location on the device 2102b is depicted. Fig. 21C As depicted, since UI element 2110 is still selected, these LEDs continue to glow.

[0238] refer to Fig. 21C , assuming that device 202 detects that UI element 2110 is not selected. In response to detecting that UI element 2110 is not selected, device 202 abandons causing LED to emit light, such as Fig.21D In some embodiments, in response to detecting that UI element 2110 is not selected, device 202 causes one or more of the LEDs to illuminate in a different manner. For example, device 202 may cause one or more of the LEDs to illuminate in a different manner than it did when UI element 2110 was selected. Fig. 21C The light emitted is different colors of light.

[0239] In some embodiments, one or more of these LEDs are caused to light up in different ways to indicate the completion or implementation of the user action. For example, device 202 can cause these LEDs to light up in different ways according to the rhythm of the media file played after the user has dragged the media file from device 2102a to device 2102b.

[0240] In some embodiments, device 202 performs the operations described above with respect to drag and drop input 2150 (e.g., FIG. 21A to FIG. 21D In some embodiments, the device 202 performs the above techniques when a drag, drop, copy, edit, crop, or move operation is detected. In some embodiments, the device 202 performs the above techniques when the device 202 receives data related to the state of an application or process, such as a state associated with the download of a file, a state associated with the playback of media, a state associated with the completion of an exercise routine, a state associated with the completion of a phased goal (e.g., a health goal, a productivity goal), etc.

[0241] FIG. 22A to FIG. 22B and Fig.25 Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0242] Fig.22A A device 202 is depicted displaying a CGR environment that includes a person 2202. The person 2202 is in the field of view of and seen by a primary display 204 and a secondary display 206.

[0243] like Fig.22A As depicted, person 2202 is frowning. Fig.22A, assume that device 202 receives data representing the emotion of person 2202, which is processed by one or more algorithms, such as one or more facial recognition algorithms. In some embodiments, the one or more algorithms may determine that person 2202 is frowning and send data representing the determination.

[0244] In response to receiving data indicating that person 2202 is frowning, device 202 determines that information related to the data is allowed to be displayed via the LED. In some embodiments, based on this determination, device 202 causes the LED to emit light in a first state. For example, Fig.22A The LED is shown in a first state (eg, the LED is filled). In some embodiments, the device 202 only updates Fig.22A A portion of the LED is depicted as solid.

[0245] In some embodiments, device 202 updates secondary display 206 with a colored overlay in certain states to indicate the emotion of person 22A (eg, based on determining that data is permitted to be displayed via secondary display 206).

[0246] Fig. 22B A device 202 is depicted displaying a CGR environment including a person 2204. Fig. 22B Depicted, with Fig.22A In contrast to person 2202, person 2204 is smiling. Fig. 22B , assume that device 202 receives data representing emotions of person 2204, which is processed by one or more algorithms, such as one or more facial recognition algorithms.

[0247] In response to receiving data representing the emotion of person 2204, device 202 determines that information related to the data is allowed to be displayed via the LED. In some embodiments, based on this determination, device 202 causes the LED to emit light in the second state instead of the first state. For example, Fig. 22B The LED is depicted in the second state by showing the LED as being shaded. Here, because the data represents a different emotion of the user, the device 202 causes the LED to illuminate in a different manner in the second state than in the first state.

[0248] In some embodiments, the same FIG. 22A to FIG. 22B The described techniques can be used to display data representing other emotions (e.g., sadness, contentment, etc.) or other types of data. For example, other types of data can include financial market conditions, weather forecasts, ratings (e.g., restaurant ratings, movie ratings, etc.).

[0249] FIG. 23A to FIG. 23B and Fig.25Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0250] Fig.23A A device 202 is depicted displaying a CGR environment including a representation 2302 of a progress indicator displayed via a secondary display 206. In some embodiments, the representation 2302 indicates one or more metrics being tracked. For example, the metric may be the number of steps taken (or required for goal achievement), download speed, number of fitness sessions in a certain time period, and / or any other metric. In some embodiments, the representation 2302 corresponds to data received from one or more health applications (such as a walking application, a training application, a strength training application, etc.).

[0251] In some embodiments, device 202 causes the LED to emit light representing the progress indicator based on determining that information related to the progress indicator is allowed to be displayed via the LED. Fig.23A As depicted, LEDs 810a and 810h-8101 are emitting light, wherein LEDs 810a and 810h-8101 correspond to the length and position of representation 2302. LEDs 810a and 810h-8101 indicate the level of the progress indicator.

[0252] In some embodiments, such as Fig.23A As depicted, LEDs 812a, 812h, 812i, 812l are illuminated. LEDs 812a, 812h, 812i, 812l are located on the right side of device 202 along the length of representation 2302. LEDs 812a, 812h, 812i, 812l also indicate the level of the progress indicator. Notably, device 202 LEDs 812j and 812k are not illuminated because they are within a certain distance from the nose of the user wearing device 202.

[0253] refer to Fig. 23B , the device 202 updates the representation 2302 via the secondary display 206 and the LEDs to indicate that the progress of the tracked metric has been completed. It is noteworthy that LEDs 812j and 812k continue to be unlit, because they are within a certain distance from the nose of the user wearing the device 202.

[0254] FIG. 24A to FIG. 24D Depicted are exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments. FIG. 24A to FIG. 24DA scenario is depicted in which device 202 causes one or more of LEDs 810a-210l and 812a-812l to sequentially illuminate in a clockwise pattern around the eyes of a user wearing device 202. In some embodiments, this pattern may indicate direction to the user and may replace or be used to supplement the above description with respect to FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11B , FIG. 12A to FIG. 12B , FIG. 13A to FIG. 13B , FIG. 14A to FIG. 14B , Fig.15 and Fig.16 An example of description.

[0255] For example, the LEDs may emit a pattern in which the LEDs are sequentially turned on and off, similar to dots moving around a circle. The circle may be animated in a clockwise or counterclockwise direction around one or more eyes of a user wearing device 202. In some embodiments, when the pattern is drawn around the user's eyes, device 202 transitions between the LEDs on the right and left sides of device 202, such as FIG. 24B to FIG. 24C As shown, the device 202 skips a portion of the LEDs (eg, 810g and 810h) in the sequence.

[0256] Fig.25 1 is a flow chart illustrating a method for managing one or more displays based on data associated with one or more processes according to some embodiments. Figure 1A and Figure 1B ) performs the method. In some embodiments, the device 202 ( Figure 2A ) performs the method. In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by a combination of one or more of system 100, device 202, and a third device or system.

[0257] At block 2502, information corresponding to a change in the state of a process is received. In some embodiments, the information may be one or more notifications, prompts, and / or one or more outputs from one or more functions, processes, applications, sensors, etc. In some embodiments, the sensors may include one or more health sensors, heart rate sensors, pedometers, thermal sensors, etc.

[0258] In some embodiments, information is calculated based on one or more environmental factors or data from one or more applications. FIG. 20A to FIG. 20DIn the process, the device 202 can calculate the start or end of the process as described above. In some embodiments, the environmental factors may include data corresponding to weather, characteristics of one or more users or devices within the environment, etc.

[0259] refer to Fig.25 At blocks 2504 and 2506, in response to receiving information corresponding to a changed state of the process and based on determining that the first criterion is satisfied, a first representation corresponding to the state of the process is displayed via the secondary display. For example, the representation may include one or more representations, such as Fig. 17B The status indicator 1702a1 in Fig.18 The color overlay displayed across the secondary display 206, Fig.19B Indicator 1912, FIG. 20A to FIG. 20D In 2004, FIG. 23A to FIG. 23B The progress indicator in 2302, etc.

[0260] In some embodiments, the first criterion includes a criterion that is satisfied when determining whether to allow the display of the object's directional information via the secondary display, as described above with respect to FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11B , FIG. 12A to FIG. 12B , FIG. 13A to FIG. 13B , FIG. 14A to FIG. 14B , Fig.15 and Fig.16 described.

[0261] refer to Fig.25 At blocks 2504 and 2506, in response to receiving information corresponding to a change in the state of the process and based on determining that the second criterion is satisfied, a second representation corresponding to the state of the process is displayed. For example, FIG. 17A to FIG. 17B , Fig.18 , FIG. 19A to FIG. 19B , FIG. 20A to FIG. 20D , FIG. 21A to FIG. 21D , FIG. 22A to FIG. 22B , FIG. 23A to FIG. 23B and FIG. 24A to FIG. 24D 810a-810l and 812a-812l.

[0262] In some embodiments, the second criterion includes a criterion that is met when determining whether to allow the object to be displayed via the tertiary display, as described above with respect to FIG. 8A to FIG. 8B , FIG. 9A to FIG. 9B , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11B , FIG. 12A to FIG. 12B , FIG. 13A to FIG. 13B , FIG. 14A to FIG. 14B , Fig.15 and Fig.16 described.

[0263] In some embodiments, the second representation is different from the first representation. In some embodiments, the first representation is a virtual object and the second representation is not a virtual object.

[0264] In some embodiments, one or more of the first representation and the second representation are associated with a third representation presented via the primary display (e.g., corresponding to one or more processes of the same application). FIG. 20A to FIG. 20D In FIG. 2 , the representations 2004 displayed via the secondary display and the LEDs (eg, a tertiary display) are both associated with the time representation 2002 displayed via the primary display 204 .

[0265] In some embodiments, when presenting the first representation, the second virtual object is displayed simultaneously with the first representation. In some embodiments, the first representation highlights at least a portion of the second virtual or physical object displayed via the secondary display. For example, Fig.19B , an indicator 1912 highlights (or brackets) an object 1910 a displayed via the secondary display 206 .

[0266] In some embodiments, one or more of the first representation and the second representation include a progress indicator. In some embodiments, the progress indicator updates to show changes in a metric associated with the device, as described above with respect to FIG. 23A to FIG. 23B As described in representation 2302.

[0267] In some embodiments, the first representation includes an application icon. In some embodiments, the process is associated with an application that includes a status indicator that is updated based on the process. FIG. 17A to FIG. 17B , status indicator 1702a1 is displayed in response to receiving data associated with the messaging application.

[0268] In some embodiments, one or more of the first representation and the second representation are displayed based on one or more sensors. In some embodiments, the one or more sensors include a heart rate sensor. In some embodiments, the received information corresponds to data detected via the heart rate sensor. In some embodiments, as a first representation (e.g., Fig.18 206) and a portion of one or more of the second representations, displaying an animation based on data detected via a heart rate sensor.

[0269] In some embodiments, the received information corresponds to a prompt corresponding to the application. In some embodiments, one or more of the first representation and the second representation are presented based on the prompt corresponding to the application, such as described above with respect to Fig.18described.

[0270] In some embodiments, the received information corresponds to a reminder corresponding to a calendar event. In some embodiments, one or more of the first representation and the second representation are presented based on the reminder corresponding to the calendar event, as described above, for example, with respect to Fig.18 described.

[0271] In some embodiments, the received information corresponds to data obtained via a facial recognition algorithm. In some embodiments, one or more of the first representation and the second representation are presented based on the data obtained via the facial recognition algorithm, such as described above, for example, with respect to Fig.22A and Fig. 22B The LEDs are described in

[0272] In some embodiments, the received information corresponds to the state of the application. In some embodiments, one or more of the first representation and the second representation are presented based on the state of the application. For example, in FIG. 21A to FIG. 21D In the example embodiment, the LED is updated based on whether a drag and drop input 2150 is detected.

[0273] In some embodiments, the received information corresponds to a detected posture of the user. In some embodiments, one or more of the first representation and the second representation are presented based on the detected posture of the user wearing the device. In some embodiments, the first representation is displayed based on the detected eye position of the user. For example, the representation can be a line representing the current posture or eye position of the user. In some embodiments, in response to detecting a change in the posture or eye position of the user, the line is updated to indicate a new posture or eye position (or alternatively, a change in posture or eye position). In some embodiments, the line is displayed only via a secondary display. In some embodiments, these LEDs can emit light in various combinations indicating a new posture or eye position (or alternatively, a change in posture or eye position).

[0274] As described above, one aspect of the present technology is the collection and use of data available from various sources to provide dedicated resource management of a device (e.g., providing a device with an additive display) using an additive display, thereby extending battery life for the user and providing dedicated content to the user of the device. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0275] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to extend the battery life of the user's device. Thus, for example, the use of such personal information data helps the system to properly manage resources to extend the battery life of the device. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals using technology to pursue health goals.

[0276] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0277] Regardless of the foregoing, the present disclosure also contemplates an example in which a user selectively blocks the use or access of personal information data. That is, the present disclosure contemplates providing hardware elements and / or software elements to prevent or block access to such personal information data. For example, in terms of managing resources for low-power devices, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for a service to participate in the collection of personal information data. In another example, a user may choose not to provide eye tracking data, such as pupil position, pupil dilation, and / or blink rate for dedicated resource management. In yet another example, a user may choose to limit the length of time that eye tracking data is maintained or completely prohibit the development of a baseline eye tracking profile. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0278] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0279] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, the present disclosure also contemplates that various examples may also be implemented without access to such personal information data. That is, various examples of the present technology will not fail to function properly due to the lack of all or part of such personal information data. For example, preferences may be inferred based on non-personal information data or an absolute minimum of personal information (such as content requested by a device associated with a user, other non-personal information available to a system controlling the device, or publicly available information), thereby managing the resources of a low-power device and selecting content (e.g., status updates and / or objects) and delivering them to the user.

Claims

1. A method comprising: At a system having one or more processors, a memory, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, and a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution: detecting an object at a first location in a computer generated reality (CGR) environment; and In response to detecting the object at the first location in the CGR environment: displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object based on determining that the first position is within a first predetermined distance outside the field of view of the primary display of the HMD device; as well as Based on determining that the first position is within the field of view of the primary display and not within the field of view of the secondary display: displaying the first actual representation via the primary display; as well as Displaying the first modified representation via the secondary display is foregone. 2 . The method of claim 1 , wherein the first modified representation has a visual content that is different from the visual content of the first actual representation. 3 . The method of claim 1 , wherein the first modified representation has a size based on a distance between the first position and the field of view of the primary display of the HMD device. 4 . 4 . The method of claim 1 , wherein the first modified representation has a brightness based on the size of the object.

5. The method according to any one of claims 1 to 2, wherein the first modified representation has a higher blur than the blur of the first actual representation.

6. The method of any one of claims 1 to 2, wherein displaying the first modified representation comprises displaying an animation.

7. The method according to any one of claims 1 to 2, further comprising: In response to detecting the object at the first location in the CGR environment: Based on determining that the first position is within the field of view of the primary display and the field of view of the secondary display: displaying, via the primary display, a second actual representation of the first portion of the object; as well as While the second actual representation of the first portion of the object is displayed via the primary display, a second modified representation of a second portion of the object is displayed via the secondary display.

8. The method of claim 7, wherein the second actual representation is displayed at a higher resolution than the second modified representation.

9. The method of claim 7, wherein the second actual representation is displayed near an edge of the primary display.

10. The method according to claim 7, wherein: displaying the second actual representation via the primary display includes applying a visual effect to the second actual representation; and The second actual representation of the object after applying the visual effect has a greater blur than the blur of the second actual representation of the object before applying the visual effect.

11. The method according to any one of claims 1 to 2, further comprising: In response to detecting the object at the first location in the CGR environment: Based on determining that the first position is within a second predetermined distance outside the field of view of the primary display of the HMD device, wherein the second predetermined distance is different from the first predetermined distance: A representation corresponding to the direction information for locating the object is displayed via the main display, wherein the representation corresponding to the direction information for locating the object is different from the first actual representation.

12. A computer-readable storage medium storing one or more programs, the one or more programs being configured to be executed by one or more processors of a head-mounted display (HMD) device, the HMD device communicating with a primary display and a secondary display, the primary display extending across a field of view and having a first resolution, the secondary display being physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 11.

13. A head mounted display (HMD) device, the HMD device being configured to communicate with a primary display and a secondary display, the primary display extending across a field of view and having a first resolution, the secondary display being physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the HMD device comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 11.

14. A head mounted display (HMD) device, the HMD device being configured to communicate with a primary display and a secondary display, the primary display extending across a field of view and having a first resolution, the secondary display being physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the HMD device comprising: Memory; and Device for carrying out the method according to any one of claims 1 to 11.

15. A computer program product, comprising one or more programs, the one or more programs being configured to be executed by one or more processors of a computer system, the computer system communicating with a primary display and a secondary display, the primary display extending across a field of view and having a first resolution, the secondary display being physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, the one or more programs including instructions for performing the method of any one of claims 1 to 11.

16. A method comprising: At a system having one or more processors, a memory, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display: receiving directional information corresponding to a position of an object outside of a field of view of the primary display of the HMD device and the secondary display of the HMD device; as well as In response to receiving the direction information: displaying, via the secondary display, a first representation of directional information based on determining that display of the directional information of the object via the secondary display is permitted; as well as displaying a second representation of directional information via the three-level display in accordance with determining that displaying the directional information of the object via the three-level display is permitted; wherein the first representation is a virtual object within the CGR environment, and Wherein the second representation is not a virtual object within the CGR environment.

17. A method comprising: At a system having one or more processors, a memory, and a head mounted display (HMD) device, the HMD device including a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display: receiving information corresponding to a change in the state of the process; as well as In response to receiving information corresponding to the change in the state of the process: based on determining that a first criterion is satisfied, displaying, via the secondary display of the HMD device, a first representation corresponding to the state of the process; as well as Based on determining that a second criterion is satisfied, a second representation corresponding to the state of the process is displayed via the tertiary display of the HMD device, wherein the second representation is different from the first representation.

Images