Management of device having additional display

Abstract

To provide one or more techniques for managing a virtual object between one or more displays.SOLUTION: According to one embodiment, an exemplary technique for displaying a virtual object is described. According to one embodiment, an exemplary technique for causing an object to transition between displays on the basis of direction information is described. According to one embodiment, an exemplary technique for displaying a corrected expression of the object is described. According to one embodiment, an exemplary technique for managing one or more displays on the basis of data related to one or more processes is described.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present disclosure generally relates to computer-generated reality environments, and more specifically to methods and techniques for managing a device having an additional display for use in a computer-generated reality environment.

[0002] (Cross-Reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 067,461, entitled "Management of Devices Having an Additional Display," filed on Aug. 19, 2020, and U.S. Provisional Patent Application No. 62 / 907,207, entitled "TECHNIQUES FOR MANAGING DEVICES HAVING ADDITIVE DISPLAYS FOR USE IN A COMPUTER-GENERATED REALITY ENVIRONMENT," filed on Sep. 27, 2019. The entire contents of the foregoing applications are incorporated herein by reference.

Background Art

[0003] Users often interact with computer-generated reality environments using wearable devices such as head-mounted display (HMD) devices. The use of wearable devices is often limited by the battery life of the wearable devices. Therefore, one or more techniques are needed to manage power usage to conserve the battery life of wearable devices.

Summary of the Invention

[0004] According to some embodiments, the method is executed in a system having one or more processors, a memory, one or more image sensors, and a head-mounted display (HMD) device. The HMD device includes a primary display that extends across the entire field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution. The method includes displaying a first portion of a virtual object via the secondary display and displaying a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0005] According to some embodiments, a head-mounted display (HMD) device includes one or more image sensors, a primary display that extends across the entire field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions to display a first portion of a virtual object via the secondary display and display a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0006] According to some embodiments, a non - transitory computer - readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, and the electronic device has one or more sensors, a primary display that extends across a field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution. The one or more programs include instructions to display a first portion of a virtual object via the secondary display and to display a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0007] According to some embodiments, a non - transitory computer - readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, and the electronic device has one or more sensors, a primary display that extends across a field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution. The one or more programs include instructions to display a first portion of a virtual object via the secondary display and to display a second portion of the virtual object via the primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0008] According to some embodiments, the system includes one or more processors. The system also includes means for displaying a first portion of a virtual object via a secondary display and means for displaying a second portion of the virtual object via a primary display, where displaying the second portion of the virtual object via the primary display includes applying a visual effect to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from an edge of the primary display.

[0009] According to some embodiments, the method is executed in a system having one or more processors, a memory, one or more image sensors, and a head-mounted display (HMD) device, the HMD device including a primary display that extends across the entire field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The method includes receiving direction information corresponding to the position of an object outside the fields of view of the primary display and the secondary display of the HMD device. The method also includes displaying, via the secondary display, a first representation of the direction information according to a determination that a first criterion including criteria that are satisfied when permission to display the direction information of the object via the secondary display is granted is satisfied, and displaying, via the tertiary display, a second representation of the direction information according to a determination that a second criterion including criteria that are satisfied when permission to display the direction information of the object via the tertiary display is granted is satisfied, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0010] According to some embodiments, a head-mounted display (HMD) device includes one or more image sensors, a primary display that extends across the entire field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors and a memory that stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of the primary display and the secondary display of the HMD device. The one or more programs also include a first criterion including criteria that are satisfied when it is permitted to display the direction information of the object via the secondary display according to a determination that the first criterion including criteria that are satisfied is satisfied, and a second criterion including criteria that are satisfied when it is permitted to display the direction information of the object via the tertiary display according to a determination that the second criterion including criteria that are satisfied is satisfied, and instructions for displaying a first representation of the direction information via the secondary display and a second representation of the direction information via the tertiary display, wherein the first representation is a virtual object within a CGR environment and the second representation is not a virtual object within a CGR environment.

[0011] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display that extends across a field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of the primary display and the secondary display of the HMD device. The one or more programs also include, in accordance with a determination that a first criterion including criteria to be satisfied when permission is given to display the direction information of the object via the secondary display is satisfied upon receiving the direction information corresponding to the position of the object outside the field of view of the primary display and the secondary display, displaying a first representation of the direction information via the secondary display, and in accordance with a determination that a second criterion including criteria to be satisfied when permission is given to display the direction information of the object via the tertiary display is satisfied, instructions for displaying a second representation of the direction information via the tertiary display, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0012] According to one embodiment, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display that extends across a field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving direction information corresponding to the position of an object outside the field of view of the primary display of the HMD device and the secondary display of the HMD device. The one or more programs also include a first criterion including criteria that are satisfied when it is permitted to display the direction information of the object via the secondary display according to a determination that the first criterion including the criteria that are satisfied is satisfied, and display a first representation of the direction information via the secondary display. According to a determination that a second criterion including criteria that are satisfied when it is permitted to display the direction information of the object via the tertiary display is satisfied, instructions for displaying a second representation of the direction information via the tertiary display are included, where the first representation is a virtual object within the CGR environment and the second representation is not a virtual object within the CGR environment.

[0013] According to some embodiments, the system includes one or more processors. The system also includes means for receiving direction information corresponding to the position of an object outside the field of view of the primary display of the head-mounted display (HMD) device and the secondary display of the HMD device. The system, in accordance with a determination that a first criterion including criteria that are satisfied when permission is given to display the direction information of the object via the secondary display is satisfied in response to receiving the direction information corresponding to the position of the object outside the fields of view of the primary display and the secondary display, displays a first representation of the direction information via the secondary display, and further includes means for displaying a second representation of the direction information via a tertiary display in accordance with a determination that a second criterion including criteria that are satisfied when permission is given to display the direction information of the object via the tertiary display is satisfied, wherein the first representation is a virtual object within a CGR environment and the second representation is not a virtual object within a CGR environment.

[0014] According to some embodiments, the method is executed in a system having one or more processors, a memory, one or more image sensors, and a head-mounted display (HMD) device, the HMD device including a primary display that extends across the entire field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution that is lower than the first resolution. The method includes detecting an object within a computer-generated reality (CGR) environment at a first position, and in response to detecting the object within the CGR environment at the first position, displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object, in accordance with a determination that the first position is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0015] According to some embodiments, a head-mounted display (HMD) device includes one or more image sensors, a primary display that extends across the entire field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution that is lower than the first resolution. The HMD device also includes one or more processors and a memory that stores one or more programs configured to be executed by the one or more processors. The one or more programs detect an object in a computer-generated reality (CGR) environment at a first location and, in response to detecting the object in the CGR environment at the first location, display, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0016] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display that extends across the entire field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution that is lower than the first resolution. The one or more programs detect an object in a computer-generated reality (CGR) environment at a first location and, in response to detecting the object in the CGR environment at the first location, display, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0017] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display that extends across a field of view and has a first resolution, and a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution. The one or more programs include instructions for detecting an object in a computer-generated reality (CGR) environment at a first location and, in response to detecting the object in the CGR environment at the first location, displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device.

[0018] According to some embodiments, a system includes one or more processors. The system also includes means for detecting an object in a computer-generated reality (CGR) environment at a first location. The system further includes means for displaying, via the secondary display of the HMD device, a first modified representation of the object that is visually distinguishable from a first actual representation of the object according to a determination that the first location is within a first predetermined distance outside the field of view of the primary display of the HMD device in response to detecting the object in the CGR environment at the first location.

[0019] According to some embodiments, the method is executed in a system having one or more processors, a memory, one or more image sensors, and a head-mounted display (HMD) device. The HMD device includes a primary display that extends across the entire field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The method includes receiving direction information corresponding to the position of an object outside the field of view of the primary display and the secondary display of the HMD device. In response to receiving information corresponding to the changed status of the process, the method also includes displaying, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that a first criterion is met, and displaying, via the tertiary display of the HMD device, a second representation corresponding to the status of the process according to a determination that a second criterion is met, where the second representation is different from the first representation.

[0020] According to some embodiments, a head-mounted display (HMD) device includes one or more image sensors, a primary display that extends across the entire field of view and has a first resolution, a secondary display that is physically and electronically coupled to the primary display and has a second resolution lower than the first resolution, and a tertiary display. The HMD device also includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for receiving information corresponding to a change in the status of a process. In response to receiving information corresponding to the changed status of the process, the one or more programs also include displaying, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that a first criterion is met, and displaying, via the tertiary display of the HMD device, a second representation corresponding to the status of the process according to a determination that a second criterion is met, where the second representation is different from the first representation.

[0021] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in the status of a process. In response to receiving information corresponding to a changed status of the process, the one or more programs also display, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that a first criterion is met, and display, via the tertiary display of the HMD device, a second representation corresponding to the status of the process according to a determination that a second criterion is met, where the second representation is different from the first representation.

[0022] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device, the electronic device having one or more sensors, a primary display extending across a field of view and having a first resolution, a secondary display physically and electronically coupled to the primary display and having a second resolution lower than the first resolution, and a tertiary display. The one or more programs include instructions for receiving information corresponding to a change in the status of a process. In response to receiving information corresponding to a changed status of the process, the one or more programs also display, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that a first criterion is met, and display, via the tertiary display of the HMD device, a second representation corresponding to the status of the process according to a determination that a second criterion is met, where the second representation is different from the first representation.

[0023] According to some embodiments, the system includes one or more processors. The system also includes means for receiving direction information corresponding to the position of an object outside the field of view of the primary display of the head-mounted display (HMD) device and the secondary display of the HMD device. The system, in response to receiving information corresponding to the change status of a process, displays, via the secondary display of the HMD device, a first representation corresponding to the status of the process according to a determination that a first criterion is met, and displays, via the tertiary display of the HMD device, a second representation corresponding to the status of the process according to a determination that a second criterion is met, where the second representation is different from the first representation.

[0024] The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. The executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. In the following description, reference is made to the accompanying drawings, which form a part hereof and illustrate several examples of the present disclosure. It is understood that other examples may be utilized and structural and operational changes may be made without departing from the scope of the present disclosure. When the same reference numerals are used in different figures, similar or identical items are indicated.

Brief Description of the Drawings

[0025]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8A

Figure 8B

Figure 9A

Figure 9B

Figure 10A

Figure 10B

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17A

Figure 17B

Figure 18

Figure 19A

Figure 19B

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 22A

Figure 22B

Figure 23A

Figure 23B

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 25

[0026] Various examples of electronic systems and techniques for using such systems in connection with various computer-generated reality techniques are described.

[0027] The physical environment (or real environment) refers to the physical world that people can perceive and / or interact with without the aid of an electronic system. Physical environments such as a physical park include physical objects (or physical objects or real objects) such as physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through senses such as vision, touch, hearing, taste, and smell.

[0028] In contrast, a computer-generated reality (CGR) environment refers to an environment that is wholly or partially simulated and with which people can perceive and / or interact through an electronic system. In CGR, a subset of a person's physical movements or their representations are tracked, and in response, one or more characteristics of one or more virtual objects simulated within the CGR environment are adjusted to conform to at least one physical law. For example, a CGR system can detect the rotation of a person's head and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. Depending on the situation (e.g., for accessibility reasons), the adjustment of the characteristics of virtual objects within the CGR environment may be made in response to a representation of physical movement (e.g., a voice command).

[0029] People can perceive and / or interact with CGR objects using any of the senses including vision, hearing, touch, taste, and smell. For example, a person can perceive and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source within a 3D space. In another example, the audio object can enable audio transparency that selectively incorporates ambient sound from the physical environment, whether or not there is computer-generated audio. In some CGR environments, people can perceive and / or interact with only audio objects.

[0030] Examples of CGR include virtual reality and mixed reality.

[0031] A virtual reality (VR) environment (or virtual environment) refers to a simulated environment designed to be completely based on computer-generated sensory inputs for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with the virtual objects in the VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0032] In contrast to a VR environment designed to be completely based on computer-generated sensory inputs, a mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory inputs or representations thereof from the physical environment in addition to including computer-generated sensory inputs (e.g., virtual objects). In the virtual continuum, an MR environment is anywhere between a completely physical environment at one end and a VR environment at the other end, but does not include these.

[0033] In some MR environments, the computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Also, some electronic systems for presenting an MR environment can track the position and / or orientation with respect to the physical environment to enable virtual objects to interact with actual objects (i.e., physical articles or representations thereof from the physical environment). For example, the system can account for movement so that a virtual tree appears stationary relative to the physical ground.

[0034] Examples of MR include augmented reality and augmented virtuality.

[0035] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person can use the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture an image or video of the physical environment, which is a representation of the physical environment. The system synthesizes the image or video with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the image or video of the physical environment and to perceive the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment shown on an opaque display is referred to as a "pass-through video," meaning that the system uses one or more image sensors to capture an image of the physical environment and uses these images to present an AR environment on the opaque display. Further alternatively, the system may have a projection system that projects virtual objects onto the physical environment, for example, as holograms or on a physical surface, whereby a person can use the system to perceive the virtual objects superimposed on the physical environment.

[0036] The AR environment also refers to a simulated environment in which the representation of the physical environment is transformed by computer-generated sensory information. For example, when providing a pass-through video, the system can transform one or more sensor images to apply a selected viewpoint (e.g., perspective) different from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment can be transformed by graphically modifying (e.g., enlarging) a part of it, such that the modified part may be a representative version rather than a realistic version of the originally captured image. As a further example, the representation of the physical environment can be transformed by graphically removing or obscuring a part of it.

[0037] An Augmented Virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but the people with faces are realistically reproduced from images of physical people. As another example, a virtual object may adopt the shape or color of a physical article imaged by one or more imaging sensors. As a further example, a virtual object can adopt a shadow that coincides with the position of the sun in the physical environment.

[0038] There are various types of electronic systems that enable a person to perceive and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to receive an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing an image or video of the physical environment and / or one or more microphones for capturing the audio of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed towards a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical coupler, an optical reflector, or any combination thereof. In one example, the transparent or translucent display may be configured to selectively become opaque. A projection-based system can employ retinal projection technology for projecting a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0039] Figures 1A and 1B illustrate an exemplary system 100 for use with various computer-generated reality technologies.

[0040] In some examples, as shown in FIG. 1A, system 100 includes device 100a. Device 100a includes various components such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch sensing surface 122. These components communicate, optionally, via communication bus 150 of device 100a.

[0041] In some examples, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, mobile device, or laptop), and other elements of system 100 are implemented in a head-mounted display (HMD) device designed to be worn by a user, and the HMD device communicates with the base station device. In some examples, device 100a is implemented in the base station device or the HMD device.

[0042] As shown in FIG. 1B, in some examples, system 100 includes two (or more) devices that communicate via a wired connection, wireless connection, etc. The first device 100b (e.g., a base station device) includes processor 102, RF circuit 104, and memory 106. These components communicate, optionally, via communication bus 150 of device 100b. The second device 100c (e.g., a head-mounted device) includes various components such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch sensing surface 122. These components communicate, optionally, via communication bus 150 of device 100c.

[0043] In some examples, system 100 is a mobile device. In some examples, system 100 is an HMD device. In some examples, system 100 is a wearable HUD device.

[0044] System 100 includes a processor 102 and a memory 106. The processor 102 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, the memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) configured to store computer-readable instructions to be executed by the processor 102 to perform the techniques described below.

[0045] System 100 includes an RF circuit 104. The RF circuit 104 optionally includes circuitry for communicating with a network such as a wireless network such as an electronic device, the Internet, an intranet, and / or a cellular network and a wireless local area network (LAN). The RF circuit 104 optionally includes circuitry for communicating using short-range communication and / or short-distance communication such as Bluetooth®.

[0046] System 100 includes a display 120. In some examples, the display 120 includes a first display (e.g., a left-eye display panel) and a second display (e.g., a right-eye display panel), and each display displays an image to the corresponding eye of the user. The corresponding images are simultaneously displayed on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, resulting in a parallax effect that provides the user with an illusion of depth of the objects on the display. In some examples, the first display can include a plurality of other displays (e.g., sub-displays) such as a primary display and a secondary display. In some embodiments, the primary display has a different resolution than the secondary display when the system 100 is operating. In some examples, the display 120 includes a single display (e.g., the first display or the second display). In some examples, the single display can include a plurality of other displays (e.g., sub-displays) such as a primary display and a secondary display. The corresponding images are simultaneously displayed on a first region and a second region of the single display for each eye of the user. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, resulting in a parallax effect that provides the user with an illusion of depth of the objects on the single display.

[0047] In some examples, the system 100 includes a touch-sensing surface 122 for receiving user input such as tap inputs and swipe inputs. In some examples, the display 120 and the touch-sensing surface 122 form a touch-sensing display.

[0048] System 100 includes an image sensor 108. The image sensor 108 optionally includes one or more visible light image sensors such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide-semiconductor (CMOS) sensor operable to acquire an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors such as a passive IR sensor or an active IR sensor for detecting infrared light from a real environment. For example, an active IR sensor includes an IR emitter such as an IR dot emitter that emits infrared light into a real environment. The image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in a real environment. The image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of a physical object from System 100. In some examples, System 100 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around System 100. In some examples, the image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in a real environment from two different viewpoints. In some examples, System 100 uses the image sensor 108 to receive user input such as hand gestures. In some examples, System 100 uses the image sensor 108 to detect the position and orientation of System 100 and / or display 120 in a real environment. For example, System 100 uses the image sensor 108 to track the position and orientation of display 120 relative to one or more fixed objects in a real environment.

[0049] In some examples, System 100 includes a microphone 112. System 100 uses the microphone 112 to detect sound from a user and / or the user's real environment. In some examples, the microphone 112 optionally includes an array of microphones (including multiple microphones) that operate in tandem, such as to identify ambient noise or to locate the position of a sound source within the space of a real environment.

[0050] System 100 includes an orientation sensor 110 for detecting the orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in the position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in the real environment. The orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0051] FIG. 2A shows device 202. In some embodiments, device 202 can include one or more components of system 100, such as a number of processors (e.g., processor 102), memory (e.g., memory 106), camera sensor (e.g., image sensor 108), or motion sensor (e.g., orientation sensor 110). For example, device 202 can be an embodiment of device 100a of system 100 shown in FIG. 1A. Alternatively, in some embodiments, device 202 can be an embodiment of the second device 100c shown in FIG. 1B.

[0052] As shown in FIG. 2A, device 202 is a wearable device that includes a first display (e.g., the left-eye display panel of device 202) and a second display (e.g., the right-eye display panel of device 202). The first display and the second display each include a primary display 204 surrounded by a secondary display 206. As shown in FIG. 2A, secondary display 206 surrounds at least a portion of primary display 204. In some embodiments, secondary display 206 extends across the entire first or second display. In some embodiments, primary display 204 overlaps only a portion (e.g., the center) of secondary display 206.

[0053] In some embodiments, the primary display 204 and the secondary display 206 are two different types of displays. In some embodiments, the primary display 204 is a waveguide display. In some embodiments, the secondary display 206 is an organic light emitting diode display.

[0054] In FIG. 2A, the primary display 204 is not only physically coupled (e.g., connected, attached) to the secondary display 206, but also electronically coupled (e.g., connected, attached) to the secondary display 206. In some embodiments, one or more regions or additional displays can be disposed between the primary display 204 and the secondary display 206.

[0055] In FIG. 2A, the primary display 204 and the secondary display 206 are shown as oval displays. However, it should be appreciated that the primary display 204 and the secondary display 206 can have various shapes such as a rectangular shape (e.g., a rectangular prism). Also, in some embodiments, the shape of the primary display 204 may be different from the shape of the secondary display 206. In some embodiments, when the primary display 204 and / or the secondary display 206 is in a rectangular shape, the device 202 can apply visual effects (e.g., blur, fade the corners of the primary display and / or the secondary display), such that the primary display 204 and / or the secondary display 206 appears round.

[0056] As shown in FIG. 2A, the edge 208 is defined between the primary display 204 and the secondary display 206. In some embodiments, the edge 208 is transparent and / or translucent. In some embodiments, the edge 208 is an edge of the primary display 204 or an edge of the secondary display 206. In some embodiments, the edge 208 is a physical boundary between the primary display 204 and the secondary display 206.

[0057] In some embodiments, the primary display 204 can operate at a maximum resolution higher than the maximum potential resolution of the secondary display 206. Since displays that can operate at a higher resolution typically require more energy, the device 202 can use more energy when displaying virtual objects (or CGR objects) via the primary display 204 than when displaying the same objects via the secondary display 206. Therefore, the device 202 can conserve battery life based on a determination to display one or more virtual objects on the secondary display 206 instead of on the primary display 204.

[0058] As shown in FIG. 2A, the device 202 includes a frame 210 disposed around the secondary display 206. The frame 210 includes a tertiary display such as a light emitting diode (LED). Alternatively, in some embodiments, the frame 210 does not include a tertiary display.

[0059] As described below, in some embodiments, the device 202 causes one or more LEDs to emit light. In some embodiments, the LEDs do not surround the interior of the secondary 206 that is near the user's nose.

[0060] In FIG. 2B, the device 202 is shown being worn by a user 250 such that the user's 250 nose is positioned below the bridge 214 of the device 202. As described below, the device 202 enables the user 250 to view a CGR environment.

[0061] Figure 2B shows a user 250 standing within a physical environment, with the user 250's eyes positioned such that the user 250's pupils 216 are aligned with a portion of the primary display 204. Thereby, in Figure 2B, the user 250's central field of view or line of sight is aligned with a portion of the field of view of the primary display 204, which indicates that an object displayed on the primary display 204 is displayed within the user's central field of view.

[0062] Furthermore, Figure 2B shows the peripheral field of view of the user 250 aligned with a portion of the field of view of the secondary display 206, which indicates that an object displayed on the secondary display 206 is displayed within the user's peripheral field of view. As will be described below, in some embodiments, when one or more LEDs are emitting light, the light is displayed outside or at the edge of the user 250's peripheral field of view. In some embodiments, one or more sensors of the device 202 can track the movement of the user's eyes, including the pupils 216. In some embodiments, the device 202 uses movement tracking to adjust or modify one or more of the techniques described below.

[0063] Figure 3 shows an exemplary virtual object according to some embodiments. In particular, Figure 3 shows the virtual object as a pencil 302 shaded in a single color (e.g., black, blue). The pencil 302 has different portions 302a - 302c.

[0064] Figure 4 shows the device 202 displaying a portion of the pencil 302. In particular, in Figure 4, the pencil 302 is shown as falling from a position outside the fields of view of the primary display 204 and the secondary display 206. Specifically, in Figure 4, portions 302a and 302b of the pencil 302 are within the field of view of the primary display 204, and portion 302c of the pencil 302 is within the field of view of the secondary display 206, but the portion of the pencil 302 shown by the dashed line in Figure 4 is shown to be in a position outside the fields of view of the primary display 204 and the secondary display 206.

[0065] As shown in FIG. 4, compared with parts 302a and 302b, part 302c is blurred. Here, the blurring of part 302c is shown by part 302c being thinner and shaded than shown in FIG. 3. In contrast, parts 302a and 302b contain little or no blurring, which is shown by parts 302a - 302b being shaded in the same color as shown in FIG. 3. Since part 302c is being displayed via the secondary display 206, part 302c has more blurring than parts 302a and 302b, and the secondary display can have a maximum resolution lower than the maximum resolution of the primary display 204 that is displaying parts 302a - 302b.

[0066] When the device 202 is worn by the user, the edge between the user's central and peripheral fields of view is typically near region 404. As shown in FIG. 4, region 404 corresponds to the regions of the primary display 204 and the secondary display 206 that display sections of parts 302b and 302c.

[0067] Since the portion of 302c is blurred compared to part 302b, hard clipping may occur at region 404 and / or at the edge between the user's central and peripheral fields of view. In some embodiments, hard clipping occurs when there is visual distortion between a portion of an object displayed on one display and a portion of an object displayed on another display. As will be further explained below, hard clipping may occur when an object transitions from one display to another.

[0068] In some embodiments, the hard clipping in region 404 is an unintended visual effect. In particular, the hard clipping can cause the user interface displayed by device 202 to distract the user wearing device 202. For example, when an object enters and exits the user's central field of view, the user interface can distract the user wearing device 202.

[0069] FIG. 5 shows an exemplary region on a device having an additional display, according to some embodiments. As further described below, hard clipping can occur between one or more exemplary regions on the device. Since the field of view of the primary display 204 is typically in the user's central field of view and the field of view of the secondary display 206 is in the user's peripheral field of view, reducing the hard clipping of objects transitioning between the primary display 204 and the secondary display 206 reduces high-frequency noise (e.g., snow blindness, random and / or sudden variations of objects, irregular movement or spatial relationships between objects) within the user's central and / or peripheral field of view. High-frequency noise can cause the user to unnecessarily view objects within the user's peripheral field of view, so reducing high-frequency noise within the user's central and / or peripheral field of view is advantageous. Thus, high-frequency noise can affect the user's ability to focus on objects within the user's central field of view, which can affect the ability of device 202 to track and / or make accurate determinations based on the movement of the user's eyes.

[0070] FIG. 5 shows a device 202 having an exemplary first region 504a and an exemplary second region 504b of the primary display 204. The first region 504a extends across the region of the primary display 204 from an edge 208 of the primary display 204 to a distance 510a from the edge 208 of the primary display 204. In some embodiments, at least a portion of the region 404 shown in FIG. 4 overlaps a portion of the first region 504a shown in FIG. 5. In some embodiments, when the user is wearing the device 202, at least a portion of the center and the edges of the peripheral vision of the user are in a portion of the first region 504a.

[0071] The second region 504b extends across the region of the primary display 204 from the center of the primary display 204 to a distance 510b toward the edge 208 of the primary display 204. In some embodiments, no portion of the region 404 shown in FIG. 4 overlaps a portion of the first region 504b shown in FIG. 5. In some embodiments, when the user is wearing the device 202, no portion of the center and the edges of the peripheral vision of the user are in a portion of the first region 504a.

[0072] In some embodiments, the first region 504a and / or the second region 504b extends across other regions of the primary display 204 not shown by FIG. 5, or does not extend across the regions shown by FIG. 5. In some embodiments, the first region 504a and / or the second region 504b extends across the entire primary display 204.

[0073] Figures 6A-6C and 7 illustrate exemplary techniques for displaying virtual objects according to some embodiments. In particular, in Figures 6A-6C and 7, the exemplary techniques and processes described below can reduce hard clipping of an object (e.g., a sudden change when an object transitions from one display to another) when an object transitions between displays having different resolutions (e.g., between primary display 204 and secondary display 206), as shown, for example, in Figure 4 above. By reducing hard clipping of an object transitioning between primary display 204 and secondary display 206, high-frequency noise within the user's central and / or peripheral field of view is reduced. Reducing high-frequency noise within the user's central and / or peripheral field of view is advantageous because high-frequency noise can affect the ability of device 202 to track and / or accurately determine based on the movement of the user's eyes.

[0074] Figure 6A shows an environment including device 202 and pencil 302. In Figure 6A, user 250 is wearing device 202 as described above in connection with Figure 2B. However, for ease of explanation, user 250 is not shown in Figure 6A.

[0075] Figure 6A shows device 202 displaying pencil 302 falling from a position outside the field of view of primary display 204 and secondary display 206. In Figures 6A-6C, the portion of pencil 302 represented by the dashed line corresponds to the portion of 302 that is not visible through primary display 204 and / or secondary display 206 while the user is wearing device 202.

[0076] As shown in FIG. 6A, the pencil 302 fell from a position outside the viewing fields of the primary display 204 and the secondary display 206 to a position within the viewing field of the secondary display 206 and outside the viewing field of the primary display 204. In FIG. 6A, the portion 302a is presented via the secondary display 206. In some embodiments, the device 202 generates a virtual representation of the portion 302a and displays the virtual representation via the secondary display 206.

[0077] In FIG. 6A, the device 202 displays a virtual representation of the portion 302a presented at a resolution corresponding to the maximum resolution of the secondary display 206. As shown in FIG. 6A, compared to the portion 302a in FIG. 3, the portion 302a is blurred (e.g., shown by its lighter shading). In some embodiments, the visual effect for creating a change in blurring is not applied to the virtual representation of the portion 302a shown in FIG. 6A.

[0078] For a period following what is shown in FIG. 6A, as the pencil 302 continues to fall, FIG. 6B shows the portion 302a that has fallen within a specific region (e.g., the region 504a described in FIG. 5 below) of the primary display 204. While the pencil 302 is being displayed via the secondary display 206 (e.g., within the viewing field of the secondary display 206), it is determined that the pencil 302 has fallen to a position within a distance 510a from the edge 208 of the primary display 204 (e.g., has moved within the primary display 204), as shown in FIG. 6B. Based on this determination, the device 202 generates a virtual representation by applying a visual effect to the portion 302a of the pencil 302, as shown in FIG. 6B, and displays the virtual representation on the primary display 204.

[0079] Here, the virtual representation corresponding to portion 302a is blurred by applying a visual effect to portion 302a. As shown in FIG. 6B, when the visual effect is applied, portion 302a is displayed as a gradient (e.g., from a light color to a dark color as shown), providing a smoother transition at the edges of the user's central and peripheral vision (and / or region 404 described in FIG. 4). In some embodiments, after applying the visual effect to portion 302a, the virtual representation corresponding to portion 302a is faded. In some embodiments, the virtual representation is faded inversely proportional to the detected amount of blur of secondary display 206 (or the amount of blur of at least one region of secondary display 206) after applying the visual effect to portion 302a. In some embodiments, the virtual representation does not include the gradient after applying the visual effect to portion 302a. In other embodiments, applying the visual effect includes applying a blurring function to portion 302a. In some embodiments, the amount of blur applied to portion 302a at edge 208 can be selected to match the blur of secondary display 206. By doing so, the appearance of virtual object 302 on both sides of edge 208 can be the same or similar, thereby reducing the hard clipping effect described above. The amount of blur applied to portion 302a (or other virtual content) within portion 504a can be decreased based on the distance from edge 208, and no blur is applied at a distance D1 from edge 208. The blur can be decreased linearly or non-linearly depending on the distance from edge 208.

[0080] As shown in FIG. 6B, portion 302b has also moved within the field of view of secondary display 206. Thus, in addition to displaying the virtual representation corresponding to portion 302a on primary display 204, device 202 also displays a virtual representation corresponding to portion 302b of pencil 302. In some embodiments, device 202 displays the virtual representation corresponding to portion 302b using the techniques described above with respect to displaying the virtual representation corresponding to portion 302a in FIG. 6A.

[0081] In particular, as shown in FIG. 6B, the virtual representations corresponding to portions 302a and 302b are blurred, and this blurring is represented by differences in the shading of the respective virtual representations corresponding to the respective portions of the pencil 302. In FIG. 6B, the virtual representation corresponding to portion 302b has more blurring than the virtual representation corresponding to portion 302a.

[0082] In FIG. 6B, the virtual representation corresponding to portion 302b is blurred because it is displayed via a secondary display 206, which is a low-resolution display that is more blurred than the primary display 204. On the other hand, the virtual representation corresponding to portion 302a is blurred because the device 202 applied a visual effect to portion 302a before displaying the virtual representation corresponding to portion 302a via the primary display 204. Thus, the virtual representation corresponding to portion 302a to which the visual effect has been applied is more similar to (e.g., has a closer blurring value to) the virtual representation corresponding to portion 302b than the virtual representation corresponding to portion 302a to which the visual effect has not been applied.

[0083] In some embodiments, the device 202 applies a visual effect to portion 302a such that the virtual representation of portion 302a appears to be visually consistent with the virtual representation corresponding to portion 302b that is displayed on the secondary display 206 (e.g., when no visual effect is applied to portion 302b). Since the secondary display 206 has a lower resolution than the primary display 204, the device 202 applies a visual effect to portion 302a to match the visual appearance of the virtual representation corresponding to portion 302a to the visual appearance of the virtual representation corresponding to portion 302b.

[0084] In some embodiments, the device 202 applies a visual effect to portion 302 to smooth the transition from displaying the virtual representation corresponding to portion 302a of FIG. 6A via a display having a higher resolution (e.g., the primary display 204) to displaying the virtual representation corresponding to portion 302a of FIG. 6B via a display having a lower resolution (e.g., the secondary display).

[0085] As shown in FIG. 6C, the pencil 302 has fallen to a new position within the field of view of the primary display 204 and the secondary display 206. As shown in FIG. 6C, the portion 302a is a distance from the edge 208 of the primary display 204 that is outside of the distance 510a (e.g., the distance that is the sum of the distance 510a and the distance 510b). Since the portion 302a is outside of the distance 510a from the edge 208 of the primary display 204, the portion 302a has entered the second region (e.g., the region 504b in FIG. 5) of the primary display 204 that is outside of the first region of the primary display 204. Accordingly, no visual effect is applied to the virtual representation of the portion 302a that is in the second region 504b (shown in FIG. 5). In some embodiments, no visual effect (or a visual effect different from the visual effect applied to the first region) is applied to the representation of the portion of the virtual object within the second region 504b of the primary display 204. In some embodiments, since the portion 302a is within the user's central field of view and the device 202 determines that the user is likely to be distracted by a virtual object (e.g., or a particular visual object) to which no visual effect is applied, no visual effect is applied. As shown in FIG. 6C, the device 202 uses the techniques described above with respect to the display of the virtual representation of the portion 302a of FIG. 6A to display a virtual representation of the portion 302c, which is a new portion of the pencil 302 that is within the field of view of the secondary display 206. Further, in FIG. 6C, since the portion 302b is within the distance 510a from the edge 208 of the primary display 204, the device 202 displays a virtual representation of the portion 302b to which a visual effect has been applied (e.g., using a technique similar to that described above with respect to the portion 302a in FIG. 6C).

[0086] In some embodiments, when the entire object (e.g., the pencil 302) is within the field of view of the primary display (e.g., no part of the object is within the field of view of the secondary display), no visual effect is applied to any part of the representation of the virtual object, including a part of the representation of the virtual object displayed in the second region 504b described above with respect to FIG. 5.

[0087] In some embodiments, device 202 can determine to display different virtual objects differently. In some embodiments, device 202 can apply a visual effect to one virtual object while not applying the visual effect to another virtual object. In some embodiments, device 202 can apply different visual effects to virtual objects. In some embodiments, device 202 uses different criteria to apply visual effects to different sets of virtual objects. In some embodiments, device 202 can apply a visual effect to one type of virtual object based on a determination that the virtual object has entered an area of the display, but can refrain from applying a visual effect to another type of virtual object based on a determination that another type of virtual object has entered the area.

[0088] In some embodiments, the operations described in FIGS. 6A-6C can be described in reverse (e.g., FIGS. 6C-6A). For example, a virtual object can transition from being within the field of view of primary display 204 and / or secondary display 206 to being outside the field of view of primary display 204 and secondary display 206 since it is within the field of view of the primary display 204 and / or secondary display 206 using techniques similar to those described above.

[0089] FIG. 7 is a flowchart illustrating a method 700 for displaying virtual objects, according to some embodiments. In some embodiments, the method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, the method is performed by device 202 (FIG. 2A). In some embodiments, the method is performed by a third device or system different from device 202 or system 100. In some embodiments, the method is performed by one or more combinations of system 100, device 202, and the third device or system.

[0090] In block 702, the first portion of the virtual object is displayed via the secondary display. For example, as described above with reference to FIG. 6C, the first portion 302c of the virtual object 302 is displayed via the secondary display 206. In some embodiments, at a first point in time, the virtual object is presented on the secondary display (e.g., secondary display 206) without being displayed on the primary display (e.g., primary display 204). In some embodiments, when the virtual object is presented on the secondary display but not on the primary display, the virtual object is within the field of view of the secondary display and the virtual object is not within the field of view of the primary display.

[0091] Referring to FIG. 7, in block 704, the second portion of the virtual object is displayed via the primary display. For example, as described above with reference to FIG. 6C, the second portion 302b of the virtual object 302 is displayed via the primary display 204. In some embodiments, when the device moves within a predetermined distance from the virtual object, the second portion of the virtual object is displayed via the primary display. In some embodiments, when the virtual object moves within a predetermined distance from the edge of the primary display or within a predetermined distance from the field of view of the primary display (e.g., the edge of the field of view), the second portion of the virtual object is displayed via the primary display.

[0092] Referring to FIG. 7, as part of displaying the second portion of the virtual object (block 704), in block 706, a visual effect is applied to the second portion of the virtual object according to a determination that the second portion of the virtual object is within a predetermined distance from the edge of the primary display. For example, as described above with reference to FIG. 6C, as part of displaying the second portion 302b of the virtual object 302, a visual effect is applied to the second portion 302b of the virtual object 302 according to a determination that the second portion 302b is within a predetermined distance 510a from the edge of the primary display 204.

[0093] In some embodiments, while a portion of one or more virtual objects is in a first region of the primary display, a visual effect is applied to the portion of the one or more virtual objects. For example, as described above in connection with FIG. 6C, a second portion 302b has a visual effect applied thereto while being displayed in a region 504a shown in FIG. 5. In some embodiments, the region 504a is within a predetermined distance from an edge of the primary display adjacent to an edge of the secondary display.

[0094] In some embodiments, the amount of visual effect applied can be the same for two or more virtual objects within a particular region. Alternatively, in some embodiments, the amount of visual effect applied to a particular virtual object is dynamic. For example, the amount of visual effect applied can be based on one or more detected characteristics associated with a user wearing the device, or based on one or more user settings. In some embodiments, user characteristics can include the user's pupil size, pupil direction, pupil position, eye dominance, blink rate, etc. at different times.

[0095] In some embodiments, the amount of visual effect applied depends on one or more characteristics of the virtual object. For example, in some embodiments, the amount of visual effect applied depends on the size, movement (e.g., speed, acceleration), color, etc. of the virtual object. In some embodiments, the amount of visual effect applied is less for virtual objects moving at a lower speed and more for virtual objects moving at a higher speed.

[0096] In some embodiments, the amount of visual effect applied depends on one or more characteristics of the device, such as the speed or acceleration of the device. In some embodiments, the amount of visual effect applied is less when the device is moving at a lower speed and more when the device is moving at a higher speed.

[0097] In some embodiments, the visual effect is applied to a portion of a virtual object when it is displayed in a region of the primary display, but the visual effect is not applied to another portion of the virtual object displayed in another region of the primary display. For example, as described above in connection with FIG. 6C, portion 302a does not have the visual effect applied while it is displayed in region 504b shown in FIG. 5.

[0098] In some embodiments, applying a visual effect to a second portion of a virtual object includes blurring the second portion of the virtual object. In some embodiments, prior to applying the visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first blurred appearance. In some embodiments, after applying the visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second blurred appearance that is different (e.g., more blurred) from the first blurred appearance.

[0099] In some embodiments, the second portion of the virtual object is blurred based on pupil characteristics aligned with a portion of the primary and / or secondary display detected via one or more image sensors. In some embodiments, a portion of the virtual object is blurred or faded in response to one or more detected pupil characteristics, while another portion of the virtual object is not. In some embodiments, the blurring or fading of a portion of the virtual object increases when the detected pupil is further away from the edge of the primary display.

[0100] In some embodiments, after blurring a second portion of the virtual object, the second portion of the virtual object includes a blurring amount based on the blurring amount of the first portion of the virtual object. In some embodiments, the blurring amount (e.g., of a portion of the virtual object) matches the blurring of the first portion of the virtual object. In some embodiments, as part of blurring the second portion of the virtual object, the blurring amount of the second portion of the virtual object varies based on a corresponding distance between the virtual object and the secondary display (e.g., continuously as the distance between the object and device 202 changes).

[0101] In some embodiments, as part of applying a visual effect to the second portion of the virtual object, the second portion of the virtual object is faded (e.g., faded in or faded out). In some embodiments, fading the second portion of the virtual object includes feathering the primary display adjacent to the second region and / or the edges of the virtual object.

[0102] In some embodiments, before applying a visual effect to the second portion of the virtual object, the second portion of the virtual object already has a visual appearance that includes a first faded appearance. In some embodiments, after applying a visual effect to the second portion of the virtual object, the second portion of the virtual object has a subsequent visual appearance that includes a second faded appearance that is different from (e.g., more faded than) the first faded appearance.

[0103] In some embodiments, the second portion of the virtual object is faded in proportion to the blurring of the secondary display (e.g., display 206) (e.g., at least one region of the secondary display). In some embodiments, the second portion of the virtual object is faded out to a level inversely proportional to the blurring of the secondary display.

[0104] In some embodiments, the second portion of the virtual object fades from a first color to a second color. In some embodiments, the second portion of the virtual object fades such that a gradient is applied to the second portion of the virtual object. In some embodiments, when a gradient is applied to the second portion of the virtual object, the second portion of the virtual object includes a gradient (e.g., a gradient that transitions from a first color to a second color).

[0105] In some embodiments, as part of applying a visual effect to the second portion of the virtual object, the corners of the second portion of the virtual object are rounded. In some embodiments, the edge between the primary display and the secondary display (and / or the first region) is rounded or modified to be rounded and displayed. In some embodiments, the corners are faded or a particular visual effect is applied (e.g., a visual effect that fades the corners is applied to the corners, but other portions of the second portion of the virtual object are not faded or do not have a particular visual effect applied). In some embodiments, when the corners are faded, a gradient is applied to the corners.

[0106] Referring again to FIG. 7, in accordance with a determination that the second portion of the virtual object is not within a predetermined distance from the edge of the primary display as part of displaying the second portion (block 704), applying a visual effect to the second portion of the virtual object is abandoned. For example, as described above with reference to FIG. 6C for displaying the second portion 302c of the virtual object 302 via the primary display 204, applying a visual effect to the second portion 302c of the virtual object 302 is abandoned in accordance with a determination that the second portion 302c of the virtual object 302 is not within a predetermined distance 510a from the edge of the primary display 204.

[0107] Figures 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, and 14A-14B illustrate exemplary techniques for transitioning objects between displays, according to some embodiments. Among other advantages, the techniques and processes described below help conserve the battery life of device 202 while continuing to maximize the ability of device 202 to present information (e.g., presenting information to user 250 of FIG. 2B). Device 202 conserves battery life by intelligently shifting the presentation of information from primary display 204 to one or more of secondary display 206 and tertiary display (e.g., the LEDs around bezel 210). Because secondary display 206 and tertiary display can have a maximum resolution lower than that of primary display 204, shifting the information maintains battery life. Thus, secondary display 206 and tertiary display can use less energy than primary display 204 to display information.

[0108] As described below, FIGS. 8A, 9A, 10A, 11A, 12A, 13A, and 14A illustrate exemplary scenarios in which user 250 interacts with object 806 while wearing device 202. However, this particular scenario is provided for illustrative purposes only. The techniques described below or similar techniques can be effectively applied to a variety of other scenarios.

[0109] As noted above, FIG. 8A shows a scenario in which user 250 is wearing device 202. In this scenario, device 202 is positioned at a distance 804a from an object 806, such as a set of keys. In some embodiments, object 806 is a physical item or physical object located within the physical environment in which user 250 is located. In some embodiments, object 806 is a virtual object located within the CGR environment presented to user 250 via device 202.

[0110] In some embodiments, object 806 moves while user 250 and / or device 202 is stationary or alternatively non - stationary.

[0111] In some embodiments, the environment includes one or more objects other than object 806. In some embodiments, device 202 detects that object 806 is in the environment shown in FIG. 8A. In some embodiments, device 202 determines that the object is a useful or important object (e.g., more useful or important than other objects in the environment). In some embodiments, device 202 can use one or more algorithms and / or one or more user preferences to determine whether an object is useful or important for an application that corresponds to the user or a particular object (e.g., an application that communicates with device 202). In some embodiments, device 202 receives a request or notification regarding the object and makes this determination based on whether the object is useful or important based on the request or notification.

[0112] FIG. 8B shows device 202 at a point in time corresponding to the scenario shown in FIG. 8A. Device 202 includes a primary display 204 and a secondary display 206 (in addition to one or more components described in FIG. 2A above). In FIG. 8B, primary display 204 and secondary display 206 do not display (or present) visual information related to the direction information corresponding to the position of object 806 relative to device 202 (direction information). In some embodiments, at least some of the visual information is not displayed via primary display 204 and / or secondary display 206 because device 202 is too far from object 806 (or distance 804a exceeds a predetermined threshold distance).

[0113] As shown in FIG. 8B, device 202 further includes LEDs 810a-810f and 812a-812f (LEDs). In FIG. 8B, the LEDs are not emitting light (e.g., emitting light to indicate direction information). In some embodiments, one or more of the LEDs are not emitting light because device 202 is too far from object 806 (or distance 804a exceeds a predetermined threshold distance).

[0114] FIG. 9A shows object 806 and / or device 202 moved from the position of FIG. 8A to the position of FIG. 9A.

[0115] FIG. 9B shows device 202 at a point in time corresponding to the scenario shown in FIG. 9A. Referring to FIGS. 9A and 9B, assume that device 202 receives direction information when it is at a distance 804b from object 806. In response to receiving the direction information, device 202 determines that object 806 is the type of object for which direction information can be displayed via one or more of the LEDs (e.g., is configured and determined to be so). In some embodiments, device 202 makes this determination because it is at a distance 804b from object 806 (or distance 804b is less than a predetermined threshold distance (e.g., a threshold distance corresponding to a maximum distance for displaying direction information)).

[0116] As shown in FIG. 9B, because device 202 is at a distance 804b from object 806 (or object 806 is the type of object for which direction information can be displayed via one or more of the LEDs), device 202 causes LEDs 812a-812f to emit light in a pattern to indicate the direction information. As shown in FIG. 9B, LEDs 812a-812f are on the right side of device 202 in an upward direction rather than a downward direction so as to be disposed on frame 210 of device 202. Thus, LEDs 812a-812f indicate that object 806 is in front of and to the right of device 202, which is consistent with the general direction of the path between device 202 and object 806 in FIG. 9A.

[0117] In FIG. 9B, LEDs 812a and 812f are hatched, while LEDs 812b to 812e are solid lines to indicate the light gradient field. Here, the light gradient field can indicate the point with a strong directional pull of the directional information between the object 806 and the device 202 from the point with a weak directional pull (e.g., the hatched LEDs 812a, 812f). In particular, the gradient field can provide more directional information to the user, which can help the user navigate more efficiently towards the object 806.

[0118] In some embodiments, the directional information is displayed via LEDs rather than on the primary display 204 and / or the secondary display 206 at a distance greater than a predetermined threshold distance (e.g., a distance greater than distance 804b). In some embodiments, displaying the directional information via LEDs instead of the primary display 204 and / or the secondary display 206 conserves battery power, so the directional information is displayed via LEDs rather than on the primary display 204 and / or the secondary display 206.

[0119] In some embodiments, the direction information is displayed via the LEDs rather than on the primary display 204 and / or the secondary display 206, based on the type of information that is permitted to be displayed when the device 202 is a particular distance away from the object 806. In some embodiments, the device 202 displays direction information that is not based on the appearance of the object (e.g., a blinking LED) when the device 202 is further away from the object 806 (or within a first set of distances from the object). In some embodiments, the device 202 displays direction information based on the appearance of the object (e.g., a visual representation of the object or a proxy object displayed via the primary display 204 and / or the secondary display 206) when the device 202 is closer to the object (or within a second set of distances from the object that is different from the first set of distances). In some embodiments, the device 202 can display direction information based on the appearance of the object and direction information that is not based on the appearance of the object (e.g., when the device 202 is within a third set of distances from the object that is between the first set of distances and the second set of distances from the object). In some embodiments, the direction information is displayed via the LEDs rather than on the primary display 204 and / or the secondary display 206, based on whether the object is within the FOV of the primary display 204 and / or the secondary display 206. In some embodiments, the direction information is displayed via the LEDs rather than on the primary display 204 and / or the secondary display 206 when the object is at a particular distance outside the FOV of the primary display 204 and / or the secondary display 206.

[0120] FIG. 10A shows the object 806 and / or the device 202 moved from the position of FIG. 9A to the position of FIG. 10A. In particular, in FIG. 10A, the device 202 is positioned at a distance 804c from the object 806, where the distance 804c is different from the distance 804b of FIG. 9A.

[0121] Figure 10B shows device 202 at a time corresponding to the scenario shown in Figure 10A. Referring to Figures 10A and 10B, assume that device 202 receives direction information. In response to receiving the direction information, using a technique similar to that described above with respect to Figures 9A and 9B, device 202 determines that object 806 is the type of object for which the direction information can be displayed via the LEDs (e.g., because device 202 is at a distance 804c from object 806). Next, based on this determination, device 202 causes LEDs 812d - 812f to emit light corresponding to the direction information and causes LEDs 812a - 812c to stop emitting light. Here, the updated LEDs of device 202 indicate the newly received direction information (e.g., indicate the position of object 806 below and the direction information of the relative path to the left of the position of device 202).

[0122] As shown in Figure 10B, device 202 reduces the number of LEDs that emit light from the number of LEDs that were emitting light in Figure 9B. In some embodiments, when device 202 is closer to object 806 than in the case of Figure 9B (e.g., in some embodiments, when user 250 is closer to the object, fewer LEDs can provide better direction information). In some embodiments, when there is a physical and / or virtual structure or object between device 202 and object 806, device 202 continuously updates a plurality of LEDs of device 202 (and / or primary display 204 and / or secondary display 206) so that user 250 can navigate around the physical and / or virtual structure or object between device 202 and object 806 (e.g., LEDs that emit light in a certain pattern).

[0123] Figure 11A shows object 806 and / or device 20 moved from the position of Figure 10A to the position of Figure 11A. In particular, in Figure 11A, device 202 is disposed at a distance 804d from object 806, where distance 804d is different from distance 804c in Figure 10A.

[0124] FIG. 11B shows device 202 at a time corresponding to the scenario shown in FIG. 11A. Referring to FIGS. 11A and 11B, assume that device 202 is closer (e.g., within a set of predetermined threshold distances) at distance 804d from object 806 (as opposed to device 202 at distance 804c from object 806 in FIG. 10B), so device 202 updates not only the LED but also secondary display 206. In some embodiments, since object 806 is within the FOV of secondary display 206, device 202 updates not only the LED but also secondary display 206.

[0125] Also assume that device 202 receives direction information. In response to receiving the direction information, using a technique similar to that described above with respect to FIGS. 9A and 9B, device 202 determines that object 806 is the type of object for which the direction information can be displayed via LED 812e (e.g., because device 202 is at distance 804d from object 806). Based on this determination, device 202 uses one or more of the techniques described above in connection with FIGS. 9B and 10B to cause LED 812e to emit light corresponding to the direction information and to stop the emission of light from LEDs 812d and 812f.

[0126] If device 202 determines that object 806 is the type of object for which the direction information can be displayed via secondary display 206, device 202 displays the direction information via secondary display 206. Here, device 202 makes this determination because it is at distance 804d from object 806.

[0127] In some embodiments, since device 202 is at a distance 804d from object 806 and / or object 806 is within a predetermined threshold distance outside the field of view of primary display 204, device 202 causes secondary display 206 to display a modified representation 826a of object 806. Here, the modified representation 826a is a representation of a proxy object or a symbolic representation of object 806 and is visually distinguishable from object 806. As shown in FIG. 11B, the modified representation 826a is a blob of a plurality of pixels at the edge of secondary display 206.

[0128] In some embodiments, when device 202 is at a distance 804d from object 806, object 806 is within the field of view of secondary display 206. In some embodiments, object 806 is at a position within the field of view of secondary display 206 corresponding to the position on secondary display 206 where the modified representation 826a is displayed.

[0129] In some embodiments, when device 202 is at a distance 804d from object 806, object 806 is outside the field of view of secondary display 206. Thus, in some embodiments, when object 806 is outside the field of view of secondary display 206, device 202 displays the modified representation 826a via secondary display 206. In some embodiments, displaying the modified representation 826a at the corresponding position on secondary display 206 when object 806 is outside the field of view of secondary display 206 provides directional information for locating an object outside the field of view of the display of device 202.

[0130] In FIG. 11B, the modified representation 826a does not resemble the actual representation of object 806. In some embodiments, the modified representation 826a does not resemble object 806 because it does not have two or more features such as the size, shape, color, texture, blur, etc. of object 806.

[0131] The modified representation 826a indicates whether an object (e.g., an object that is useful or important as described above in connection with FIG. 8A) is within or outside the user's peripheral vision (e.g., the peripheral vision of the user that receives visual information presented or displayed via the secondary display 206). The modified representation 826a can also indicate the general direction of the object's position relative to the position of the device 202. In some embodiments, the general direction is indicated by placing the modified object representation 826a at a specific position on the secondary display 206.

[0132] In some embodiments, the modified representation 826a is static (e.g., does not move, does not pulsate, is not animated), but the light emitted from one or more LEDs is not static (e.g., moves, pulsates, is animated), or vice versa. In some embodiments, both the modified representation 826a and the LEDs 812a - 812l are static (e.g., or not static).

[0133] FIG. 12A shows the object 806 and / or the device 202 that has moved from the position in FIG. 11A to the position in FIG. 12A. In particular, in FIG. 12A, the device 202 is positioned at a distance 804e from the object 806, where the distance 804e is different from the distance 804d in FIG. 11A.

[0134] FIG. 12B shows the device 202 at a point in time corresponding to the scenario shown in FIG. 12B. Referring to FIGS. 12A and 12B, assume that the device 202 receives new direction information. Here, based on the new direction information, the device 202 updates the primary display 204 and the secondary display 206 to display a representation of the direction information and abandons displaying the direction information via the LEDs.

[0135] In some embodiments, since the device 202 is at a distance 804e from the object 806, the device 202 determines that the object 806 is not the type of object for which the direction information can be displayed via the LEDs 812a - 812l. For example, the device 202 can determine that the distance 804e is a distance at which the device 202 is too close to the object 806. In some embodiments, when the direction information or one or more types of the direction information (e.g., the representation 828) is displayed via the primary display 204 and / or the secondary display 206, the device 202 stops displaying the direction information via the LEDs.

[0136] In some embodiments, in response to receiving the direction information, the device 202 uses a technique similar to that described above with respect to FIG. 11B to determine that the object 806 is the type of object for which the direction information of the object can be displayed via the secondary display 512 (e.g., because the device 202 is at a distance 804e from the object 806 and / or the object 806 is within a predetermined threshold distance outside the field of view of the primary display 204). For this determination, the device 202 causes the modified representation 8'26b to be displayed on the secondary display 206. <{

[0137] In FIG. 12B, since the device 202 is closer to the object 806 in FIG. 12B than in the case of FIG. 1%B, the modified representation 826b is larger than the modified representation 826a. As shown in FIG. 12B, the modified representation 826b is displayed at a new position on the secondary display 206 closer to the primary display 204 than the position of the modified representation 826a as shown in FIG. 11A. Since the object 806 is closer to the field of view of the primary display 204, the modified representation 826b is displayed at the new position.

[0138] In some embodiments, the modified representation 826b occupies a portion of the secondary display 206 that the modified representation 826a occupied. In some embodiments, the modified representation 826b grows from the modified representation 826a (e.g., via an animation). In some embodiments, the device 202 displays the modified representation 826b and / or the modified representation 826a with a brightness based on the size of the object 806. For example, the device 202 displays the modified representation of a large object with a lower or darker brightness than the modified representation of a small object.

[0139] In some embodiments, in response to receiving direction information, the device 202 determines that the object 806 is the type of object whose direction information can be displayed via the primary display 204. In FIG. 12B, the device 202 makes this determination based on the determination that the distance 804e is close enough to the object 806. Based on this determination, the device 202 causes the representation 828 to be displayed via the primary display 204.

[0140] In FIG. 12B, the representation 828 is an arrow indicating the position of the object 806. In some embodiments, the device 202 displays the representation 828 to indicate that the user 250 can adjust the field of view of the primary display 204 (e.g., by moving the user 250's head while standing still), whereby the object 806 can be positioned within the field of view of the primary display 204. In some embodiments, the device 202 animates the representation 828. In some embodiments, the device 202 causes the representation 828 (or a representation corresponding to the representation 828) to be displayed via the secondary display 206.

[0141] FIG. 13A shows the object 806 and / or device 202 moved from the position of FIG. 12A to the position of FIG. 13A. In particular, in FIG. 13A, the device 202 is disposed at a distance 804f from the object 806, where the distance 804f is different from the distance 804e of FIG. 12A. Further, as shown in FIG. 13A, the object 806 is within the field of view of the primary display 204 of the device 202.

[0142] FIG. 13B shows the device 202 at a point in time corresponding to the scenario shown in FIG. 13A. Referring to FIGS. 13A and 13B, assume that the device 202 receives new orientation information. Since the device 202 is at a distance 804f from the object 806 and / or at least a portion of the object 806 is within the field of view of the primary display and within the field of view of the secondary display, the device 202 displays an actual representation of a portion of the object 806 via the primary display 204 and a modified representation 826c of the object 806 via the secondary display 206. In some embodiments, the actual representation of a portion of the object is similar to the object 806 (e.g., having two or more features similar to the object 806), but the modified representation is not similar thereto. In some embodiments, the actual representation of a portion of the object is displayed using a first model of the object, but the modified representation is displayed using a second model of the object. In some embodiments, the first model of the object is different from the second model of the object because the second model of the object is simpler than the first model of the object. In some embodiments, the second model of the object is simpler than the first model of the object because the second model has less detail than the first model (e.g., a 3D model with fewer polygons).

[0143] In some embodiments, using techniques similar to those described above with respect to FIG. 7 (e.g., to reduce clipping of object 806 transitioning between primary display 204 and secondary display 206), by applying a visual effect to a virtual representation of a portion of object 806, a portion of modified representation 826b moves within primary display 204. In some embodiments, using techniques similar to those described above with respect to FIG. 7, when device 202 transitions an actual representation of a portion of object 806 that is displayed via primary display 204, device 202 replaces the actual portion of object 806 on secondary display 206 with modified representation 826b or 826c.

[0144] FIG. 14A shows object 806 and / or device 202 moved from the position of FIG. 13A to the position of FIG. 14A. In particular, in FIG. 14A, device 202 is disposed at a distance 804g from object 806, where distance 804g is different from distance 804f of FIG. 13A. Further, as shown in FIG. 14A, the entirety of object 806 is within the field of view of primary display 204 of device 202.

[0145] FIG. 14B shows device 202 at a point in time corresponding to the scenario shown in FIG. 14A. Referring to FIGS. 14A and 14B, device 202 stops displaying an actual representation of object 806 (or, in some embodiments, a portion of object 806) that is displayed via primary display 204 and stops displaying a modified representation of object 806 via secondary display 206. In some embodiments, since device 202 is within the field of view of primary display 204 and not within the field of view of secondary display 206, device 202 causes an actual representation of object 806 to be displayed via primary display 204 and stops displaying a modified representation of object 806 via secondary display 206.

[0146] In some embodiments, in FIGS. 13B and 14B, instead of displaying an actual representation of object 806 via primary display 204, device 202 can display a proxy representation of object 806 via secondary display 206. In some embodiments, secondary display 206 can overlap primary display 204 such that secondary display 206 covers most (or all) of primary display 204. Thus, in some of these embodiments, the proxy representation of object 806 can be shown via secondary display 206 over a portion of primary display 204 until device 202 receives a request to interact with object 806. For example, a request to interact with object 806 can be received when an input (e.g., voice input, physical input) is determined to be directed towards object 806 and / or when a process is initiated to perform an operation in which one or more components of object 806 and / or objects associated with object 806 are (e.g., indirectly or directly) involved. In some embodiments, in response to receiving a request to interact with object 806, device 202 stops displaying the proxy representation of object 806 via secondary display 206 and displays an actual representation of object 806 via primary display 204. Thus, in embodiments where device 202 uses more battery power to display an object via primary display 204 than via secondary display 206, battery power of device 202 is conserved because primary display 204 is not used to display a representation of object 806 until it is determined that object 806 can be interacted with.

[0147] In some embodiments, in FIGS. 13B and 14B, instead of displaying an actual representation of object 806 via primary display 204, device 202 can display a proxy representation of object 806. In some embodiments, the proxy representation of object 806 has less visual detail than the actual representation of object 806 and / or requires less power to display. In some embodiments, while displaying the proxy representation of object 806 via primary display 204, device 202 receives a request to interact with object 806. For example, a request to interact with object 806 can be received when an input (e.g., voice input, physical input) is determined to be directed at object 806 and / or when a process is initiated to perform an operation involving one or more components of object 806 and / or objects associated with object 806. In some embodiments, in response to receiving a request to interact with object 806, device 202 replaces the display of the proxy representation of object 806 with an actual representation of object 806. Thus, in some embodiments, the actual representation of object 806 is only displayed after a determination has been made that object 806 can be interacted with, thereby conserving battery power and / or reducing the number of processes executed by device 202. In some embodiments, the LED is inactive (or not included as part of device 202) while the proxy representation of object 806 and / or the actual representation of object 806 is being displayed.

[0148] FIG. 15 is a flowchart showing a method of transitioning an object between displays based on direction information, according to some embodiments. In some embodiments, this method is performed by system 100 (FIGS. 1A and 1B). In some embodiments, this method is performed by device 202 (FIG. 2A). In some embodiments, this method is performed by a third device or system different from device 202 or system 100. In some embodiments, this method is performed by one or more combinations of system 100, device 202, and the third device or system.

[0149] In block 1502, direction information (e.g., 804a - 804d) corresponding to the position of an object outside the field of view of the primary display (e.g., 204) of the device and the secondary display of the device (e.g., north, south, east, west, right, left, behind, in front, other cardinal directions, or any combination thereof) is received. In some embodiments, the object is a physical object or article in a physical environment, and in some embodiments, the object is a virtual object in a virtual environment.

[0150] In some embodiments, the primary display, secondary display, and / or tertiary display are different types of displays. In some embodiments, the primary display is a waveguide display. In some embodiments, the secondary display is an organic light - emitting diode display. In some embodiments, the tertiary display is a plurality of light - emitting diodes.

[0151] In blocks 1504 and 1506, in response to receiving direction information corresponding to the position of an object outside the field of view of the primary and secondary displays, and according to a determination that a first criterion is met, a first representation of the direction information is displayed via the secondary display.

[0152] In some embodiments, the first criterion includes a criterion that is satisfied when the object is of a type for which the orientation information of the object can be displayed on the secondary display. In some embodiments, when the object is at a predetermined distance from or beside a certain part of the object (e.g., the primary display, the secondary display, the frame of the device), it is permitted to display the orientation information of the object via the secondary display. In some embodiments, when orientation information is received from an application or an object corresponding to the application, it is permitted to display the orientation information of the object via the secondary display, and when the object has one or more characteristics or features (e.g., size, shape, distance) that enable the object to be displayed via the secondary display, the object is configured by an application (or other process) to be displayed via the secondary display. In some embodiments, when the object is not in a specific position or orientation, it is permitted to display the orientation information of the object via the secondary display, or the representation of the object is displayed adjacent to a part of the device (e.g., inside the frame of the device, near the bridge of the device, near the user's nose) by the object.

[0153] In blocks 1504 and 1508, in response to receiving orientation information corresponding to the position of an object outside the visual fields of the primary display and the secondary display, and according to the determination that the second criterion is satisfied, a second representation (e.g., 812a - 812g) of the orientation information (e.g., 804b - 804d) is displayed via the tertiary display.

[0154] In some embodiments, the second criterion includes a criterion that is satisfied when the object is of a type for which the direction information of the object can be displayed on the three-dimensional display. In some embodiments, when direction information is received from an application or an object corresponding to the application, if the object is at a predetermined distance from or beside a part of the device (e.g., the primary display, the secondary display, the frame), the direction information of the object is permitted to be displayed via the three-dimensional display, or if the object has one or more characteristics or features (e.g., size, shape, distance) that enable it to be displayed via the three-dimensional display, if the object is not in a particular position or orientation, or if the object causes the representation of the object to be adjacent to a part of the device (e.g., inside the frame of the device, near the bridge of the device, near the user's nose) or displayed on a part of the three-dimensional display, the object is configured by the application (or other process) to be displayed via the three-dimensional display.

[0155] In some embodiments, the first representation is a virtual object (e.g., the first representation is a modified representation of an object), and the second representation is not a virtual object within the CGR environment. In some embodiments, the first modified representation of the virtual object can be visually distinguished from the actual representation of the object. For example, the first modified representation can have a visual appearance that does not resemble or is dissimilar to the actual object, while the actual representation of the object can have a visual appearance that resembles or is similar to the actual object. In some embodiments, the device cannot identify the actual object when processing the modified representation of the object using image recognition technology, but can identify the actual object when processing the actual representation of the object. In some embodiments, the first modified representation can be a representation that does not resemble the actual object (e.g., does not have two or more identical features such as color, shape, texture, size, brightness, thickness, fade, etc.), while the actual representation is a representation that resembles the actual object.

[0156] In some embodiments, the first representation is a static representation. For example, a static representation is a representation that does not move or become animated over a period of time (e.g., does not change, does not pulsate, does not have a predetermined sequence or pattern, etc.). In some embodiments, the second representation is a dynamic representation. For example, a dynamic representation is a representation that moves or becomes animated over a period of time (e.g., pulsates, is displayed in a predetermined sequence or pattern, changes color over a period of time). In some embodiments, the first representation is a different color from the second representation. In some embodiments, the first representation is a different size from the second representation. In some embodiments, the first representation is not displayed while the second representation is displayed, and vice versa. In some embodiments, at least one of the first representation and the second representation includes a gradient (e.g., a gradient field indicating a direction (e.g., solid lines and hatched LEDs)).

[0157] In some embodiments, the primary display and the secondary display can display different representations of an object. In some embodiments, the first representation of the direction information is the first representation of the object displayed via the secondary display. In some embodiments, while the first representation of the direction information is being displayed via the secondary display, the object is detected within the field of view of the primary display of the HMD device. In some embodiments, in response to detecting that the object is within the field of view of the primary display, the second representation of the object is displayed via the primary display. In some embodiments, the first representation of the object displayed via the secondary display is displayed based on the first model of the object. In some embodiments, the second representation of the object displayed via the primary display is displayed based on the second model of the object. In some embodiments, the second model of the object is simpler than the first model of the object (e.g., has more visual features, more visual details).

[0158] In some embodiments, the tertiary display surrounds a first portion of the secondary display. In some embodiments, the tertiary display surrounds a first portion of the secondary display but does not surround a second portion of the secondary display. In some embodiments, the primary display overlaps the secondary display.

[0159] In some embodiments, a representation of the object different from the first and second representations of the direction information is displayed via the primary display, and the first representation is displayed via the secondary display. In some embodiments, the representation of the direction information is associated with the position of the object. For example, the representation of the direction information can indicate the position of the device relative to the position of the object.

[0160] In some embodiments, the representation of the object may not be displayed via the primary display until interacting with the object. In some embodiments, direction information corresponding to the position of the object within the field of view of the primary display of the HMD device and the secondary display of the HMD device is received. In some embodiments, in response to receiving the direction information corresponding to the position of the object within the field of view of the primary display of the HMD device and the secondary display of the HMD device, the fourth representation of the object is displayed via the secondary display without displaying the fifth representation of the object via the primary display. In some embodiments, a request to interact with the object (e.g., select the object, initiate a process in which the object is involved) is received. In some embodiments, in response to receiving the request to interact with the object, the fifth representation of the object is displayed via the primary display, and the display of the fourth representation of the object via the secondary display is stopped.

[0161] In some embodiments, a request to interact with the object is received. In some embodiments, the representation of the object is displayed via the primary display in response to receiving the request to interact with the object.

[0162] In some embodiments, while the first representation and the second representation are being displayed, the first representation is displayed via the secondary display at a first position in a corresponding direction with respect to the primary display, and the second representation is not displayed via the tertiary display at a second position in a corresponding direction with respect to the primary display.

[0163] FIG. 16 is a flowchart showing a method of displaying a modified representation of an object according to some embodiments. In some embodiments, this method is executed by system 100 (FIGS. 1A and 1B). In some embodiments, this method is executed by device 202 (FIG. 2A). In some embodiments, this method is executed by a third device or system different from device 202 or system 100. In some embodiments, this method is executed by a combination of one or more of system 100, device 202, and the third device or system.

[0164] At block 1602, an object (e.g., a virtual object, a virtual object that is a representation of a physical object, a physical object in a physical environment) within a computer-generated reality (CGR) environment is detected at a first location (or a location). In some embodiments, the object is determined to be important or useful. For example, the object can be determined to be important or useful by receiving user input related to the object, by receiving user interaction related to the object, or by receiving data that identifies the object as important or useful. In some embodiments, the data that identifies the object as important or useful is received from one or more applications that are communicating with or being executed by device 202.

[0165] At blocks 1604 and 1606, in response to detecting an object within the CGR environment at the first location and in accordance with the determination that the first location is within a first predetermined distance outside the field of view of the primary display of the device, a first modified representation of the virtual object is displayed.

[0166] The first modified representation of the virtual object can be visually distinguished from the actual representation of the object. For example, the first modified representation can have a visual appearance that does not resemble or is dissimilar to the actual object, while the actual representation of the object can have a visual appearance that resembles or is similar to the actual object. In some embodiments, the device may not be able to identify the actual object when processing the modified representation of the object using image recognition technology, but can identify the actual object when processing the actual representation of the object. In some embodiments, the first modified representation can be a representation that does not resemble the actual object (e.g., does not have two or more of the same characteristics such as color, shape, texture, size, brightness, thickness, fade, etc.), while the actual representation is a representation that resembles the actual object.

[0167] In some embodiments, the first modified representation has visual content (e.g., data representing the displayed representation) that is different from the visual content of the first actual representation. In some embodiments, the modified representation of the object has a structure that is different from the structure of the actual object, while the actual representation of the object has the same structure as the object. For example, if the actual object is a tree, the actual representation of the object visually appears like a tree, but the first modified representation appears like a blob of pixels that is not in the shape of a tree.

[0168] In some embodiments, the first modified representation has a size based on the distance between the first position and the field of view of the primary display of the device. In some embodiments, when the object is at a first distance outside the primary display, the first modified representation is a first size. In some embodiments, when the object is at a second distance outside the primary display, the first modified representation is a second size. In some embodiments, when the first distance is closer to the primary display than the second distance, the first modified representation displayed at the first size is larger than the first modified representation displayed at the second size. In some embodiments, the size of the first modified representation increases when the object is detected at a position closer to the field of view of the primary display of the device than the first position.

[0169] In some embodiments, the first modified representation has a luminance based on the size of the object (e.g., such that larger objects are made darker). In some embodiments, when the object is a first size, the first modified representation has a first luminance. In some embodiments, when the object is a second size, the first modified representation has a second luminance. In some embodiments, when the first size of the object is smaller than the second size of the object, the first modified representation displayed at the first luminance is brighter than the first modified representation displayed at the second luminance.

[0170] In some embodiments, the first modified representation has a luminance based on the size of the object relative to another object that is displayed via the device. For example, the luminance of the first modified representation can increase as the size of the object becomes larger than the size of another object, and vice versa. In some embodiments, if the size of the object is substantially equal to the size of another object, the luminance of the modified representation of the object can be equal to the luminance of the representation of the other object. Thus, in some embodiments, as the user moves through the environment, the luminance of the modified representation of the object can vary based on other objects in the environment.

[0171] In some embodiments, the first modified representation has a higher blur than the blur of the first actual representation. In some embodiments, the first modified representation is more faded than the first actual representation.

[0172] In some embodiments, as part of displaying the first modified representation, an animation can be displayed. For example, the animation can be a pulsating animation, an animation that shifts or moves the first modified representation, an animation that converts the first actual representation of the object to the first modified representation, an animation that fades in or fades out the first modified representation, and the like.

[0173] In some embodiments, the second actual representation of the first portion of the object is displayed via the primary display in response to detecting the object in the CGR environment at the first position and in accordance with the determination that the first position is within the field of view of the primary display and within the field of view of the secondary display. For example, as shown in FIG. 13B, the second actual representation of the first portion of the object (e.g., 806) can be displayed via the primary display if the device is within a threshold distance within the field of view of the secondary display.

[0174] While displaying a second actual representation of a first portion of an object via a primary display, a second modified representation of a second portion of the object is displayed via a secondary display. For example, as shown in FIG. 13B, the second modified representation (e.g., 826c) is displayed via the secondary display.

[0175] In some embodiments, the second portion of the object is a portion of the object that is different from the first portion of the object. In some embodiments, the first portion of the object is within the field of view of the primary display but not within the field of view of the secondary display. In some embodiments, the second portion of the object is within the field of view of the secondary display but not within the field of view of the primary display. In some embodiments, the second actual representation is displayed at a higher resolution than the second modified representation. In some embodiments, the second actual representation is displayed adjacent to an edge of the primary display adjacent to the secondary display. In some embodiments, the second modified representation is displayed adjacent to an edge of the secondary display adjacent to the primary display.

[0176] In some embodiments, as part of displaying a second actual representation of the object via the primary display, a visual effect is applied to the second actual representation of the object. In some embodiments, the second actual representation of the object after the visual effect is applied has a greater blur than the blur of the second actual representation of the object before the visual effect is applied. In some embodiments, one or more of the second actual representation of the first portion of the object displayed via the primary display and the second modified representation of the second portion of the object displayed via the secondary display are displayed using one or more of the techniques described above in connection with FIG. 7.

[0177] In some embodiments, a representation corresponding to direction information for locating an object is displayed via a primary display in response to detecting the object within the CGR environment at a first location and according to a determination that the first location is within a second predetermined distance outside the field of view of the primary display of the device. For example, as shown in FIGS. 12A and 12B, unlike the case where the user 250 and the object 806 are separated from each other by a distance 804d in FIG. 11A, when the distance between the user 250 and the object 806 is a distance 804e at which they are separated from each other, a representation (e.g., 828) corresponding to direction information for locating the object is displayed via the primary display.

[0178] In some embodiments, a representation corresponding to direction information for locating an object is different from an actual representation. A representation corresponding to direction information for locating an object can provide an indication of the object's position, the object's proximity to the device, and whether the device is moving away from or approaching the object.

[0179] In some embodiments, while a representation corresponding to the position of an object is being displayed via a primary display, a modified representation of the object is displayed via a secondary display.

[0180] In some embodiments, in response to detecting an object at a first location and according to a determination that the first location is within the field of view of the primary display and not within the field of view of the secondary display, a first actual representation (e.g., 806) of the object is displayed via the primary display, but the display via the secondary display of a first modified representation of the object is stopped. In some embodiments, a portion of the first actual representation of the object is displayed. In some embodiments, the entire object is displayed.

[0181] Figures 17A - 17B, 18, and 25 illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments. In Figure 17A, device 202 is shown to have one or more components, as described above with respect to Figures 2A - 2B and 8A.

[0182] Figure 17A shows device 202 displaying a plurality of application icons. The plurality of application icons includes a messaging application icon 1702a, a health application icon 1702b, a phone application icon 1702c, and an email application icon 1702d.

[0183] Referring to Figure 17A, assume that device 202 receives a new message via the messaging application corresponding to messaging icon 1702a. In response to receiving the new message, device 202 can determine whether information related to the new message is to be displayed via a particular display (e.g., primary display 204, secondary display 206, LED, etc.).

[0184] In some embodiments, in response to receiving the new message, device 202 determines that it is permitted (or configured) for information related to the new message to be displayed via secondary display 206. As shown in Figure 17B, device 202 updates messaging application icon 1702a to include message status indicator 1702a1 for this determination. As shown in Figure 17B, message status indicator 1702a1 is a "1" adjacent to messaging icon 1702a, indicating that one additional new or unread message has been received. Unread messages may include messages that have not received user interaction, such as voicemail messages not listened to, unread text or email messages, notifications or alerts from applications not yet opened, notifications or alerts not yet dismissed, etc.

[0185] In some embodiments, in response to receiving a new message, device 202 determines that information related to the new message is permitted (or configured) to be displayed via a tertiary display, such as an LED. As shown in FIG. 17B, device 202 causes the LED to emit light for this determination. In some embodiments, device 202 causes only a portion of the LED to emit light.

[0186] One or more other types of alerts, notifications, sensor data, requests or data from applications, incoming calls, etc. can be received. In some embodiments, device 202 performs similar operations in response to their reception as described above with respect to the new messages received in FIGS. 17A - 17B. For example, device 202 determines that health data detected by a heart rate sensor (or one or more other sensors that detect personal or health data) is above or below a threshold level and that information related to that data is permitted to be displayed via the LED, device 202 can cause the LED to emit light.

[0187] Device 202 can also perform other operations in response to the reception of one or more messages, alerts, notifications, sensor data, requests or data from applications, incoming calls, etc. For example, as shown in FIG. 18, device 202 displays a color across at least a portion of secondary display 206 based on a determination that information related to a new message is permitted to be displayed via secondary display 206. In some embodiments, a portion of primary display 204 is updated for this determination.

[0188] In some embodiments, device 202 can display an animation via one or more displays. In some embodiments, the animation can include one or more pulsating colors, one or more moving objects, one or more objects that change shape or transition to one or more other objects, etc. For example, device 202 can display an animation of pulsating colors on primary display 204 and / or secondary display 206 based on a determination that the user's health data is above / below a threshold level and that information related to the health data is permitted to be displayed on the corresponding display.

[0189] Figures 19A - 19B and 25 illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0190] Figure 19A shows device 202 displaying a CGR environment that includes representations of multiple objects 1910a - 1910d. Objects 1910a - 1910d are automobiles within the field of view of primary display 204 and secondary display 206. Here, objects 1910a - 1910d represent physical automobiles within a physical environment that can be seen through primary 204 and secondary display 206 of device 202. In some embodiments, objects 1910a - 1910d represent virtual objects displayed by primary 204 and secondary display 206 of device 202.

[0191] Referring to FIG. 19A, assume that device 202 receives data from an application indicating that object 1910a is associated with the user of device 202. For example, in FIG. 19A, the application can be a car-sharing application, a taxi application, a vehicle location application, etc. In some embodiments, device 202 determines that object 1910a is associated with the user of device 202 without receiving data from a dedicated application, using one or more user settings, data stored in memory, one or more machine learning algorithms, etc.

[0192] In response to receiving data from the application, device 202 can determine whether information related to the data from the application is to be displayed via a specific display. In some embodiments, in response to receiving data from the application, device 202 determines that it is permitted for information related to the data to be displayed via secondary display 206. As shown in FIG. 19B, based on this determination, device 202 displays indicator 1912 around object 1910a on secondary display 206.

[0193] Indicator 1912 is displayed around object 1910a to visually distinguish object 1910a from objects 1910b - 1910d. In some embodiments, device 202 visually distinguishes object 1910a from other objects in other ways. For example, device 202 can stop the display of objects 1910b - 1910d, minimize objects 1910b - 1910d, enlarge object 1910a, highlight object 1910a, etc.

[0194] In some embodiments, in response to receiving data from an application, device 202 determines that information related to the data is permitted to be displayed via the LED. As shown in FIG. 19B, based on this determination, device 202 causes one or more LEDs to emit light.

[0195] FIGS. 20A-20D and 25 illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0196] FIG. 20A shows device 202 displaying a CGR environment that includes a representation of time 2002. The representation of time 2002 is displayed via primary display 204 and indicates that the process has 60 seconds remaining. In some embodiments, the representation of time 2002 represents a process related to a timer application, a delivery application, a workout application, a meeting application, a productivity application, etc. In some embodiments, since a determination has been made that the representation of time 2002 is permitted to be displayed via primary display 204, the representation of time 2002 is displayed via primary display 204.

[0197] Simultaneously with the representation of time 2002, a representation of time 2004 is displayed via secondary display 2006. The representation of time 2004 includes a colored overlay that covers the entire secondary display 206 to indicate the remaining 60 seconds. In some embodiments, the representation of time 2004 indicates the time represented by the representation of time 2002. In other words, in some embodiments, the representation displayed via one display corresponds to or can represent the representation displayed via another display.

[0198] In some embodiments, since a determination has been made that the representation of time 2004 is permitted to be displayed via secondary display 206, the representation of time 2002 is displayed via secondary display 206.

[0199] As shown in FIG. 20A, the representation of time 2004 indicates the percentage of the remaining time when compared to the start time of the timer or process. For example, assuming the process starts and ends in 60 seconds, the representation 2004 indicates that the percentage of the remaining time is 100%.

[0200] Simultaneously with the representation of time 2002 and the representation of time 2004, as shown in FIG. 20A, the device 202 causes the LED to emit light. Similar to the representation of time 2004, the LED indicates the time represented by the representation of time 2002 and thus corresponds to the representation of time 2002. Here, assuming the process starts and ends in 60 seconds, since the percentage of the remaining time is 100%, all the LEDs are emitting light.

[0201] As shown in FIG. 20B, assume that the device 202 receives an indication that the remaining time of the process is 30 seconds. In response to receiving the indication, the device 202 updates the representation of time 2002 and the representation of time 2004. In FIG. 20B, the representation of time 2004 includes a colored overlay that covers approximately half of the area of the secondary display 206. In some embodiments, the representation 2004 in FIG. 20B represents that the remaining time (e.g., 30 seconds) is 50% of the original time (e.g., 60 seconds in FIG. 20A) set for the process. Further, the device 202 also stops the emission of light from half of the LEDs (e.g., 810a - 810l and 812a - 812l) in response to the remaining time being at least 50% of the original time.

[0202] As shown in FIGS. 20C - 20D, the remaining times are 15 seconds and 5 seconds respectively. As shown in FIGS. 20C - 20D, the device 202 continues to update the primary display 204, the secondary display 206, and the LEDs (e.g., 810a - 810l and 812a - 812l) based on the remaining time of the process (e.g., 15 seconds in FIG. 20C and 5 seconds in FIG. 20D).

[0203] However, in contrast to FIGS. 20A - 20C, device 202 causes more of the LED (e.g., more than 1 / 12 of the LED) to emit light in FIG. 20D. Here, device 202 causes all of the LEDs to emit light to provide an indication that the remaining time is less than a predetermined minimum time.

[0204] In some embodiments, device 202 animates (e.g., pulsates, brightens) one or more LEDs and / or secondary display 206 as the remaining time approaches zero. In some embodiments, secondary display 206 and / or the LEDs change to different colors and / or shades when a predetermined amount of time remains. For example, secondary display 206 can change from translucent to red when a predetermined amount of time remains. In some embodiments, one or more LEDs and / or secondary display 206 animate, while one or more other displays do not. In some embodiments, secondary display 206 can overlap primary display 204, such that the animation shown in the overlapping portion of secondary display 206 is shown in the center of the display of device 202 (e.g., the area corresponding to around the user's pupil).

[0205] FIGS. 21A - 21D and FIG. 25 illustrate exemplary techniques for managing one or more displays based on data associated with one or more processes, according to some embodiments.

[0206] FIG. 21A shows device 202 displaying a CGR environment that includes devices 2102a and 2102b. As shown in FIG. 21A, devices 2102a and 2102b are within the field of view of primary display 204 and secondary display 206.

[0207] As shown in FIG. 21A, the user interface (UI) element 2110 is displayed on the device 2102a. For example, the UI element 2110 can be one or more icons, files, application shortcuts, characters, etc.

[0208] Referring to FIG. 21B, the device 202 detects a drag-and-drop input 2150 on the UI element 2110. In response to detecting the drag-and-drop input 2150 on the UI element 2110, the device 202 determines whether information related to the status of the drag-and-drop input can be displayed via the LED. As shown in FIG. 21B, the device 202 causes the LED to emit light for this determination.

[0209] Referring to FIG. 21B, assume that the device 202 detects the movement of the drag-and-drop input 2150. In response to detecting the movement of the drag-and-drop input 2150, the device 202 displays the UI element 2110 that moves from a position on the device 2102a to a position on the device 2102b.

[0210] FIG. 21C shows the UI element 2110 at a position on the device 2102b. As shown in FIG. 21C, since the UI element 2110 remains selected, the LED continues to emit light.

[0211] Referring to FIG. 21C, assume that the device 202 detects that the UI element 2110 is not selected. In response to detecting that the UI element 2110 is not selected, the device 202 abandons causing the LED to emit light, as shown in FIG. 21D. In some embodiments, in response to detecting that the UI element 2110 is not selected, the device 202 causes one or more LEDs to emit light differently. For example, the device 202 can cause one or more LEDs to emit light of a color different from the color previously emitted in FIG. 21C while the UI element 2110 is selected.

[0212] In some embodiments, causing one or more LEDs to emit light differently indicates the completion or effect of a user action. For example, device 202 can cause the LEDs to emit light differently according to the rhythm of a media file that is played after a user drags a media file from device 2102a to device 2102b.

[0213] In some embodiments, device 202 performs the techniques described above with respect to a drag and drop input 2150 (e.g., in FIGS. 21A-21D) to indicate other user interactions and / or the state of an application. For example, in some embodiments, device 202 performs the above techniques when a drag, release, copy, edit, clip, or move operation is detected. In some embodiments, device 202 performs the above techniques when it receives data regarding the state of an application or process, such as a state related to downloading a file, a state related to playing media, a state related to completion of an exercise routine, a state related to completion of a periodic goal (e.g., a health goal, a productivity goal), etc.

[0214] FIGS. 22A-22B and FIG. 25 illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0215] FIG. 22A shows device 202 displaying a CGR environment that includes person 2202. Person 2202 is within the field of view of primary display 204 and secondary display 206 and is visible through them.

[0216] As shown in FIG. 22A, person 2202 is frowning. Referring to FIG. 22A, assume that device 202 receives data representing the mood of person 2202 that is processed by one or more algorithms, such as one or more face recognition algorithms. In some embodiments, the one or more algorithms can determine that person 2202 is frowning and transmit data representing this determination.

[0217] In response to receiving data indicating that person 2202 is frowning, device 202 determines that information related to that data is permitted to be displayed via the LED. In some embodiments, in accordance with this determination, device 202 causes the LED to emit light in a first state. For example, FIG. 22A shows the LED in a first state (e.g., the LED is filled). In some embodiments, device 202 updates only a portion of the LED shown by the solid line in FIG. 22A.

[0218] In some embodiments, device 202 updates secondary display 206 with a colored overlay in a state that suggests the mood of person 22A (e.g., in accordance with a determination that data is permitted to be displayed via secondary display 206).

[0219] FIG. 22B shows device 202 displaying a CGR environment that includes person 2204. As shown in FIG. 22B, person 2204 is smiling, in contrast to person 2202 in FIG. 22A. Referring to FIG. 22B, assume that device 202 receives data representing the mood of person 2204 that is processed by one or more algorithms, such as one or more face recognition algorithms.

[0220] In response to receiving data representing the mood of person 2204, device 202 determines that information related to that data is permitted to be displayed via the LED. In some embodiments, in accordance with this determination, device 202 causes the LED to emit light in a second state, as opposed to the first state. For example, FIG. 22B shows the LED in the second state by showing that the LED is hatched. Here, device 202 causes the LED to emit light differently in the second state than in the first state because the data represents a different mood of the user.

[0221] In some embodiments, data representing other moods (e.g., sad, satisfied, etc.) or other types of data can be displayed using techniques similar to those described above in connection with FIGS. 22A-22B. For example, other types of data can include the state of the financial markets, weather forecasts, ratings (e.g., restaurant ratings, movie ratings, etc.).

[0222] FIGS. 23A-23B and 25 illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments.

[0223] FIG. 23A shows a device 202 displaying a CGR environment that includes a representation 2302 of a progress indicator being displayed via a secondary display 206. In some embodiments, the representation 2302 indicates one or more metrics being tracked. For example, the metric can be the number of steps taken (or remaining to achieve a goal), download speed, number of workouts over a period of time, and / or any other metric. In some embodiments, the representation 2302 corresponds to data received from one or more health applications, such as a walking application, a training application, a strength training application, etc.

[0224] In some embodiments, the device 202 causes the LEDs to emit light representing the progress indicator based on a determination that information related to the progress indicator is permitted to be displayed via the LEDs. For example, as shown in FIG. 23A, LEDs 810a and 810h-810l are emitting light, and LEDs 810a and 810h-810l correspond to the length and position of the representation 2302. LEDs 810a and 810h-810l indicate the level of the progress indicator.

[0225] In some embodiments, as shown in FIG. 23A, LEDs 812a, 812h, 812i, 812l emit light. LEDs 812a, 812h, 812i, 812l are on the right side of device 202 along the length of representation 2302. LEDs 812a, 812h, 812i, 812l also indicate the level of the progress indicator. In particular, LEDs 812j and 812k of device 202 do not emit light because they are within a specific distance from the nose of the user wearing device 202.

[0226] Referring to FIG. 23B, device 202 updates representation 2302 via secondary display 206 and LEDs to indicate that the progress of the tracked metric has been achieved. In particular, LEDs 812j and 812k of device 202 remain non-emitting because they are within a specific distance from the nose of the user wearing device 202.

[0227] FIGS. 24A-24D illustrate exemplary techniques for managing one or more displays based on data related to one or more processes, according to some embodiments. FIGS. 24A-24D show a scenario where device 202 sequentially illuminates one or more of LEDs 810a-210l and 812a-812l in a clockwise pattern around both eyes of the user wearing device 202. In some embodiments, the pattern can indicate a direction to the user and can also replace or be used in addition to the examples described above in connection with FIGS. 8A-8B, FIGS. 9A-9B, FIGS. 10A-10B, FIGS. 11A-11B, FIGS. 12A-12B, FIGS. 13A-13B, FIGS. 14A-14B, FIG. 15, and FIG. 16.

[0228] For example, an LED can emit a pattern, in which case the LED is sequentially turned on and off, which is similar to a point moving around a circle. The circle can be animated around one or more eyes of a user wearing the device 202 in a clockwise or counterclockwise direction. In some embodiments, when drawing a pattern around both eyes of the user, the device 202 transitions between the right and left LEDs of the device 202 as shown in FIGS. 24B-24C, and the device 202 skips some of the LEDs in the sequence (e.g., 810g and 810h).

[0229] FIG. 25 is a flow diagram illustrating a method for managing one or more displays based on data associated with one or more processes according to some embodiments. In some embodiments, this method is performed by the system 100 (FIGS. 1A and 1B). In some embodiments, this method is performed by the device 202 (FIG. 2A). In some embodiments, this method is performed by a third device or system different from the device 202 or the system 100. In some embodiments, this method is performed by one or more combinations of the system 100, the device 202, and the third device or system.

[0230] At block 2502, information corresponding to a change in the status of a process is received. In some embodiments, the information can be one or more notifications, alerts, and / or one or more outputs from one or more functions, processes, applications, sensors, etc. In some embodiments, the sensors can include one or more health sensors, heart rate sensors, pedometers, thermal sensors, etc.

[0231] In some embodiments, the information is calculated based on data from one or more environmental factors or one or more applications. For example, in FIGS. 20A-20D, the device 202 can calculate the start or end of a process as described above. In some embodiments, the environmental factors can include data corresponding to weather, characteristics of one or more users or devices in the environment, etc.

[0232] Referring to FIG. 25, in blocks 2504 and 2506, in response to receiving information corresponding to the change status of the process and in accordance with the determination that the first criterion is satisfied, the first representation corresponding to the status of the process is displayed via the secondary display. For example, the representation can include one or more representations such as the representation status indicator 1702a1 of FIG. 17B, the colored overlay displayed on the entire secondary display 206 of FIG. 18, the indicator 1912 of FIG. 19B, the representation 2004 of FIGS. 20A-20D, and the representation 2302 of the progress indicator of FIGS. 23A-23B.

[0233] In some embodiments, the first criterion includes the criterion that is satisfied when it is determined that, as described above in connection with FIGS. 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, 14A-14B, 15, and 16, it is permitted to display the orientation information of the object via the secondary display.

[0234] Referring to FIG. 25, in blocks 2504 and 2506, in response to receiving information corresponding to the change in the status of the process and in accordance with the determination that the second criterion is satisfied, the second representation corresponding to the status of the process is displayed. For example, 810a-810l and 812a-812l of FIGS. 17A-17B, 18, 19A-19B, 20A-20D, 21A-21D, 22A-22B, 23A-23B, and 24A-24D.

[0235] In some embodiments, the second criterion includes the criterion that is satisfied when it is determined that, as described above in connection with FIGS. 8A-8B, 9A-9B, 10A-10B, 11A-11B, 12A-12B, 13A-13B, 14A-14B, 15, and 16, it is permitted to display the object via the tertiary display.

[0236] In some embodiments, the second representation is different from the first representation. In some embodiments, the first representation is a virtual object and the second representation is not a virtual object.

[0237] In some embodiments, one or more of the first representation and the second representation are associated with a third representation presented via a primary display (e.g., corresponding to one or more processes of the same application). For example, in FIGS. 20A-20D, both representations 2004 displayed via a secondary display and an LED (e.g., a tertiary display) are associated with the representation of time 2002 displayed via the primary display 204.

[0238] In some embodiments, while presenting the first representation, a second virtual object is displayed simultaneously with the first representation. In some embodiments, the first representation highlights at least a portion of a second virtual or physical object displayed via a secondary display. For example, in FIG. 19B, the indicator 1912 highlights (or brackets) the object 1910a displayed via the secondary display 206.

[0239] In some embodiments, one or more of the first representation and the second representation include a progress indicator. In some embodiments, the progress indicator is updated to indicate a change in a metric associated with the device, as described above with respect to the representation 2302 in FIGS. 23A-23B.

[0240] In some embodiments, the first representation includes an application icon. In some embodiments, the process is associated with an application that includes a status indicator that is updated based on the process. For example, in FIGS. 17A-17B, in response to receiving data related to a messaging application, a status indicator 1702a1 is displayed.

[0241] In some embodiments, one or more of the first representation and the second representation are displayed based on one or more sensors. In some embodiments, the one or more sensors include a heart rate sensor. In some embodiments, the received information corresponds to data detected via the heart rate sensor. In some embodiments, as part of displaying one or more of the first representation (e.g., 206 in FIG. 18) and the second representation based on at least one of the one or more sensors, an animation is displayed based on data detected via the heart rate sensor.

[0242] In some embodiments, the received information corresponds to an alert corresponding to an application. In some embodiments, one or more of the first representation and the second representation are presented based on an alert corresponding to an application, as described above in connection with, for example, FIG. 18.

[0243] In some embodiments, the received information corresponds to an alert corresponding to a calendar event. In some embodiments, one or more of the first representation and the second representation are presented based on an alert corresponding to a calendar event, as described above in connection with, for example, FIG. 18.

[0244] In some embodiments, the received information corresponds to data obtained via a face recognition algorithm. In some embodiments, one or more of the first representation and the second representation are presented based on data obtained via a face recognition algorithm, as described above in connection with the LEDs of FIGS. 22A and 22B, for example.

[0245] In some embodiments, the received information corresponds to the state of an application. In some embodiments, one or more of the first representation and the second representation are presented based on the state of the application. For example, in FIGS. 21A - 21D, the LED is updated based on whether a drag - and - drop input 2150 is detected.

[0246] In some embodiments, the received information corresponds to the detected posture of the user. In some embodiments, one or more of the first representation and the second representation are presented based on the detected posture of the user wearing the device. In some embodiments, the first representation is displayed based on the detected position of the user's eyes. For example, the representation can be a line representing the user's current posture or eye position. In some embodiments, in response to detecting a change in the user's posture or eye position, the line is updated to indicate the new posture (or alternatively, the change in posture) or eye position. In some embodiments, the line is displayed only via a secondary display. In some embodiments, the LEDs can emit light in various combinations to indicate the new posture (or alternatively, the change in posture) or eye position.

[0247] As described above, one aspect of the present technology is to collect and use data available from various sources to provide specialized resource management for a device having an additional display (e.g., a device having an additional display), save battery life for the user, and provide specialized content to the user of the device. The present disclosure contemplates that in some instances, this collected data may include personally identifiable information data that can be used to uniquely identify a particular person or to contact or locate a particular person. Such personally identifiable information data includes demographic data, location-based data, phone numbers, email addresses, Twitter (registered trademark) IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0248] The present disclosure recognizes that the use of such personal information data in the present technology can be used for the benefit of the user. For example, personal information data can be used to save the battery life of the user's device. Thus, for example, the use of such personal information data helps the system to manage resources appropriately to save the battery life of the device. Further, other uses of personal information data that provide benefits to the user are also contemplated by the present disclosure. For example, health and fitness data can be used to provide insights into the user's overall health or can be used as positive feedback to individuals pursuing health goals using technology.

[0249] This disclosure contemplates that entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data comply with established privacy policies and / or privacy practices. In particular, such entities should implement and consistently use privacy policies and practices that meet, or exceed, generally recognized industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be readily available to users and should be updated if the data collection and / or use is changed. Personal information from users should be collected for legitimate and reasonable use by the entity and should not be shared or sold outside of those legitimate uses. Further, such collection / sharing should occur after receiving informed consent from the user. Further, such entities should consider taking all necessary measures to protect and secure access to such personal information data and to ensure that others with access to the personal information data comply with the privacy policies and procedures. Further, such entities can be subject to third-party evaluations to demonstrate compliance with widely accepted privacy policies and practices. Further, the policies and practices should be tailored to the specific types of personal information data being collected and / or accessed and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be regulated by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Thus, different privacy practices need to be maintained for different personal data types for each country.

[0250] Notwithstanding the foregoing, the present disclosure also contemplates instances where a user selectively blocks the use or access to personal information data. That is, the present disclosure contemplates that it can provide hardware and / or software elements to prevent or block access to such personal information data. For example, when managing the resources of a low-power device, the technology can be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data either during or at any time after registration of the service. In another example, a user can select not to provide eye-tracking data such as pupil position, pupil dilation, and / or blink rate for dedicated resource management. In yet another example, a user can select to limit the length of time that eye-tracking data is maintained or to completely prohibit the development of a baseline eye-tracking profile. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notice regarding access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an app and may be notified again immediately prior to the personal information data being accessed by the app.

[0251] Furthermore, it is the intent of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. The risk can be minimized by restricting the collection of data and deleting data that has become unnecessary. Additionally, anonymization of data can be used to protect a user's privacy where applicable, including for certain health-related applications. Anonymization can be facilitated, as needed, by removing certain identifiers (e.g., date of birth, etc.), controlling the amount or specificity of the data stored (e.g., collecting location data at the city level rather than the address level), controlling how the data is stored (e.g., aggregating data across users), and / or other means.

[0252] Accordingly, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, the present disclosure also contemplates that the various examples can be implemented without the need to access such personal information data. That is, the various examples of the present technology will not become inoperable due to the absence of all or part of such personal information data. For example, it is possible to manage the resources of a low-power device and select and deliver content (e.g., status updates and / or objects) to a user by inferring preferences based on non-personal information data such as content requested by a device associated with the user, other non-personal information available to a system that controls the device, or publicly available information, or minimal personal information.

Claims

1. 1. A method performed in a system, comprising: The system includes one or more processors, a memory, and a head-mounted display (HMD) device including a first display, the first display including a first set of one or more lights and a second set of one or more lights arranged around a frame of the HMD device, the frame including a first surface separating the first set of one or more lights from the second set of one or more lights, the first set of one or more lights and the second set of one or more lights configured to be simultaneously within a field of view of a user wearing the HMD device; receiving information corresponding to a change in the state of the application; in response to receiving information corresponding to the change in the state of the application; displaying, via the first display of the HMD device, a first representation corresponding to the first state of the application in accordance with determining that the application changes to a first state; and displaying, via the first display of the HMD device, a second representation corresponding to the second state of the application, the second representation being different from the first representation, in accordance with a determination that the application changes to a second state different from the first state.

2. the first state corresponds to a state in which the application receives information that an object is at a first position relative to the HMD device; The method of claim 1 , wherein the second state corresponds to a state in which the application receives information that the object is at a second position relative to the HMD device, the second position being different from the first position.

3. the frame of the HMD device at least partially surrounds a display area; the first state corresponds to a state in which the application receives information that an object is present at a first distance outside the display area; The method of claim 1 or 2, wherein the second state corresponds to a state in which the application receives information that the object is at a second distance outside the display area, the second distance being different from the first distance.

4. The method of claim 3 , wherein the HMD device is configured to communicate with a second display different from the first display, and the display area includes the second display.

5. The method of any one of claims 1 to 4, wherein one or more of the first representation and the second representation includes a progress indicator.

6. The method of claim 5 , wherein the progress indicator is updated to indicate a change in a metric associated with the HMD device.

7. in response to receiving information corresponding to the change in the state of the application; in response to determining that the application changes to the first state; emitting light from the first set of one or more lights; emitting light from the second set of one or more lights; in response to determining that the application changes to the second state; The method of any one of claims 1 to 6, wherein light is emitted from the first set of one or more lights without emitting light from the second set of one or more lights.

8. the first representation includes a first color; The method of claim 7 , wherein the second representation includes a second color different from the first color.

9. The method of any one of claims 1 to 7, wherein the HMD device is configured to communicate with one or more sensors, and one or more of the first representation and the second representation are displayed based on information detected via the one or more sensors.

10. the one or more sensors include a heart rate sensor; the information detected via the one or more sensors corresponds to data detected via a heart rate sensor; The method of claim 9 , wherein one or more of the first representation and the second representation are displayed based on information detected via the heart rate sensor.

11. the information corresponds to an alert corresponding to the application; The method of any one of claims 1 to 10, wherein one or more of the first representation and the second representation are displayed based on an alert corresponding to the application.

12. the information corresponds to an alert corresponding to a calendar event; The method of any one of claims 1 to 11, wherein one or more of the first representation and the second representation are displayed based on an alert corresponding to the calendar event.

13. the information corresponds to data obtained via a facial recognition algorithm; The method of any one of claims 1 to 12, wherein one or more of the first representation and the second representation are displayed based on the data obtained via a facial recognition algorithm.

14. The method of any one of claims 1 to 13, wherein the application is a text messaging application, an email application, a phone application, a calendar application, a health application, a car sharing application, a taxi application, or a car location application.

15. the information corresponds to a detected posture of a user wearing an HMD device; The method of any one of claims 1 to 14, wherein one or more of the first representation and the second representation are displayed based on the detected posture of a user wearing the HMD device.

16. The method of any preceding claim, wherein the first representation is displayed based on a detected eye position of a user.

17. A computer program for causing a system to perform the method according to any one of claims 1 to 16, comprising: The system includes a head-mounted display (HMD) device having a first display; the first display includes a first set of one or more lights and a second set of one or more lights arranged around a frame of the HMD device; The frame includes a first surface separating the first set of one or more lights from the second set of one or more lights.

18. a memory for storing the computer program of claim 17; one or more processors capable of executing the computer programs stored in the memory; a head-mounted display (HMD) device, the HMD device comprises a first display; the first display includes a first set of one or more lights and a second set of one or more lights arranged around a frame of the HMD device; The system, wherein the frame comprises a first surface separating the first set of one or more lights from the second set of one or more lights.

19. 1. A system including a head-mounted display (HMD) device with a first display, the first display includes a first set of one or more lights and a second set of one or more lights arranged around a frame of the HMD device; the frame includes a first surface separating a first set of the one or more lights from a second set of the one or more lights, and a second surface separating one or more lights of the second set; The system comprises means for carrying out the method according to any one of claims 1 to 16.

Images