Head-mounted display with low light operation

Abstract

The present disclosure relates to head-mounted displays with low-light operation. A disclosed display system includes a controller and a head-mounted display. The head-mounted display includes a display, a head support coupled to the display for supporting the display on a head of a user for viewing by the user, and a sensor coupled to the head support for sensing an environment from the head-mounted display unit in low light. The sensor includes one or more of an infrared sensor for sensing the environment by infrared electromagnetic radiation, or a depth sensor for sensing a distance to an object of the environment, and further includes an ultrasonic sensor for sensing the environment by ultrasonic waves. The controller determines graphical content from the sensing of the environment by the one or more of the infrared sensor or the depth sensor, and by the ultrasonic sensor, and operates the display to provide the graphical content in parallel with the sensing of the environment.

Description

[0001] This application is a divisional application of patent application No. 202010295847.2, filed on April 15, 2020, entitled "Head-mounted Display with Low-Light Operation". Technical Field

[0002] This disclosure relates to display systems, and more specifically, to head-mounted display units and their operation. Background Technology

[0003] The human eye has varying sensitivities under different lighting conditions. Photopic vision is human vision with high levels of ambient light (e.g., brightness of approximately 10 to 10^8 cd / m^2), such as during the day. Photopic vision is provided by the cone cells of the eye, which provide sensitivity to different colors (i.e., wavelengths) of light. Scotopic vision is human vision with low levels of ambient light (e.g., brightness of approximately 10^-6 to 10^-3.5 cd / m^2), such as at night when the sky is overcast (e.g., without moonlight). Scotopic vision is provided by the rod cells of the eye. Mesovision is human vision with ambient light levels between photopic and scotopic (e.g., brightness of approximately 10^-3 to 10^0.5 cd / m^2), such as at night when the sky is not overcast (e.g., with moonlight) to early twilight. Mesovision is provided by both cone cells and rod cells. Compared to photopic vision, scotopic vision, or even mesopic vision, can lead to loss of color vision, altered sensitivity to different wavelengths of light, reduced acuity, and more severe motion blur. Therefore, under poor lighting conditions, such as when relying on scotopic vision, a person's ability to observe their environment is lower compared to well-lit conditions. Summary of the Invention

[0004] This document discloses specific implementations of a display system, including a head-mounted display unit and a method for providing content. In the specific implementation, the display system includes a controller and a head-mounted display unit. The head-mounted display unit includes a display, a head support coupled to the display for supporting the display on a user's head for viewing by the user, and sensors coupled to the head support for sensing the environment from the head-mounted display unit in low light. The sensors include one or more of an infrared sensor for sensing the environment via infrared electromagnetic radiation, or a depth sensor for detecting the distance to objects in the environment, and also include an ultrasonic sensor for sensing the environment via ultrasound. The controller determines graphic content based on the sensing of the environment via the infrared sensor or depth sensor and via the ultrasonic sensor, and operates the display to provide graphic content concurrently with the sensing of the environment.

[0005] In a specific implementation, the display system includes a controller and a head-mounted display unit. The head-mounted display unit includes a display for showing graphic content to a user wearing the unit and sensors for sensing the environment from the unit. These sensors include infrared sensors, depth sensors, ultrasonic sensors, and a visible light camera. Under high-light conditions, the sensors sense the environment to acquire first sensor data stored in a storage device. This first sensor data includes first visible light sensor data acquired by the visible light camera and first non-visible light sensor data acquired from one or more of the infrared, depth, or ultrasonic sensors. After the first sensor data is stored, under low-light conditions, the sensors sense the environment to acquire current sensor data, and the controller determines the graphic content based on the current sensor data and the first visible light sensor data.

[0006] In a specific implementation, a method for providing graphic content through a display system includes sensing the environment, processing sensor data, determining graphic content, and outputting the graphic content. The sensing includes acquiring sensor data in low light by sensing the environment through sensors. The sensors are coupled to a head-mounted display unit of the display system and include an infrared sensor, a depth sensor, and an ultrasonic sensor. The processing includes processing the sensor data through a controller. The controller performs the determination of the graphic content based on the processing. The graphic content includes an ultrasonic graphic component and one or more of the following: an infrared graphic component based on sensor data acquired by the infrared sensor, a depth graphic component based on sensor data acquired by the depth sensor, or a combined graphic component based on sensor data acquired by both the infrared sensor and the depth sensor. The output of the graphic content is performed through a display of the head-mounted display unit. Attached Figure Description

[0007] Figure 1 A top view of the head-mounted display unit on the user's head (H).

[0008] Figure 2 This is a schematic diagram of a head-mounted display unit.

[0009] Figure 3 A schematic diagram of an example hardware configuration for the controller of a head-mounted display unit.

[0010] Figure 4 This is a schematic diagram of the sensor in a head-mounted display unit.

[0011] Figure 5 To display a block diagram of the system.

[0012] Figure 6 For use through Figure 5 The flowchart of the process by which the display system provides graphical content.

[0013] Figure 7 To display a block diagram of the system.

[0014] Figure 8 For use through Figure 7 The flowchart of the process by which the display system determines the graphic content.

[0015] Figure 9 To display a block diagram of the system.

[0016] Figure 10 For use through Figure 9 A flowchart of another process by which the display system determines the graphic content.

[0017] Figure 11 To display a block diagram of the system.

[0018] Figure 12 For use through Figure 11 The display system provides another process flowchart for providing graphical content. Detailed Implementation

[0019] This document discloses an embodiment of a head-mounted display unit configured to operate in low-light conditions, such as at night and / or when a user might otherwise use dark vision or mesovision to view their environment. More specifically, the head-mounted display unit includes one or more sensors configured to observe the environment and / or detect objects in low-light conditions, said sensors including one or more of infrared sensors, depth sensors, and / or ultrasonic sensors. The head-mounted display unit provides content based on the sensors, which may include providing graphical content (e.g., displaying one or more of stored images, renderings of objects, and graphical indicators).

[0020] See Figure 1 and Figure 2The display system 100 includes a head-mounted display unit 102. The display system 100 can be configured to provide computer-generated reality, as described below. The display system 100 typically includes a head support 110, one or more displays 120, and one or more sensors 130 coupled to the head support 110, which cooperatively form the head-mounted display unit 102. The head support 110 may, for example, include a base 112 and a head engagement mechanism 114 coupled to the base 112. The one or more displays 120 and the one or more sensors 130 are coupled to the base 112, while the head engagement mechanism 114 engages a user's head H to support the displays 120 for displaying graphic content to the user's eyes. The one or more displays 120 may each be configured as a display panel (e.g., a liquid crystal display (LCD), a light-emitting diode display (LED), an organic light-emitting diode display (e.g., OLED)) or configured as a projector (e.g., which projects light onto a reflector back to the user's eyes), and may also be considered to include any associated optical components (e.g., lenses or reflectors). Sensor 130 is configured to sense the environment, and is referred to below. Figure 4 Let's have a discussion.

[0021] The display system 100 also includes a controller 140 and other electronic devices 150. The controller 140 and the other electronic devices 150 may be coupled to the head-mounted display unit 102 (e.g., coupled to a base), provided separately from and operatively connected to the head-mounted display unit 102 (e.g., wired or wirelessly connected to transmit signals and / or power between them), or partially coupled to the head-mounted display unit 102 (e.g., multiple components of the controller 140 and / or electronic devices 150 are coupled to the head-mounted display unit 102 and other components are operatively connected thereto). The controller 140 controls various operations of the display system 100, such as sensing various conditions via sensor 130 and providing content via display 120. An exemplary hardware configuration of the controller 140 is referenced below. Figure 3 The discussion will proceed. The other electronic devices 150 may include, for example, power electronic devices (e.g., batteries), communication devices (e.g., modems and / or radio components for wireless communication with other devices), and / or other output devices (e.g., speakers for auditory output, haptic devices for tactile output).

[0022] See Figure 3The controller 140 is a computing device configured to implement the systems and methods described herein. The controller 140 typically includes a processor 342, a memory 344, a storage device 346, a communication interface 348, and a bus 349 that allows communication between them. The processor 342 may be any suitable processor, such as a central processing unit (CPU) configured to execute instructions. The memory 344 is short-term volatile memory (e.g., a random access memory module). The storage device 346 is long-term non-volatile memory (e.g., a hard disk or solid-state drive) that stores software-programmed instructions executed by the processor 342. The communication interface 348 sends and receives signals to and from the controller 140, such as to and from other electronic components of the display system 100 (e.g., a display 120, a sensor 130, and / or other electronic devices 150). Although the controller 140 is illustrated as a single device, its various components may be provided in any suitable manner, and the controller 140 may include fewer and / or more components. For example, controller 140 may be considered as including processing devices or other electronic hardware specifically associated with each sensor.

[0023] See Figure 4 Sensor 130 includes one or more infrared sensors 432, one or more depth camera sensors 434, and / or one or more ultrasonic sensors 436, which are oriented outward from the head-mounted display unit 102 to observe the environment E. Sensor 130 also includes one or more visible light cameras 438 oriented outward from the head-mounted display unit 102 to observe the environment, and one or more motion sensors 440 to detect the position, orientation, and / or movement of the head-mounted display unit 102.

[0024] Infrared sensor 432 detects infrared light in the environment. Infrared sensor 432 can be any suitable type of infrared sensor for detecting electromagnetic radiation in the infrared frequency range. In one example, infrared sensor 432 is an infrared camera that uses an image sensor (such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor) and other suitable hardware components (e.g., an image processor) to capture images (e.g., video) in the infrared frequency range. The images captured by infrared sensor 432 may be referred to as IR images. Infrared sensor 432 can be a passive sensor for observing infrared radiation emitted or reflected by objects in the environment. Alternatively, infrared sensor 432 may include an infrared illuminator that emits electromagnetic radiation in the infrared frequency range, thereby illuminating the environment and objects therein.

[0025] (For example, after processing) Information about the environment acquired and / or derived from infrared sensor 432 may be referred to as infrared sensor data 552. As discussed in further detail below, infrared sensor data 552 may be processed, stored, and / or used to determine graphic content in various ways.

[0026] Depth sensor 434 detects the environment, and more specifically, detects the depth (e.g., distance) of objects in the environment from it. Depth sensor 434 typically includes an illuminator 434a and a detector 434b. Illuminator 434a emits electromagnetic radiation (e.g., infrared light) from head-mounted display unit 102 into the environment. Detector 434b observes the electromagnetic radiation reflected from objects in the environment. In two specific examples, depth sensor 434 is a depth camera using structured light or time-of-flight. In the case where depth sensor 434 is a structured light sensor, illuminator 434a projects a pattern of electromagnetic radiation in the infrared frequency range (e.g., a grid or array of infrared dots, such as tens of thousands of dots), while detector 434b is a camera that captures an image of the pattern of infrared light reflected by objects in the environment, which may be referred to as a structured light image. The structured light image is then (e.g., by controller 140) analyzed to determine the depth (e.g., distance) from depth sensor 434 to an object (or a point on it) in the environment, for example, by analyzing the deformation of the light pattern reflected from the object. Detector 434b may be a camera, including an image sensor such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor, and other suitable hardware components (e.g., an image processor). In the case where depth sensor 434 is a time-of-flight camera, illuminator 434a projects electromagnetic radiation, and detector 434b is a suitable camera that measures the time of flight of the electromagnetic radiation emitted from illuminator 434a and reflected from the object to determine its distance. Depth sensor 434 may operate in a frequency range of an electromagnetic radiation spectrum different from that of infrared sensor 432, so as not to detect the electromagnetic radiation of the other or otherwise be sensitive to it (e.g., using suitable filters, camera image sensors, and / or illuminators, and projector 434a within a suitable frequency range). In other examples, depth sensor 434 may be a radar-detected ranging sensor (RADAR) or a light-detected ranging sensor (LIDAR). It should be noted that one or more types of depth sensors 434 may be used, such as combining one or more of structured light sensors, time-of-flight cameras, RADAR sensors, and / or LIDAR sensors. In a preferred embodiment, the depth sensor 434 includes only a structured light sensor.

[0027] (For example, after processing) the information about the environment acquired and / or derived from depth sensor 434 may be referred to as depth sensor data 554. As discussed in further detail below, depth sensor data 554 may be processed, stored, and / or used to determine graphic content in various ways.

[0028] Ultrasonic sensor 436 utilizes ultrasonic waves to detect the environment. Ultrasonic sensor 436 may, for example, include an ultrasonic transmitter that transmits ultrasonic waves and an ultrasonic receiver that detects those ultrasonic waves reflected by physical objects in the environment, or alternatively, an ultrasonic transceiver may be used and included to perform the functions of both the ultrasonic transmitter and the ultrasonic receiver. The ultrasonic waves are then processed, such as by controller 140 or another suitable processor, to identify and / or locate features of the environment. Advantageously, ultrasonic sensor 436 can detect objects that cannot be otherwise observed by infrared sensor 432 and / or depth sensor 434, such as shards of glass (e.g., shards of glass in windows or doors).

[0029] (For example, after processing) Information about the environment acquired and / or derived from the ultrasonic sensor 436 may be referred to as ultrasonic sensor data 556. As discussed in further detail below, the ultrasonic sensor data 556 may be processed, stored, and / or used to determine graphic content in various ways.

[0030] The one or more visible light cameras 438 detect visible light in the environment. The visible light camera 438 includes an image sensor such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor, as well as other suitable hardware components (e.g., an image processor) and optical components (e.g., a lens).

[0031] (For example, after processing) Information about the environment acquired and / or derived from the visible light camera 438 may be referred to as visible light sensor data 558. Visible light sensor data 558 may include images (e.g., video), which may be referred to as visible light images. As discussed in further detail below, visible light sensor data 558 may be processed, stored, and / or used to determine graphic content in various ways.

[0032] One or more (e.g., two) of each of the infrared sensor 432, depth sensor 434, ultrasonic sensor 436, and / or visible light camera 438 may be coupled to the head-mounted display unit 102, for example, to stereoscopically sense the environment E from the head-mounted display unit 102. The one or more infrared sensors 432, depth sensor 434, ultrasonic sensor 436, and / or visible light camera 438 may provide the same or different fields of view, which in... Figure 1The term is typically indicated by a dashed arrow radiating from sensor 130. In the systems and processes discussed below, various types of sensors are mentioned singularly, but it should be understood that the systems and processes can be applied to multiple such sensors (e.g., to stereoscopically determine and provide graphic content).

[0033] The one or more motion sensors 440 detect the position, orientation, and / or motion of the head-mounted display unit 102. The one or more motion sensors 440 may include, for example, a global positioning system sensor (e.g., GPS), one or more accelerometers, one or more gyroscopes, and / or an inertial measurement unit (IMU) for determining the position, orientation, and / or motion of the head-mounted display unit 102.

[0034] (For example, after processing) the information acquired and / or derived from motion sensor 440 may be referred to as motion sensor data 560. As discussed in further detail below, motion sensor data 560 may be processed, stored, and / or used to determine graphic content in various ways.

[0035] See Figure 5 The display system 100 outputs graphic content to the user, enhancing the user's ability to see in low-light conditions. Low-light conditions may be defined, for example, based on ambient light conditions (e.g., brightness below 10^-3.5 cd / m^2) and / or the type of human vision used otherwise to directly observe the environment (e.g., when scotopic vision is primarily or substantially solely used by humans). Graphic content may include, for example, renderings of objects that enhance the environment, detected objects, or other indicators, and / or images with contrast between captured images. Rendering is a computer-generated graphic representation of the detected objects generated based on sensor information. Rendering may be an accurate representation of the detected objects (e.g., with corresponding texture, color, size, etc.), a representational representation that alters various features of the detected objects (e.g., becoming an "animated" form with uniform and / or different colors, different textures, contrasting contours), or a highlighting rendering (e.g., overlaying objects). As an exemplary and non-limiting example of representational representation, a wooden table may have an accurate representation depicting the variations in color and grain of the wood and emphasizing shadows on different surfaces, or a representational representation describing uniform color and black contours between surfaces. For example, highlighting a table could involve highlighting it in green by providing a colored, semi-transparent outline over other graphic content on the table. The highlight could be of the environment (e.g., objects or structures that define the environment, such as the walls of a room) or objects within the environment.

[0036] Infrared sensor 432, depth sensor 434, ultrasonic sensor 436, and visible light camera 438 sense the environment E from head-mounted display unit 102, while motion sensor 440 senses the position, orientation, and / or motion of head-mounted display unit 102 on the user's head. Sensor data (including infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, visible light sensor data 558, and / or motion sensor data 560) are transmitted from the respective sensors to controller 140 via one or more sensor signals 550.

[0037] Controller 140 receives sensor signals 550, processes the sensor data received therefrom, determines graphic content based on the sensor data, and then sends one or more signals to display 120, which may be referred to as display signals 570. Display 120 receives display signals 570 and outputs the graphic content. As described above, controller 140 may include one or more local or distributed processing devices, such as processors that may be associated with different sensors 130, for processing sensor data and / or determining graphic content. However, the processing and / or determination may be performed by different processors and / or different controllers.

[0038] See Figure 6 Process 600 provides graphical content via a display system including head-mounted display units, such as display system 100 and head-mounted display unit 102. Process 600 typically includes sensing the environment in low light 610 via sensors to generate sensor data, processing the sensor data 620, determining the graphical content 630, and outputting the graphical content 640. Process 600 may also include another operation 615 of sensing motion of the head-mounted display unit.

[0039] Sensing the environment 610 to generate sensor data is performed by sensors (such as sensor 130) facing outwards from a head-mounted display unit (such as head-mounted display unit 102). Sensors include one or more of, for example, an infrared sensor 432, a depth sensor 434, an ultrasonic sensor 436, and / or a visible light camera 438. Sensor data (such as infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, visible light sensor data 558, and / or motion sensor data 560) is transmitted to a processor, such as processor 342 of controller 140, via one or more signals (such as sensor signal 550). Different combinations of acquiring and / or transmitting sensor data are possible. For example, the visible light camera 448 and / or motion sensor 450 may be omitted and / or not operated.

[0040] The processing of sensor data 620 is performed by a processor, for example, by the processor 342 of the controller 140 and / or other processing devices specifically associated with one or more of the sensors (e.g., an image processor associated with a camera). The processing of sensor data can be performed in various ways, as described below. Figures 7 to 12 The systems and processes discussed in more detail are as follows.

[0041] The determination of the graphic content 630 is performed, for example, by a processor, such as processor 342 of controller 140 and / or other processing devices specifically associated with display 120. The determination of the graphic content can be performed in various ways, as described below. Figures 7 to 12 The system and processes are discussed in more detail below. Processing 620 and determining 630 can be performed as a single operation, for example, when simply converting infrared light into visible light. Graphic content can be transmitted, for example, via display signal 570.

[0042] The output of graphical content 640 (e.g., display) is performed by one or more displays, such as display 120, based on display signals 570 received from controller 140. Output 640 is performed substantially in parallel with sensing 610, such that the graphical content is displayed in parallel (e.g., in real-time) with the environment sensed by the head-mounted display. Parallel or substantially parallel should be understood to take into account the latency associated with sensing, processing, determining, and / or transmitting operations of display system 100.

[0043] Repeated sensing 610, processing 620, determining 630, and outputting 640 are performed to provide graphical content to the user as an image stream (e.g., a video stream).

[0044] See Figure 7 and Figure 8The display system 100 processes sensor data from each sensor independently and can further determine graphic content components independently. For example, the display system 100 can display graphic content including an infrared graphic component 772 based on infrared sensor data 552 from infrared sensor 432, a depth graphic component 774 based on depth sensor data 554 from depth sensor 434, an ultrasonic graphic component 776 based on ultrasonic sensor data 556 from ultrasonic sensor 436, and / or a visible light graphic component 778 based on visible light sensor data 558 from visible light camera 438. The infrared graphic component 772, depth graphic component 774, ultrasonic graphic component 776, and / or visible light graphic component 778 can be displayed simultaneously and in parallel with the detection of associated information, for example, overlapping each other and having appropriate transparency for viewing each component. The graphic content is displayed substantially in parallel with its sensing, allowing the user to observe the environment in real time via the graphic content. Auditory content (e.g., output from a speaker of the other electronics 350) may also be provided, for example, based on an ultrasonic graphic component 776 based on ultrasonic sensor data 556 from ultrasonic sensor 436 to indicate distance to an object.

[0045] Infrared sensor data 552 (such as an IR image captured by infrared sensor 432) can be processed in various ways to determine graphic content. In one example, infrared sensor data 552 is processed to convert the infrared image from infrared light into visible light, which forms the infrared graphic component 772. Alternatively, infrared sensor data 552 is processed to enhance the contrast of the infrared image. Alternatively, infrared sensor data 552 is processed to detect objects in the environment using, for example, suitable computer vision or other object detection programming or algorithms. Detection may include locating, characterizing, and / or identifying objects, or other object recognition functions relevant to the environment. Locating typically refers to determining the position of an object or its features in a physical coordinate system and / or relative to the head-mounted display unit 102. Characterizing typically refers to determining the characteristics of a physical object, such as size, shape, and / or color. Identifying typically refers to identifying the type of object (e.g., door, wall, chair) or uniquely identifying a specific object (e.g., a door in a room of a house).

[0046] The infrared graphic component 772 of the graphic content may, for example, include an IR image converted to visible light and / or with enhancement. Alternatively or otherwise, the infrared graphic component 772 may include other graphics, such as a rendering generated based on infrared sensor data 552.

[0047] Depth sensor data 554 (such as structured light images captured by depth sensor 434) can be processed in various ways to determine graphic content, and more specifically, to determine the distance from depth sensor 434 on head-mounted display unit 102 to a location (e.g., a point) on an object in the environment. Such distances can be represented by a depth map, which is a mathematical and / or visual representation of such distances. The depth map or other information derived from depth sensor 434 can be further processed (e.g., analyzed) to detect (e.g., locate, characterize, and / or identify) objects in the environment. Depth maps or other depth information can also be processed to determine the relative position, orientation, and / or motion of head-mounted display unit 102 relative to the environment and / or the relative position, orientation, and / or motion of objects in the environment.

[0048] The depth graphics component 774 of the graphic content may include, for example, a structured light image. Alternatively or otherwise, the depth graphics component 774 includes rendering determined (e.g., generated) based on the depth sensor data 554, such as rendering of the depth map itself, rendering of the environment, rendering of objects therein, and / or rendering for highlighting detected objects.

[0049] The ultrasonic sensor data 556 detected by the ultrasonic sensor 436 is processed to determine the graphics or other content output by the head-mounted display unit 102. Specifically, the ultrasonic sensor data 556 is processed to detect (e.g., locate, characterize, and / or identify) objects in the environment, such as by determining the distance from the head-mounted display unit 102 (e.g., its ultrasonic sensor 436) to objects in the environment and / or the relative motion between them.

[0050] The ultrasonic graphic component 776 of the graphic content may, for example, include a graphical representation of the distance to the detected object or a graphical representation of the detected object (e.g., a glass panel or wall). The ultrasonic graphic component 776 of the graphic content may, conversely, include another graphical indicator, such as a numerical or color indicator, indicating the distance to the object. Alternatively or in addition, auditory ultrasonic indicators, such as speech indicators or sound indicators, may be provided to indicate the type and / or distance to the object.

[0051] Visible light sensor data 558 detected by visible light camera 438 (e.g., visible light image) can be processed in various ways to determine graphic content. In one example, the visible image can be processed in a suitable manner for display to a user. Alternatively or otherwise, the visible image can be processed to enhance contrast. Alternatively or otherwise, the visible light sensor data can be processed to detect (e.g., locate, characterize, and / or identify) physical objects in the environment, which may include physical features of the environment and / or physical objects in the environment, such as through object recognition programming (e.g., computer vision software).

[0052] The visible light graphics component 778 of the graphic content may include, for example, a visible light image and / or rendering determined based on visible light information, such as the rendering of the environment and / or the rendering of objects therein. The visible light graphics component 778 may also be omitted under low-light conditions.

[0053] When determining the graphic content, the controller 140 causes the display 120 to output the graphic content. For example, as described above, the controller 140 sends a display signal 570 to the display 120. The display signal 570 may include each component of the graphic content, such as infrared graphic component 772, depth graphic component 774, ultrasonic graphic component 776, and / or visible light graphic component 778, which may form a combined or parallel data stream sent to the display 120. As described above, some of the graphic components may be omitted; for example, the visible light graphic component 778 may be omitted under low-light conditions.

[0054] See Figure 8 A process 800 is provided for processing sensor data and determining graphic content based thereon. This process can be performed as part of method 600 to process sensor data 620 and determine graphic content based thereon 630. Therefore, process 800 can be performed before and after ambient sensing 610 and graphic content output 640. Process 800 typically includes processing operations 822, 824, 826, and / or 828 for processing infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and visible light sensor data 558, respectively. Process 800 also typically includes graphic content determination operations 832, 834, 836, and 838 for determining infrared graphic components 572, depth graphic components 574, ultrasonic graphic components 576, and visible light graphic components 578, respectively. Process 800 can be performed, for example, under low-light conditions.

[0055] The processing 822 of the infrared sensor data 552 is performed by a processor such as processor 342 of controller 140. As described above relative to... Figure 7 The infrared sensor data 552 can be processed to convert the infrared sensor data 552 into visible light, enhance contrast, and / or detect (e.g., locate, characterize, and / or identify) objects in the environment.

[0056] The determination 832 of the infrared graphic component 772 is performed by a processor (such as processor 342 of controller 140) based on the processing 822 of the infrared sensor data 552. As described above, the infrared graphic component 772 may include a visible light image converted from an infrared image, an image with enhanced contrast, and / or a rendering of a detected object or a rendering associated with a detected object. The infrared graphic component 772 is then sent to a display 120 for output therefrom, such as via display signal 570.

[0057] The processing 824 of depth sensor data 554 is performed by a processor such as processor 342 of controller 140. (As mentioned above...) Figure 7 The device can process depth sensor data 554, for example, to generate a depth map to detect (e.g., locate, characterize, and / or identify) objects in the environment, and / or to determine the relative position, orientation, and / or motion of the head-mounted display unit 102 relative to the environment and / or the relative position, orientation, and / or motion of objects in the environment.

[0058] The determination 834 of the depth graphics component 774 is performed by a processor (such as processor 342 of controller 140) based on the processing of depth sensor data 554. As described above, the depth graphics component 774 may include the rendering of a structured light image, a depth map, and / or the rendering of detected objects or rendering associated with detected objects. The depth graphics component 874 is then sent to display 120 for output therefrom, such as via display signal 570.

[0059] The processing 826 of the ultrasonic sensor data 556 is performed by a processor such as processor 342 of controller 140. As described above relative to... Figure 7 The device can process ultrasonic sensor data 556, for example, to detect (e.g., locate, characterize, and / or identify) objects in the environment.

[0060] The determination 836 of the ultrasonic graphic component 776 is performed by a processor (such as processor 342 of controller 140) based on the processing of ultrasonic sensor data 556. As described above, the ultrasonic graphic content may include a graphical representation of the detected object (e.g., a wall or glass panel) and / or alphanumeric and / or color indicators of the distance to the head-mounted display unit 102 (e.g., ultrasonic sensor 436) and the object. The ultrasonic graphic component 776 is then sent to display 120 for output by it, such as via display signal 570.

[0061] The processing 828 of the visible light sensor data 558 is performed by a processor such as processor 342 of controller 140. As described above relative to... Figure 7 The device can process visible light sensor data 558, for example, to enhance contrast and / or locate and / or characterize objects in the environment.

[0062] The determination 838 of the visible light graphic component 778 is performed by a processor (such as processor 342 of controller 140) based on the processing of visible light sensor data 558. As described above, the visible light graphic component 778 may include a visible light image, an image with enhanced contrast, and / or the rendering of objects in the environment. The visible light graphic component 778 is then sent to display 120 for output therefrom, such as via display signal 570.

[0063] When determining each component of the graphic content, layering and / or transparency may also be determined. For example, ultrasound graphic component 776 may be highly transparent and cover other graphic components.

[0064] See Figure 9 and Figure 10 The display system 100 processes sensor data from two or more of the sensors collaboratively and can further collaboratively determine graphic content components. For example, the display system 100 can display graphic content including a combined graphic component 971, which may be based on a combination of infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558. The display system 100 can simultaneously display an ultrasonic graphic component 776 (based on ultrasonic sensor data 556) and the combined graphic component 971, for example, the ultrasonic graphic component 776 being overlaid by the combined graphic component.

[0065] Infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558 can be processed collaboratively in any suitable manner. For example, infrared sensor data 552, depth sensor data 554, and visible light sensor data 558 can first be processed independently, such as to detect (e.g., locate, characterize, and / or identify) objects in the environment (e.g., using computer vision or other suitable object detection programming) and / or generate a depth map in the presence of depth sensor data 554. The location, characterization, and / or identification of objects in the independently determined environment can then be processed collaboratively, such as according to a suitable sensor fusion algorithm, to provide more accurate and / or more complete data than can be obtained individually from the sensors. As another example, infrared images, depth maps, and visible light images can be processed collaboratively (e.g., by sensor fusion algorithms) and subsequently evaluated to detect (e.g., locate, identify, and / or characterize) objects in the environment. In a specific example, infrared sensor data 552 and depth sensor data 554 are processed collaboratively without visible light sensor data 558.

[0066] Based on the collaborative processing of infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558, a combined graphics component 971 is determined. For example, the combined graphics component 971 may include the rendering of objects in the environment. The combined graphics component 971 is then sent to a display 120 for output therefrom, such as via a display signal 570.

[0067] The ultrasonic sensor data 556 can be processed in the manner described above and the ultrasonic pattern components 776 can be determined.

[0068] See Figure 10A process 1000 is provided for processing sensor data and determining graphic content based thereon. This process can be performed as part of process 600 to process sensor data 620 and determine graphic content based thereon 630. Therefore, process 1000 can be performed before and after environmental sensing 610 and the output of graphic content 640. Process 1000 typically includes a combined processing 1021 of sensor data and a determination 1031 of combined graphic content based thereon. Process 1000 may optionally also include processing ultrasonic sensor data 556 826 and determining ultrasonic graphic components 776 based thereon 836, as previously described. Although not shown, process 1000 may additionally include separate processes 822, 824, 828 and determinations 832, 834, 838 associated with one or more of infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558. Process 1000 can be performed, for example, under low-light conditions.

[0069] The processing 1021 of a combination of two or more of the infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558 is performed by a processor such as processor 342 of controller 140. (As mentioned above...) Figure 9 The combined processing 1021 includes processing two or more of the infrared sensor data 552, depth sensor data 554, and / or visible light sensor data 558 in a collaborative manner, such as through a sensor fusion algorithm.

[0070] The determination 1031 of the combined graphic component 971 is performed by a processor (such as processor 342 of controller 140) based on the combined sensor data. (Refer to the above) Figure 9 The combined graphics component 971 may include the rendering of objects in the environment.

[0071] The ultrasonic sensor data 556 can be processed in the manner described above and the ultrasonic pattern components 776 can be determined.

[0072] See Figure 11 and Figure 12 The display system 100 senses the environment E and stores sensor data, and then determines graphic content based on the previously stored sensor data. The display system 100 may store visible light sensor data 558 captured in the environment under bright conditions (brightness is ambient light with a luminance higher than 10 cd / m²), and then determine graphic content based on the stored visible light sensor data 558 under low-light conditions, for example, displaying a visible light image, portions thereof, and / or a rendering derived therefrom. For example, a visible light image may be captured and stored when a user previously moves through the environment.

[0073] At one or more prior times (e.g., a first time) under high-light conditions, the display system 100 senses the environment via sensors 130, including a visible light camera 438 and one or more of an infrared sensor 432, a depth sensor 434, an ultrasonic sensor 436, and / or a motion sensor 440. Visible light sensor data 558 is stored in association with infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560, which are captured in parallel therewith.

[0074] Visible light sensor data 558 may be stored in a generally unprocessed format (e.g., as a raw image) and / or after processing, such as being formatted for later display and / or object identification of the environment and / or objects therein (e.g., locating, characterizing and / or identifying such objects or their features).

[0075] Other sensor data (including one or more of infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560) may be processed and stored in association with visible light sensor data 558 detected in parallel with it. Infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560 may be processed and stored individually and / or in combination (e.g., as described above in processes 800 and 1000, such as using sensor fusion). In a specific example, infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560 are processed to determine position information (e.g., position and / or orientation) of the head-mounted display unit 102 associated with visible light sensor data 558 and / or other sensor data (e.g., infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560). In other examples, the location, characteristics, and / or identity of objects and / or features in the storage environment may be referred to as object information.

[0076] In one or more subsequent times (e.g., the current time) under low-light conditions, the display system 100 senses the environment via one or more of infrared sensors 432, depth sensors 434, and / or ultrasonic sensors 436, and / or motion sensors 440 senses the position, orientation, and / or motion of the head-mounted display unit 102. Sensor data (e.g., infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560) is processed and compared with sensor data from previous times, for example, processed in the same or similar manner as previous times (e.g., individually and / or collaboratively) for appropriate comparison. Based on the comparison, visible light sensor data 558 may be invoked and graphic content may be determined at least in part based on it. In a specific example, current location information is compared with stored location information to invoke visible light sensor data 558 associated with stored location information that matches the current location or is otherwise agreed upon for comparison with the current location. In other examples, object information from the current time can be compared with object information from a previous time to retrieve visible light sensor data 558.

[0077] See Figure 12 Process 1200 provides graphical content through a display system including head-mounted display units, such as display system 100 and head-mounted display unit 102. Process 1200 typically includes sensing the environment via sensors at a previous time 1210, processing sensor data from the previous time 1220, storing the previous sensor data 1230, sensing the environment via sensors at the current time 1240, processing sensor data from the current time 1250, comparing the current sensor data with the previous sensor data 1260, recalling previous visible light sensor data 1270, determining graphical content based on the previous visible light sensor data 1280, and outputting the graphical content at the current time 1290. Parts of process 1200 may be performed under high-light conditions (e.g., sensing 1210), while other parts of process 1200 may be performed under low-light conditions (e.g., sensing 1240 to output 1290) and repeated to provide continuous content to the user.

[0078] The sensing 1210 of the previous time is performed by sensor 130 and can be performed as described above for sensing 610 of process 600. Sensing 1210 is performed in the previous time and under high light conditions.

[0079] Processing 1220 is executed by a processor 342, such as controller 140, and may include parallel operations for processing visible light sensor data 558 and the other sensor data (i.e., non-visible light sensor data, including infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, and / or motion sensor data 560). Processing 1220 of visible light sensor data 558 may be executed as described above, for example, by processing 828 of visible light sensor data 558 in process 800. Processing 1220 of non-visible light sensor data may be executed independently (e.g., by processes 822, 824, and / or 826 in process 800) and / or collaboratively (e.g., by process 1021 in process 1000) for different types of sensor data.

[0080] Storage 1230 is performed via a storage device (such as storage device 346 of controller 140 or other suitable storage device). Storage 1230 includes storing visible light sensor data 558 (before and after processing), such as visible light images, in association with non-visible light sensor data (before and / or after processing), and / or storing the visible light sensor data 558 (before and / or after processing) for association with non-visible light sensor data (before and / or after processing). The stored visible light sensor data 558 may be referred to as stored visible light sensor data. The stored non-visible light sensor data may be referred to as stored non-visible light sensor data, which may include location and / or object information. The stored visible light sensor data and the stored non-visible light sensor data are stored and / or stored in association with each other, such that the stored visible light sensor data can be retrieved by identifying the stored non-visible light sensor data associated with it. For example, visible light sensor data 558 may be stored in a public location (e.g., a database) and / or in a different location (e.g., a different database) with public metadata (e.g., timestamps) suitable for association and retrieval, in association with non-visible light sensor data.

[0081] Sensing of the current time 1240 is performed by sensors such as infrared sensor 432, depth sensor 434, ultrasonic sensor 436 and / or motion sensor 440. Sensing 1240 may also include sensing by visible light camera 438.

[0082] Processing 1250 of the sensor data derived therefrom (e.g., infrared sensor data 552, depth sensor data 554, ultrasonic sensor data 556, visible light camera data 438, and / or motion sensor data 560) is performed by a processor such as processor 342 of controller 140. Processing 1250 may be performed in the same or similar manner as the aforementioned processing 1220, such that the current non-visible sensor data can be compared with stored non-visible sensor data.

[0083] A comparison 1260 between the current non-visible sensor data and the stored non-visible sensor data is performed by a processor 342, such as a controller 140.

[0084] The retrieval 1270 of the stored visible light sensor data 558 is performed via a processor 342 of a processor such as controller 140 and a storage device such as storage device 346 of controller 140. For example, when the current non-visible light sensor data matches or is concurringly compared to (e.g., closest to) the stored non-visible light sensor data, the visible light sensor data 558 associated with the stored non-visible light sensor data is retrieved (e.g., retrieved from the storage device).

[0085] The determination of the graphic content 1280 is performed by a processor 342, such as the controller 140. The graphic content includes at least one of visible light graphic components 778 and / or combined components 1279, each determined based on stored visible light sensor data 558 invoked in call 1270. For example, the graphic content may include a visible light image, a visible light image with enhanced contrast, or a portion thereof, or a rendering otherwise derived from the stored visible light sensor data 558. The graphic content may include other individual or combined components, as previously described.

[0086] Process 1200 may optionally also include processing 826 ultrasonic sensor data 556 and determining 836 ultrasonic pattern components 776 based thereon, as previously described.

[0087] The output of the graphical content 1290 is performed via the display 120, for example, based on a display signal 570 containing the graphical content received from a processor (e.g., controller 140).

[0088] The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0089] Conversely, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of characteristics of virtual objects in a CGR environment can be done in response to a representation of physical motion (e.g., a voice command).

[0090] Humans can use any of their senses to sense and / or interact with CGR objects, including sight, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, audio objects can enable audio transparency, which selectively introduces ambient sound from the physical environment, with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects.

[0091] Examples of CGR include virtual reality and mixed reality.

[0092] A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. Examples of virtual objects include trees, buildings, and computer-generated images representing human avatars. A person can sense and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment, and / or through a simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0093] Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments are simulated environments designed to incorporate sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between a purely physical environment as one end and a virtual reality environment as the other end, but not including either end.

[0094] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment, enabling virtual objects to interact with real objects (i.e., physical objects or representations of them from the physical environment). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0095] Examples of mixed reality include augmented reality and augmented virtual reality.

[0096] Augmented reality (AR) environments are simulated environments in which one or more virtual objects are overlaid on a physical environment or its representation. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via images or videos of the physical environment and perceive the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto a physical environment, such as as a hologram or on a physical surface, so that a person can use the system to perceive the virtual objects superimposed on the physical environment.

[0097] Augmented reality environments also refer to simulated environments where the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, a system can transform images from one or more sensors to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. Alternatively, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portion is a representative but not realistic version of the original captured image. Furthermore, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0098] Augmented virtual (AV) environments are simulated environments in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more features of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could adopt shadows that correspond to the sun's position within the physical environment.

[0099] Many different types of electronic systems enable people to sense and / or interact with a variety of CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0100] As described above, one aspect of this technology involves collecting and using data from various sources to provide graphical content to users based on the current physical environment. This disclosure anticipates that, in some instances, the collected data may include personal information that uniquely identifies or can be used to contact or locate specific individuals. Such personal information may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0101] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be beneficial to users. For example, personal information data can be used to deliver graphic content to users relating to their current and past movements. Furthermore, this disclosure also contemplates other uses of personal information data that are beneficial to users. For example, health and fitness data can be used to provide insights into a user's overall health status or as positive feedback for individuals using the technology to pursue health goals.

[0102] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Additionally, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0103] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, when providing graphic content, this technology can be configured to allow users to opt-in or opt-out at any time during or after registering for the service. As another example, users may choose not to store sensor data. In addition to providing "opt-in" and "opt-out" options, this disclosure envisions providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0104] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0105] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, graphic content may be based on non-personal information data or a minimal amount of personal information, such as using current sensor data to provide graphic content.

Claims

1. A display system, comprising: Controller; and A head-mounted display unit includes a display, a head support coupled to the display, a visible light camera coupled to the head-mounted display unit for sensing the environment via visible light in high light conditions, and a sensor coupled to the head support. The head support is used to support the display on a user's head for viewing by the user. The sensor is used to sense the environment from the head-mounted display unit in low light conditions, and the sensor includes: One or more of the following: an infrared sensor for sensing the environment by infrared electromagnetic radiation, a depth sensor for sensing the distance to a physical object in the environment, or an ultrasonic sensor, wherein the ultrasonic sensor transmits ultrasonic waves and detects the ultrasonic waves reflected by a physical object in the environment to sense the physical object in the environment. In high-light conditions, sensor data obtained by one or more of the infrared sensor, the depth sensor, or the ultrasonic sensor is stored in association with a visible light image of the environment from the visible light camera. In low-light conditions, the controller retrieves one or more of the visible light images of the environment from the memory of the display system. The one or more sensors in the visible light image are determined by comparing the stored sensor data with sensor data obtained by sensing the environment through one or more of the infrared sensor, the depth sensor, or the ultrasonic sensor. The controller operates the display to provide graphic content of one or more of the visible light images in parallel with the sensing of the environment, and The graphic content includes composite graphic components, which include rendering of the environment based on the visible light image and are collaboratively determined based on sensing of the environment by the visible light camera, the infrared sensor, and the depth sensor.

2. The display system according to claim 1, wherein the visible light image is further determined based on the current position of the head-mounted display unit.

3. The display system according to claim 1, wherein the graphic content includes an infrared graphic component determined by the infrared sensor, a depth graphic component determined by the depth sensor, and an ultrasonic graphic component determined by the ultrasonic sensor.

4. The display system according to claim 3, wherein the infrared graphic component, the depth graphic component, and the ultrasonic graphic component are determined independently of each other.

5. The display system of claim 1, wherein the visible light camera is configured to sense the environment in high light by visible light before sensing the environment in low light, wherein visible light sensor data obtained from the visible light camera is stored in the memory, and wherein the controller determines the combined graphic components based on the visible light sensor data in the memory.

6. The display system of claim 5, wherein the depth sensor is a structured light sensor that senses the environment via structured light.

7. The display system of claim 5, wherein the visible light sensor data includes the visible light image, and the graphic content includes the visible light image.

8. The display system according to claim 5, wherein, Non-visible light sensor data obtained from the infrared sensor, the depth sensor, and the ultrasonic sensor in high light are stored in association with the visible light sensor data obtained in high light.

9. The display system according to claim 1, wherein the low light is ambient light with a brightness of less than 10^-3.5 cd / m^2.

10. A display system, comprising: Controller; and A head-mounted display unit having a display and sensors, the display for displaying graphic content to a user wearing the head-mounted display unit, and the sensors for sensing the environment from the head-mounted display unit, the sensors including a visible light camera and a non-visible light sensor, wherein the non-visible light sensor includes an infrared sensor configured to sense the environment using infrared electromagnetic radiation, a depth sensor configured to sense the distance to physical objects in the environment, an ultrasonic sensor configured to detect ultrasonic waves reflected by physical objects in the environment, and / or a motion sensor; Under high light conditions, a visible light image of the environment is obtained by the visible light camera, and non-visible light sensor data including information describing the position and orientation of the head-mounted display unit is obtained by the non-visible light sensor. The visible light image and the non-visible light sensor data, which are stored as stored non-visible light sensor data, are stored in a memory. In low-light conditions, additional non-visible light sensor data is obtained from the non-visible light sensor. Based on a comparison of this additional non-visible light sensor data and stored non-visible light sensor data associated with the visible light image, the visible light image is retrieved from the memory. The controller determines the graphical content, including rendering of the environment, based on the visible light image retrieved from the memory, and outputs the graphical content using the display. The graphic content includes a combination of graphic components determined collaboratively based on the sensing of the environment by the visible light camera, the infrared sensor, and the depth sensor.

11. The display system of claim 10, wherein the controller determines that the graphic content includes the visible light image.

12. The display system of claim 10, wherein the bright light condition occurred at a previous time and the dark light condition occurred at the current time.

13. The display system of claim 10, wherein the position and orientation of the head-mounted display unit are matched with stored non-visible light sensor data stored in association with the visible light image.

14. A method for providing graphical content through a display system, comprising: The environment is sensed in low light by sensors coupled to the head-mounted display unit to obtain low-light sensor data. The sensors include an infrared sensor, a depth sensor, and an ultrasonic sensor. The ultrasonic sensor transmits ultrasonic waves to the environment and detects the ultrasonic waves reflected by physical objects in the environment to sense the physical objects. A visible light image of the environment obtained by a visible light camera in high light is recalled, wherein the image of the environment obtained in high light is stored in association with high light sensor data obtained by the sensor in high light, and wherein the image is recalled based on a comparison between the low light sensor data and the high light sensor data; as well as When the head-mounted display unit is in low light, the image of the environment is output through the display of the head-mounted display unit. The graphic content includes composite graphic components, which include rendering of the environment based on the visible light image and are collaboratively determined based on sensing of the environment by the visible light camera, the infrared sensor, and the depth sensor.

15. The method of claim 14, wherein sensing the environment includes sensing information about the physical object, the information including the identity of the physical object.

16. The method of claim 15, wherein the image is invoked based on a comparison of information about the physical object obtained in the low light with information about the physical object obtained in the high light.

17. The method of claim 14, wherein the sensor includes a motion sensor for detecting the position and orientation of the head-mounted display unit, and sensing the environment includes sensing information about the physical object, the information including the position of the physical object.

18. The method of claim 17, wherein the image is invoked based on a comparison of the position of the physical object in the low light and the position of the physical object in the high light.

19. The method of claim 18, wherein the image is invoked when the position of the physical object in the low light matches the position of the physical object in the high light.

Images