System for automatic lighting of wearable devices

Abstract

Intelligently illuminating an environment includes receiving, at a head-mounted device, information indicative of ambient lighting conditions in the environment. In accordance with a determination that the ambient lighting condition does not meet the brightness criteria, one or more first luminaires on the head-mounted device are activated to project light in a first spectrum. When the one or more first luminaires are activated, image data of the environment is captured. A region of interest is determined in the environment based on the captured image data, and a second one or more illuminators are activated to project light in a visible spectrum different from the first spectrum.

Description

Technical Field

[0001] This disclosure relates generally to lighting. More specifically, but not in a limiting way, this disclosure relates to techniques and systems for monitoring the physical environment and triggering lighting in glasses in response to situations in the physical environment. Background Technology

[0002] Performing certain activities in specific environments can be difficult. For example, viewing a screen in a dark room or reading in the dark can be problematic for someone attempting to do so. Depending on some implementations, users navigating an environment may have difficulty in low-light conditions.

[0003] Furthermore, simply turning on the overhead light may not be an ideal solution. For example, other people in the environment might be bothered by the light. Additionally, the overhead light might be too bright for user activity. Therefore, improvements to smart lighting are needed. Attached Figure Description

[0004] Figure 1 A diagram illustrating an environment utilizing variations of this disclosure according to one or more embodiments is shown.

[0005] Figure 2 A flowchart is shown of a technique for managing the operation of a device to intelligently activate a luminaire, according to one or more embodiments.

[0006] Figure 3 An example process for ambient light conditions for lighting-based operations, according to one or more implementation schemes, is illustrated in flowchart form.

[0007] Figure 4 An example process for performing lighting operations based on ambient light conditions and an object of interest, according to one or more implementation schemes, is illustrated in flowchart form.

[0008] Figure 5 An example network diagram based on one or more implementation schemes is shown in block diagram form.

[0009] Figure 6 A mobile device according to one or more embodiments is shown in block diagram form. Detailed Implementation

[0010] Generally, the embodiments described herein relate to a technique for adjusting the lighting operation of a device in response to detected environmental conditions. In some embodiments, luminaires on and / or away from the device may be activated in response to conditions such as ambient light, an object of interest, or predefined mapping information.

[0011] According to one or more embodiments, the lighting operation is performed by a low-power device and is designed to require minimal power or other resources. For example, the device may be a wearable device, such as a head-mounted device, which is intended to be worn for extended periods and is therefore likely to be power-limited. Because the device may be power-limited, it may rely on low-power sensors, such as ambient light sensors, and may illuminate strategically, such as only in a defined direction or only when it is determined that the lighting conditions do not meet a brightness criterion.

[0012] According to one or more embodiments, a user may wear a head-mounted device or other wearable device. The head-mounted device may include illuminators that are intelligently activated based on characteristics of the surrounding environment, such as ambient light determined by sensor data collected by an ambient light sensor. In some embodiments, the wearable device may include illuminators that, when activated, illuminate the surrounding environment using light in a first spectrum (such as infrared light) that may be invisible to the user. When the illuminators are activated, an image of the environment may be captured. The device may determine a region of interest in the environment based on the image data. The device may then activate one or more additional illuminators in the visible spectrum toward the region of interest, making or increasing the visibility of the region of interest to the user. For example, the device may include a perspective display through which the real environment is visible to the user as is. That is, compared to a perspective display in which a camera feed of the environment is captured and presented to the user as image data, a perspective display allows the user to view the actual physical environment through a perspective display. Thus, by illuminating the region of interest in the environment, the physical components of the environment become more visible to the user's eyes.

[0013] In one or more embodiments, the system can intelligently illuminate the environment to draw attention to obstacles or unexpected objects in the physical environment. For example, the device may access predefined mapping data of the physical environment, such as a point cloud or other geometric representation of the environment. It can be determined whether the environment includes an object of interest. An object of interest may include, for example, obstacles near the user as indicated by the mapping data, or objects or obstacles not identified in the mapping data and therefore unexpectedly near the user. The device may trigger illuminators toward the object of interest to draw attention to it.

[0014] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is merely illustrative of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention; therefore, it is necessary to resort to the claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with the implementation is included in at least one implementation of the disclosed subject matter, and multiple references to “an implementation” or “implementation” should not be construed as referring to the same implementation in its entirety.

[0015] It should be understood that in the development of any actual implementation (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement multimodal processing systems in benefit from this disclosure.

[0016] It describes various examples of electronic systems related to various technologies and the technologies that use such systems.

[0017] As used herein, a physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, and / or virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As an example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting an XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., audio commands).

[0018] Go to Figure 1 This presents an exemplary environment diagram according to one or more embodiments. According to some embodiments, user 106 can view environment 100 via device 102. Device 102 may include a perspective display 120 through which user 106 can see objects in the environment, as shown in 122. Thus, lamp 104 and book 108 are visible via perspective display 110.

[0019] In some embodiments, device 102 may include various sensors that acquire sensor data of the environment and / or the user. The sensors may be oriented in any direction. For example, sensor 114 may include an ambient light sensor, a motion detection sensor such as an inertial sensor, a microphone, a flash sensor, an image capture sensor, etc. The ambient light sensor may include a subsystem for measuring and reporting ambient light levels (ALS), which include the relative levels of different wavelengths / bands of visible light. Sensor 114 may additionally include a motion detection sensor, such as a gyroscope, accelerometer, etc., configured to measure the user's rotation and / or head posture. The image capture sensor may include one or more cameras located on the device to capture image data of a view in front of the user. In some embodiments, device 102 may include network connectivity through which the device can control the functionality of other devices in the environment. This may include, for example, Wi-Fi signals, Bluetooth connectivity, pairing signals, etc.

[0020] According to one or more embodiments, device 102 is communicatively coupled to one or more remote devices in the environment and can obtain additional environmental data from the remote devices. Remote devices may include, for example, additional electronic devices, laptop computers, desktop computers, Internet of Things (IoT) devices (such as thermostats), smart lighting (such as lamp 104), and other devices. Therefore, in some embodiments, device 102 may aggregate environmental data and / or functionality from multiple devices.

[0021] In one or more embodiments, device 102 may include one or more illuminators 112 that are intelligently activated based on characteristics of the surrounding environment, such as ambient light determined by sensor data collected by sensor 114. The illuminators may include one or more light sources and optical elements for controlling the diffusion and direction of light output and its perceived color. In some embodiments, the wearable device may include illuminators capable of producing light of different spectra. For example, the illuminator may produce light of a first spectrum invisible to user 106, and a second spectrum of visible light different from the first spectrum. The illuminator may be an outward-facing illuminator on the device. Thus, when the illuminator is activated and produces light in the spectrum invisible to the user, an image of the environment can be captured. The device may determine a region of interest in the environment based on the image data. The device may then activate one or more additional illuminators in the visible spectrum toward the region of interest, making the region of interest visible to the user or increasing its visibility. For example, auxiliary illuminators on device 102 that produce visible light may be activated. Additionally or alternatively, if lamp 104 is communicatively coupled to device 102, device 102 may trigger other light sources in the environment, such as lamp 104. In some implementations, the specific illuminators (or illuminators) activated can be based on the spatial relationship between device 102 (or the illuminators on device 102) and the region of interest. That is, device 102 can intelligently activate illuminators in the direction of the region of interest, thereby reducing resource consumption by not activating unnecessary illuminators.

[0022] In one or more embodiments, device 102 may intelligently illuminate the environment to draw attention to obstacles or unexpected objects in the physical environment. For example, the device may access predefined mapping data of the physical environment 100, such as a point cloud, 3D model, or other geometric representation of environment 100. It may be determined whether the environment includes an object of interest. An object of interest may include, for example, obstacles near the user as indicated by the mapping data, or objects or obstacles near the user that are not identified in the mapping data and are therefore unexpected. For example, if environment 100 darkens and the user is walking toward table 126, the device may trigger a illuminator toward table 126 to draw attention to the table. For example, device 102 may activate a illuminator on device 102, or a illuminator on a remote device in the environment, such as lamp 104. In some embodiments, device 102 may generate a notification or other visual indication to be presented on display 110 near the field of vision of table 126, so that the user is aware of the table.

[0023] Now go to Figure 2 This presents a flowchart of a technology for managing the operation of a device to intelligently activate a lighting fixture, according to one or more embodiments. The flowchart begins at 200, showing... Figure 1An alternative view of environment 100. Thus, 200 shows user 106 reading book 108 in a dark room. User 106 is reading via device 102, which may include a perspective display so that book 108 is visible to the user through device 102.

[0024] The flowchart continues at box 205, where ambient light conditions are determined. According to one or more embodiments, ambient light conditions can be determined based on ambient light levels reported from the device's ambient light system. According to one or more embodiments, it is determined whether the device is in a low-light environment based on the determined ambient light conditions. For example, it can be determined that the ambient light conditions do not meet a predefined brightness standard. A brightness standard can be defined, for example, as a single predefined brightness value, or it can be based on the user or device context. For example, the brightness standard used by the device in an outdoor environment may differ from the brightness standard used in an indoor environment. Furthermore, the brightness standard can be user-defined or can be dynamically changed based on environmental or device conditions.

[0025] According to some implementation schemes, a region of interest is determined in the environment, as shown in box 210. The region of interest can be determined in several ways. For example, it can be defined based on a portion of the environment containing obstacles or unexpected objects, based on predefined mapping data of the environment. As another example, the region of interest can be determined based on sensor data captured at that region, such as by a depth sensor, image sensor, etc.

[0026] The flowchart continues to box 215, where a specific illuminator is activated based on ambient light conditions and / or the area of ​​interest. As described above, the calibration operation can be performed by a local device and / or by directing one or more additional devices to perform the operation. The illumination operation thereby induces an opportunity for ambient light in the environment. According to one or more embodiments, the activated illuminator can be based on the spatial relationship between the illuminator and the object of interest. Additionally or alternatively, the activated illuminator can be selected based on the head posture of the user wearing the device.

[0027] Additional considerations can be used when selecting which luminaires to activate. For example, if one or more additional people are detected in the environment, luminaires can be activated to avoid those people, preventing light from directly illuminating other people in the environment. As another example, luminaires can be activated to match the color temperature of the ambient light.

[0028] In some implementations, as shown in 220A, device 226A can activate or turn on a lamp 224A, illuminating the book 228A in the environment 230. For example, lamp 224A may be part of a connected "smart home" network operable from device 226A. As another example, as shown in 220B, device 226B can engage an externally facing illuminator 232 included in the device to illuminate the book 228B, illuminating the book 228B in the environment while lamp 224B remains off. In some implementations, a combination of on-device and remote illuminators can be used. When the illuminator is activated, the book 228B becomes more visible than it was initially presented at 200.

[0029] Figure 3 An example process for ambient light conditions used in lighting-based operations is illustrated in flowchart form. For illustrative purposes, [further details will be provided]. Figure 1 The following steps are described in the context of this scenario. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required.

[0030] Flowchart 300 begins at block 305, obtaining ambient lighting conditions from environmental sensor data. In some embodiments, the environmental sensor data may include data from sensors on a local device or data obtained from sensors on a remote device. Sensors may be embedded in the device and may include sensors such as cameras, ambient light sensors, microphones, flicker sensors, etc.

[0031] At box 310, it is determined whether the ambient lighting conditions meet the luminance criterion. The luminance criterion can be defined, for example, as a single predefined luminance value, or it can be based on the user or device context. For example, the luminance criterion used by the device in an outdoor environment may differ from the luminance criterion used in an indoor environment. Furthermore, the luminance criterion can be user-defined or can be dynamically changed based on environmental or device conditions. If the ambient lighting conditions meet the luminance criterion, no further action is taken, and flowchart 300 returns to box 305, and the device continues to track the ambient lighting conditions.

[0032] Returning to box 310, if it is determined that the ambient lighting conditions do not meet the brightness threshold, the flowchart proceeds to box 315. At box 315, an on-device illuminator is activated in the first spectrum. For example, the illuminator could be an outward-facing illuminator that produces light in the first spectrum invisible to the user. As an example, the illuminator could include an infrared (IR) lamp invisible to humans. Depending on one or more embodiments, an illuminator can be selected that faces the same direction as the user or that can flood the environment.

[0033] Flowchart 300 proceeds to box 320, where image data is captured when the illuminator is activated in the first spectrum. According to one or more embodiments, capturing the image data may include activating one or more computer vision cameras or other computer vision processors on the device. That is, images captured using infrared light allow for object detection without requiring the environment to be illuminated with visible light.

[0034] At box 325, a region of interest (ROI) is detected in the image data. According to one or more embodiments, the ROI in the environment can be determined by using computer vision to determine objects or object classifications (such as walls, furniture, etc.) within the environment. In some embodiments, the ROI may be additionally or alternatively determined based on head pose. For example, motion capture sensors in the device can be used to determine the device's pose and thus the direction the user is facing. Therefore, head pose data can be used to determine the area of ​​the environment the user is facing.

[0035] The flowchart proceeds to box 330, where a second illuminator is selected based on the region of interest and ambient lighting conditions. The second illuminator can be selected to illuminate the region of interest and can be chosen based on several considerations. At box 335, the spatial relationship between the region of interest and the device is determined. In one or more embodiments, the spatial relationship can be determined based on the relative orientation between the illuminator on the device and the region of interest. In some embodiments, the orientation of the device may also be considered. For example, if the device is tilted downwards, the user may be looking down, and the illuminator can be activated to provide light in the downward direction.

[0036] The flowchart continues to box 340, where the target lighting settings are determined based on spatial relationships. In one or more embodiments, the target lighting settings may include a subset of available illuminators on the device and / or in the environment, the temperature of the light, the brightness of the light, etc. For example, a target lighting temperature may be selected to match the lighting temperature detected by an ambient light sensor. As another example, the target brightness may be deterministic. For example, the target brightness may depend on the context. As an example, the target brightness may differ during the day from that at night. As another example, the target brightness may be location-specific. Thus, the target brightness in a home may be brighter than the target brightness in a public place to avoid unintentionally disturbing others in the environment with excessive light. As another example, the target lighting settings may be based on the user context. For example, if user activity is detected using fine motor skills such as threading a needle, brighter light may be provided than when the user is simply navigating in a dark environment.

[0037] At box 345, a second illuminator is selected based on spatial relationships, lighting setup, and / or environmental context. That is, one or more illuminators can be selected based on the relative position of the area of ​​interest and the device, and the characteristics of the illuminator can be selected for activation based on target parameters. According to one or more embodiments, the illuminator can be selected from available illuminators on the device and / or illuminators in the environment that can be activated by the device.

[0038] In one or more embodiments, the device can communicate with other devices in the environment that also have illuminators. According to one or more embodiments, two or more head-mounted devices integrated with illuminators can be in the same environment, such as two people using a separate system in a room. In this embodiment, the two devices can transmit location information to each other to collaboratively determine the lighting settings. Therefore, one or more illuminators on the second device can be selected. In some embodiments, one or more illuminators on the second device are selected based on the head posture or device orientation of the local device, the head posture or device orientation of the second device, and the area of ​​interest.

[0039] Flowchart 300 ends at 350, where the second illuminator is activated in the visible spectrum. In some embodiments, the device includes a see-through or transparent display. When the second illuminator, which projects visible light, is activated, the surrounding environment becomes more visible to the user through the see-through or transparent display.

[0040] In one or more implementations, the system can intelligently illuminate the environment by utilizing data about the environment to draw attention to obstacles or unexpected objects in the physical environment. Figure 4 An example process for performing lighting operations based on ambient light conditions and an object of interest, according to one or more implementation schemes, is illustrated in flowchart form. For illustrative purposes, further details will be provided. Figure 1 The following steps are described in the context of this scenario. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required.

[0041] Flowchart 400 begins at block 405, obtaining ambient lighting conditions from environmental sensor data. In some embodiments, the environmental sensor data may include data from sensors on a local device or data obtained from sensors on a remote device. Sensors may be embedded in the device and may include sensors such as cameras, ambient light sensors, microphones, flicker sensors, etc.

[0042] At box 410, it is determined whether the ambient lighting conditions meet the luminance criterion. The luminance criterion can be defined, for example, as a single predefined luminance value, or it can be based on the user or device context. For example, the luminance criterion used by the device in an outdoor environment may differ from the luminance criterion used in an indoor environment. Furthermore, the luminance criterion can be user-defined or can be dynamically changed based on environmental or device conditions. If the ambient lighting conditions meet the luminance criterion, no further action is taken, and flowchart 400 returns to box 405, and the device continues to track the ambient lighting conditions.

[0043] Returning to box 410, if it is determined that the ambient lighting conditions do not meet the brightness threshold, the flowchart proceeds to box 415. At box 415, environmental mapping data is obtained. Depending on one or more embodiments, the environmental mapping data may include data related to the layout of the environment. As an example, point clouds, meshes, etc., can be used to indicate the relative positions of walls, floors, and objects in the environment. As an example, the environment may have been previously scanned by the device and registered as a known environment. The head-mounted device may access the environmental mapping data, for example, from a cloud storage device.

[0044] The flowchart proceeds to box 420 and detects objects of interest based on the mapping data. The object of interest can be any object detected in the environment, which should be highlighted to the user using one or more illuminators. As an example, as shown in optional box 425, detecting the object of interest may include identifying unexpected objects based on the mapping data. For example, a depth sensor or camera may indicate objects in the environment at locations not expected in the environment, according to the mapping data. That is, the device's positioning information can be determined and compared with the environmental mapping data to identify expected objects in the environment.

[0045] Additionally or alternatively, as shown in optional box 430, obstacles can be identified based on the mapping data. For example, the device's location information can be determined and compared with environmental mapping data to determine whether a user is approaching an obstacle, such as a wall or object present in the mapping data.

[0046] It can be determined whether the environment includes an object of interest. An object of interest may include, for example, obstacles near the user as indicated by mapping data, or objects or obstacles near the user that are not identified in the mapping data and are therefore unexpected. The device can trigger a illuminator to be directed toward the object of interest to draw attention to it.

[0047] The flowchart proceeds to box 435, where a illuminator is selected based on the object of interest and ambient lighting conditions. An illuminator can be selected to illuminate the object of interest, and the selection can be based on multiple considerations. At box 445, the spatial relationship between the object of interest and the device is determined. In one or more embodiments, the spatial relationship can be determined based on the relative orientation between the illuminator on the device and the object of interest. In some embodiments, the orientation of the device can also be considered. For example, if the device is tilted in a direction toward a specific portion of the object of interest, that orientation can be determined based on the device's posture information.

[0048] The flowchart continues to box 450, where the target lighting setting is determined based on spatial relationships. In one or more embodiments, the target lighting setting may include a subset of available illuminators on the device and / or in the environment, the temperature of the light, the brightness of the light, etc. For example, a target lighting temperature may be selected to match the lighting temperature detected by an ambient light sensor. As another example, the target brightness may be deterministic. For example, the target brightness may depend on the context. As an example, the target brightness may differ during the day from that at night. As another example, the target brightness may be location-specific. Thus, the target brightness in a home may be brighter than the target brightness in a public place to avoid unintentionally disturbing others in the environment with excessive light.

[0049] At box 455, illuminators are selected based on spatial relationships, lighting setup, and / or environmental context. That is, one or more illuminators can be selected based on the relative position of the area of ​​interest and the device, and the characteristics of the illuminators can be selected for activation based on target parameters. According to one or more embodiments, the illuminators can be selected from available illuminators on the device and / or illuminators in the environment that can be activated by the device.

[0050] Flowchart 400 ends at 460, where the illuminator is activated according to the selection. Optionally, at box 465, the device triggers the activation of a remote illuminator. For example, the device may be part of a connected "smart home" network operable from the device. Furthermore, in one or more embodiments, the device may communicate with other devices in the environment that also have illuminators. According to one or more embodiments, two or more head-mounted devices integrated with the illuminator may be in the same environment, such as two people using a separate system in a room. In this embodiment, the two devices may transmit location information to each other to collaboratively determine and execute lighting settings. Other remote devices with illuminators can therefore be communicatively connected to the device and thus can be activated remotely, for example, from the head-mounted device. In some embodiments, the device includes a see-through display. When the illuminator is activated, objects of interest become more visible to the user through the see-through display.

[0051] Figure 5A network diagram of the system is depicted, and various implementations of this disclosure are practiced through this system. Specifically, Figure 5 An electronic device 500 for a computer system is illustrated. Electronic device 500 may be part of a multi-functional device such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system as described herein. Electronic device 500 may be connected across network 505 to other devices, such as accessory electronic device 510, mobile device, tablet device, desktop device, and remote sensing device, as well as network storage device 515. Accessory device 510 may include, for example, an attached laptop computer, desktop computer, mobile device, wearable device, and other devices communicatively coupled to electronic device 500. In some embodiments, accessory device 510 may include an IoT device communicatively coupled to electronic device 500 and having one or more sensors capable of capturing environmental data and / or one or more lights triggerable by electronic device 500. Network storage device 515 may be any kind of electronic device communicatively coupled to electronic device 505 across network 505 via network interface 545. In some embodiments, network storage device 515 may include cloud storage devices, etc. Network 505 may include one or more types of networks across which various electronic components can be communicatively coupled. Exemplary networks include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational local area networks (LANs), and wide area networks (such as the Internet).

[0052] Electronic device 500, accessory electronic device 510, and / or network storage device 515 may additionally or alternatively include one or more additional devices (such as server equipment, base stations, accessory devices, etc.), in which various functionalities can be accommodated or distributed across these devices. It should be understood that the various components and functionalities within electronic device 500, accessory electronic device 510, and network storage device 515 may be distributed differently across devices or across additional devices.

[0053] Electronic device 500 may include processor 520. Processor 520 may be a system-on-a-chip, such as those present in mobile devices, and includes one or more central processing units (CPUs), dedicated graphics processing units (GPUs), or both. Furthermore, processor 520 may include multiple processors of the same or different types. Electronic device 500 may also include memory 550. Memory 550 may include one or more different types of memory that can be used in conjunction with processor 520 to perform device functions. For example, memory 550 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 550 may store various programming modules during execution, such as a low-light assist module configured to detect low-light conditions via sensor data, for example, captured by ambient light sensor 540. In some embodiments, low-light assist module 552 may determine lighting settings and accordingly trigger the activation of local illuminators 570 and remote illuminators. Furthermore, memory 550 may include one or more additional application programs 558. In some implementations, the low-light assist module 552 can use the state of the application 558 to determine user activity, device context, etc. In some implementations, the context is determined based on image data captured by the camera 525.

[0054] Electronic device 500 may also include storage device 530. Storage device 530 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Storage device 530 may be used to store various data and structures that can be used to mitigate triggering conditions in a perspective environment. For example, storage device 530 may include an environmental data repository 535, which may include predefined geometric and / or layout information for one or more physical environments. Additionally or alternatively, environmental data may be stored remotely, for example, as an environmental data repository 590 as a network storage device 515.

[0055] Electronic device 500 allows a user to interact with an XR environment. Many electronic systems enable individuals to interact with and / or perceive various XR scenes. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smartphone). The head-mounted system may have one or more imaging sensors and / or microphones for capturing images / videos of the physical scene and / or capturing audio of the physical scene. The head-mounted system may also have a transparent or semi-transparent see-through display 560. The transparent or semi-transparent display may be combined with a substrate through which light representing the image is guided to the individual's eyes. The display may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate transmitting light may be an optical waveguide, an optical combiner, a light reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or semi-transparent display may selectively switch between an opaque state and a transparent or semi-transparent state. As another example, the electronic system may be a projection-based system. Projection-based systems can use retinal projection to project images onto an individual's retina. Alternatively, projection systems can project virtual objects onto a physical set (e.g., onto a physical surface or as a hologram). Other examples of XR systems include head-up displays, car windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, headphones or earpieces, speaker arrangements, input mechanisms (e.g., controllers with or without haptic feedback), tablet computers, smartphones, and desktop or laptop computers.

[0056] Now for reference Figure 6 This document illustrates a simplified functional block diagram of an exemplary multi-functional electronic device 600 according to one embodiment. Each electronic device in the electronic device may be a multi-functional electronic device, or may have some or all of the components of the multi-functional electronic device described herein. The multi-functional electronic device 600 may include a processor 605, a display 610, a user interface 615, graphics hardware 620, device sensors 625 (e.g., proximity / ambient light sensors, accelerometers, and / or gyroscopes), a microphone 630, an audio codec 635, a speaker 640, communication circuitry 645, digital image capture circuitry 650 (e.g., including a camera system), a memory 660, a storage device 665, and a communication bus 670. The multi-functional electronic device 600 may be, for example, a mobile phone, a personal music player, a wearable device, a tablet computer, etc.

[0057] Processor 605 can execute necessary instructions to implement or control the operation of various functions performed by device 600. Processor 605 may, for example, drive display 610 and receive user input from user interface 615. User interface 615 allows the user to interact with device 600. For example, user interface 615 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, or touchscreen. Processor 605 may also be, for example, a system-on-a-chip, such as those present in mobile devices, and includes a dedicated graphics processing unit (GPU). Processor 605 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 620 may be dedicated computing hardware for processing graphics and / or assisting processor 605 in processing graphics information. In one embodiment, graphics hardware 620 may include a programmable GPU.

[0058] Image capture circuit 650 may include one or more lens assemblies, such as 680A and 680B. The lens assemblies may have various combinations of characteristics, such as different focal lengths. For example, lens assembly 680A may have a shorter focal length relative to the focal length of lens assembly 680B. Each lens assembly may have a separate associated sensor element 690. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 650 may capture still images, video images, and enhanced images, etc. The output from image capture circuit 650 may be processed at least in part by a video codec 655, a processor 605, graphics hardware 620, and / or a dedicated image processing unit or pipeline incorporated within circuit 645. Images thus captured may be stored in memory 660 and / or storage device 665.

[0059] Memory 660 may include one or more different types of media used by processor 605 and graphics hardware 620 to perform device functions. For example, memory 660 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 665 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 665 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tape, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Memory 660 and storage device 665 may be used to tangibly hold computer program instructions or computer-readable code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 605, such computer program code may implement one or more of the methods described herein.

[0060] Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto the human retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as holograms or onto a physical surface.

[0061] It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figures 3 to 4 The specific arrangement of the steps or actions shown or Figures 1 to 2 and Figures 5 to 6 The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by referring to the appended claims and the full scope of their equivalents. In the appended claims, the terms "comprising" and "wherein" are used as common Chinese equivalents to the corresponding terms "comprising" and "characterized in".

Claims

1. A method, the method comprising: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Activate one or more first illuminators on the head-mounted device to project light in a first spectrum; When one or more of the first illuminators are activated, image data of the environment is captured; The region of interest in the environment is determined based on the captured image data; as well as Activate a second or more illuminators to project light in the visible spectrum that is different from the first spectrum.

2. The method of claim 1, wherein the second or more illuminators are selected from a plurality of outward-facing illuminators based on the spatial relationship between the head-mounted device and the region of interest and the ambient lighting conditions.

3. The method of claim 1, wherein the environment is visible through a transparent display of the head-mounted device.

4. The method of claim 1, wherein activating the second or more illuminators comprises: Based on the region of interest, determine the remote devices including the second or more illuminators; as well as Trigger the remote device to activate the second or more illuminators.

5. The method of claim 1, wherein activating the second or more illuminators comprises: Detect one or more people in the environment; as well as Choose the second or more luminaires to avoid illuminating the one or more people in the environment.

6. The method of claim 1, wherein the first at least one of the second or more illuminators is included in the head-mounted device.

7. The method of claim 6, wherein the second at least one of the second or more illuminators is included in the second head-mounted device, and Activating the second or more illuminators includes: The lighting setup is determined based on the region of interest, the lighting setup including the first at least one illuminator among the second or more illuminators and the second at least one illuminator among the second or more illuminators; Activate the second at least one of the second or more illuminators; as well as Trigger the second head-mounted device to activate the first at least one of the second or more illuminators.

8. A non-transitory computer-readable medium comprising computer-readable code, said computer-readable code being executable by one or more processors to: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Activate one or more first illuminators on the head-mounted device to project light in a first spectrum; When one or more of the first illuminators are activated, image data of the environment is captured; The region of interest in the environment is determined based on the captured image data; as well as Activate a second or more illuminators to project light in the visible spectrum that is different from the first spectrum.

9. The non-transitory computer-readable medium of claim 8, wherein the second or more illuminators are selected from a plurality of outward-facing illuminators based on the spatial relationship between the head-mounted device and the region of interest and the ambient lighting conditions.

10. The non-transitory computer-readable medium of claim 8, wherein the environment is visible through a transparent display of the head-mounted device.

11. The non-transitory computer-readable medium of claim 8, wherein the computer-readable code for activating the second or more illuminators comprises computer-readable code for the following operations: Based on the region of interest, determine the remote devices including the second or more illuminators; and Trigger the remote device to activate the second or more illuminators.

12. The non-transitory computer-readable medium of claim 8, wherein the computer-readable code for activating the second or more illuminators comprises computer-readable code for the following operations: Detecting one or more people in the environment; and Choose the second or more luminaires to avoid illuminating the one or more people in the environment.

13. The non-transitory computer-readable medium of claim 8, wherein at least one of the second or more illuminators is included in the head-mounted device.

14. The non-transitory computer-readable medium of claim 13, wherein at least one of the second or more illuminators is included in the second head-mounted device, and The computer-readable code for activating the second or more illuminators includes computer-readable code for the following operations: The lighting setup is determined based on the region of interest, the lighting setup including the first at least one illuminator among the second or more illuminators and the second at least one illuminator among the second or more illuminators; Activate the second at least one of the second or more illuminators; as well as Trigger the second head-mounted device to activate the first at least one of the second or more illuminators.

15. A system comprising: One or more processors; and One or more computer-readable media, the one or more computer-readable media including computer-readable code, the computer-readable code being executable by the one or more processors to: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Activate one or more first illuminators on the head-mounted device to project light in a first spectrum; When one or more of the first illuminators are activated, image data of the environment is captured; The region of interest in the environment is determined based on the captured image data; as well as Activate a second or more illuminators to project light in the visible spectrum that is different from the first spectrum.

16. The system of claim 15, wherein the second or more illuminators are selected from a plurality of outward-facing illuminators based on the spatial relationship between the head-mounted device and the region of interest and the ambient lighting conditions.

17. The system of claim 15, wherein the environment is visible through a transparent display of the head-mounted device.

18. The system of claim 15, wherein the computer-readable code for activating the second or more illuminators comprises computer-readable code for: Based on the region of interest, determine the remote devices including the second or more illuminators; and Trigger the remote device to activate the second or more illuminators.

19. The system of claim 15, wherein the computer-readable code for activating the second or more illuminators comprises computer-readable code for: Detecting one or more people in the environment; and Choose the second or more luminaires to avoid illuminating the one or more people in the environment.

20. The system of claim 15, wherein at least one of the second or more illuminators is included in the head-mounted device.

21. The system of claim 20, wherein at least one of the second or more illuminators is included in the second head-mounted device, and The computer-readable code for activating the second or more illuminators includes computer-readable code for the following operations: The lighting setup is determined based on the region of interest, the lighting setup including the first at least one illuminator among the second or more illuminators and the second at least one illuminator among the second or more illuminators; Activate the second at least one of the second or more illuminators; as well as Trigger the second head-mounted device to activate the first at least one of the second or more illuminators.

22. A method, the method comprising: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Obtain predefined mapping data for the environment; Based on the mapping data, one or more objects of interest can be determined; as well as Activate one or more illuminators on the head-mounted device to project light toward the one or more objects of interest.

23. The method of claim 22, wherein the ambient lighting conditions include an illumination temperature, and wherein the first or more illuminators are activated to match the illumination temperature.

24. The method of claim 22, wherein the predefined mapping data of the environment includes a three-dimensional model of the environment.

25. The method of claim 22, wherein determining one or more objects of interest based on the mapping data includes detecting unexpected objects in the environment.

26. The method according to claim 22, further comprising: Based on the object of interest, determine the remote device including a second or more illuminators; as well as Trigger the remote device to activate the second or more illuminators.

27. The method of claim 22, wherein activating one or more first illuminators comprises: Determine the spatial relationship between the head-mounted device and the object of interest; as well as The first one or more illuminators are selected from a plurality of illuminators based on the spatial relationship.

28. The method of claim 22, wherein activating the first one or more illuminators comprises: Detect one or more people in the environment; as well as Select the first one or more luminaires to avoid illuminating the one or more people in the environment.

29. The method according to claim 22, further comprising: An indicator of the object of interest is displayed on the perspective display of the head-mounted device.

30. A non-transitory computer-readable medium comprising computer-readable code, said computer-readable code being executable by one or more processors to: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Obtain predefined mapping data for the environment; Based on the mapping data, one or more objects of interest can be determined; as well as Activate one or more illuminators on the head-mounted device to project light toward the one or more objects of interest.

31. The non-transitory computer-readable medium of claim 30, wherein the ambient lighting conditions include an illumination temperature, and wherein the first or more illuminators are activated to match the illumination temperature.

32. The non-transitory computer-readable medium of claim 30, wherein the predefined mapping data of the environment includes a three-dimensional model of the environment.

33. The non-transitory computer-readable medium of claim 30, wherein the computer-readable code for determining one or more objects of interest based on the mapping data includes computer-readable code for detecting unexpected objects in the environment.

34. The non-transitory computer-readable medium of claim 30, further comprising computer-readable code for the following operations: Based on the object of interest, determine remote devices including a second or more illuminators; and Trigger the remote device to activate the second or more illuminators.

35. The non-transitory computer-readable medium of claim 30, wherein the computer-readable code for activating the one or more first illuminators comprises computer-readable code for the following operations: Determine the spatial relationship between the head-mounted device and the object of interest; and The first one or more illuminators are selected from a plurality of illuminators based on the spatial relationship.

36. The non-transitory computer-readable medium of claim 30, wherein the computer-readable code for activating the one or more first illuminators comprises computer-readable code for the following operations: Detecting one or more people in the environment; and Select the first one or more luminaires to avoid illuminating the one or more people in the environment.

37. The non-transitory computer-readable medium of claim 30, further comprising computer-readable code for the following operations: An indicator of the object of interest is displayed on the perspective display of the head-mounted device.

38. A system comprising: One or more processors; and One or more computer-readable media, the one or more computer-readable media including computer-readable code, the computer-readable code being executable by the one or more processors to: Receive information indicating ambient lighting conditions in the environment at the head-mounted device. Based on the determination that the ambient lighting conditions do not meet the brightness standard: Obtain predefined mapping data for the environment; Based on the mapping data, one or more objects of interest can be determined; as well as Activate one or more illuminators on the head-mounted device to project light toward the one or more objects of interest.

39. The system of claim 38, wherein the ambient lighting conditions include an illumination temperature, and wherein the first or more illuminators are activated to match the illumination temperature.

40. The system of claim 38, wherein the predefined mapping data of the environment includes a three-dimensional model of the environment.

41. The system of claim 38, wherein the computer-readable code for determining one or more objects of interest based on the mapping data includes computer-readable code for detecting unexpected objects in the environment.

42. The system of claim 38, further comprising computer-readable code for the following operations: Based on the object of interest, determine remote devices including a second or more illuminators; and Trigger the remote device to activate the second or more illuminators.

43. The system of claim 38, wherein the computer-readable code for activating the one or more first illuminators comprises computer-readable code for: Determine the spatial relationship between the head-mounted device and the object of interest; and The first one or more illuminators are selected from a plurality of illuminators based on the spatial relationship.

44. The system of claim 38, wherein the computer-readable code for activating the one or more first illuminators comprises computer-readable code for: Detecting one or more people in the environment; and Select the first one or more luminaires to avoid illuminating the one or more people in the environment.

45. The system of claim 38, further comprising computer-readable code for the following operations: An indicator of the object of interest is displayed on the perspective display of the head-mounted device.

Images