Electronic devices with displays designed for low-light conditions

Abstract

Operating an outward-facing display device for low-light conditions may include receiving information indicating ambient lighting conditions of the environment in which the display device is located; determining a set of pixels in the outward-facing display device corresponding to a region of interest in the environment based on the determination that the ambient lighting conditions do not meet a brightness standard; and driving the set of pixels at a specific brightness to improve the illumination of the region of interest.

Description

Background Technology

[0001] This disclosure relates in its entirety to display operation. More specifically, but not as a limitation, this disclosure relates to techniques and systems for automatically operating outward-facing display devices in low-light conditions.

[0002] Display devices can be used in a variety of environmental conditions, such as bright sunlight, top fluorescent lighting, and dim or low-light lighting. While many electronic devices include light sources to enable flash functionality, what is needed are improved technologies for illuminating the environment of outward-facing display devices. Attached Figure Description

[0003] Figure 1 An exemplary system comprising various outward-facing display devices is illustrated in block diagram form for use in low-light conditions, including extended reality environments.

[0004] Figure 2 A schematic diagram illustrating an exemplary operating environment of an electronic device according to one or more embodiments, the electronic device including an outward-facing display.

[0005] Figure 3 An exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions.

[0006] Figures 4A to 4B An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 3 A schematic diagram of an exemplary operating environment for the exemplary process shown.

[0007] Figure 5 Another exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions.

[0008] Figure 6 An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 5 A schematic diagram of an exemplary operating environment for the exemplary process shown.

[0009] Figure 7 Another exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions.

[0010] Figures 8A to 8B An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 7 A schematic diagram of an exemplary operating environment for the exemplary process shown.

[0011] Figure 9 An exemplary process for determining ambient lighting conditions according to one or more embodiments is illustrated in flowchart form.

[0012] Figure 10 An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 9 A schematic diagram of an exemplary operating environment for the exemplary process shown.

[0013] Figures 11A to 11B An exemplary computer system according to one or more embodiments is shown in block diagram form. Detailed Implementation

[0014] This disclosure relates to systems, methods, and computer-readable media for operating an outward-facing display device to illuminate an environment. Specifically, information indicating ambient lighting conditions for the outward-facing display device can be received. The ambient lighting conditions may be determined to be unmet brightness criteria. Based on this determination, a set of pixels in the outward-facing display device corresponding to a region of interest in the environment can be identified. This set of pixels in the outward-facing display device can then be driven at a specific brightness to improve illumination in the region of interest.

[0015] According to some embodiments, the information indicating ambient lighting conditions includes the brightness and color temperature of the ambient lighting. The set of pixels can be driven according to the color temperature to match the ambient lighting. In some embodiments, the information indicating ambient lighting conditions includes information about multiple areas in the environment, and determines whether at least one of these areas fails to meet a brightness standard.

[0016] In some implementations, the region of interest can be determined based on indicators from an eye-tracking user interface and / or an input area from a gesture-based user interface. In some implementations, an outward-facing camera can capture an image of the environment, and information indicating ambient lighting conditions can be determined based on that image. The outward-facing display device can be positioned away from the device user, such that the outward-facing display device emits light into the surrounding environment when positioned away from the user. In some implementations, the outward-facing display device can emit a color sequence, and the outward-facing camera can capture an image of the environment for each color in the color sequence. The color of interest in the environment can be determined based on this set of images, and the set of pixels can be driven with a specific color based on that color of interest. In some implementations, the outward-facing camera is a monochrome camera, and the color of interest in the environment can be determined by identifying a specific image that includes the strongest signal and the corresponding color of the color sequence. In some implementations, media items associated with a brightness standard can be displayed on an inward-facing display device. The inward-facing display device can be oriented towards the device user, such that the projection from the inward-facing display device and the projection from the outward-facing display device do not intersect.

[0017] Various examples of electronic systems and technologies for using such systems in relation to various extended reality technologies are described.

[0018] Humans can interact with and / or perceive the physical environment or physical world without the aid of electronic devices. The physical environment can include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest that includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means, such as hearing, vision, taste, touch, and smell. In contrast, humans can use electronic devices to interact with and / or perceive a fully or partially simulated extended reality (XR) environment. This XR environment can include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and so on. Using an XR system, a person's physical movements, or some of their representations, can be tracked, and in response, the characteristics of virtual objects simulated in the XR environment can be adjusted in a manner consistent with at least one physical law. For example, the XR system can detect movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). For example, the XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) presenting the XR environment and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in a physical environment). In some cases, the XR system can adjust the characteristics of the graphical content in response to other inputs such as representations of physical motion (e.g., voice commands).

[0019] Many different types of electronic systems enable users to interact with and / or perceive XR environments. A non-exclusive list of examples includes head-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), head-mounted receivers / headsets, input systems with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have opaque displays and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (e.g., smartphones). Head-mounted systems may include one or more image sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays, rather than opaque displays. Transparent or semi-transparent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof, or other similar technologies can be used as the medium. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects graphic images onto a user's retina. Projection systems can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0020] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, as part of this specification, some of the drawings of this disclosure are provided in the form of flowcharts. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, thereby resorting to the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with that implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring to all of the same implementation.

[0021] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0022] refer to Figure 1This document depicts a simplified block diagram of an electronic device 100, communicatively connected via a network 105 to an additional electronic device 110 and a network storage device 115, according to one or more embodiments of this disclosure. The electronic device 100 may be part of a multi-functional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system as described herein. The electronic device 100, the additional electronic device 110, and / or the network storage device 115 may additionally or alternatively include one or more additional devices (such as server devices, base stations, accessory devices, etc.), in which various functions may be included, or various functions may be distributed across these additional devices. Exemplary networks such as network 105 include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 100 is used to illuminate the environment of the device 100. It should be understood that the various components and functions within the electronic device 100, the additional electronic device 110, and the network storage device 115 may be distributed differently on the device or on the additional device.

[0023] Electronic device 100 may include one or more processors 125, such as a central processing unit (CPU). Processor 125 may include a system-on-a-chip (such as those present in mobile devices) and may include one or more dedicated graphics processing units (GPUs). Additionally, processor 125 may include multiple processors of the same or different types. Electronic device 100 may also include memory 135. Memory 135 may include one or more different types of memory that can be used in conjunction with processor 125 to perform device functions. For example, memory 135 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 135 may store various programming modules for execution by processor 125, including an outward lighting module 165, an ambient lighting module 170, and various other applications 175. Electronic device 100 may also include a storage device 130. Storage device 130 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (e.g., CD-ROMs and digital video optical discs (DVDs)), and semiconductor storage devices (e.g., electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). According to one or more embodiments, storage device 130 may be configured to store ambient lighting data 160.

[0024] Electronic device 100 may also include one or more cameras 140 or other sensors 145, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 140 may be a conventional RGB camera or a monochrome camera. Additionally, cameras 140 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc. Sensors 145 may also include sensors such as accelerometers, gyroscopes, or other motion sensors. Electronic device 100 may also include a display 155. Display 155 includes an outward-facing display device. In some embodiments, display 155 may additionally include an inward-facing display device. The outward-facing and inward-facing displays may be located in electronic device 100 such that the inward-facing display faces the user and the outward-facing display faces away from the user and towards the environment. For example, projections from the inward-facing display and the outward-facing display may not intersect. Display 155 may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one implementation, a transparent or semi-transparent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto a person's retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or onto a physical surface.

[0025] Storage device 130 can be used to store various data and structures that can be used to determine the ambient lighting conditions of electronic device 100 in order to operate an outward-facing display device to illuminate a dimly lit environment. Storage device 130 may include, for example, an ambient lighting data repository 160. Ambient lighting data repository 160 can be used to store information indicating the ambient lighting conditions of the current physical environment of electronic device 100, or information indicating a predetermined set of ambient lighting conditions associated with the environment in which electronic device 100 may be used. In one or more embodiments, ambient lighting data may include information indicating low-light ambient lighting conditions associated with a dark room, overhead fluorescent ambient lighting conditions associated with an office environment, bright sunlight ambient lighting conditions associated with a sunny outdoor environment, etc., which can be used by outward-facing lighting module 165 to drive pixels in display 155 to illuminate the environment. In one or more embodiments, ambient lighting data may be stored occasionally, periodically, or in response to triggers, such as threshold changes in the position of electronic device 100 detected by sensor 145, etc. In one or more embodiments, ambient lighting data may be stored locally in each system (such as electronic device 100 and additional electronic device 110), and / or geometric information may be stored in a global ambient lighting data repository 120 as part of network storage device 115.

[0026] According to one or more embodiments, memory 135 may include one or more modules comprising computer-readable code executable by processor 125 to perform functions. The memory may include, for example, an outward illumination module 165 that can be used to drive an outward-facing display device to illuminate the environment of electronic device 100. Outward illumination module 165 can determine the ambient lighting conditions of the environment of electronic device 100 and whether the ambient lighting conditions meet a brightness standard in order to drive specific pixels of the display device at a specific brightness. In some embodiments, outward illumination module 165 obtains information indicating ambient lighting conditions from ambient lighting module 170. The memory may also include, for example, other applications 175 that can be used to determine an appropriate brightness standard for outward illumination module 165. For example, other applications 175 may include a media management application that determines an appropriate brightness standard based on media items displayed on an inward-facing display device, etc.

[0027] Although electronic device 100 is described as including the numerous components described above, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. Additionally, additional components may be used, and certain combinations of the functions of any components may be combined.

[0028] Figure 2 A schematic diagram illustrating an exemplary operating environment of an outward-facing display device according to one or more embodiments is provided. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the examples in the specific embodiments disclosed herein. Therefore, as a non-limiting example, operating environment 200 includes an optional outward-facing optical sensor 220.

[0029] like Figure 2 As shown, user 250 is depicted using an electronic device having an outward-facing display device 210 and an outward-facing optical sensor 220. In one or more embodiments, the electronic device includes a mobile device, such as a handheld device, a tablet device, a wearable device, etc. The optical sensor 220 may be configured to determine the ambient lighting conditions of the operating environment 200 in which the outward-facing display device 210 is located, and may include, for example, a photodiode, a camera, an ambient light sensor, etc., or some combination thereof. The outward-facing display device 210 may be configured to illuminate a field of view 230 in the operating environment 200 to present information to a user near user 250, or both.

[0030] Figure 3An exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions. For illustrative purposes, [further details will be provided]. Figure 1 and Figure 2 The following steps are described in the context of [the relevant document / document]. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required. Various actions can be performed remotely by server equipment, by a single electronic device, and / or distributed among multiple electronic devices.

[0031] The flowchart begins at 310, where the outward illumination module 165 receives information indicating the ambient lighting conditions of the environment in which the outward display device is located (e.g., in environment 200 of the outward display device 210). The outward illumination module 165 may receive information indicating the ambient lighting conditions from the ambient lighting module 170, the ambient lighting data repository 160 in storage device 130, the global ambient lighting data repository 120 in network storage device 115, optical sensor 220, etc. At 320, the outward illumination module 165 determines whether the ambient lighting conditions meet a brightness standard. The brightness standard may represent a threshold brightness of the environment that allows the user 250 to see and move around in environment 200, an object recognition application included in application 175 to track objects in environment 200, and a gesture-based user interface to recognize gestures performed by the user 250, etc. The brightness standard may include other lighting parameters besides or replacing the measured ambient lighting intensity level, such as lighting color, direction, electromagnetic spectrum characteristics, etc. Based on the determination that the ambient lighting conditions meet the brightness standard, the outward lighting module 165 can return to the frame 310 and continuously, periodically at fixed intervals, and in response to triggers, receive information indicating the updated ambient lighting conditions in the environment.

[0032] Based on the determination in block 320 that the ambient lighting conditions do not meet the brightness standard, the outward lighting module 165 can determine a set of pixels in the outward display device 210 corresponding to a region of interest in the environment in block 330. The region of interest can be a relevant object, an area of ​​the environment in which the user 250's hand moves in response to a gesture-based interface, an area of ​​the environment in which the user 250 is moving towards, etc. The set of pixels corresponding to the relevant object can be pixels that illuminate that relevant object. The set of pixels corresponding to the area of ​​the environment in which the user 250's hand moves can be pixels that illuminate that area along the side of the outward display device 210. The set of pixels corresponding to the area of ​​the environment in which the user 250 is moving towards can be pixels that illuminate the ground and walking path of the user 250 along the bottom of the outward display device 210.

[0033] Then, the outward illumination module 165 drives the set of pixels at a specific brightness at 340 to improve the illumination in the region of interest. Improving the illumination in the region of interest may include increasing the brightness of the region of interest, changing the color of the illumination in the region of interest, changing the direction of the ambient light toward the region of interest, etc. The outward illumination module 165 can drive the set of pixels with specific brightness values, specific color values, specific light frequency ranges, etc., to improve the illumination in the region of interest. The brightness of the ambient lighting in the environment 200 and the light brightness from the driven set of pixels illuminate the environment 200, enabling the user 250 to see and move within the environment 200 based on a specific brightness standard, allowing the object recognition application in the application 175 to track objects in the environment 200, and enabling the gesture-based user interface to recognize gestures, etc. By driving only the set of pixels corresponding to the region of interest, the outward illumination module 165 ensures that the region of interest is adequately illuminated without consuming excessive power by driving every pixel in the outward display device 210, or causing the user 250 to be overwhelmed by a suddenly bright environment. In addition, the remaining pixels can still be used to present information to other users near the user 250.

[0034] Figures 4A to 4B An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 3 This is a schematic diagram of an exemplary operating environment 400 for an exemplary process. Figure 4A In the side view 400A shown, the optical sensor 220 acquires information indicating ambient lighting conditions in environment 400, and the outward illumination module 165 determines that the ambient lighting conditions do not meet a brightness standard. In response, the outward illumination module 165 can drive the group of pixels 450 indicated by the area at a specific brightness to illuminate a sub-field of view 460 within the larger field of view 230 of the outward display device 210. The group of pixels 450 and the sub-field of view 460 are selected to illuminate a relevant object 420 on the table 410 in environment 400. Figure 4B The front view 400B shows which pixels of the outward-facing display device 210 are included in area 450 for illumination. Figure 4A The subfield of view 460 and associated object 420 are shown in the side view 400A. The group of pixels 450 includes pixels in the middle portion of the outward-facing display device 210, while pixels at the top and bottom of the outward-facing display device 210 remain dark or can be used for other purposes.

[0035] Figure 5Another exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions. In one or more embodiments, certain actions occur as part of determining a set of pixels corresponding to a region of interest in the environment. Other actions include additional functions. However, various actions may be... Figure 5 It occurs elsewhere within the flowchart. For illustrative purposes, it will be... Figure 1 and Figure 2 The following steps are described in the context of [the relevant document / document]. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required. Various actions can be performed remotely by server equipment, by a single electronic device, and / or distributed among multiple electronic devices.

[0036] In some embodiments, the display device also includes an inward-facing display. At 510, media items associated with a brightness standard are optionally displayed on the inward-facing display. For example, a movie may be displayed on the inward-facing display, and the movie may be associated with a brightness standard representing a dark threshold brightness, which is similar to the darkness level of a movie theater. At 520, the outward-facing illumination module 165 receives information indicating the ambient lighting conditions of the outward-facing display device 210.

[0037] In some embodiments, the information 530 indicating ambient lighting conditions includes luminance information 535 indicating the brightness of the ambient lighting conditions and color information 540 indicating the color of the ambient lighting conditions. For example, luminance information 535 may include a luminance level, and color information 540 may include a color temperature. Ambient lighting information 530 for a sunny outdoor environment may include luminance information 535 of approximately 3,500 lumens and color information 540 of approximately 5,500 Kelvin. In contrast, ambient lighting information 530 for an indoor space illuminated only by lamps may include luminance information 535 of approximately 450 lumens and color information 540 of approximately 2,700 Kelvin.

[0038] At 550, the outward illumination module 165 determines whether the ambient lighting conditions meet a brightness standard associated with the media item displayed in frame 510. In some embodiments, the outward illumination module 165 may compare brightness information 535 with a brightness standard to determine whether the brightness standard is met. Based on the determination that the ambient lighting conditions do not meet the brightness standard, the outward illumination module 165 may identify a set of pixels in the outward display device 210 corresponding to the region of interest in frame 560.

[0039] Determining this group of pixels within frame 560 may optionally include frames 565 and / or 570. At frame 565, the outward illumination module 165 may determine the region of interest based on an indicator from a gaze-tracking user interface. For example, the outward illumination module 165 may receive an indicator from the gaze-tracking user interface indicating that the user 250 is looking at a relevant object 420 on a table 410 in the environment 400. At frame 570, the outward illumination module 165 may determine the region of interest based on an input area from a gesture-based user interface. For example, the gesture-based user interface may provide the outward illumination module 165 with an indicator of an input area in which the user 250's hand is gesturing.

[0040] At 580, the outward illumination module 165 can drive the set of pixels defined in frame 560 with brightness 585 and color 540 to improve illumination in the area of ​​interest of the environment. Driving the set of pixels in frame 580 may optionally include frame 590, where the outward illumination module 165 can drive the set of pixels defined in frame 560 with brightness 585, such that the brightness standard associated with the media item displayed in frame 510 is met. Since the ambient lighting information 530 includes color information 540, the outward illumination module 165 can drive the set of pixels in the outward display device 210 with a matching color 540, allowing the user 250 to experience more consistent lighting in the environment. Additionally, the outward illumination module 165 can determine a minimum brightness 585 for driving the pixels, such that the combination of the ambient lighting brightness 535 and the brightness 585 of the outward display device meets the brightness standard without causing the colors in the environment to feel "bleached" or "faded".

[0041] Figure 6 An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 5This is a schematic diagram of an exemplary operating environment 600 for the exemplary process 500 shown. The display device includes an outward-facing display device 210A and an inward-facing display device 210B, as well as an optical sensor 220. The inward-facing display device 210B and the outward-facing display device 210A are positioned such that the projection from the inward-facing display device 210B and the projection from the outward-facing display device 210A do not intersect. That is, the inward-facing display device 210B and the outward-facing display device 210A face different directions in the environment. In the exemplary operating environment 600, the inward-facing display device 210B faces the user 250, and the outward-facing display device 210A faces away from the user 250 and towards the environment. In other operating environments, the inward-facing display device 210B and the outward-facing display device 210A may be located on opposite sides or approximately opposite sides of the display device, on different sides of the display device, etc. The inward-facing display device 210B may display media items associated with a specific brightness standard, such as movie media items and associated brightness standards for a dark cinema environment. Optical sensor 220 can acquire information indicating ambient lighting conditions in environment 600, outward lighting module 165 can determine that the ambient lighting conditions do not meet the associated brightness standard, and accordingly drive selected pixels of outward display device 210A.

[0042] For example, while a brightness standard associated with a movie media item might correspond to a dark movie theater, a gesture-based user interface might have an associated brightness standard representing a threshold brightness below which the gesture-based user interface cannot recognize gestures. An outward illumination module 165 might drive a set of pixels along the bottom of the outward display device 210A to illuminate the input area 660 of the gesture-based user interface and the user 250's hand 260. Alternatively, an eye-tracking interface might determine that the user 250's gaze 650 is pointing towards a relevant object 620 on a table 610. Then, the outward illumination module 165 might drive a set of pixels in the middle portion of the outward display device 210A to illuminate the relevant object 620. Furthermore, the display device might be included in an HMD. An inward display device 210B might be used to present content to the user 250, and an outward display device 210A might be used to present information to people near the user 250. The HMD may also include an eye-tracking system that tracks the gaze 650 of the user 250, and an external camera or other sensor 220 located on the same surface as the outward-facing display device 210A to determine the ambient lighting conditions of the outward-facing display device 210A. The outward-facing lighting module 165 may determine, based on information from the eye-tracking system in the HMD, that the gaze 650 is pointing towards a relevant object 620 on the table 610, and drive a set of pixels in the middle portion of the outward-facing display device 210A to illuminate the relevant object 620. The upper and lower portions of the outward-facing display device 210A remain available for presenting information to people near the user 250.

[0043] Figure 7A further exemplary process according to one or more embodiments is illustrated in flowchart form for operating an outward-facing display in an electronic device under low-light conditions. In one or more embodiments, certain actions occur as a result of receiving information indicating the ambient lighting conditions of the outward-facing display device and determining a portion of a set of pixels in the outward-facing display device corresponding to a region of interest. Other actions include additional functions. However, various actions may be... Figure 7 It occurs elsewhere within the flowchart. For illustrative purposes, it will be... Figure 1 and Figure 2 The following steps are described in the context of [the relevant document / document]. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required. Various actions can be performed remotely by server equipment, by a single electronic device, and / or distributed among multiple electronic devices.

[0044] The flowchart begins at 710, where the outward-facing illumination module 165 receives information indicating ambient lighting conditions for the outward-facing display device. Receiving information indicating ambient lighting conditions in block 710 may optionally include blocks 715 and / or 720. At block 715, an optical sensor 220, including an outward-facing camera, captures images of the environment, and the outward-facing illumination module 165 may determine the ambient lighting conditions based on the images captured from the optical sensor 220. Alternatively, an ambient lighting module 170 may determine the ambient lighting conditions based on the images captured from the optical sensor 220 and provide information indicating the ambient lighting conditions to the outward-facing illumination module 165. At block 720, the outward-facing illumination module 165 receives information indicating ambient lighting conditions for multiple areas of the environment. For example, the outward-facing illumination module 165 may receive information indicating ambient lighting conditions for each of the four quadrants of the environment.

[0045] At 730, the outward-facing lighting module 165 determines whether the ambient lighting conditions meet a brightness criterion. In an embodiment including block 720, step 730 includes determining whether each area meets the brightness criterion. For example, a light in a corner of a room illuminates three of the four quadrants of the environment. The ambient lighting conditions in three of the four quadrants meet the brightness criterion, but the fourth quadrant does not. Based on the determination that one or more areas do not meet the brightness criterion, the outward-facing lighting module 165 may determine a set of pixels in the outward-facing display device 210 corresponding to a region of interest in the environment.

[0046] The group of pixels at frame 740 may optionally include frames 745 and / or frames 750 to 755. At frame 745, the outward illumination module 165 determines the region of interest based on one or more specific areas in the environment that do not meet a brightness standard. For example, when the region of interest is not included in a specific area, the outward illumination module 165 may determine that the region of interest requires less additional illumination and includes fewer pixels in the group of pixels compared to when the region of interest is included in the specific area and requires more additional illumination. At frame 750, the outward illumination module 165 determines a change in position of the outward-facing display device 210 and determines the group of pixels in frame 755 based on the determined change in position. For example, an accelerometer included in sensor 145 may determine that the user 250 holding the outward-facing display device 210 has moved from a seated position to a standing position. The outward illumination module 165 may then determine that the region of interest is the ground and the path that the user 250 expects to move along. The outward illumination module 165 determines the set of pixels corresponding to the area of ​​interest, including pixels in the bottom of the outward display device 210 that illuminate the ground in front of the user 250's feet.

[0047] At 760, the outward illumination module 165 drives the set of pixels at a specific brightness to improve illumination in the area of ​​interest. In an embodiment where the display device also includes an inward display device, at block 770, media items may optionally be displayed on the inward display device based on the set of pixels and a determined change in position. Returning to the example of user 250 moving from a seated to a standing position, media items indicating a recommended path through the environment may be displayed on the inward display device. Alternatively, media items containing warnings of obstacles identified in the environment may be displayed on the inward display device.

[0048] Figures 8A to 8B An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 7 A schematic diagram of an exemplary operating environment 800 for the exemplary process shown. Figure 8A In view 800A, corresponding to the perspective of user 250, the environment is divided into two areas: area 820 to the left of user 250 and area 830 to the right of user 250. Area 830 includes a window 810 through which sunlight can illuminate the room. The ambient lighting conditions in area 830 meet the brightness standard, while the ambient lighting conditions in area 820, which has no window, do not meet the brightness standard. Figure 8B The front view 800B is shown, which shows which pixels of the display are included in area 850 for illumination. Figure 8A The area 820 shown in view 800A. The group of pixels 850 includes pixels along the right side of the display, while pixels along the left side of the display and pixels corresponding to area 830 and window 810 remain dark.

[0049] Figure 9 An exemplary process 900 for determining ambient lighting conditions according to one or more embodiments is illustrated in flowchart form. In one or more embodiments, certain actions occur as part of determining the ambient color of interest. Other actions include additional functions. However, various actions may be... Figure 7 It occurs elsewhere within the flowchart. For illustrative purposes, it will be... Figure 1 and Figure 2 The following steps are described in the context of [the relevant document / document]. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, depending on the implementation, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be required. Various actions can be performed remotely by server equipment, by a single electronic device, and / or distributed among multiple electronic devices.

[0050] At 910, outward illumination module 165 drives outward display device 210 with a specific color from the color sequence. For example, the color sequence may include a sequence of red, orange, yellow, green, blue, and purple. At 920, outward illumination module 165 causes an outward camera (such as optical sensor 220) to capture an image of the environment corresponding to a color in the color sequence driven at 910. In some embodiments, optical sensor 220 includes a monochrome camera, which is more sensitive in low-light conditions than a red / green / blue camera. At 930, outward illumination module 165 determines whether any remaining color exists in the color sequence. If a remaining color exists, outward illumination module 165 returns to 910 and drives outward display device 210 with another color from the color sequence.

[0051] Once images corresponding to each color in the color sequence have been captured, the outward illumination module 165 may optionally determine the ambient color of interest based on the set of images within block 940. For example, the outward illumination module 165 may determine the color temperature of ambient lighting in the environment, the color gamut of objects in the environment, etc. Determining the ambient color of interest within block 940 may optionally include block 945. In an embodiment where the optical sensor 220 includes a monochrome camera and the set of images comprises a set of monochrome images, the outward illumination module 165 identifies at 945 the image containing the strongest signal and which color in the color sequence corresponds to that particular image. Therefore, the system gains the benefit of improved monochrome camera sensitivity without sacrificing color information.

[0052] At 950, the outward illumination module 165 determines whether the ambient lighting conditions meet a brightness standard. Based on the determination that the ambient lighting conditions do not meet the brightness standard, the outward illumination module 165 can identify a set of pixels in the outward display device 210 corresponding to the region of interest within frame 960. The outward illumination module 165 then drives this set of pixels at a specific brightness at frame 970 to improve illumination in the region of interest.

[0053] In an embodiment where the outward-facing lighting module 165 determines the ambient color of interest in box 940, box 975 may optionally be included as the location where the set of pixels is driven at a specific brightness. In box 975, the outward-facing lighting module 165 may drive the set of pixels with a specific color based on the ambient color of interest. For example, the ambient color of interest may indicate that furniture in the environment is blue, and ambient lighting conditions may indicate that the room is dark. Although the outward-facing lighting module 165 could default to driving the set of pixels with red in a dark environment so that the user 250's dark vision is not impaired, red light might be absorbed by the blue furniture rather than reflected, making it impossible for the user 250 to see the furniture. Instead, the outward-facing lighting module 165 could specifically drive the set of pixels with blue to ensure that the user 250 can see the blue furniture.

[0054] Figure 10 An outward-facing display in an electronic device according to one or more embodiments is shown performing... Figure 9 This is a schematic diagram of an exemplary operating environment 1000 for an exemplary process. In environment 1000, a table 1010 and a display case 1020 are made of light-colored wood and placed on a dark carpet 1030. An optical sensor 220 may include an outward-facing camera that captures a set of images of the environment 1000 corresponding to each color in a color sequence. An outward-facing lighting module 165 may determine the ambient color of interest corresponding to the colors of the table 1010 and the display case 1020 based on the set of images from the optical sensor 220. A user 250 may change from a seated to a standing position, and the outward-facing lighting module 165 may drive a set of pixels in an outward-facing display device 210 to illuminate the user 250's walking path. The outward-facing lighting module 165 may drive the set of pixels with specific colors reflected from the table 1010 and the display case 1020, allowing the user 250 to clearly see the table and display case and avoid them as they walk through environment 1000.

[0055] Figures 11A to 11B An exemplary computer system according to one or more embodiments is shown in block diagram form. Figure 11A and Figure 11B An exemplary system 1100 for various extended reality technologies is depicted.

[0056] In some examples, such as Figure 11AAs shown, system 1100 includes device 1100a. Device 1100a includes various components such as processor 1102, RF circuitry 1104, memory 1106, image sensor 1108, orientation sensor 1110, microphone 1112, position sensor 1116, speaker 1118, display 1120, and touch-sensitive surface 1122. Power supply 1175 may include a rechargeable battery (e.g., a lithium-ion battery) or other electrical connection to a power source (e.g., a mains power supply) for managing the electronics and associated circuitry of device 1100a and / or providing power to the electronics and associated circuitry of the device. These components optionally communicate via a communication bus 1150 of device 1100a.

[0057] In some examples, components of system 1100 are implemented in a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop computer), and other components of system 1100 are implemented in a second device (e.g., a head-mounted device). In some examples, device 1100a is implemented in either the base station device or the second device.

[0058] like Figure 11B As shown, in some examples, system 1100 includes two (or more) communicating devices, such as via wired or wireless connections. A first device 1100B (e.g., a base station device) includes a processor 1102, RF circuitry 1104, and memory 1106. These components optionally communicate via a communication bus 1150 of device 1100B. A second device 1100C (e.g., a head-mounted device) includes various components such as processor 1102, RF circuitry 1104, memory 1106, image sensor 1108, orientation sensor 1110, microphone 1112, position sensor 1116, speaker 1118, display 1120, and touch-sensitive surface 1122. These components optionally communicate via a communication bus 1150 of device 1100C.

[0059] System 1100 includes processor 1102 and memory 1106. Processor 1102 includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, memory 1106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 1102 to perform the techniques described below.

[0060] System 1100 includes RF circuitry 1104. RF circuitry 1104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 1104 optionally includes circuitry for using near-field communication and / or short-range communication (such as Bluetooth). ® The circuit that enables communication.

[0061] System 1100 includes a display 1120. Display 1120 may be opaque. Display 1120 may also be transparent or translucent, and may be combined with a substrate through which light representing an image is directed to an individual's eye. Display 1120 may incorporate LED, OLED, digital light projector, laser scanning light source, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate may be an optical waveguide, optical combiner, light reflector, holographic substrate, or any combination of these substrates. In one example, the transparent or translucent display may selectively switch between an opaque state and a transparent or translucent state. Other examples of display 1120 include head-up displays, automotive windshields capable of displaying graphics, windows capable of displaying graphics, lenses capable of displaying graphics, tablet computers, smartphones, and desktop or laptop computers. Alternatively, system 1100 may be designed to receive an external display (e.g., a smartphone). In some examples, system 1100 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical environment (e.g., onto a physical surface or as a hologram).

[0062] In some examples, system 1100 includes a touch-sensitive surface 1122 for receiving user input, such as tap and swipe input. In some examples, display 1120 and touch-sensitive surface 1122 form a touch-sensitive display.

[0063] System 1100 includes an image sensor 1108. Image sensor 1108 optionally includes one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors, operable to acquire images of physical elements from a physical environment. One or more image sensors also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from a physical scene. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the physical scene. Image sensor 1108 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical environment. Image sensor 1108 also optionally includes one or more depth sensors configured to detect the distance between physical elements and system 1100. In some examples, system 1100 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical environment surrounding system 1100. In some examples, image sensor 1108 includes a first image sensor and a second image sensor. The first and second image sensors are optionally configured to capture images of physical elements in a physical setting from two different perspectives. In some examples, system 1100 uses image sensor 1108 to receive user input, such as gestures. In some examples, system 1100 uses image sensor 1108 to detect the position and orientation of system 1100 and / or display 1120 in the physical environment. For example, system 1100 uses image sensor 1108 to track the position and orientation of display 1120 relative to one or more fixed elements in the physical environment.

[0064] In some examples, system 1100 includes microphone 1112. System 1100 uses microphone 1112 to detect sound from a user and / or the user's physical environment. In some examples, microphone 1112 includes a microphone array (comprising multiple microphones) that optionally cooperate to identify ambient noise or locate sound sources in the space of the physical environment.

[0065] System 1100 includes an orientation sensor 1110 for detecting the orientation and / or movement of system 1100 and / or display 1120. For example, system 1100 uses orientation sensor 1110 to track changes in the position and / or orientation of system 1100 and / or display 1120, such as relative to physical objects in the physical environment. Orientation sensor 1110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0066] The technologies defined in this document take into account options for obtaining and utilizing users' personal information. For example, such personal information may be used to provide multi-user communication sessions on electronic devices. However, the extent to which such personal information is collected should be based on the user's informed consent, enabling the user to be aware of and control the use of their personal information.

[0067] Parties authorized to access personal information will use it only for lawful and reasonable purposes and will comply with privacy policies and practices that at least meet appropriate laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, personal information may not be distributed, sold, or otherwise shared for any purpose other than a lawful and reasonable one.

[0068] However, users can limit the extent to which parties can access their personal information. The processes and devices described herein may allow for changes to settings or other preferences that enable users to control access to their personal information. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. For example, a user's personal information may be obscured or otherwise generalized so that it cannot identify a specific user from whom the information was obtained.

[0069] It should be understood that the above description is intended to be exemplary and not restrictive. The material has been presented to enable any person skilled in the art to make and use the disclosed matters protected by the claims, and is provided in the context of a particular embodiment, variations of which will be readily apparent to a person skilled in the art (e.g., some embodiments of the disclosed embodiments may be used in combination with each other). Therefore, Figure 3 , Figure 5 , Figure 7 and Figure 9 The specific arrangement of the steps or actions shown or Figure 1 , Figure 2 The arrangement of the elements shown in Figure 11 should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “comprising” and “therein” are used as common English equivalents of the corresponding terms “including” and “wherein”.

Claims

1. A display method, comprising: Information indicating ambient lighting conditions in a physical environment is received at a head-mounted device, wherein the head-mounted device includes an outward-facing display device pointing towards the physical environment and an inward-facing display device pointing towards the user, and wherein the projection from the inward-facing display device and the projection from the outward-facing display device do not intersect; Based on the determination that the ambient lighting conditions in the physical environment do not meet the brightness standard, a set of pixels in the outward display device is determined to be configured to illuminate the region of interest in the physical environment; as well as The set of pixels is driven at a specific brightness to project illumination toward the region of interest.

2. The display method according to claim 1, wherein the information indicating ambient lighting conditions includes the brightness of the ambient lighting conditions and the color temperature of the ambient lighting conditions.

3. The display method according to claim 2, wherein driving the group of pixels with the specific brightness includes driving the group of pixels with the specific brightness and the light color.

4. The display method according to any one of claims 2 to 3, wherein: Receiving the information indicating the ambient lighting conditions includes receiving information indicating the ambient lighting conditions of multiple areas of the environment; Determining that the ambient lighting conditions do not meet the luminance standard includes determining that the ambient lighting conditions in at least one of the plurality of areas of the environment do not meet the luminance standard; and The region of interest corresponds to at least one of the plurality of regions of the environment.

5. The display method of claim 1 further includes determining the region of interest based on indicators from the eye-tracking user interface.

6. The display method of claim 1, wherein receiving the information indicating ambient lighting conditions comprises capturing an image of the environment by an outward-facing camera.

7. The display method of claim 6, wherein the outward-facing camera is arranged to face the same direction as the outward-facing display device.

8. The display method according to any one of claims 6 to 7, further comprising: The color sequence is emitted by the outward display device; The environment is captured by the outward-facing camera for each color in the color sequence to obtain a set of images of the environment; and Determining the color of interest environment based on the set of images, wherein driving the set of pixels with the specific brightness includes driving the set of pixels with the specific brightness and the specific color based on the color of interest environment, wherein the outward-facing camera includes a monochrome camera, and wherein determining the color of interest environment includes identifying a specific image from the set of images, the specific image including the strongest signal and the corresponding color of the color sequence.

9. The display method according to claim 1, wherein the area of ​​interest corresponds to an input area of ​​the environment, wherein the input area includes a gesture-based user interface.

10. The display method of claim 1, further comprising displaying a media item associated with the brightness standard on the inward display device.

11. The display method of claim 1, further comprising determining a positional change of the outward-facing display device based on a signal from an orientation sensor, wherein determining the set of pixels is further based on the determined positional change.

12. The display method according to claim 1, further comprising: Receive information indicating updated ambient lighting conditions for the outward-facing display device; If the combination of the specific brightness and the updated ambient lighting conditions does not meet the brightness standard, adjust the specific brightness and at least one of the group of pixels. as well as Drive the set of pixels or the adjusted set of pixels at the specified brightness or the adjusted specific brightness.

13. The display method according to claim 12, wherein: The information indicating the ambient lighting conditions includes the color of light under those conditions; The information indicating the updated ambient lighting conditions includes the updated color temperature of the updated ambient lighting conditions; Driving the group of pixels with the specific brightness includes driving the group of pixels with both the specific brightness and the color of light; Adjusting the specific brightness and at least one of the set of pixels includes adjusting at least one of the specific brightness, the set of pixels, and the updated light color; and Driving the set of pixels or the adjusted set of pixels with the specified brightness or the adjusted specific brightness includes driving the set of pixels or the adjusted set of pixels with the specified brightness or the adjusted specific brightness and with the light color or the updated light color.

14. A non-transitory computer-readable medium comprising computer code, said computer code being executable by one or more processors to: Information indicating ambient lighting conditions in a physical environment is received at a head-mounted device, wherein the head-mounted device includes an outward-facing display device pointing towards the physical environment and an inward-facing display device pointing towards the user, wherein the projection from the inward-facing display device and the projection from the outward-facing display device do not intersect; Based on the determination that the ambient lighting conditions in the physical environment do not meet the brightness standard, a set of pixels in the outward display device is determined to be configured to illuminate the region of interest in the physical environment; as well as The set of pixels is driven at a specific brightness to project illumination toward the region of interest.

15. The non-transitory computer-readable medium of claim 14, wherein the information indicating ambient lighting conditions includes the brightness and color of the ambient lighting conditions, and wherein the computer code for driving the set of pixels at the specific brightness includes computer code for driving the set of pixels at the specific brightness and the color.

16. The non-transitory computer-readable medium according to claim 15, wherein: The computer code for receiving the information indicating the ambient lighting conditions includes computer code for receiving the information indicating the ambient lighting conditions of multiple areas of the environment; Determining that the ambient lighting conditions do not meet the luminance standard includes determining that the ambient lighting conditions in at least one of the plurality of areas of the environment do not meet the luminance standard; and The region of interest corresponds to at least one of the plurality of regions of the environment.

17. The non-transitory computer-readable medium of claim 16, wherein the computer code for receiving the information indicating ambient lighting conditions includes computer code for capturing an image of the environment by an outward-facing camera, wherein the outward-facing camera is arranged to face the same direction as the outward-facing display device.

18. The non-transitory computer-readable medium of claim 17, further comprising computer code for performing the following operations: The color sequence is emitted by the outward display device; The environment is captured by the outward-facing camera for each color in the color sequence to obtain a set of images of the environment; and The set of images determines an interest environment color, wherein the computer code for driving the set of pixels at the specific brightness includes computer code for driving the set of pixels at the specific brightness and with a specific color based on the interest environment color.

19. The non-transitory computer-readable medium of claim 14, wherein the region of interest corresponds to an input region of the environment, wherein the input region includes a gesture-based user interface.

20. An electronic system comprising: Head-mounted device, the head-mounted device comprising: Outward-facing display devices that point to the physical environment; An inward-facing display device pointing towards the user, wherein the projection from the inward-facing display device and the projection from the outward-facing display device do not intersect; One or more processors; and One or more computer-readable media including computer code, said computer code being executable by said one or more processors to: Receive information indicating ambient lighting conditions in the physical environment; Based on the determination that the ambient lighting conditions in the physical environment do not meet the brightness standard, a set of pixels in the outward-facing display device configured to illuminate the region of interest in the physical environment is determined; and The set of pixels is driven at a specific brightness to project illumination toward the region of interest.

21. A computer program product comprising a computer program, said computer program including computer code capable of being executed by one or more processors to perform the display method according to any one of claims 1 to 13.

Images