Dynamic brightness compensation in display assembly

Abstract

The embodiment of the invention relates to a display assembly with dynamic brightness compensation. The display assembly includes a display element, a controller, and a backlight device. The controller obtains gaze information of an eye of a user, generates a transmit instruction based at least in part on the gaze information, and dynamically allocates at least a portion of a plurality of light sources in the backlight device to a plurality of regions based at least in part on the gaze information. The plurality of light sources emit light through the display element, and the display element spatially modulates light received from the backlight device to generate image light. Each subset of the plurality of light sources dynamically assigned to a respective area emits a portion of the light having a brightness adjusted by a respective adjustment value according to an emission instruction.

Description

Technical Field

[0001] The present disclosure relates generally to a display device, and in particular, to dynamic brightness compensation in a display assembly. Background Art

[0002] Common display technologies include liquid crystal displays (LCD), organic light emitting diode (OLED) displays, and projection systems. Some display technologies include a backlight that illuminates a spatial light modulator (SLM). LCD is an example of an SLM. Displays with backlights can also be integrated into head-mounted displays (HMDs), for example for artificial reality applications. However, due to the lenses and lens technology used in HMDs, the display brightness in HMDs decreases off-axis. Therefore, it is necessary to compensate and adjust (e.g., increase or decrease) the display brightness in displays with backlights. Summary of the invention

[0003] Embodiments of the present disclosure relate to a display component with dynamic brightness compensation. According to a first aspect, a display component is provided, the display component comprising: a display element; a controller configured to: obtain gaze information of a user's eyes, generate an emission instruction based at least in part on the gaze information, and dynamically allocate at least a portion of a plurality of light sources to a plurality of regions based at least in part on the gaze information; and a backlight device, the backlight device comprising the plurality of light sources emitting light passing through the display element, each subset of the plurality of light sources dynamically allocated to a corresponding region being configured to emit a portion of the light, the portion of the light having a brightness adjusted by a corresponding adjustment value according to the emission instruction.

[0004] The controller may also be configured to set the corresponding adjustment value to a higher value for a corresponding area that is farther from the gaze point of the user's eyes included in the gaze information. Alternatively or additionally, the controller may also be configured to set the corresponding adjustment value to a lowest value among a plurality of adjustment values ​​for a corresponding area corresponding to the gaze point of the user's eyes included in the gaze information. Alternatively or additionally, the controller may also be configured to set the corresponding adjustment value based on the distance from the geometric center of the corresponding area to the gaze point of the user's eyes included in the gaze information.

[0005] The controller may also be configured to adjust the size of the corresponding area based at least in part on the gaze information. Alternatively or additionally, the controller may also be configured to adjust the size of the corresponding area according to a distance from a geometric center of the corresponding area to a gaze point of the user's eye included in the gaze information. The size of the area including the gaze point in the plurality of areas may be fixed.

[0006] The controller may be further configured to load a plurality of adjustment values ​​from a non-transitory computer-readable medium coupled to the controller based at least in part on the gaze information. The controller may be further configured to generate a firing instruction using the loaded plurality of adjustment values.

[0007] Gaze information may include information about the location of the pupils of the user's eyes.

[0008] Each of the plurality of light sources may include a light emitting diode (LED). The plurality of light sources may include a two-dimensional array of light emitting diodes (LEDs).

[0009] The display assembly may be part of a head mounted device (headset). The head mounted device may also include an eye tracker configured to: determine gaze information; and provide the gaze information to the controller.

[0010] According to a second aspect, a method is provided, the method comprising: obtaining gaze information of a user's eyes; generating an emission instruction based at least in part on the gaze information; dynamically allocating at least a portion of a plurality of light sources in a backlight device to a plurality of areas based at least in part on the gaze information; and emitting light from the backlight device through a display element so that each subset of the plurality of light sources dynamically allocated to a corresponding area emits a portion of the light, the portion of the light having a brightness adjusted by a corresponding adjustment value according to the emission instruction.

[0011] The method may further include: for a corresponding area that is farther from a gaze point of the user's eyes included in the gaze information, setting the corresponding adjustment value to a higher value. Alternatively or additionally, the method may further include: for a corresponding area corresponding to a gaze point of the user's eyes included in the gaze information, setting the corresponding adjustment value to a lowest value among a plurality of adjustment values. Alternatively or additionally, the method may further include: setting the corresponding adjustment value based on a distance from a geometric center of the corresponding area to a gaze point of the user's eyes included in the gaze information.

[0012] The method may further include adjusting the size of the corresponding area based at least in part on the gaze information. Alternatively or additionally, the method may further include adjusting the size of the corresponding area according to a distance from a geometric center of the corresponding area to a gaze point of the user's eye included in the gaze information.

[0013] According to a third aspect, a head-mounted device is provided, comprising: a display component according to the first aspect; an eye tracker configured to determine gaze information of a user's eyes; and an optical component configured to guide image light to an eye-box of the head-mounted device corresponding to the position of the user's eyes; wherein a controller is configured to obtain the gaze information of the user's eyes by receiving the gaze information from the eye tracker; and wherein the display element is configured to spatially modulate the light received from the backlight device to generate image light. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A and Figure 1B is a schematic diagram of a head mounted display (HMD) including a near-eye display (NED).

[0015] Figure 2A yes Figure 1A or Figure 1B A cross-sectional view of the HMD is shown in FIG.

[0016] Figure 2B is a cross-sectional view of a display assembly having a backlight device, which display assembly may be Figure 1A or Figure 1B A portion of the HMD is shown in FIG.

[0017] Figure 3 An example backlight device having an array of backlight light emitting diodes is shown.

[0018] Figure 4 An example backlight device having multiple dimming zones is shown.

[0019] Figure 5A A first example backlight apparatus is shown having different brightness zones for brightness compensation relative to a first gaze position of a user's eye.

[0020] Figure 5B A second example backlight apparatus is shown having different brightness zones for brightness compensation relative to a second gaze position of a user's eye.

[0021] Figure 5CA third example backlighting apparatus is shown having different brightness zones for brightness compensation relative to a third gaze position of a user's eye.

[0022] Figure 6 is a flow chart illustrating a process of performing dynamic brightness compensation at a display component.

[0023] Figure 7 is a block diagram of a system environment including an HMD.

[0024] The figures depict various example arrangements for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternatives to the structures and methods shown herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0025] The brightness of light emitting diodes (LEDs) in a backlight device of a display assembly is controlled to compensate for the decrease in brightness of light incident on a user's pupil when the user's pupil gazes away from the center of a display element (i.e., a display panel), and also to compensate for uneven brightness on the display element due to the placement of optical devices between the backlight device and the user's pupil. The backlight device can be dynamically assigned to multiple areas, including a central area and peripheral areas of the user's gaze that change based on the direction of the user's gaze. The subset of LEDs in the backlight device assigned to these areas is dynamically updated based on the user's gaze.

[0026] The display components proposed in this article can be integrated into a wearable device (e.g., a head-mounted display (HMD) or a head-mounted device), a mobile device, or any other hardware platform capable of providing artificial reality content to a user.

[0027] Embodiments of the present disclosure may include an artificial reality system, or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and the artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of the video, audio, tactile feedback, or some combination thereof may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or otherwise used in artificial reality (e.g., performing activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including an HMD connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0028] Figure 1A and Figure 1B 1 is a schematic diagram of a head mounted display (HMD) 100 including a near eye display (NED) 110. The NED 110 can present media to a user. Examples of media that can be presented by the NED 110 include one or more images, videos, audio, or some combination thereof. In some examples, audio can be presented via an external device (e.g., a speaker and / or headphones) that receives audio information from the HMD 100, a console (not shown), or both, and presents audio data to the user based on the audio information. The HMD 100 is typically configured to operate as a VR HMD. However, in some examples, the HMD 100 can be modified to also operate as an AR HMD, an MR HMD, or some combination thereof. For example, in some examples, the HMD 100 can use computer-generated elements (e.g., static images, videos, sounds, etc.) to enhance the view of a physical, real-world environment.

[0029] like Figure 1A or Figure 1BThe HMD 100 shown may include a frame 105 and a display 110. The frame 105 may include one or more optical elements that together display media to a user. That is, the display 110 may be configured to allow a user to view content presented by the HMD 100. Figure 2A As discussed, display 110 may include at least one source component to generate image light to present optical media to a user's eyes. The source component may include, for example, a source, an optical system, or some combination thereof.

[0030] Figure 1A and Figure 1B These are merely examples of virtual reality systems, and the display systems described herein may be incorporated into other such systems.

[0031] Figure 2A yes Figure 1A or Figure 1B 2. Cross-section 200 of HMD 100 is shown in FIG. Cross-section 200 may include at least one display component 210, eye tracker 215, and exit pupil 230. Exit pupil 230 is where eye 220 may be located when a user wears HMD 100. In some examples, frame 105 may represent the frame of eye-wear glasses. For illustration purposes, Figure 2A The cross section 200 is shown associated with a single eye 220 and a single display assembly 210, but in an alternative example not shown, Figure 2A Another display assembly separate from or integrated with the display assembly 210 shown may provide image light to the user's other eye.

[0032] The display assembly 210 can direct the image light to the eye 220 through the exit pupil 230. The display assembly 210 can be composed of one or more materials (e.g., plastic, glass, etc.) having one or more refractive indices that effectively reduce the weight of the HMD 100 and widen the field of view of the HMD 100.

[0033] Eye tracker 215 tracks the position of eye 220. Although Figure 2A Only one eye 220 of the user is shown in FIG, but in some examples, the eye tracker 220 can track the position of each eye 220 of the user. In some examples, a separate eye tracker 215 can be implemented for each eye 220 of the user.

[0034] In an alternative configuration, HMD 100 may include one or more optical elements (not shown) between display assembly 210 and eye 220. As various examples, these optical elements may be used to correct aberrations of image light exiting from display assembly 210, amplify image light exiting from display assembly 210, perform some other optical adjustment to image light exiting from display assembly 210, or a combination of the above operations. Example optical elements may include an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, or any other suitable optical element that can affect image light.

[0035] In some examples, the display component 210 may include a light source component for generating image light to present media to the user's eye. The light source component may, for example, include a light source, an optical device system, or some combination thereof. According to various examples, the light source component may include a light emitting diode (LED), such as an organic light emitting diode (OLED). According to various examples, the display component 210 may be implemented as a liquid crystal display (LCD) component, an OLED display component, or some other type of display component configured to emit color image light to the eye 220. The following combination Figure 2B Further details are provided regarding the structure and operation of display assembly 210 .

[0036] Figure 2B 2 is a cross-sectional view 240 of display 110. Cross-sectional view 240 shows components of display assembly 210 and eye tracker 215, and controller 270 coupled to display assembly 210 and eye tracker 215. Display assembly 210 may include backlight device 245, optical element 252, display element 255, and optical element 257.

[0037] The backlight device 245 can emit light 247 that passes through the display element 255. The backlight device 245 may include a plurality of light sources 250, which, for example, generate light 247 based at least in part on an emission instruction from a controller 270. These light sources 250 may be organized as a one-dimensional array or a two-dimensional array of light sources. The light source 250 may be an LED, a mini-LED, a superluminescent LED (SLED), an OLED, some other type of light source, or a combination thereof. The light source 250 may be configured to generate the light 247 as white light (visible light). The light source 250 may include LEDs that include a yellow phosphor that is illuminated by a blue LED dye to generate white light. Light source 250 may include red-green-blue (RGB) LED dyes included in the same electronic package, or may include discrete red LEDs, green LEDs, and blue LEDs in separate electronic packages that are positioned close enough so that a "mixing distance" of red, green, and blue light emitted from the red, green, and blue LEDs mixes into white light before light 247 encounters display element 255. In some embodiments, light source 250 is a blue LED configured to illuminate quantum dots to generate white light.

[0038] Optical element 252 can be disposed between backlight 245 and display element 255, for example, to diffuse and / or polarize light 247 emitted from backlight 245. Optical element 252 can include one or more optical layers, such as one or more optical diffuser films and / or one or more polarizer films.

[0039] The display element 255 can be a display panel that spatially modulates the light 247 received from the backlight device 245 to generate image light (e.g., content). The display element 255 can be an LCD, an active-matrix liquid crystal display (AMLCD), or some other type of display that spatially modulates the received light. In some examples, the display element 255 includes a color filter array covering a liquid crystal array. The color filter array can be, for example, a Bayer pattern or some other color pattern. The light 247 from the backlight device 245 can be modulated by the color filter array and the liquid crystal array to form image light. In some examples, the optical element 257 includes the color filter array. The display element 255 emits image light toward the optical element 257. In some examples, the display element 255 includes a light source 250 (e.g., an OLED) that directly emits image light (content). In this case, the display element 255 can be, for example, an OLED display, and the backlight device 245 can be omitted from the display component 210.

[0040] The optical element 257 may amplify the image light received from the display element 255, correct the optical errors associated with the image light, and present the corrected image light to the user of the HMD 100. The optical element 257 directs the amplified and / or corrected image light toward the exit pupil 230 for presentation to the user wearing the HMD 100. In various examples, the optical element 257 may be implemented as one or more optical elements. Example optical elements included in the optical element 257 may include an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light.

[0041] The eye tracker 215 that tracks the position of the eye 220 may include a light emitter 260 and a camera 265. The light emitter 260 emits light toward the exit pupil 230. The light emitter 260 may, for example, illuminate the eye 220 with infrared (IR) light, and the IR light reflected / scattered by at least one surface (e.g., the pupil) of the eye 220 may be captured by the camera 265. In some examples, the light emitter 260 is implemented to illuminate the eye 220 so that the flash on the cornea of ​​the eye 220 can be captured by the camera 265. In some examples, the light emitter 260 emits structured light. In an alternative example, the light emitter 260 emits light pulses for determining depth via time of flight. The light emitter 260 may include one or more light sources, such as one or more LEDs, one or more organic LEDs, one or more microLEDs, some other type of light source, or some combination thereof. The camera 265 captures an image of the eye 220. The camera 265 can be, for example, a (one-dimensional or two-dimensional) array of photodiodes, a charge coupled display (CCD) array, some other device capable of detecting light, or some combination thereof. The camera 265 can be configured to detect light in the IR band. The camera 265 can capture images based on instructions received from the controller 270.

[0042] Controller 270 may control components of display assembly 210 and eye tracker 215. Controller 270 may generate tracking instructions for light emitter 260 and camera 265 of eye tracker 215. In some examples, controller 270 receives one or more images of eye 220 captured by camera 265. Controller 270 may use the images from camera 265 to determine eye tracking information for eye 220. For example, controller 270 identifies the location of reflections of light from light emitter 260 in the image of eye 220. Controller 270 determines the location and orientation of eye 220 based on the shape and / or location of the identified reflections. The determined location and orientation of eye 220 may be referred to herein as "gaze information" or "gaze location" and may include information about the location and / or orientation of one or more surfaces (e.g., pupil) of eye 220. In the case where the target area of ​​the eye 220 is illuminated by the structured light pattern emitted from the light emitter 260, the controller 270 can detect the distortion of the structured light pattern projected onto the eye 220 and estimate the gaze position of the eye 220 based on the detected distortion. The controller 270 can also estimate the pupil axis, the gaze angle (e.g., corresponding to the foveal axis), the translation of the eye 220, the torsion of the eye 220, and the current shape of the eye 220 based on one or more images captured by the camera 265. Alternatively, the eye tracker 215 includes a controller coupled to the light emitter 260 and the camera 265. The internal controller of the eye tracker 215 can determine the gaze information of the eye 220 and provide the gaze information (i.e., information about the gaze position) to the controller 270.

[0043] The controller 270 may also generate emission instructions for the display assembly 210 based at least in part on the gaze information of the eye 220. The controller 270 may, for example, provide emission instructions to the light source 250 of the backlight device 245. The emission instructions from the controller 270 may include an electrical signal (e.g., a voltage signal or a current signal) that controls the light emission from the light source 250 at a specific brightness level. For example, an electrical signal with a higher amplitude level (e.g., a higher voltage level or a higher current level) generated by the controller 270 and provided to a corresponding subset of the plurality of light sources 250 may cause the light source 250 of the corresponding subset to emit a portion of the light 247 with a higher brightness level. And, conversely, an electrical signal with a lower amplitude level generated by the controller 270 may cause the light source 250 of the corresponding subset to emit a portion of the light 247 with a lower brightness level.

[0044] In some examples, the controller 270 dynamically allocates at least a portion of the plurality of light sources 250 to a plurality of brightness regions using emission instructions. Each subset of the light sources 250 dynamically allocated to a corresponding brightness region according to the emission instructions (i.e., based on the gaze information of the eye 220) can emit a portion of the light 247 having a brightness adjusted by a corresponding adjustment value according to the emission instructions. The corresponding adjustment value can be, for example, in the range of 0% to 120% of a predetermined maximum brightness value of a single light source 250.

[0045] The controller 270 may be coupled to a non-transitory computer-readable medium (i.e., a memory) that is used, for example, to store a look-up table (LUT) having a list of adjustment values. A set of adjustment values ​​from the list may be a function of gaze information (e.g., information about the gaze position or pupil position of the user's eye). Alternatively, the LUT having the list of adjustment values ​​may be stored in an internal memory of the controller 270 (e.g., a non-transitory computer-readable medium). Once the controller 270 loads a particular set of adjustment values ​​from the LUT, the controller 270 may map the loaded set of adjustment values ​​to appropriate emission instructions (e.g., appropriate voltage signal levels and / or current signal levels) to control each subset of the multiple light sources 250 dynamically assigned to the corresponding area to emit a portion of the light 247 having a brightness adjusted to the corresponding adjustment value. The following references Figure 5A , Figure 5B and Figure 6 Additional details are provided regarding dynamically controlling the brightness of portions of light 247 emitted from backlight 245 based on gaze information of eye 220 .

[0046] Figure 3 An example backlight device 300 having an array of backlight LEDs 310 is shown. The backlight device 300 can be used as the backlight device 245, and the backlight LEDs 310 can be used as the light source 250. The array of backlight LEDs 310 can be arranged on the backlight device 300 as a two-dimensional array of LEDs. The operation of the backlight LEDs 310 can be controlled by a controller 315 by providing emission instructions that are generated based at least in part on the gaze information obtained from the eye tracker 320. The controller 315 can be configured as a drive circuit for driving the two-dimensional array of LEDs 310 using a voltage signal and / or a current signal. The controller 315 can be the controller 270, and the eye tracker 320 can be the eye tracker 215.

[0047] The array of backlight LEDs 310 may be used in a direct-lit display architecture in which a two-dimensional array of backlight LEDs 310 is arranged to illuminate a display pixel array (e.g., an LCD display pixel array) that is similar in shape to the backlight device 300. Figure 3 3. Although shown as approximately square in FIG. 3, backlight device 300 may be rectangular, octagonal, hexagonal, or otherwise shaped to illuminate a corresponding LCD (eg, display element 255) with illumination light (eg, as shown in FIG. 3). Figure 4 and FIG. 5A to FIG. 5C ) in a suitable configuration. The two-dimensional array of backlight LEDs 310 can be separated into multiple rows and columns. For example, the two-dimensional array of backlight LEDs 310 can be configured as a 25×20 matrix of 500 LEDs in total. However, matrices of other sizes can also be used. In some examples, the density of the two-dimensional array of backlight LEDs 310 is substantially uniform on the backlight device 300. In some other examples, the density of the two-dimensional array of backlight LEDs 310 is uneven on the backlight device 300. In some examples, the two-dimensional array of LEDs 310 includes one or more white LED strings (emitting white visible light), or includes red / green / blue LEDs that are separated by a sufficient mixing distance to allow red light, green light, and blue light to mix into white light to illuminate the display pixel array (e.g., display element 255).

[0048] Figure 4 An example backlight device 400 having multiple dimming zones is shown. The backlight device 400 may be the backlight device 245 or the backlight device 300. The backlight device 400 may emit light having a predetermined brightness distribution profile, which may be characterized by a set of dimming zones, such as Figure 4 Dimming zones 405A, 405B, 405C, 405D, 405E, 405F, 405G, 405H are shown. Each dimming zone 405A to 405H can be associated with a specific dimming (or brightness) level of light emitted from a corresponding portion of the light source in the backlight device 400. For example, each dimming zone 405A to 405H can be independently adjusted based on an emission instruction from a controller 410 coupled to the backlight device 400. In some examples, the controller 410 can use a specific algorithm to determine the adjustment value of each dimming zone 405A to 405H to improve the uniformity appearance of the light generated by the backlight device 400 (e.g., the appearance in the user's eyes). The controller 410 can be the controller 270 or the controller 315. Figure 4 4 shows one possible distribution of dimming zones in the backlight device 400. However, other distributions of dimming zones and Figure 4Different numbers of dimming zones than those shown are possible.

[0049] Figure 5A An example 500 of a backlight device 505 is shown, which has different brightness areas for brightness compensation relative to a user's gaze location 507 (e.g., pupil location). The backlight device 505 can be the backlight device 245, the backlight device 300, or the backlight device 400. Without adjusting the brightness of the light emitted from the brightness device 505, the brightness distribution profile of the light emitted from the backlight device 505 can be the same as the brightness distribution profile of the brightness device 400. The operation of the backlight device 505 can be controlled by the controller 512 by providing emission instructions, which are generated at least in part based on the gaze location 507. The gaze location 507 can be determined by the controller 512 based on one or more images of the user's eyes collected by the eye tracker 515. Alternatively, the eye tracker 515 can determine the gaze location 507 and provide information about the gaze location 507 to the controller 512. The controller 512 can be the controller 270, and the eye tracker 515 can be the eye tracker 215.

[0050] The controller 512 may allocate at least a portion of the plurality of light sources in the backlight device 505 to a plurality of brightness regions based on the information about the gaze position 507, that is, Figure 5A The brightness area 510A, brightness area 510B, brightness area 510C, brightness area 510D and brightness area 510E shown. Each subset of the multiple light sources in the backlight device 505 assigned to the corresponding brightness area 510A to 510E can emit such light: the light has a brightness adjusted by the corresponding adjustment value according to the emission instruction from the controller 512. The controller 512 can use the emission instruction to set the corresponding adjustment value to a higher value for the corresponding brightness area 510A to 510E farther from the gaze point of the user's eyes (i.e., the gaze position 507). In addition, the adjustment value of the brightness area 510A including the gaze position 507 can be the lowest value in a group of adjustment values ​​for the brightness areas 510A to 510E. The controller 512 can use the emission instruction to set the corresponding adjustment value of the corresponding brightness area 510A to 510E based on the distance from the geometric center of the corresponding brightness area 510A to 510E to the gaze point of the user's eyes (i.e., to the gaze position 507).

[0051] Figure 5BAn example 520 of a backlight device 505 is shown, which has different brightness zones for brightness compensation relative to a new gaze position 522 of the user's eye. Information about the new gaze position 522 is available at the controller 512. The controller 512 can assign different portions of the multiple light sources in the backlight device 505 to the multiple brightness zones based on the information about the new gaze position 522, i.e., Figure 5B The brightness area 525A, the brightness area 525B, the brightness area 525C, the brightness area 525D and the brightness area 525E are shown. Each subset of the plurality of light sources in the backlight device 505 assigned to the corresponding brightness area 525A to 525E can emit a portion of light having a brightness adjusted by a corresponding adjustment value according to an emission instruction from the controller 512.

[0052] Each brightness region 525A to 525E can represent an updated version of the corresponding brightness region 510A to 510E, where the size, position within the backlight device 505, and / or corresponding adjustment value of the corresponding brightness region can be updated (or adjusted) based on information about the new gaze position 522. The controller 512 can use the emission instructions to adjust the size of each brightness region 510A to 510E to obtain the corresponding brightness region 525A to 525E. The controller 512 can adjust the size of each brightness region 510A to 510E according to the distance from the geometric center of each brightness region 510A to 510E to the new gaze position 522 to obtain the corresponding brightness region 525A to 525E. For example, the size of brightness region 510B can be effectively reduced to obtain brightness region 525B, while the size of brightness region 510C can be effectively increased to obtain brightness region 525C because the geometric center of brightness region 510B is closer to the gaze position 522 than the geometric center of brightness region 510C. In this way, a uniform appearance of light emitted from the backlight device 505 is achieved, which light is later used to generate image light for presentation to a user.

[0053] Figure 5C An example 530 of a backlight device 505 having different brightness zones for brightness compensation relative to a new gaze position 532 is shown. Information about the new gaze position 532 is available at the controller 512. The controller 512 can allocate another portion of the plurality of light sources in the backlight device 505 to the plurality of brightness zones based on the information about the new gaze position 532, i.e., Figure 5CThe brightness area 535A, the brightness area 535B, the brightness area 535C, the brightness area 535D and the brightness area 535E are shown. Each subset of the plurality of light sources in the backlight device 505 assigned to the corresponding brightness area 535A to 535E can emit a portion of light having a brightness adjusted by a corresponding adjustment value according to an emission instruction from the controller 512.

[0054] Each brightness region 535A to 535E can represent an updated version of the corresponding brightness region 525A to 525E, where the size, position within the backlight device 505, and / or the corresponding adjustment value of the corresponding brightness region can be updated (or adjusted) based on the information about the new gaze position 532. The controller 512 can use the emission instructions to effectively adjust the size of each brightness region 525A to 525E to obtain the corresponding brightness region 535A to 535E. The controller 512 can effectively adjust the size of each brightness region 525A to 525E according to the distance from the geometric center of each brightness region 510A to 510E to the new gaze position 532 to obtain the corresponding brightness region 535A to 535E. For example, the size of brightness region 525B can be effectively increased to obtain brightness region 535B, while the size of brightness region 525C can be effectively reduced to obtain brightness region 535C because the geometric center of brightness region 525B is farther away from the gaze position 532 than the geometric center of brightness region 525C. In this way, a uniform appearance of light emitted from the backlight device 505 is achieved, which light is later used to generate image light for presentation to a user. FIG. 5A to FIG. 5C It can be observed that the size of the brightness region including the gaze point (ie, the gaze position) among the plurality of brightness regions is fixed, ie, the sizes of the brightness regions 510A, 525A, and 535A are the same.

[0055] Figure 6 is a flow chart illustrating a process 600 of performing dynamic brightness compensation at a display component. Figure 6 The process 600 shown in FIG. 600 may be performed by a component of a display assembly (e.g., a component of display assembly 210). In other examples, other entities may perform Figure 6 Each example may include different steps and / or additional steps, or perform the steps in a different order.

[0056] The display component obtains 605 gaze information of the user's eyes (e.g., via the controller). The display component can be part of a head-mounted device, and the head-mounted device can also include an eye tracker configured to determine gaze information and provide the gaze information to the controller of the display component. The gaze information can include information about the pupil position of the user's eyes.

[0057] The display component generates 610 a firing instruction based at least in part on the gaze information (e.g., via the controller). The display component can load a plurality of adjustment values ​​from a non-transitory computer-readable medium coupled to the controller (e.g., via the controller) based at least in part on the gaze information. The display component can generate the firing instruction using the loaded plurality of adjustment values ​​(e.g., via the controller).

[0058] The display component dynamically allocates 615 at least a portion of a plurality of light sources (e.g., in a backlight device or in a display element) to a plurality of regions (e.g., via a controller) based at least in part on the gaze information. Each of the light sources in the backlight device may include an LED. The plurality of light sources may include a two-dimensional array of LEDs or a two-dimensional array of OLEDs. The display component may adjust the size of the corresponding region (e.g., via a controller) based at least in part on the gaze information. The display component may adjust the size of the corresponding region (e.g., via a controller) based on the distance from the geometric center of the corresponding region to the gaze point of the user's eye included in the gaze information. The size of the region including the gaze point in the regions may be fixed.

[0059] The display component emits 620 light from the multiple light sources. In some examples, the backlight device includes a light source that emits light through the display element. The display element can spatially modulate the light received from the backlight device to generate image light (content). In some other examples, the light source is part of the display element, and the light source directly emits image light (content). In this case, the display component may not include a backlight device. Each subset of these light sources dynamically assigned to the corresponding area can emit a portion of light, and the portion of the light has a brightness adjusted according to the emission instruction. The display component can set the corresponding adjustment value (e.g., via a controller) to a higher value for the corresponding area farther from the gaze point of the user's eyes included in the gaze information. The display component can set the corresponding adjustment value (e.g., via a controller) to the lowest value among multiple adjustment values ​​for the corresponding area corresponding to the gaze point of the user's eyes included in the gaze information. The display component can set the corresponding adjustment value (e.g., via a controller) based on the distance from the geometric center of the corresponding area to the gaze point of the user's eyes included in the gaze information.

[0060] System environment

[0061] Figure 7 700 is a block diagram of a system environment including an HMD. The system 700 may operate in an artificial reality environment, such as a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof. Figure 7The illustrated system 700 includes an HMD 705, and an input / output (I / O) interface 715 coupled to a console 710. Figure 7 The example system 700 is shown to include one HMD 705 and one I / O interface 715, but in other examples, any number of these components can be included in the system 700. For example, there can be multiple HMDs 705, each with an associated I / O interface 715, where each HMD 705 and I / O interface 715 communicates with the console 710. In alternative configurations, different and / or additional components can be included in the system 700. Additionally, in some examples, the system 700 can be combined with a plurality of HMDs 705, each with an associated I / O interface 715, and each HMD 705 and I / O interface 715 can communicate with the console 710. Figure 7 The functions described in one or more of the components shown in the figure may be combined with Figure 7 The described methods are distributed among the components in different ways. For example, some or all functions of the console 710 are provided by the HMD 705.

[0062] The HMD 705 presents content to the user, including virtual and / or augmented views of a physical real-world environment with computer-generated elements (e.g., two-dimensional or three-dimensional images, two-dimensional or three-dimensional video, sound, etc.). In some examples, the presented content includes audio presented via an external device (e.g., a speaker and / or a headset) that receives audio information from the HMD 705, the console 710, or both, and presents audio data based on the audio information. The HMD 705 may include one or more rigid bodies that may be rigidly or non-rigidly coupled together. Rigid coupling between rigid bodies may allow the coupled rigid bodies to act as a single rigid entity. In contrast, non-rigid coupling between rigid bodies allows the rigid bodies to move relative to each other. An example of an HMD 705 is Figure 1A Another example of an HMD 705 is Figure 1B HMD 100 in the.

[0063] HMD 705 may include a display component 720, an optical block 725, one or more position sensors 730, an inertial measurement unit (IMU) 735, an eye tracker 740, and a controller 750. Some examples of HMD 705 have a Figure 7 Different components and / or additional components than those described. In addition, in other examples, by combining Figure 7 The functionality provided by the various components described may be distributed differently among the components of the HMD 705 .

[0064] Display assembly 720 displays a two-dimensional image or a three-dimensional image to the user based on the data received from console 710. In various examples, display assembly 720 includes a single display or multiple displays (e.g., one display for each eye of the user). Display assembly 720 may include backlight device 721, display element 723, and one or more optical layers 724. An example of display assembly 720 is display assembly 210.

[0065] Backlight device 721 emits light that passes through display element 723. Backlight device 721 may include multiple light sources (e.g., a two-dimensional array of LEDs). At least a portion of the multiple light sources may be dynamically assigned to multiple regions based at least in part on gaze information determined by eye tracker 740. Each subset of the light sources dynamically assigned to a corresponding region may emit a portion of light having a brightness adjusted to a corresponding adjustment value according to an emission instruction (e.g., provided by controller 750). Examples of backlight device 721 are backlight device 245, backlight device 305, backlight device 400, or backlight device 505.

[0066] Display element 723 can spatially modulate light received from backlight device 721 to generate image light for presentation to a user wearing HMD 705. An example of display element 723 operating as a spatial light modulator is a liquid crystal display (LCD). An example of display element 723 is display element 255.

[0067] The one or more optical elements 724 can process the light emitted from the backlight device 721 before it reaches the display element 723. For example, the one or more optical elements 724 can diffuse and / or polarize the light emitted from the backlight device 721. The one or more optical elements 724 can also optically correct the image light generated by the display element 723. Examples of the one or more optical layers 724 are optical element 252 and / or optical element 257.

[0068] The optical block 725 amplifies the image light received from the display assembly 720, corrects the optical errors associated with the image light, and presents the corrected image light to the user of the HMD 705. In various examples, the optical block 725 includes one or more optical elements. Example optical elements included in the optical block 725 include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflective surface, or any other suitable optical element that affects the image light. In addition, the optical block 725 may include a combination of different optical elements. In some examples, one or more of the optical elements in the optical block 725 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

[0069] Amplification and focusing of the image light by the optical block 725 allows the display assembly 720 to be physically smaller, lighter, and consume less power than a larger display. Additionally, the amplification can increase the field of view of the content presented by the display assembly 720. For example, the field of view of the displayed content is such that the displayed content is presented using nearly all (e.g., approximately 110 degrees diagonal) of the user's field of view, and in some cases, the displayed content is presented using the entire field of view of the user's field of view. Additionally, in some examples, the amount of amplification can be adjusted by adding or removing optical elements.

[0070] In some examples, the optical block 725 can be designed to correct one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include: spherical aberration; chromatic aberration; or errors due to lens field curvature, astigmatism; or any other type of optical error. In some examples, the content provided to the electronic display for display is pre-distorted, and the optical block 725 corrects the distortion when it receives image light generated based on the content from the electronic display.

[0071] The IMU 735 is an electronic device that generates data indicating the position of the HMD 705 based on measurement signals received from one or more of the position sensors 730. The position sensor 730 generates one or more measurement signals in response to the movement of the HMD 705. Examples of the position sensor 730 include one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects movement, a type of sensor used for error correction of the IMU 735, or some combination thereof. The position sensor 730 can be located outside the IMU 735, inside the IMU 735, or some combination thereof. An example of the position sensor 735 is the position sensor 130.

[0072] The eye tracker 740 can track the eye position of the user wearing the HMD 705. In one or more examples, the eye tracker 740 captures images of the user's eyes and provides the captured images to the controller 750 to determine the gaze position of the user's eyes. In one or more other examples, an internal controller of the eye tracker 740 determines the gaze position of the user's eyes. The information about the gaze position can include information about the pupil position of the user's eyes. An example of the eye tracker 740 is the eye tracker 215.

[0073] The controller 750 can control components of the display component 720 and the eye tracker 740. The controller 750 can generate tracking instructions for the eye tracker. In some examples, the controller 750 receives one or more images of the user's eye 220 collected by the eye tracker 740, and uses the collected images to determine eye tracking information (i.e., gaze information or gaze position). The controller 750 can also generate emission instructions for the display component 720 based at least in part on the gaze information. The controller 750 can provide emission instructions to the backlight device 721. The emission instructions from the controller 750 may include an electrical signal (e.g., a voltage signal or a current signal) that controls the light emission from the backlight device 721 of a specific brightness distribution. In some examples, the controller 750 uses the emission instructions to dynamically allocate at least a portion of the multiple light sources in the backlight device 721 to multiple areas. Each subset of the plurality of light sources in the backlight device 721 dynamically allocated to a corresponding area according to an emission instruction from the controller 750 can emit a portion of light having a brightness adjusted by a corresponding adjustment value according to the emission instruction. An example of the controller 750 is the controller 270 or the controller 315.

[0074] The I / O interface 715 is a device that allows a user to send an action request to the console 710 and receive a response from the console 710. An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end the acquisition of image data or video data, or may be an instruction to perform a specific action within an application. The I / O interface 715 may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and transmitting these action requests to the console 710. The action request received by the I / O interface 715 is transmitted to the console 710, which performs an action corresponding to the action request. In some examples, the I / O interface 715 includes an IMU 735 that collects calibration data, which indicates an estimated position of the I / O interface 715 relative to the initial position of the I / O interface 715. In some examples, the I / O interface 715 may provide tactile feedback to the user based on the instructions received from the console 710. For example, tactile feedback is provided when an action request is received, or the console 710 transmits instructions to the I / O interface 715 causing the I / O interface 715 to generate tactile feedback when the console 710 performs an action.

[0075] The console 710 provides content to the HMD 705 for processing based on information received from one or more of the following: the eye tracker 740, the controller 750, and the I / O interface 715. Figure 7In the example shown, the console 710 includes an application repository 760, a tracking module 765, and an engine 770. Some examples of the console 710 have Figure 7 Similarly, the functions described further below may be combined with Figure 7 The described methods are distributed among the components of the console 710 in different ways.

[0076] The application repository 760 stores one or more applications for execution by the console 710. An application is a set of instructions that, when executed by a processor, generates content for presentation to a user. The content may be generated by an application in response to input received from a user via movement of the HMD 705 or movement of the I / O interface 715. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

[0077] The tracking module 765 uses one or more calibration parameters to calibrate the system 700, and can adjust one or more calibration parameters to reduce errors in determining the position of the HMD 705 or the position of the I / O interface 715. For example, the tracking module 765 transmits calibration parameters to the eye tracker 740 to adjust the focus of the eye tracker 740 to more accurately determine the gaze position of the user's eyes. The calibration performed by the tracking module 765 also takes into account information received from the IMU 735 in the HMD 705, and / or from the IMU included in the I / O interface 715. In addition, if tracking of the HMD 705 is lost, the tracking module 765 can recalibrate some or all of the system 700.

[0078] The tracking module 765 tracks the movement of the HMD 705 or the movement of the I / O interface 715 using information from the following components: one or more position sensors 730, IMU 735, or some combination thereof. For example, the tracking module 765 determines the position of the reference point of the HMD 705 in the map of the local area based on the information from the HMD 705. The tracking module 765 can also use data indicating the position of the HMD 705 from the IMU 735, or use data indicating the position of the I / O interface 715 from the IMU 735 included in the I / O interface 715 to determine the position of the reference point of the HMD 705 or the position of the reference point of the I / O interface 715, respectively. In addition, in some examples, the tracking module 765 can use part of the data indicating the position of the HMD 705 from the IMU 725 to predict the future position of the HMD 705. The tracking module 765 provides the estimated or predicted future position of the HMD 705 or the I / O interface 715 to the engine 770.

[0079] The engine 770 generates a three-dimensional map of the area around the HMD 705 (i.e., the "local area") based on the information received from the HMD 705. In some examples, the engine 770 determines depth information of the three-dimensional map of the local area, which is related to the technology used in calculating the depth. The engine 770 can calculate the depth information using one or more technologies for calculating depth, such as stereo-based technologies, structured light illumination technologies, and time-of-flight technologies. In various examples, the engine 770 uses the depth information to update a model of the local area, for example, and generates content based in part on the updated model.

[0080] The engine 770 also executes applications within the system 700 and receives position information, acceleration information, velocity information, predicted future position, or some combination thereof of the HMD 705 from the tracking module 765. The engine 770 determines, based on the received information, the content provided to the HMD 705 for presentation to the user. For example, if the received information indicates that the user is looking to the left, the engine 770 generates the following content for the HMD 705: the content reflects the user's movement in the virtual environment or in an environment where the local area is enhanced using additional content. In addition, the engine 770 performs an action within an application executed on the console 710 in response to an action request received from the I / O interface 715, and provides feedback to the user that the action has been performed. The feedback provided may be visual feedback or auditory feedback via the HMD 705, or tactile feedback via the I / O interface 715.

[0081] Additional configuration information

[0082] The foregoing description has been presented for purposes of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0083] Some parts of this specification sheet describe multiple embodiments with regard to the algorithm and symbolic representation of the operation of information. The technical personnel in the field of data processing usually use these algorithmic descriptions and representations to effectively convey the essence of their work to other technical personnel in the field. Although these operations are described functionally, computationally or logically, these operations are understood to be realized by computer programs or equivalent circuits, or microcodes, etc. In addition, without losing universality, sometimes it is also proved to be convenient to refer to the arrangement of these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0084] Any steps, operations or processes described herein may be performed or implemented individually or in combination with other devices using one or more hardware modules or software modules. In one example, a software module is implemented using a computer program product including a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0085] The embodiments may also be directed to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or the apparatus may include a general-purpose computing device that is selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

[0086] Embodiments may also be directed to a product produced by the computing process described herein. Such a product may include information obtained from the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0087] Finally, the wording used in the specification is selected primarily for readability and instructional purposes, and the wording may not be selected to describe or limit the patent rights. Therefore, it is intended that the scope of the patent rights is not limited by the specific implementation, but by any claims published on this application based thereon. Therefore, the disclosure of the embodiments is intended to illustrate, not limit, the scope of the patent rights listed in the appended claims.

Claims

1. A display assembly, comprising: Display components; A controller, the controller being configured to: Get the user's eye gaze information, generating a firing instruction based at least in part on the gaze information, and dynamically assigning at least a portion of a plurality of light sources to a plurality of zones based at least in part on the gaze information; as well as A backlight device, comprising the plurality of light sources emitting light passing through the display element, wherein each subset of the plurality of light sources dynamically assigned to a corresponding area is configured to emit a portion of the light, the portion of the light having a brightness adjusted by a corresponding adjustment value according to the emission instruction.

2. The display assembly according to claim 1, wherein: The controller is further configured to set the corresponding adjustment value to a higher value for the corresponding area that is farther from a gaze point of the user's eyes included in the gaze information.

3. The display assembly according to claim 1 or 2, wherein: The controller is further configured to set the corresponding adjustment value to a lowest value among a plurality of adjustment values ​​for the corresponding area corresponding to a gaze point of the user's eyes included in the gaze information.

4. A display assembly according to any preceding claim, wherein: The controller is further configured to set the corresponding adjustment value based on a distance from a geometric center of the corresponding area to a gaze point of the user's eyes included in the gaze information.

5. A display assembly according to any preceding claim, wherein: The controller is further configured to adjust a size of the corresponding area based at least in part on the gaze information.

6. A display assembly according to any preceding claim, wherein: The controller is further configured to: adjust the size of the corresponding area according to the distance from the geometric center of the corresponding area to the gaze point of the user's eyes included in the gaze information; Preferably, a size of an area including the gaze point among the plurality of areas is fixed.

7. A display assembly according to any preceding claim, wherein: The controller is further configured to: loading a plurality of adjustment values ​​from a non-transitory computer-readable medium coupled to the controller based at least in part on the gaze information; and The firing instruction is generated using the loaded plurality of adjustment values.

8. A display assembly according to any preceding claim, wherein: The gaze information includes information about the position of the pupil of the user's eye.

9. A display assembly according to any preceding claim, wherein: Each of the plurality of light sources includes a light emitting diode (LED).

10. A display assembly according to any preceding claim, wherein: The plurality of light sources comprises a two-dimensional array of light emitting diodes (LEDs).

11. A display assembly according to any preceding claim, wherein: The display assembly is part of a head mounted device, and the head mounted device further comprises an eye tracker, the eye tracker being configured to: determining the gaze information; and The gaze information is provided to the controller.

12. A method comprising: Get the user's eye gaze information; generating a firing instruction based at least in part on the gaze information; dynamically assigning at least a portion of a plurality of light sources in a backlight device to a plurality of zones based at least in part on the gaze information; as well as Light is emitted from the backlight device through the display element so that each subset of the plurality of light sources dynamically assigned to a corresponding area emits a portion of the light having a brightness adjusted by a corresponding adjustment value according to the emission instruction.

13. The method according to claim 12, further comprising one or more of the following steps: For the corresponding area that is farther from the gaze point of the user's eyes included in the gaze information, setting the corresponding adjustment value to a higher value; For the corresponding area corresponding to the gaze point of the user's eyes included in the gaze information, setting the corresponding adjustment value to a lowest value among a plurality of adjustment values; and The corresponding adjustment value is set based on a distance from a geometric center of the corresponding area to a gaze point of the user's eyes included in the gaze information.

14. The method according to claim 12 or 13, further comprising: adjusting a size of the corresponding area based at least in part on the gaze information; and / or The size of the corresponding area is adjusted according to the distance from the geometric center of the corresponding area to the gaze point of the user's eyes included in the gaze information.

15. A head mounted device comprising: The display assembly according to any one of claims 1 to 11; an eye tracker configured to determine gaze information of a user's eyes; as well as an optical assembly configured to direct image light toward an eye zone of the head mounted device corresponding to a position of the user's eyes; Wherein, the controller is configured to obtain the gaze information of the user's eyes by receiving the gaze information from the eye tracker; and The display element is configured to spatially modulate the light received from the backlight device to generate image light.

Images