Selective illumination of spatial light modulators

Abstract

The display system includes a segmented light source containing multiple separately controlled segments for emitting light, optical elements for directing the light, and a spatial light modulator (SLM) for modulating the light from the light source, wherein light from one or more of the multiple segments from the segmented light source is directed to the SLM through the optical elements such that a spatial mapping from the segments to the SLM is maintained.

Description

Technical Field

[0001] (Related Application) This application claims priority to U.S. Provisional Patent Application No. 63 / 506,832, filed Jun. 7, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] (Field of the Invention) This application relates to image display, and more particularly to selective illumination of a spatial light modulator.

Background Art

[0003] Spatial light modulator (SLM)-based displays are often color sequential and have many advantages compared to conventional emissive displays. Key advantages include a high fill factor per pixel and the ability to tightly control the illumination / radiation angle of light from the display. However, a significant drawback of SLM displays is that the entire SLM needs to be illuminated regardless of the number of pixels used or the content being displayed.

Brief Description of the Drawings

[0004] The present invention is shown by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1A] Shows an embodiment of a typical illumination system (DMD or LCoS with LEDs). [Figure 1B] Shows an embodiment of a typical illumination system with a DMD element. [Figure 2] Shows a perspective view of an embodiment of a portion of an architecture using a non-segmented illumination source. [Figure 3A] Shows a perspective view of an embodiment of a portion of an architecture showing a segmented illumination source and spatial mapping. [Figure 3B] Shows an embodiment of an illumination system with a monochromatic segmented illumination source. [Figure 3C] Shows an embodiment of an illumination system with a multi-color segmented illumination source. [Figure 3D] This document illustrates one embodiment of a double-pass system having a multi-color segmented illumination source and a combination of illumination and projection optical elements. [Figure 4] This shows the difference between a standard non-segmented LED and a segmented LED array. [Figure 5] Exemplary embodiments of two types of LED arrays are shown. [Figure 6A] This shows one embodiment of a segmented LED including a central LED and outer LEDs. [Figure 6B] This shows one embodiment of a segmented LED having an LED stripe. [Figure 7A] An embodiment of an LED array having a foveal zone is shown. [Figure 7B] An embodiment of an LED array having zones of different shapes is shown. [Figure 8] This document describes one embodiment of a diffraction holographic optical element that can be used in conjunction with a segmented light source. [Figure 9] This document describes one embodiment of a photonic crystal that can be used with a segmented light source. [Figure 10] This shows one embodiment of a refractive lens used with a segmented light source. [Figure 11] This shows one embodiment of a reflective lens used with a segmented light source. [Figure 12] This document describes one embodiment of a reflective-refracting optical system having a combination of reflective and refractive lenses used in conjunction with a segmented light source. [Figure 13] An embodiment of a free-form lens, including reflective and refractive elements used in conjunction with a segmented light source, is illustrated. [Figure 14] This shows one embodiment of a refractive microlens used with a segmented light source. [Figure 15A] This shows one embodiment of a reflective microlens used with a segmented light source. [Figure 15B]This document describes one embodiment of a reflective-refracting optical system having a combination of reflective microlenses and refractive microlenses used in conjunction with a segmented light source. [Figure 16] This is a flowchart of one embodiment using a segmented illumination source to display an image. [Figure 17A] This is a flowchart of one embodiment of adjusting the display for dynamic mode or binary mode. [Figure 17B] This is a flowchart of one embodiment of dynamic blending. [Figure 18] This figure shows one embodiment of a display that may be used with this system. [Figure 19] This figure shows one embodiment of a differential illumination mode. [Figure 20] This document illustrates one embodiment of a different lighting power control mechanism for different light source segments. [Figure 21] This document illustrates one embodiment of different lighting timing and power control for adjacent segments within a single frame. [Figure 22] This is a block diagram showing one embodiment of a computer system that may be used in conjunction with this system. [Modes for carrying out the invention]

[0005] A spatial light modulator display with selective illumination is described, where a portion of a segmented light source may be used to selectively illuminate a portion of the spatial light modulator. The system is designed to maintain a spatial mapping between the portion of the segmented light source and the corresponding portion of the spatial light modulator. The illumination system may have various configurations, with variations in the illumination source, optical elements, electronics, control, and other attributes. The illumination source is spatially mapped to the SLM. A portion of the illumination is mapped to a portion of the SLM. In one embodiment, the illumination portion has the same aspect ratio as the SLM portion being illuminated.

[0006] Typical illumination systems for spatial light modulators (SLMs), such as digital micromirror devices (DMDs) or liquid crystal on silicon (LCoS) devices, use three individual light sources with one light source for each color to illuminate the entire SLM. These light sources can be lamp-based, such as incandescent, UHP (metal halide), halogen, mercury, and xenon bulbs. The light source can also be LED- or laser-based. The SLM and illumination configuration can be used in both sequential color and non-sequential color formats.

[0007] In one embodiment, the selective illumination system can operate in two modes, namely, a binary mode where each zone is either on or off, and a dynamic mode where the color channel luminance can be adjusted for each zone based on the content. In one embodiment, the system also applies smoothing and blending to ensure that there are no gaps or visible lines between zones.

[0008] The following detailed description of embodiments of the present invention refers to the accompanying drawings that illustrate specific embodiments in which like reference numerals indicate like elements. The description of these embodiments is detailed enough to enable those skilled in the art to implement the present invention. Those skilled in the art will understand that other embodiments may be utilized and that logical, mechanical, electrical, functional, and other changes may be made without departing from the scope of the present invention. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the present invention is defined only by the appended claims.

[0009] Figure 1A shows an embodiment of a typical lighting system having a spatial light modulator, which may be a digital mirror display (DMD) or liquid crystal on silicon (LCoS) having light-emitting diodes (LEDs) as a light source. This design includes three separate LEDs 110A, 110B, and 110C, one for each color, and optical elements 115A, 115B, and 115C associated with each LED. In one embodiment, two color-selection mirrors 120A and 120B can be used to combine colors. The light is then directed through an additional optical element 130 to a beam-splitting cube 140, which directs the light to an LCoS microdisplay 150. The light is modulated by the LCoS 150, passes through the beam-splitting cube 140, and is directed through a final optical element 155 to the display.

[0010] A typical SLM lighting system illuminates the entire SLM display panel, regardless of the content being displayed or the pixels being used on the panel. The drawbacks of this are higher power consumption and lower contrast, which can result from illuminating unused pixels on the SLM.

[0011] Figure 1B shows one embodiment of a typical DMD architecture system. This design includes three separate color LEDs 160A, 160B, and 160C, one for each color, and optical elements 165A, 165B, and 165C associated with each LED 160A, 160B, and 160C. In one embodiment, two color-selection mirrors can be used to combine colors. The light can then be directed through an additional optical element 170 and redirected to the DMD 190 through a mirror 175 and further optical elements. In one embodiment, a pair of prisms 185A and 185B direct the light to the DMD 190 and direct the modulated light through a final optical element 195. In this configuration, the entire DMD 190 is illuminated regardless of the proportion of the display area that has content relevant to the display.

[0012] Figure 2 shows a perspective view of one embodiment using a non-segmented illumination source. The non-segmented illumination source 210 outputs light of one color, which passes through the first optical element 220 before illuminating the spatial light modulator 230. The spatial light modulator (SLM) 230 has on-pixels and off-pixels to generate an image. In this example, the entire SLM 230 is illuminated, but parts of the SLM 230 are off, and therefore the light sent to those parts of the SLM 230 is not used in generating the image. In this example, the area around the image of the Dalmatian dog's face is blank and not used. Therefore, that part of the light emitted by the light source 210 is not used by the SLM. This wasted light uses additional power and reduces the contrast of the image.

[0013] Figure 3A shows one embodiment of a partial SLM system that enables partial illumination. In this example, selective illumination is provided using a segmented illumination source 310 which includes multiple segments that can be selectively illuminated 314 or not illuminated 312. The light passes through an optical element 316 to a spatial light modulator 325. The light may pass through additional optical elements or be redirected using mirrors or other elements (not shown). By using the segmented illumination source 310, it is possible to illuminate only a portion of the spatial light modulator 325 with illumination light 320. Depending on the portion of the screen with content, one or more segments may be illuminated.

[0014] As seen on the SLM 325, each segment in the segmented illumination source 310 corresponds to an area of ​​the SLM 325, and therefore an area on the final image. This allows the system to not illuminate the parts of the SLM 325 that do not have content. The diagram in Figure 3A shows only one set of illumination source and optical elements. In one embodiment, the SLM system includes a separate illumination source 310 and associated optical elements 316 for each color. In one embodiment, the system configuration shown in Figure 3B or Figure 3C may be used with the segmented illumination source of Figure 3A.

[0015] Using selective lighting, which allows control over the areas of illumination, can reduce the overall power consumption of the display system and increase image contrast. Controlling the areas of illumination means that some or all of the spatial light modulator is illuminated based on the content being displayed. Such control over spatial and / or angular positioning is called maintaining spatial mapping. Spatial mapping means that instead of mixing light from the entire light source for uniformity, distinct segments of light are kept separate, and as a result, illuminating one segment of the light source always illuminates the same corresponding area of ​​the SLM. This makes it possible to map light source segments to image content, reducing the amount of power used by the light source and increasing contrast because light is not directed to "off-pixel" areas of the SLM.

[0016] Figure 3B shows one embodiment of an LCoS architecture having a segmented light source. This design includes three separate segmented color LEDs 330A, 330B, and 330C, one for each color, and optical elements 335A, 335B, and 335C associated with each LED. In one embodiment, two color-selection mirrors 340A and 340B can be used to combine colors. The light can then be directed through additional optical elements and redirected to the SLM 360 through mirrors and further optical elements 350. In one embodiment, a beam splitting cube 345 is used to direct the light to the SLM 350 and direct the modulated light through a final optical element 355. The optical elements 335A, 335B, 335C, 340A, 340B, 345, and 355 are designed to maintain a spatial mapping of light from a specific segment of the light source to the corresponding portion of the SLM 350. For this mapping, in this configuration, only the portion of the SLM 350 relevant to the displayed content is illuminated.

[0017] Figure 3C shows one embodiment of an LCoS architecture having a multi-color segmented LED array. In one embodiment, the multi-color segmented LED array 360 contains all the colors used in the system. Optical element 362 directs light from the multi-segment LED array 360 to a beam splitting cube 364 and then to an SLM 368 via an additional optical element 366. The SLM 368 modulates the light and outputs the modulated light through a final optical element 370. The modulated light may be shown to the user via a waveguide or redirected via additional optical elements. The optical paths 362, 364, 366, and 370 are designed to maintain a spatial mapping of light from a specific segment of the light source 360 ​​to the corresponding portion of the SLM 368. In one embodiment, the multi-color segmented LED array 360 uses a time-series display in which the colors are displayed in sequence. In one embodiment, a “segment” can include multiple LED portions. For example, a segment of the multi-color LED array 360 could include a red LED, a blue LED, and two green LEDs. In one embodiment, the optical element mixes light sources within such segments while maintaining separation and spatial mapping between segments.

[0018] Selective illumination can have a variety of light sources. One embodiment may use LEDs. Another embodiment may use mini-LEDs. Another embodiment may use micro-LEDs. Another embodiment may use monolithic segmented LEDs. These LEDs, mini-LEDs, and micro-LEDs can be made from superluminescent light-emitting diodes (SLEDs), color-changing LEDs, quantum dot LEDs, resonant cavity LEDs, organic LEDs, quantum dot organic LEDs, stacked RGB LEDs, color-adjustable LEDs, or other types of LEDs. In some embodiments, non-LED-based light sources may be used.

[0019] Figure 3D shows one embodiment of a double-pass optical system having a multi-color segmented illumination source and combined illumination and projection optical elements. The segmented color LED 380 provides illumination. As described above, each segment may be on or off. Furthermore, in one embodiment, the brightness level of each segment may be adjusted based on the image content. Although the figure illustrates LED 380, the light source may be a micro-LED array or another configuration. LED optical element 383 is associated with LED 380. The image components from each of the LEDs 380 are combined by a combiner 386. In one embodiment, the combiner 380 is a partial mirror.

[0020] The coupled light from LED 380 is optionally directed through a diffuser 388 and an optical element. In one embodiment, one or more optical elements, such as a turner 390, may also be present to allow for a flexible layout of the system. The light then, in one embodiment, travels through a cross polarizer 392 and a double-pass optical element 394, and is subsequently modulated by the LCoS microdisplay 396. The modulated image may illuminate only a subsection of the LCoS display 396 as described above, passing through the double-pass optical element 394 and the cross polarizer 392. In one embodiment, the polarizer 392 is circular rather than crossed. The modulated image may pass through one or more additional optical elements and may be displayed to the user through a waveguide (not shown). The double-pass optical element 394 combines the illumination and projection optical elements into a single module, eliminating the need for a separate illumination and projection optical element optical module or a polarization beam splitter such as a PBS cube. These double-pass optical elements maintain the angular mapping from the segmented LED light input to the modulated output.

[0021] Figure 3D shows one embodiment in which the optical elements associated with segmented LEDs are designed to maintain the spatial relationships between LED segments without using separate illumination and projection systems by using a double-pass optical system that combines illumination and projection optical elements into a single module. This eliminates the need for polarizing beam splitter elements and separate illumination and projection optical systems. Light from segmented LED 380 is collected by LED optical element 383 and directed to double-pass optical element 394. This light can pass through any diffuser element 388 and polarizer 392. The double-pass optical element 394 maintains the angular mapping from the LEDs between the input and output of the double-pass system. Those skilled in the art will understand that in the LCoS microdisplay 396, the spatial mapping of the LEDs is preserved. The image, when it hits the SLM, still has the same spatial relationships as the original image produced by the LEDs. Since only a portion of the spatial light modulator receives the image data, portions of the SLM corresponding to portions of the segmented LEDs that are off can also be turned off. The SLM modulates image data and returns the modulated image through a double-pass optical element. In one embodiment, the output of the optical engine is coupled to a combiner waveguide and output via the waveguide's output coupler. The output coupler may be implemented in glasses or goggles for a head-mounted display for virtual reality, augmented reality, reality, or mixed reality systems. The above description refers to “image” data, but in this context, images may include video, or other streams of image or text data.

[0022] Figure 4 shows the difference between a standard non-segmented LED 410 and segmented LED arrays 420, 430. Segmented LED arrays 420, 430 divide an LED into segments that can be controlled individually. In one embodiment, the segmented LED is a monolithically constructed array 430 of individually controllable LEDs. In another embodiment, the segmented LED is constructed from individual LED chips 420 assembled in close proximity to each other. While the term "LED" is used throughout, in this context, it refers to any diode-based light source, including mini-LEDs, micro-LEDs, etc.

[0023] LED arrays can vary in the size of individually controlled LEDs from 1x2 to 100x100. In one embodiment, the LEDs can be segmented into a 3x3 array of smaller LEDs. In another embodiment, the LEDs can be segmented into a 5x5 array. The dimensions of the array do not have to be limited to a square, i.e., an equal number of LEDs in the horizontal and vertical dimensions. The array can have a layout that is rectangular or any other configuration. For example, in some types of displays, the array can be laid out in a triangular, circular, or other format based on the display area designed to be covered by the image to be displayed. For example, in the case of an augmented reality (AR) display, the relevant display area may be only the bottom 1 / 3 of the visual area, which would optimally use a rectangular array such as 10x3. In the case of an AR display using goggles with elliptical or circular lenses with hard borders, an LED array whose shape matches the shape of the display area may be used.

[0024] The size of individual LED segments can vary from 1 μm to 500 μm. In one embodiment, the LED segment size is 200 μm. In another embodiment, the LED segment size is 100 μm. In yet another embodiment, the LED segment size is 20 μm. In one embodiment, the individual segments within the LED array do not have to be the same size.

[0025] As described above, each segment of the LED can be controlled individually. In one embodiment, the LED array may be located on a passive backplane, and the LED segments are individually driven by an external LED driver. In another embodiment, the LEDs are located on an active backplane, and the LED segments can be addressed externally. Other methods may be used to control the LED segments individually.

[0026] An LED array may be a monochromatic or multichromatic array. An exemplary array is shown in Figure 5. In the monochromatic LED array 510, each LED is the same color. In one embodiment, in a monochromatic array, separate LED segments may use slightly different wavelengths to improve the efficiency of reflecting or transmitting values ​​across various angles of incidence (AOI) using dichroic mirrors / filters. The use of slightly different wavelengths may also improve the efficiency of waveguide coupling across different angles of incidence. However, in a monochromatic segmented LED configuration, all slightly different wavelengths are the same color.

[0027] In one embodiment, arrays of different colors may have different numbers of segments. For example, a green array may have more segments than a red or blue array. In one embodiment, there may be twice as many green segments as there are red or blue segments. Other configurations may be used.

[0028] In one embodiment, a single multicolor LED array 520 has red, green, and blue LED segments. Slightly different wavelengths may span the LED segments within the single multicolor LED. In one embodiment, the number of segments for each color is not identical. In one embodiment, there are twice as many green LEDs as red or blue LEDs in the array. In one embodiment, the array has a repeating 2x2 LED pattern where two green LEDs are surrounded by blue and red LEDs. Other configurations may be used.

[0029] The light source may be segmented into zones rather than an array. These zones may vary in size, shape, position, color, and number. In one embodiment, the light source may have a central zone surrounded by outer zones. In another embodiment, the light source may have horizontally or vertically striped zones.

[0030] Figure 6A shows one embodiment of a segmented LED including a central zone surrounded by an outer zone. The composite LED 610 includes a central LED 620 surrounded by an outer LED 630. This configuration allows the display to show the central image portion by using only the central LED 620 while the outer frame is not lit, and to show the frame portion without illuminating the outer LED 630 without illuminating the central LED 620. In one embodiment, the central LED 620 and the outer LED 630 are the same color. In another embodiment, they may be different colors.

[0031] Figure 6B shows one embodiment of a composite segmented LED having horizontally or vertically striped zones. In this embodiment, the composite LED 710 has three zones 720A, 720B, and 720C, as shown. In another embodiment, a different number of zones can be used. This configuration allows, for example, display on one segment of the screen across the bottom, while no image is displayed on the other parts of the screen.

[0032] Figure 7A shows one embodiment of an LED array having a foveal zone. The LED array has zones of non-uniform size such that the zones near the central region are smaller compared to the zones in the lateral region. This allows for clearer distinction between the zones in the foveal region. This provides a better quality image, as the image is perceived with higher resolution in the central region of vision.

[0033] Figure 7B shows alternative embodiments of LED areas with different zone sizing configurations. In one embodiment, the size and position of the zones may be selected based on the intended use of the system. For example, in the case of a head-up display showing warning text in the lower right, the area may be a single segment. Other configurations may be used, including configurations where the center is larger and the outer edges are smaller, or configurations where the segments are not rectangular.

[0034] These selective illumination sources may consist of refractive or reflective optical systems. The reflective optical system may include elements that use a composite parabolic condenser (CPC) or internal total internal reflection (TIR). These refractive or reflective elements may be combined in a reflective-refracting configuration. In one embodiment, this reflective-refracting configuration may be a single optical element that combines a TIR and a refractive surface, such as a TIR CPC lens or a TIR Fresnel lens.

[0035] Figures 10–15B show various exemplary lenses that may be used with segmented light sources. The lenses are designed to maintain spatial mapping of the light source segments. The lenses described below can be used in any combination, and a single system may contain multiple lens types. In one embodiment, each light source has the same sequence of lenses in a particular embodiment, but different embodiments may use different combinations of lenses.

[0036] Figure 10 shows one embodiment of a refractive lens used with a segmented light source.

[0037] Figure 11 shows one embodiment of a reflective lens used with a segmented light source.

[0038] Figure 12 shows one embodiment of a reflective-refracting optical system having a combination of reflective and refractive lenses used with a segmented light source.

[0039] Figure 13 illustrates one embodiment of a freeform lens, including reflective and refractive elements used with a segmented light source. The illumination source may be constructed using a freeform optical system. These freeform optical systems may have one or more freeform surfaces. The freeform surfaces may be refractive, reflective, or reflective-refracting.

[0040] Figure 14 shows one embodiment of a refractive microlens used with a segmented light source. The illumination source may consist of microoptical elements. These microoptical elements may be refractive or reflective. In one embodiment, there is one microoptical element per illumination zone or segment. In one embodiment, the microoptical elements are incorporated into the illumination source package. In one embodiment, there may be a separate microoptical element 1420 for each separately controlled LED segment 1410.

[0041] Figure 15A shows one embodiment of a reflective microlens used with a segmented light source. The reflective microlens 1520 is used to direct the light from each LED segment 1510.

[0042] Figure 15B shows one embodiment of a reflective-refracting optical system having a combination of reflective and refractive microlenses used with a segmented light source. In this embodiment, some of the LED segments 1550 have refractive microlenses 1570, while others have reflective microlenses 1560.

[0043] The illumination source may consist of diffractive or holographic optical elements. In one embodiment, these diffractive or holographic optical elements are located less than 1 mm from the surface of the illumination source. In another embodiment, the diffractive or holographic optical elements are incorporated into the illumination source package.

[0044] Figure 8 shows one embodiment of a diffracting or holographic optical element used with a segmented light source. The DOE or HOE may be used to shape or redirect light emitted from the light source.

[0045] Figure 9 shows one embodiment of a photonic crystal used with a segmented light source. The photonic crystal may be used to shape or redirect light emitted from the light source.

[0046] The illumination source may consist of a photonic crystal. In one embodiment, the photonic crystal is located less than 1 mm from the surface of the illumination source. In another embodiment, the photonic optical element is incorporated into the package of the illumination source.

[0047] In conventional SLM-based lighting designs, overall efficiency and uniformity are key metrics for design optimization. In this conventional approach, non-imaging design methods are preferable to imaging design methods because they prioritize optimizing optical throughput and uniformity over image quality. Therefore, conventional lighting designs often shy away from spatially resolved mapping of heterogeneous light sources, as they focus on achieving high uniformity. However, in segmented lighting systems, achieving spatially resolved mapping of light sources to SLMs is necessary to achieve efficient selective illumination of segments.

[0048] In one embodiment, uniformity can be improved by projecting an illumination source having various optical elements onto a microlens array (MLA) or an engineered diffuser. In one embodiment, the MLA or engineered diffuser restricts mixing or diffusion to individual illumination segments, so that the segments remain spatially separated and mapped to appropriate areas of the SLM. In another embodiment, there is no MLA or diffuser element, and the focus of the illumination system is designed to control illumination roll-off between adjacent zones. In one embodiment, segments can also be converted to appropriate polarization states. In one embodiment, improved uniformity and polarization control can be achieved in a single step. In one embodiment, this is achieved using an engineered diffuser with an integrated polarizer. In one embodiment, polarization control is achieved using a polarization conversion / recycling system. In one embodiment, polarization control is achieved using polarization control based on orbital angular momentum. In one embodiment, the engineered diffuser is a holographic element. In one embodiment, polarization control is a holographic element. In one embodiment, a single holographic element performs polarization recycling and improved uniformity.

[0049] The illumination source may include an element for polarization recycling, in which all unpolarized light is converted to a single polarized state. In one embodiment, the polarization recycling element increases the etenduance of the light source so that individual LED segments are mapped to a larger spatial area on the SLM.

[0050] When only a portion of the SLM is illuminated, there are various driving modes in which the SLM can operate to save power or increase efficiency. In one embodiment, the portion of the SLM that is not illuminated is shut off to save power.

[0051] SLM typically runs in color sequential mode, using a combination of red, green, and blue subframes to construct a full-color frame. SLM can also run in reduced color mode, which may be alternating RGB subframes, or a two-color or one-color cycle. Reduced color mode can be selected to increase brightness, improve efficiency, or reduce color breakup.

[0052] Different parts of the SLM may be illuminated using different color modes. In one embodiment, one or more parts of the SLM may be illuminated using a full-color sequential mode, while one or more other parts may be illuminated using a subtractive color mode. In one embodiment, the color mode of each part of the SLM may be selected to enhance brightness and efficiency. Furthermore, in some embodiments, adjacent segments may operate in different modes. For example, one segment may be time-series, while adjacent segments may be continuously on.

[0053] Figure 19 shows one embodiment of a differential illumination mode in which different segments of a segmented light source may have different timings and sequences.

[0054] Within a single segmented LED 1910, different LED segments may be driven at different power levels to provide different illumination levels to different parts of the display. In one embodiment, these various power levels are generated by driving the LED segments at different current levels. In one embodiment, these different current levels are generated by using a single anode voltage and driving the low-side driver in a linear regime. In one embodiment, these different current levels are generated by using a single anode voltage and driving the low-side driver in a high-frequency pulse-width modulation (PWM) mode so that the time-averaged power supplied to the LED is reduced. In one embodiment, these various power levels are generated by driving the LED segments at the same current level but over different lengths of time, without using a high-frequency PWM mode.

[0055] Figure 20 shows one embodiment of a different illumination power control mechanism for different light source segments. This embodiment shows control of illumination power using high-frequency pulse width modulation (PWM) and a non-PWM method—time-averaged power by high or low current. The figure also shows different duty cycles that may be used. The portion of the frame on which the LEDs are lit can be considered the duty cycle. In one embodiment, a segmented LED 2010 may have different duty cycles applied to each segment. This applies to the monolithic LED 2010 shown here, as well as other types of segmented LEDs. In one embodiment, a reduced duty cycle may be used to reduce color crosstalk.

[0056] Figure 21 shows one embodiment of a different illumination control mechanism for different light source segments. This embodiment demonstrates that each color segment can operate at different timings than adjacent segments. Segments can be illuminated sequentially or simultaneously within a single frame, at the same or different power levels. This perspective image of the entire system, including monolithic segmented LEDs for each color, does not include any additional optical elements that may be present in the complete system for the sake of simplification. Those skilled in the art will understand that additional optical elements and light refraction may be added. Those skilled in the art will also understand that a non-PBS cube system or a combined projection / illumination optical system that does not require separate projection and illumination optical modules may be used.

[0057] In one embodiment, when different segments of an LED are driven at different brightness levels, the panel controller adjusts the dynamic range of each associated lighting zone to compensate for the different illumination levels. This can be beneficial in situations where different parts of an image have different brightness levels, potentially increasing contrast across the image, saving overall display power, and increasing the dynamic range in parts of the display. For example, if zone A is illuminated at maximum brightness and zone B is illuminated at 50% brightness, zone B needs to have twice the digital control level of each pixel in the zone compared to zone A to compensate for the decrease in brightness. By reducing the brightness, the dynamic range of that zone increases. This is particularly useful in augmented reality displays that enable the display of high dynamic range images, as contrast is set segment by segment rather than for the entire image.

[0058] Figure 16 is a flowchart of one embodiment using a segmented illumination source to display an image. The process begins in block 1610.

[0059] In block 1620, the system receives image data for display. The image data may be a sequence of still images or images to be displayed in a row for video or other content.

[0060] In block 1630, the system determines whether display zone information is available with the image data. Display zone information may be provided in metadata or other data associated with the image data. In one embodiment, for a generated image, display zone information may be provided by the image generation system. In one embodiment, an offline system may preprocess the image data before it is sent to the display system in order to determine the display zones. In one embodiment, this data may be associated with the image data and stored as metadata. Display zone information identifies the lighting segments used in the displayed image. In one embodiment, the display zone information may also specify the brightness of the light source for each segment and the color mode for each segment.

[0061] If display zone information is available, in block 1640 the data is passed to the segmented lighting source controller. In block 1650, the image is displayed using the selected light source segments in the selected brightness and color mode. The process then ends in block 1680. In one embodiment, this process is used for each frame of an image, video, or other set of displays. In one embodiment, when the system identifies the presence or absence of display zone information for a given piece of content, the query in block 1620 is skipped because the status is consistent for the content.

[0062] If no display zone information is found to be available in block 1630, the process proceeds to block 1660.

[0063] In block 1660, the process utilizes content analysis to analyze image content and identify display zone information within the image. In one embodiment, content analysis logic within the processor analyzes the content of the displayed image. The image content within the frame is analyzed to identify the segment used to display the image. In one embodiment, the system can also select a color sequence based on the displayed content. For example, if the content is best displayed in monochrome, the system can adjust the color sequence to monochrome. In one embodiment, a segment of the display may have a different color sequence than other segments of the display. For example, in a display with a persistent clock in one area, that area may utilize a different color scheme than the portion of the display that has image data. Similarly, closed captions may be displayed in monochrome, while the image above them is displayed as a full-spectrum color image.

[0064] In block 1670, the display zone information is associated with the frame. The process then proceeds to block 1640, where the image data is displayed.

[0065] In one embodiment, different lighting modes for each SLM section can dynamically adjust the color balance to improve the system's color uniformity or lighting uniformity. In one embodiment, selective lighting modules are used in a light engine for augmented reality. In one embodiment, the image output may be via one or more waveguides. The lighting section can compensate for the color or lighting uniformity of the waveguides.

[0066] Naturally, this is shown as a flowchart, but in one embodiment, the order of operations is not restricted to the illustrated order, as long as the processes are not dependent on each other.

[0067] Figure 17A is a flowchart of one embodiment of tuning the display for dynamic or binary mode. The process begins in block 1705. These reduced-power modes are advantageous because they save power when generating an image, instead of generating the image and then discarding some of the light.

[0068] In block 1710, an operating mode is selected. In one embodiment, the system may operate in one of two operating modes: dynamic or binary. Binary mode means that the portion of the light source corresponding to the portion of the display without content is turned off, while the rest remains lit as usual. Dynamic mode means that the color channel brightness is set based on the content being displayed. In one embodiment, dynamic mode may include binary mode, where a portion of the light source in zones where no content is shown is turned off, and another portion is adjusted based on the content.

[0069] If the operating mode is dynamic, as determined in block 1715, the process in block 1720 selects the luminance of each color channel within each zone based on the content. In one embodiment, the process identifies the brightest component in a zone and adjusts the luminance of each color channel based on that component. In another embodiment, the process identifies the brightest component for each color in a zone separately and adjusts the color channel based on the color in that zone. In yet another embodiment, the system analyzes a histogram of luminance across zones to determine the color channel adjustment. In yet another embodiment, the average luminance is used to determine the adjustment. In one embodiment, the process first identifies zones that have no content and sets the luminance to zero for those zones. In one embodiment, the luminance is set by changing the current to the LEDs or other light sources.

[0070] In block 1722, the process adjusts the color levels for uniformity across the zones. Since differences in brightness are perceived as differences in color by the human eye, the system adjusts the displayed colors so that changes in brightness between zones do not affect the colors perceived by the user. The process then proceeds to block 1730.

[0071] In block 1715, if the system determines that the operating mode is binary, in block 1725, the process identifies zones that have no content and turns them off. The process then proceeds to block 1730.

[0072] In block 1730, in one embodiment, global blending is applied to blend the edges between zones. In one embodiment, a global blend filter is applied to the image.

[0073] In block 1735, in one embodiment, digital edge blending is applied to blend edges between illumination zones. In one embodiment, digital edge blending provides dynamic blending. In one embodiment, a different gamma map is used for each illumination zone. In one embodiment, a different gamma map is used within a single illumination zone. In one embodiment, the system provides roll-off between illumination zones to smooth transitions.

[0074] In another embodiment, only one of the two blending approaches may be used. In yet another embodiment, different blending mechanisms may be used. In one embodiment, blending may be performed in the rendering engine or the display.

[0075] In one embodiment, content is displayed in block 1740. In block 1745, the process determines whether there is further content to display. If so, the process returns to block 1715 to determine the mode. In another embodiment, the mode does not change between images / frames, and the system proceeds directly to block 1720 or 1725, depending on the previously selected mode. Otherwise, the process terminates in block 1747. In one embodiment, the system preprocesses the content, and as a result, analysis of color channel luminance and blending options is performed separately from the display process.

[0076] Figure 17B is a flowchart of one embodiment of dynamic blending. The process begins in block 1750.

[0077] In block 1755, in one embodiment, the process analyzes the luminance of each zone. In one embodiment, the process identifies the brightest pixel within each zone. In one embodiment, the luminance is evaluated for each color. In one embodiment, the frame is analyzed based on a histogram, which is used to determine the luminance of each zone.

[0078] In block 1760, the process determines whether the zone has no content. If so, in block 1765, the brightness is set to zero for all colors. This turns off the light source and, for example, does not send current through the LED / light source. As mentioned above, this leads to power savings. In one embodiment, the portion of the SLM associated with the no-content zone is also turned off. The process then proceeds to block 1775.

[0079] If a zone has content, in block 1770, the process determines the drive current for each light source within each zone. The drive current controls the brightness of the light. In one embodiment, a lookup table is used to determine the drive current based on the determined brightness setting. In one embodiment, there is a unique lookup table for each color and each lighting zone. In one embodiment, the required brightness for each zone and each color is algorithmically calculated based on device performance.

[0080] In block 1775, alpha mapping is applied across frames to smooth transitions between zones. Colors are remapped with gamma values ​​to match the intended perceived colors. This is done so that the perceived color does not change across zones.

[0081] In block 1780, digital blending is applied based on the zone content.

[0082] Next, the process terminates at block 1785.

[0083] Figure 18 shows one embodiment of a display system that can use the selective illumination system of the present invention. The optical engine 1810 generates an image for display to the user. The optical engine 1810 includes an illumination system 1820 and a projection system 1830. The optical engine 1810 is driven by a processing system 1870. In one embodiment, the processing system 1870 includes a processor such as a central processing unit (CPU) and a graphics processing unit (GPU). In one embodiment, the processing system 1870 may include a local computer or server, a remote computer or server, and / or a distributed set of one or more computers or servers.

[0084] The lighting system includes one or more segmented LEDs, the outputs of which are combined to output a segmented image. The segmented LEDs receive control data from a segmented lighting source controller 1875 that controls which LED segments are on and off. The segmented lighting source controller 1875 controls the power to the LEDs. In some embodiments, the segmented lighting source controller 1875 may be a processor, CPU, or GPU. In other embodiments, it may be a dedicated hardware element. The segmented lighting source controller 1875 controls the LED segments based on display zone information 1880 that indicates which parts of the display area contain image data and therefore which parts are illuminated. In one embodiment, the segmented lighting source controller 1875 uses a lookup table 1890 to convert selected light levels into current levels for each light segment. The display zone information 1880 may be pre-calculated offline by a computer system or calculated on the fly by a processor (not shown). In one embodiment, the processing system 1870 also includes an alpha blender 1885 for mixing light levels between zones. The processor or computer system may be part of the display system 1800, or it may be separate from the system.

[0085] Optical elements associated with segmented LEDs are designed to maintain the spatial relationships between LED segments. One or more intermediate optical elements between the illumination system 1820 and the projection system 1830 also maintain these spatial relationships. In the projection system 1830, a beam splitting cube directs the image to a spatial light modulator (SLM). When the image hits the SLM, it still has the same spatial relationships as the original image produced by the LEDs. In one embodiment, since only a portion of the spatial light modulator receives image data, the portion of the SLM corresponding to the portion of the segmented LEDs that is off can also be turned off. The SLM modulates the image data and passes the modulated image to the final optical element via the beam splitting cube. In one embodiment, the output of the optical engine 1810 is coupled to a combiner waveguide 1840 and output via the waveguide's output coupler 1860. The output coupler 1860 may be implemented in glasses or goggles for a head-mounted display for a virtual reality, augmented reality, reality, or mixed reality system. The output coupler 1860 may be implemented on the windshield as a head-up display. Other configurations may be used to enable the display of images from the optical engine 1810 to the user. Although the above description refers to “image” data, images in this context may include video, or other streams of image or text data.

[0086] The processing system shown in Figure 18 can be used with any configuration of the optical engine 1810, including any lighting and projection systems. For example, the processing system can be used with an embodiment of a beam-splitting cube segmented monochromatic LED optical engine shown in Figure 3B, a segmented multichromatic LED array with a beam-splitting cube shown in Figure 3C, and a double-pass optical engine shown in Figure 3D.

[0087] Figure 22 is a block diagram of one embodiment of a computer system for a specific purpose. However, it will be apparent to those skilled in the art that other alternative systems with various system architectures may also be used.

[0088] The computer system shown in Figure 22 includes a bus or other internal communication means 2240 for transmitting information and a processing unit 2210 coupled to the bus 2240 for processing information. The processing unit 2210 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processor (GPU), or another type of processing unit 2210.

[0089] The system further includes, in one embodiment, a memory 2220 which may be a random access memory (RAM) or other storage device 2220 coupled to the bus 2240 for storing information and instructions executed by the processor 2210. The memory 2220 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processing unit 2210.

[0090] In one embodiment, the system also includes a read-only memory (ROM) 2250 and / or static storage device 2250 connected to the bus 2240 for storing static information and instructions for the processor 2210.

[0091] In one embodiment, the system also includes data storage devices 2230, such as magnetic disks or optical disks and their corresponding disk drives, flash memory, or other storage devices that can store data when the system is not powered. In one embodiment, the data storage devices 2230 are connected to a bus 2240 for storing information and instructions.

[0092] In some embodiments, the system may be further connected to an output device 2270, such as a computer screen, speaker, or other output mechanism connected to bus 2240 via bus 2260, for outputting information. The output device 2270 may be a visual output device, an audio output device, and / or a tactile output (e.g., vibration) device.

[0093] An input device 2275 may be connected to bus 2260. Input device 2275 may be an alphanumeric input device, such as a keyboard, including alphanumeric and other keys, to enable the user to transmit information and command selections to the processing unit 2210. Additional user input devices 2280 may be included. One such user input device 2280 is a cursor control device 2280, such as a mouse, trackball, stylus, cursor directional keys, or touch screen, which may be connected to bus 2240 via bus 2260 to transmit directional information and command selections to the processing unit 2210 and control movement on the display device 2270.

[0094] Another device that may optionally be connected to the computer system 2200 is a network device 2285 for accessing other nodes of the distributed system via a network. The communication device 2285 may include any of the following: Ethernet, Token Ring, the Internet, or some commercially available networking peripherals used to connect to a wide area network, personal area network, wireless network, or other methods for accessing other devices. The communication device 2285 may further be a null modem connection or any other mechanism that provides connectivity between the computer system 2200 and the outside world.

[0095] It should be noted that any or all of the components and related hardware of this system illustrated in Figure 22 may be used in various embodiments of the present invention.

[0096] Those skilled in the art will understand that a particular machine embodying the present invention may be configured in various ways according to a particular embodiment. The control logic or software for implementing the present invention may be stored in the main memory 2220, the mass storage device 2230, or other storage medium accessible locally or remotely to the processor 2210.

[0097] It will be apparent to those skilled in the art that the systems, methods, and processes described herein may be implemented as software stored in main memory 2220 or read-only memory 2250 and executed by processor 2210. This control logic or software may also reside on a product including computer-readable media, in which computer-readable program code is embodied and readable by mass storage device 2230, causing processor 2210 to operate according to the methods and teachings herein.

[0098] The present invention may also be embodied in a portable device that includes a subset of the computer hardware components described above. For example, a handheld device may be configured to include only a bus 2240, a processor 2210, and memory 2250 and / or 2220. The portable device may be configured to include a set of buttons or a set of input signaling components that the user can select from a set of available options. These may be considered as a first input device 2275 or a second input device 2280. The handheld device may also be configured to include an output device 2270, such as a liquid crystal display (LCD) or a display element matrix, for displaying information to the user of the handheld device. Conventional methods may be used to implement such a handheld device. Implementations of the present invention for such devices will be obvious to those skilled in the art, given the disclosures of the present invention as provided herein.

[0099] The present invention may also be embodied in a dedicated device that includes a subset of the above-described computer hardware components, such as a head-mounted display or other dedicated display system. For example, the device may include a processing unit 2210, a data storage device 2230, a bus 2240, and memory 2220 and a display, but may not include an input mechanism or may include only a rudimentary communication mechanism, such as a small touch screen that allows a user to communicate with the device in a basic manner. Generally, the more specialized the purpose of the device, the fewer elements are essential for it to function. In some devices, communication with the user may occur through a touch-based screen or similar mechanism. In one embodiment, the device may not provide any direct input / output signals but may be configured and accessed through a website or through other network-based connections via a network device 2285.

[0100] It will be understood by those skilled in the art that any configuration of a particular machine implemented as a computer system may be used according to its particular implementation. Control logic or software implementing the present invention may be stored on a machine-readable medium accessible locally or remotely by the processor 2210. The machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, the machine-readable medium includes read-only memory (ROM), random access memory (RAM), magnetic disk storage medium, optical storage medium, flash memory device, or other storage medium that may be used for temporary or permanent data storage. In one embodiment, the control logic may be implemented as transmittable data such as electrical, optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.).

[0101] Furthermore, in one embodiment, the system may be implemented on a distributed computing system. In a distributed computing system, processing may be performed on one or more remote computer systems from the operator's location. The system may use computer system 2200 to provide local processing and further utilize one or more remote systems for storage and / or processing. In one embodiment, the system may further utilize distributed computers. In one embodiment, computer system 2200 may represent client and / or server computers on which the software is executed. Other configurations of processing systems that perform the processes described herein may be utilized without departing from the scope of this disclosure.

[0102] In the aforementioned specification, selective lighting systems are described with reference to specific exemplary embodiments thereof. However, it will be apparent that various modifications and changes may be made without departing from the broader spirit and scope of this disclosure as set forth in the appended claims. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

Claims

1. It is a display system, A segmented light source including multiple separately controlled segments for emitting light, An optical element for directing the aforementioned light, The system comprises a spatial light modulator (SLM) that modulates the aforementioned light, A display system in which light from one or more of the segments of the segmented light source is directed to the SLM through the optical element such that spatial mapping from the segments to the SLM is maintained.

2. The system according to claim 1, wherein the segmented light source includes an array of light-emitting diodes (LEDs).

3. The system according to claim 2, wherein the array includes a single color.

4. The aforementioned system, The system according to claim 3, further comprising a red array, a green array, and a blue array, and associated optical elements of each of the arrays.

5. The system according to claim 2, wherein the array includes a red LED, a green LED, and a blue LED.

6. The system according to claim 5, wherein the LEDs of different colors are of different sizes.

7. The system according to claim 1, wherein the segmented light source is a monolithic segmented light-emitting diode (LED).

8. The system according to claim 1, wherein the optical element comprises one or more of a reflective lens, a refractive lens, and a free-form lens including a reflective surface and a refractive surface.

9. The system according to claim 1, wherein the optical element comprises a microlens, and the microlens corresponds to a segment of the segmented light source.

10. Each of the separately controlled segments corresponds to a display zone, and the system, The system according to claim 1, further comprising a controller for identifying one or more display zones in an image that do not have image components, and for turning off one or more corresponding segments of the light source that are separately controlled.

11. The system according to claim 10, further comprising a controller for identifying the brightness of the image in each display zone and setting the illumination level of each of the separately controlled segments of the light source based on the brightness.

12. The system according to claim 11, wherein the brightness and illumination level are set for each color.

13. Each of the separately controlled segments corresponds to a display zone, and the system, The system according to claim 1, further comprising a blender for applying blending between the display zones.

14. The system according to claim 1, wherein the optical element is a double-pass optical element such that the light passes through the optical element to the SLM, and the modulated light from the SLM passes through the optical element before it is displayed to the user.

15. A system for selective illumination of a spatial light modulator, It is an optical engine, A segmented light source comprising a plurality of separately controlled segments for emitting light, wherein the plurality of separately controlled segments correspond to display zones, A double-pass optical element for directing the light from the segmented light source to a spatial light modulator, An optical engine comprising a spatial light modulator (SLM) for modulating the aforementioned light, A system in which light from one or more of the segments of the segmented light source is directed to the SLM through the double-pass optical element so as to maintain spatial mapping from the segments to the SLM, and modulated light from the SLM passes through the double-pass optical element before being output as an image.

16. The system according to claim 15, a processing system for analyzing the content of the image for display and providing the display zone information to the optical engine for controlling the segmented light source, wherein the display zone information indicates light settings for the display zones in the image based on the content of the image.

17. The system according to claim 15, wherein one or more of the display zones are turned off.

18. The system according to claim 17, wherein the brightness in one or more of the display zones is set based on the content of the image.

19. A method for selectively illuminating a spatial light modulator (SLM) to output an image, wherein the method is: Controlling a segmented light source that includes multiple separately controlled segments for emitting light, Directing the aforementioned light through an optical element, This includes modulating the light using the spatial light modulator (SLM), A method wherein light from one or more of the segments of the segmented light source is directed to the SLM through the optical element such that a spatial mapping from the segments to the SLM is maintained.

20. The method according to claim 19, wherein the segmented light source includes an array of light-emitting diodes (LEDs).

21. Each of the separately controlled segments corresponds to a display zone, and the method is Identifying one or more display zones within an image that does not contain image components, The method according to claim 19, further comprising turning off one or more of the separately controlled segments of the light source.

22. To identify the brightness of the image in each display zone, The method according to claim 19, further comprising setting the illumination level of each of the separately controlled segments of the light source based on the luminance.

23. The method according to claim 22, wherein the luminance and the illumination level are set for each color.

Images