Multi-Mode Display

Abstract

The multi-mode display includes a mode selector that selects one of a plurality of modes, each of the modes having a different light configuration, one mode including a reduced color space mode, and one or more light sources controlled by the mode selector, the one or more light sources being used to display content to a user using the multi-mode display.

Description

[Technical field]

[0001] (Related Applications) This application claims priority to and incorporates in its entirety U.S. Provisional Patent Application No. 63 / 362,519, filed April 5, 2022.

[0002] FIELD OF THEINVENTION The present invention relates to displays, and more particularly to an architecture for multi-mode displays with varying illumination types. [Background technology]

[0003] Color sequential displays, typically spatial light modulators, have many advantages over traditional emissive displays. The main advantages include a high fill factor per pixel and the ability to tightly control the illumination / emission angle of light from the display. However, the main disadvantage of color sequential displays is the loss of brightness and light efficiency, since each color in the system is only on for a portion of the total frame time. [Brief description of the drawings]

[0004] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: [Figure 1] 1 illustrates an embodiment of a multi-mode LCOS-based display system. [Figure 2A] 1 illustrates exemplary light timing for two exemplary modes of a three light source system: color sequential and non-color sequential. [Figure 2B] 1 illustrates exemplary light timing for two exemplary modes of a three light source system: color sequential and non-color sequential. [Figure 3A] FIG. 1 is a diagram of one embodiment of an opto-electronic and electronic driving configuration for a common anode three-light source multi-mode display system. [Figure 3B]FIG. 1 is a diagram of one embodiment of an opto-electronic driving configuration for a discrete anode three-light source multi-mode display system using a single controller IC. [Figure 3C] FIG. 1 is a diagram of one embodiment of an opto-electronic driving configuration for a discrete anode three-light source multi-mode display system using a separate controller IC for each light source. [Figure 3D] FIG. 1 is a diagram of one embodiment of an opto-electronic driving configuration for a light array-based display system. [Figure 4A] FIG. 1 is a diagram of one embodiment of an optical system configuration for a four-light source multimode display system. [Figure 4B] FIG. 1 is a diagram of one embodiment of an optical system configuration for a four-light source multimode display system. [Figure 4C] 1 illustrates an exemplary configuration of a multi-LED package. [Figure 4D] 1 illustrates an exemplary configuration of a multi-LED package. [Figure 4E] 1 illustrates an exemplary configuration of a multi-color light array. [Figure 4F] 1 illustrates an exemplary configuration of a multi-color light array. [Diagram 5] FIG. 1 is a perspective view of another embodiment of an optical system configuration for a four light source multimode display system. [Figure 6A] 1 illustrates exemplary light timing for two exemplary modes of a four light source system: color sequential and non-color sequential. [Figure 6B] 1 illustrates exemplary light timing for two exemplary modes of a four light source system: color sequential and non-color sequential. [Figure 7A] 13 illustrates some example light timing for various restricted light option embodiments. [Figure 7B] 13 illustrates some example light timing for various restricted light option embodiments. [Figure 7C] 13 illustrates some example light timing for various restricted light option embodiments. [Figure 7D]13 illustrates some example light timing for various restricted light option embodiments. [Figure 7E] 1 shows some example light timings for a light display with multiple red, blue, and green light sources. [Figure 7F] 1 shows some example light timings for a light display with multiple red, blue, and green light sources. [Figure 8] FIG. 1 is a block diagram of one embodiment of a multi-mode display system. [Figure 9] 13 is a flow chart of one embodiment of an automatic mode change trigger. [Figure 10] 1 is a flow chart of one embodiment of on-the-fly display optimization. [Figure 11] FIG. 1 is a block diagram illustrating one embodiment of a computer system that may be used with the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] A display is described that can operate in multiple modes. Each mode has a different light configuration in which the light sources, their duty cycles, their power levels, and / or combinations thereof are varied. In one embodiment, the modes include a first mode with full color RGB and a second mode with reduced color space (RCS). In one embodiment, the modes include a color sequential mode of operation and a non-color sequential (NCS) mode of operation. In another embodiment, the modes include a full color RGB mode (which may or may not be color sequential) and a reduced color space mode (which may or may not be color sequential). The reduced color space mode sacrifices color (the user perceives less color space, or only a single color), but gains the benefit of increased brightness and / or lower system power consumption. The RCS and NCS modes can be advantageous for many use cases, including augmented reality (AR) head-mounted displays (HMDs). Augmented reality HMDs may be worn outdoors, where the display brightness needs to be visible even in the presence of sunlight. By providing a multi-mode display, a full color space can be provided when operating in color sequential mode, and additional brightness and reduced power can be provided in RCS mode. Thus, the system can be optimized for both use cases without the need for a separate display. The multi-mode display can also provide an optimized nighttime display.

[0006] A reduced color space (RCS) mode is a mode in which a display changes its light sources such that the color space that can be represented in a single image frame is reduced compared to a full-color RGB mode. In a full-color RGB mode, the color space can typically be represented by a triangle, with the red, green, and blue light sources each representing a vertex of the triangle. In one embodiment, the RCS mode only displays a single color per image frame, in which case only a single point in the color space is represented per image frame. In another embodiment, the RCS mode displays two color frames per image frame, in which case the color space of the system becomes a one-dimensional line.

[0007] In one embodiment, the RCS mode can change the color selected in a subsequent frame or series of frames. In one frame, the system color space is a one-dimensional line, and in a subsequent frame or series of frames, the system color space is a different one-dimensional line. Drive current adjustments can be used to provide such a change. This allows the user the experience of perceiving one image containing two colors, and then a subsequent image containing two different colors. This provides the benefits mentioned above, more brightness and reduced power. In one embodiment, these different color selections are made based on the content being displayed. In another embodiment, the color selection is based on the environment.

[0008] The multi-mode display system allows for on-the-fly changing of color sequences. In one embodiment, the system can be set to one of multiple pre-configured sequences, where the sequences are designed to address various internal and external factors. In one embodiment, a machine learning / artificial intelligence system is used to analyze possible color sequences for the display and generate settings to optimize the color display. In one embodiment, the color sequence may be optimized based on various factors, including one or more of battery usage (efficiency), background, image frame content, ambient light levels, time of day, waveguide spectral response, temperature effects on LEDs or other light sources, etc. In one embodiment, the system may prioritize these customization factors based on user preferences, system settings, and / or relevance. In another embodiment, the system may select or optimize a pre-configured sequence based on sensor input on the device, such as, but not limited to, light sensors, cameras, inertial measurement units, motion sensors, eye tracking sensors, touch sensors, audio sensors, or user input via buttons or other mechanisms. Thus, the system can provide smart adjustments to provide customized color mode displays, where the modes can change based on various factors.

[0009] In one embodiment, the system utilizes an illumination array. The illumination array may be an N×M array of LEDs. The use of the array allows for partial illumination of a spatial light modulator (SLM). This allows, in one embodiment, different portions of the display to have different color sequences. In one embodiment, a separate array is used for each color. In another embodiment, a single array may include all available colors. In one embodiment, the colors are RGB LEDs. In one embodiment, the colors include RGB and a fourth color, amber or white, as described below.

[0010] The following detailed description of embodiments of the present invention refers to the accompanying drawings, in which like reference numerals indicate like elements and which show, by way of illustration, specific embodiments embodying the present invention. The description of these embodiments is in sufficient detail to enable those skilled in the art to practice the invention. Those skilled in the art will understand that other embodiments may be utilized and that logical, mechanical, electrical, functional, and other changes may be made without departing from the scope of the present invention. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

[0011] 1 illustrates one embodiment of a multi-mode LCOS-based display system for displaying images, including still and / or moving images, to a user. This is merely an exemplary embodiment, and the system may be implemented in a variety of display systems that utilize sequential color displays. For example, the system may be implemented in a head-up display, a projection display, a head-mounted display (HMD), a virtual reality display, an augmented reality display, a head-up display, etc.

[0012] The display system 100 displays the subframes using red, green, and blue light sources. In one embodiment, the light sources are LEDs 110. Using the red, green, and blue light sources 110 allows for a wide enough color gamut to be reproduced, providing a full-color display experience to the user. In one embodiment, the light sources may be light-emitting diodes (LEDs), lasers, micro LEDs, organic LEDs (OLEDs), superluminescent diodes (SLEDs), phosphors, or quantum dots. In one embodiment, the light sources may be segmented LEDs or an array of LEDs. In one embodiment, the spatial light modulator (SLM) may be liquid crystal on silicon (LCoS), digital light processing (DLP), digital micromirror device (DMD), or liquid crystal display (LCD).

[0013] The light from these sources passes through illumination optics 115A-115C and in one embodiment is combined using a combiner such as an X-cube 120 or other optical element. The combiner 120 used may be different and may be a dichroic plate, a holographic optical element, or any other type of combiner that ensures that the light from the different light sources 110 is accurately positioned. In one embodiment, the light passes through a microlens array 130 and intermediate optics 135. They are then reflected to a spatial light modulator (SLM) such as a liquid crystal on silicon (LCoS) 140. The modulated light then passes through a final optics 150 before being displayed to a user. In one embodiment, the output of the final optics 150 is coupled into a combiner waveguide 160. The combiner waveguide 160 in one embodiment is used to display an image via glasses, goggles, or another mechanism.

[0014] Typically, such LCoS and DLP systems use color sequential displays. In a color sequential display, the total brightness perceived by the user is the fractional sum of the duty cycle of each color. The duty cycle is the percentage of the frame time that a particular color is on. If the system was operated such that each color subframe was on for one-third of the total frame time, the total system brightness would be one-third that of a system in which all three colors were on simultaneously for the entire frame. Figure 2A shows an exemplary color sequential display in which the red, green, and blue LEDs each have a duty cycle of one-third of the frame.

[0015] The multimode system of the present invention provides a solution for applications where the ability to adjust color sequence and timing is beneficial. One such application is situations where color breadth (full color) is less important than brightness, power efficiency, and / or visual contrast. A multimode display can be operated with different color sequences and timings, which in some embodiments are referred to as reduced color space (RCS) modes. In non-color sequential modes, only a single color is displayed per image frame. In one embodiment, in non-color sequential modes, one or more light sources may be enabled simultaneously to produce a single combined color. This allows for tuning of the color displayed to the user by varying the relative drive strength of each light source. In one embodiment, red, green, and blue light sources are enabled simultaneously such that the user perceives white. FIG. 2B shows an exemplary non-color sequential mode display with all three colors on at the same time. In another embodiment, green and red are enabled simultaneously such that the user perceives amber or yellow. Any other color combination may be used.

[0016] In one embodiment, only a single color is displayed to the user for the full frame. In one embodiment, the single color is green, which provides maximum photopic efficiency to the user's eyes. In one embodiment, the single color is red, which may allow the user to maintain scotopic / night vision. In one embodiment, the user may select the color. In one embodiment, the color may be automatically selected based on various factors including the environment and conditions of use.

[0017] In another embodiment, the color timing may vary based on the multimode preference. For example, a frame may consist of an RGBRGB sequence where each on-segment is 1 / 6 of the frame. A frame may also be RG / BG where the red and green lights are on at the same time, then the blue and green lights are on in another segment. Additionally, the duty cycle of the combined lights may be less than the length of the frame. For example, the red, green, and blue lights may all be on for less than 1 / 3 of the entire frame, resulting in no lights being on during a portion of the frame. The length that each light source is on may vary. As mentioned in the RG / BG example, in some cases, one or more lights may be on for the entire frame and other lights may be on for a portion of the frame.

[0018] In one embodiment, multi-mode is enabled by adjusting the drive architecture of the light sources. In a typical color sequential display system shown in FIG. 3A, the drive electronics for the light sources use a common anode drive architecture. Multi-mode display embodiments that can enable two or more colors simultaneously utilize an architecture that allows individual control of the light sources. Because light sources of different colors typically have different forward voltages, the drive architecture may need to support multiple forward voltages simultaneously. In one embodiment, this is accomplished by using a single anode power supply and varying the cathode voltage for each color. FIG. 3A can be operated in a mode that enables this use case. This approach can save space, but is generally inefficient as power is lost due to the low-side transistors of each color operating in their linear region to maintain the forward voltage of each color.

[0019] In another embodiment, the system supports separate anode voltages for each electro-optic light source but uses a single controller IC. An exemplary diagram of this configuration is shown in FIG. 3B. This architecture may require more PCB space, since adjusting multiple anode voltages requires filtering electronics for each anode voltage line. However, having separate anode voltages typically allows for more efficient operation per color when operating in RCS mode compared to systems where the low-side drive transistors operate in the linear region. In another embodiment, the system supports separate anode voltages for each electro-optic light source and uses separate control ICs for each light source. An exemplary configuration is shown in FIG. 3C. In another embodiment, the light sources are designed to have sufficiently similar forward voltages, so that a single anode voltage can be used for all light sources, while also allowing each low-side transistor to operate either in saturation or in the minimum linear region where little power is consumed by the linear operation of the transistors. This configuration is shown in FIG. 3A. In one embodiment, sufficiently similar forward voltages per light source are achieved by using the same diode design to drive each color and changing the emission color of some of the diodes. In one embodiment, phosphor conversion can be used to change the color of the diode.

[0020] FIG. 3D illustrates an embodiment in which a multi-mode system is supported by an LED array 355 containing N×M LEDs. In one embodiment, the LED arrays 335 are all monochromatic and there are three separate arrays, one for each color. In one embodiment, where each LED array 355 is monochromatic, there may be three separate controller ICs 350 and associated LED arrays 355. In another embodiment, a single controller IC 350 may control two or more LED arrays 355. Any of the configurations described above in FIGS. 3A-3C may be used with the LED arrays.

[0021] In one embodiment, rather than a monochromatic array, the LED array 355 includes multiple colors. In one embodiment, there are the same number of LEDs of each color in the array 355. In another embodiment, the number of LEDs of each color may vary. For example, there may be more green LEDs than blue or red LEDs. In one embodiment, the LEDs of various colors are distributed so that a portion of the display can be illuminated with all colors. In one embodiment, the LED array 355 is controlled by a controller IC 350. In one embodiment, a multiplexer is used to address the individual LEDs in the array. In another embodiment, there may be separate horizontal and vertical addresses controlled by separate portions of the controller IC 350. Other methods of controlling the individual LEDs in the array 355 may be used.

[0022] In one embodiment, the LED array may have a passive matrix backplane. In another embodiment, the LED array may have an active matrix backplane.

[0023] 4A and 4B are diagrams of one embodiment of an optical system configuration for a four-light source multimode display system, showing the example RGB light paths of FIG. 4A and the example NCS illumination path.

[0024] As shown in FIG. 4A, in one embodiment, the system includes four LEDs in two LED packages 410, 460. The first LED package 410 includes red, green, and amber LEDs. The second LED package 460 includes a blue LED. The LED packages used may vary, for example, red / blue may be in one package and green in another package. Alternatively, one package may include red / green and the other package may include blue / green. Other configurations may be used. In one embodiment, the color combination used depends on the use case. In one embodiment, the LED packages may be LED arrays, as described in more detail below.

[0025] The illumination optics 430, 470 include red / green / amber illumination optics 430 through which light from the red / green / amber LED package 410 passes, and blue illumination optics 470 through which light from the blue LED 460 passes. In one embodiment, the illumination optics may include a compound parabolic concentrator. Other types of optics may be used.

[0026] The light is combined by a combiner 480. The combiner 480 includes two dichroic plates 490, 495. The blue light passes through both plates towards the projection optics (not shown), the green light is reflected by one dichroic plate 490 towards the projection optics, and the red light is reflected by the other dichroic plate 495 towards the projection optics. Although shown spaced apart, in an actual configuration the distance between the back of the green reflector plate 480 and the front of the red reflector plate 495 is minimized. In one embodiment, the light reflecting from the plate closer to the red and green LEDs 410 reflects light from the back, while the further plate reflects light from the front, minimizing the displacement. In one embodiment, the dichroic plates 490, 495 of the combiner 480 may be steerable to shift angles when reflecting the red / green LEDs versus the phosphor-converted LEDs. In one embodiment, an actuator is used to shift the dichroic plates 490, 495 between two positions, which has the advantage of better aligning the illumination pupil between full color and NCS modes.

[0027] Combiner 480, which includes a dichroic plate, may be replaced by another type of combiner, such as an X-cube (which is also dichroic), or a combiner that uses refractive, dichroic, diffractive, freeform, catadioptric, metalens, or holographic optical elements.

[0028] 4B shows the path when the first LED package 410 uses a fourth light source, in this example an amber LED. The first LED package 410 includes an amber (or white) LED that travels through red / green / amber illumination optics 430 and is reflected by green and red reflecting dichroic plates 490, 495.

[0029] This is merely an example configuration of these elements. In other configurations, the fourth color light source may be separate. In another embodiment, the light sources may come from three separate locations, for example, the fourth light source may have separate optics. Other optical configurations may be used. Although this illustration refers to "LEDs," each of the light sources may be an LED array. In another embodiment, a single LED array may provide multiple color light sources.

[0030] 4C and 4D show an example configuration of a multi-LED package. In this example layout, a green LED (G) and a red LED (R) are side-by-side, and a phosphor-converted fourth color LED (P) is placed toward the center above the red and green. As shown in FIG. 4D, the phosphor-converted fourth color LED (P) may be smaller than the green LED (G) and the red LED (R). This configuration has an RCS mode with a smaller field of view than the RGB mode. This may be useful in some cases, for example, in an AR display when information is displayed on only a portion of the screen.

[0031] In one embodiment, the fourth color is combined with the other colors using one or more polarizing elements, such as, but not limited to, liquid crystal rotators, liquid crystal elements, polarizing reflectors, geometric phase lenses, polarizing beam splitters, and wave plates. In one embodiment, the light may be polarized such that the RGB light has a first polarization and the fourth light has the opposite polarization.

[0032] In one embodiment, the fourth color is combined with a dichroic element, such as, but not limited to, a dichroic filter, a dichroic combiner, and a dichroic reflector.

[0033] In one embodiment, the colors are combined and selected using mechanical elements such as, but not limited to, moveable mirrors, moveable optics, moveable filters, shutters, MEMS devices, deformable mirrors, adjustable lenses, or moveable prisms.

[0034] In one embodiment, the system has multiple inputs and multiple outputs for illumination, the inputs and outputs corresponding to one or more colors, hi another embodiment, the multiple inputs / outputs are angle selective.

[0035] In one embodiment, the illumination color combination uses one or more elements, which may include any of the listed polarization, dichroic, mechanical and / or multiple elements, or other elements that may be used to combine illumination.

[0036] FIG. 4E illustrates one embodiment of a 4×4 LED array. The LED array 430 includes multiple LEDs of each color. In this example, there are four clusters 435 of four LEDs, one red (R), one blue (B), and two green (G). However, this is only an exemplary configuration in which the colors are clustered with two greens. In another embodiment, the LED cluster 435 may include a fourth color, which may instead be white, yellow, or amber. Other methods of clustering the light may be used. In some embodiments, the shape of the individual light sources may vary. Furthermore, it is not necessary for all colors to have equal areas or shapes. For example, the green light sources in the array 430 may be larger in some embodiments because the human eye is most sensitive to green. Furthermore, the layout of the array may vary.

[0037] 4F illustrates an N×M LED array. The LED array 440 includes M rows of LEDs with N LEDs per row. As discussed above, the color configuration can include one, two, three, or four colors in the array. In one embodiment, each LED represents an array element 445 having all three colors RGB. Although the diagram illustrates the colors in each array element 445 as being the same size, the actual size of the color elements may vary by color and / or across the array.

[0038] In addition to arrays, the present application can use segmented LEDs. Segmented LEDs are monochromatic LEDs with two or more segments that can be controlled separately. This can be used, for example, to illuminate a portion of a display area with a specific color. This can be particularly useful for use cases where there is an area that consistently displays limited color data. For example, for a heads-up display where the bottom third of the display shows speed, temperature, and similar numerical data, it can be efficient to have segmented LEDs that can provide a full color high color resolution image in the top two-thirds of the display and a reduced color display in the bottom third. Arrays can also be used for this purpose. Segmented LEDs can be considered, in one embodiment, as a monochromatic 1×N array.

[0039] 5 is a perspective view of another embodiment of an optical system configuration for a four-light source multimode display system. In this configuration, the full-color mode illumination block 510 is separate from the RCS mode illumination block. The full-color mode illumination block 510 has light sources that are combined by a combiner 520 and reflected by a mirror 530 towards the projection optics 570. The RCS mode light source 550 passes through a diffractive / holographic combiner 560, which in one embodiment is a waveguide, on its way to the projection optics 570. The projection optics 570 is shared by the RCS mode light 550 and the full-color mode light.

[0040] 6A and 6B show exemplary light timing for two exemplary modes of a four light source system, color sequential and non-color sequential. As seen in FIG. 6A, the color sequential mode shows each color for 1 / 3 of a full frame. In contrast, for the non-color sequential mode, in one embodiment, the fourth LED (RCS_0) is on for the entire frame. In another embodiment, the fourth LED is on for a portion of the frame. In one embodiment, there is some blanking time between frames. In another embodiment, all four LEDs are on at the same time, maximizing the brightness outside the system.

[0041] 7A-7D show light timing for various limited light configuration embodiments. FIG. 7A shows example timing for a green only display. In one embodiment, the green light source is on for the entire frame, shown as Green_0, full brightness. In another embodiment, Green_0 * As shown in Figure 1, the green light source is on for 50% of the frame. This is called the enhanced contrast mode. Because LCOS takes some time to switch on and off, having blanking time between frames produces a more accurate reproduction of the image and provides higher contrast. Although lower duty cycles may be used, a fully on system is typically more efficient as it allows for lower drive currents and allows the LEDs to operate in a more efficient operating region. Alternatively, rather than having the green light source on for half the frame and off for the other, a fully on system could be used instead, as shown in Figure 1. ** shows a distributed green LED on time where the green light is on for two pulses in a single frame. This can reduce flicker. Other pulsing schemes may be used. Although this is described with respect to green light, similar pulsed illumination may be used for any color. Additionally, this type of pulsed illumination may be used with displays that include more than two colors.

[0042] 7B shows example timing for a red-only display. It is shown with the red light on for the entire frame, but one of ordinary skill in the art will understand that the red light may be lit for only a portion of the time.

[0043] FIG. 7C illustrates an example timing for a red / green display. In one embodiment, red and green LEDs are enabled simultaneously and the drive current to each LED is varied to achieve the desired shade of amber. In one embodiment, the red and green LEDs are driven sequentially and a combination of drive currents and duty cycles is used to achieve the desired shade of amber. In this embodiment, the displayed content can achieve any color between red and green. In another embodiment, the red and green LEDs may be enabled simultaneously for only a portion of the frame, allowing one of the colors to be on for a different duration. FIG. 7D illustrates an example timing for a display in which a first color sequence showing red and green simultaneously is combined with a second color sequence showing blue and green simultaneously.

[0044] Although the figures show various duty cycles for different examples, it should be understood that any of the configurations can have any of the duty cycles shown. To maximize brightness, the duty cycle may be 100%. However, a reduced duty cycle may be used, for example, for battery conservation or when full brightness is not required. Additionally, the frame rate may be varied for different light configurations. In one embodiment, the frame rate for a full spectrum display is 60Hz (60 frames per second). In one embodiment, for a reduced color space option, the frame rate may be reduced or increased. In one embodiment, for example, for a head-up display, the frame rate may be varied between 30Hz and 360Hz, or at the highest frame rate available for the spatial light modulator. In one embodiment, a variable frame rate may be used to display persistent information at a frame rate less than 60Hz. For example, for a clock showing hours and minutes, the frame rate may be reduced to one frame per minute, for example 0.016Hz.

[0045] 7E shows one embodiment of timing for an LED array with two LEDs for each color. In this example, both sets of LEDs are illuminated sequentially for the first frame Frame0, but only the second set of LEDs is illuminated for the second frame Frame0. In one embodiment, this may allow the second frame to illuminate only a portion of the display.

[0046] For example, if the system is outputting an augmented reality image and only a portion of the screen has content, this allows the system to conserve power and still provide a high quality image for the portion of the frame that has data. This may also be useful for displaying things like closed captions, road signs, or a persistent clock for a portion of the display. In one embodiment, the second half of the frame may contain only a single color or a subset of the colors being displayed.

[0047] FIG. 7F illustrates one embodiment of the timing of an LED array with two LEDs for each color. In this example, during a single frame, the first set of RGB LEDs (Red 0, Green 0, and Blue 0) are illuminated in sequence during the entire frame. In this example, they are each illuminated in a color-sequential manner for about one-third of the frame. The other set of LEDs (Red 1, Green 1, and Blue 1) are also illuminated in sequence throughout the frame, but some of the LEDs are illuminated for longer periods than others. In this example, the red LEDs are on for one-sixth of the frame, the green LEDs are on for half the frame, and the blue LEDs are illuminated for the remaining two-sixths of the frame. Thus, the duty cycle can vary by color and by the portion of the display that is illuminated. In this case, the display panel is synchronized with each illumination subsection. Of course, this is merely an example, and one of ordinary skill in the art will understand that the relative length of on-time of each LED may be varied and there may be periods when none of the lights are illuminated.

[0048] 8 is a block diagram of one embodiment of a multi-mode display system. The mode selection system 810 provides display settings to a display system 870 that utilizes the color sequences identified by the mode selection system 810 when displaying content. The mode selection system 810 in one embodiment utilizes data from a sensor 830. In one embodiment, the display settings may be specified on a frame-by-frame basis. In another embodiment, the display settings may be transmitted only when changed from the current settings. In one embodiment, the mode selection system 810 receives pre-computed color sequences from an AI / ML system 890.

[0049] In one embodiment, the mode selection system 810 is implemented by a computer system including a processor and memory. The mode selection system 810, in one embodiment, is a computer system that provides instructions to the display system 870. The processor(s) providing the functionality of the mode selection system 810 may be split across multiple devices. For example, some or all of the processing may occur in the glasses or other display system itself. Other parts of the processing may occur on a remote device. In one embodiment, the precomputed sequence 857, the mode selector 855, the environmental background analyzer 840, and the content analysis 850 may be part of the display system 870. In one embodiment, one or more of the sensors 830 may be in the display system 870. The mode selection system 810 may not be a separate device from the display system 870.

[0050] The AI / ML system 890 includes one or more processors that receive data from and provide data to the mode selection system 810. In one embodiment, the AI / ML system 890 may be a remote server. In one embodiment, the AI / ML system 890 may be implemented in the cloud using distributed processing and storage.

[0051] The display settings from the mode selection system 810 are transmitted by the display controller 860 based on a determination of one or more factors. As mentioned above, in one embodiment, one of the factors is user preferences received via the user interface 815.

[0052] A particular mode may be automatically enabled in the display system by mode selection system 810 based on the use case or environmental conditions. In one embodiment, ambient light sensor 832 is turned on when the system is in an AR format or a heads-up format, or another format in which image data is displayed over other external data, as identified by display format identifier 820.

[0053] In one embodiment, the mode selection system 810 selects the RCS mode when the ambient light sensor 832 detects high ambient light intensity. In one embodiment, the mode selection system 810 enables the RCS mode when a thermal sensor 835 in the system detects that the system is too warm and should operate in a more power-efficient mode. The thermal sensor 835 in one embodiment indicates when the temperature of the display system 870 or the processing system exceeds a threshold. In one embodiment, when the temperature exceeds a threshold, a reduced color display may be selected to reduce power consumption. In one embodiment, as the temperature of the LEDs changes, the color space may be adjusted to compensate for color drift due to temperature. In one embodiment, this may be detected based on data from the thermal sensor 835. In one embodiment, a display sensor may be used to monitor the output of the lighting system 880 and adjust for color drift based on the detected color spectrum versus the intended color spectrum.

[0054] In one embodiment, the RCS mode is enabled when the environmental background analyzer 840 detects visual content that may be more advantageously shown in monochrome or reduced color space format. In one embodiment, this visual content is, for example, a text or speech-based application, or a simple image. In one embodiment, the RCS mode is enabled when the ambient light sensor 832 detects low ambient light and the clock 825 indicates that it is nighttime. In one embodiment, when the clock indicates that it is nighttime, the system may operate in a nighttime mode that better preserves the user's scotopic / night vision. In one embodiment, this may include utilizing a red-only display. In one embodiment, the RCS mode is enabled in a battery-powered system as a power saving mechanism when a low battery is detected by the battery sensor 845. In one embodiment, a preference or goal may be set for a battery threshold below which the system will switch to a power saving mode. In one embodiment, if the display system is displaying a movie or other content of a known length, the system may switch to a reduced color space mode to ensure that the entirety of the movie or other content can be displayed with the remaining battery power.

[0055] In one embodiment, the sensor 830 also includes an eye tracker 867 to determine where the user's gaze is looking. In one embodiment, the system may utilize different color spectrums for the focal area and peripheral areas of the user's gaze.

[0056] In one embodiment of the AR HMD, the system includes an outward-facing AR background sensor 837 that images the same field of view as the display shown to the user. An environmental background analyzer 840 analyzes the background data, and a mode selection system 810 selectively enables a particular display mode to optimize the display for the background of the user's environment. In one embodiment, the optimization is designed to provide the highest possible contrast between the display and the environmental lighting and color conditions. For example, if the user's field of view that matches the AR display contains light that is primarily red in color, the system can switch to an operating mode that uses more blue and green light to provide a high-contrast virtual image compared to the red light the user sees.

[0057] In one embodiment, the field of view selector 865 selects the display field of view. In some embodiments, there may be an RCS mode that has a smaller field of view than the RGB mode.

[0058] A user may manually put the display system into a particular mode using the user interface 815. The user interface 815 may be a button, dial, screen, voice interface, or another means that allows the user to select a display mode. In one embodiment, the user may put the system into a scotopic / night vision mode to preserve night vision. In one embodiment, the user may put the system into a night mode that reduces or completely eliminates blue light emission from the system. In one embodiment, the user may put the system into an RCS mode because the user is watching content that is better represented in a monochrome format. In one embodiment, the user may select an RCS mode because the system is used in a high brightness environment. In one embodiment, the user may put a battery-operated system into an RCS mode as a power saving mechanism to preserve or extend battery life. The user may also set general preferences via the user interface 815. Such general preferences may be saved as user preferences 817. Preferences may be global, e.g., the user prefers high color fidelity for lower power consumption, or time-based, e.g., the user prefers warmer colors with lower brightness and less blue light at night. Other preferences may be set by the user. Target selection 852 may be based on a combination of user preferences 817 and data from the sensors and image data from content analysis 850.

[0059] The color or color combination used for the selected display mode may be selected by the mode selector 855 based on the use case, user preferences, content display, and / or environmental conditions. In one embodiment, the system may select a preconfigured sequence or optimize a sequence based on sensor data from the sensors 830. In one embodiment, the sensors 830 may be on the device. The sensors may include, but are not limited to, user interface elements 815 such as light sensors 832, eye tracking sensors 857, thermal sensors 835, touch sensors, audio sensors, or other types of user input sensors, as well as other sensors 869 such as cameras, inertial measurement units, motion sensors, etc. The mode selector 855 selects the color, color sequence, and timing. In one embodiment, the mode selector 855 utilizes one of multiple precomputed sequences 857. The precomputed sequences 857 in one embodiment are received from the AI / ML system 890.

[0060] The AI / ML system 890 in one embodiment is a remote server system that calculates potential sequences to be used by the mode selection system 810. In one embodiment, the AI / ML system 890 is an offline system that pre-calculates such modes for various lighting, environmental conditions, display content, and other factors. In one embodiment, the AI / ML system 890 utilizes a machine learning system 896 that receives data from data collection 894 with various potential sequences and available sensor data and calculated data. In one embodiment, the machine learning system 896 is first trained with data from expert users that classify color quality and visibility in various scenarios, for example, with various AR backgrounds and real-world scene content. The trained machine learning system 896 can then be used to create pre-calculated sequences that are optimized for various scenarios. This data is passed to the mode selection system 810, and the mode selector 855 can select a pre-calculated sequence 857 based on data from the sensors 830, the environmental background analyzer 840, and other data sources.

[0061] In one embodiment, a system with an ambient light sensor 832 detects a high brightness environment and shows red, green, and blue simultaneously to provide high brightness white visual content to the user. In one embodiment, a system with an ambient light sensor 832 detects a low brightness environment and shows red only in RCS mode to preserve the user's scotopic / night vision. In one embodiment, a battery-powered display detects a low battery 845 and shows only green to maximize photopic efficiency and minimize power consumption so that battery life may be extended.

[0062] In one embodiment, a thermal sensor 835 in the system detects that the system is getting too warm and an RCS mode showing only green is enabled to maximize photopic efficiency and minimize the amount of heat still produced by the system. In one embodiment, green and red light sources are enabled in RCS mode to provide high photopic efficiency to the user while providing a more pleasing visual color versus pure green.

[0063] In one embodiment, Content Analysis 850 analyzes the content of the image being displayed. In one embodiment, the system can optimize the color sequence based on the content being displayed. For example, if the content is best displayed in monochrome, the system can adjust the color sequence to monochrome. In one embodiment, for a display with a light source array, one portion of the display may have a different color sequence than another portion of the display. For example, in a display with a persistent clock in one area, that area may utilize a different color scheme than the portion of the display that has the image. Similarly, closed captions may be displayed in monochrome, while the image above is displayed in a full spectrum color image.

[0064] In one embodiment, the alternative operating mode has two color sequences selected by the mode selector 855. In one embodiment, the two color sequences are red and green color sequences. In one embodiment, the first color sequence shows red and green simultaneously and the second color sequence shows blue and green simultaneously. In one embodiment, the two selected colors can be changed on a frame-by-frame basis. In that case, the user will see a first frame with two colors followed by a subsequent frame with two different colors. For such color switching, in one embodiment, the color selection is based on the content.

[0065] These embodiments are for illustration and not for limitation to two color sequences. Operating in two color sequences allows for non-monochrome displays while also allowing some of the benefits of a single color mode (increased brightness and / or power efficiency).

[0066] Display system 870 includes a buffer / memory 872 for storing display settings from mode selection system 810. The display system further includes optics 876, a spatial light modulator 874, optionally a waveguide 878, and an illumination system 880. Illumination system 880 may have separate light sources 882, 884 for the full color mode versus the reduced color space mode. In one embodiment, the full color mode uses red, green, and blue light sources 882, and the RCS mode uses a white light source 884. In one embodiment, the white light source is a white LED.

[0067] In one embodiment, the full color mode uses red, green, and blue light sources 882, and the RCS mode uses a phosphor-converted green / amber LED 884 located adjacent to the red and green LEDs in the illumination subsystem. In one embodiment, the RCS light sources are combined using a combiner 888. The combiner 888 may include reflective, refractive, dichroic, diffractive, or holographic optical elements. In one embodiment, the combiner 888 enables the use of a shared optical element 876 between the RGB color sources 882 and the fourth color source 884.

[0068] In illumination systems 880 that use a dichroic plate as the color combining combiner 892, this configuration has the advantage that the same dichroic film combiner can be efficiently used for both full color and RCS modes. In one embodiment, the spatial light modulator 874 may be an LCoS, DLP, or LCD.

[0069] In one embodiment, the RCS can use a different light source than the full color mode light source 882. The different light source 884 in one embodiment may have a different illumination angle than the full color mode light source. In one embodiment, the RCS mode has a different field of view than the full color mode. In one embodiment, the light source may be an LED, laser, micro LED, OLED, SLED, phosphor, or quantum dot.

[0070] In one embodiment, display system 870 has an LCoS spatial light modulator 874 and a polarizing beam splitter as part of combiner 888. In another embodiment, display system 870 has an LCoS spatial light modulator and a non-polarizing beam splitter as part of combiner 888. In one embodiment, the display system has a DLP spatial light modulator with a prism. In another embodiment, the display system has a DLP spatial light modulator without a prism.

[0071] In one embodiment, the display system uses different gamma tables for different types of sequences, including RCS sequences. In one embodiment, the system has separate gamma tables for red, green, and blue, respectively, and another gamma table for modes where red, green, and blue are on simultaneously. These gamma tables can be custom tuned to get the best contrast, best brightness, to compensate for thermal changes in the LEDs and panel, or according to some other system performance metric.

[0072] 9 is a flow chart of one embodiment of an automatic mode change trigger. The process begins at block 910. In one embodiment, the mode change may be on the fly and the mode may be changed every frame.

[0073] At block 915, the process determines whether a user has selected a mode. In one embodiment, the user can manually select any mode and override the automatic mode selection. If the user has selected a mode, at block 920, the display mode is set according to the user's selection. In one embodiment, the display modes may include a full color RGB display and various reduced color space modes, which may be non-sequential or sequential.

[0074] If the user has not selected a mode, in block 925 the system determines whether the system is being used in AR / heads-up mode. If the system is not in AR / heads-up mode, in block 930 the process determines whether low battery or high heat is an issue. If there is low battery or high heat, in block 940 a display mode is selected to reduce power. In one embodiment, this may be a green only display mode. Alternatively, another monochromatic or reduced color space display mode may be selected. If battery or heat is not an issue, in block 935 a full color space mode is selected. If AR mode is identified in block 925, the process continues to block 945.

[0075] Environmental factors are identified at block 945. These environmental factors may include, for example, the brightness, time of day, and color spectrum of the real-world scene being viewed.

[0076] At block 950, an appropriate mode is selected based on the determination. The process then ends. As noted above, in one embodiment, the user may override the automatic mode determination. Additionally, in one embodiment, the user may trigger a re-evaluation of the selected mode.

[0077] 10 is a flow chart of one embodiment of adjusting a color sequence based on various factors. The process begins at block 1010. At block 1015, a default display mode is selected. In one embodiment, the default display mode is a full color display.

[0078] At block 1020, the process determines whether a waveguide adjustment should be performed. In one embodiment, the waveguide adjustment is performed at the initial configuration of the system. If the waveguide adjustment has not yet been performed, at block 1025, the system adjusts the color display based on the spectral characteristics of the waveguide in the actual device. This allows for color consistency across different waveguides. At block 1027, the default mode is updated based on the optimization. The process then returns to block 1030 to continue the process.

[0079] At block 1030, the process determines whether the user has set fixed preferences. If so, at block 1035, the user's preferred color settings are applied. Then, at block 1037, the process updates the default mode. Then, at block 1075, the process ends since the user's setting preferences override other configuration changes. If the user has not set fixed preferences, the process proceeds to block 1040 to continue adjustments.

[0080] At block 1040, the process determines whether the display is an augmented reality display. If so, at block 1045, the process evaluates the real-world scene in which the display is shown and sets a color mode based on environmental factors. The process then returns to block 1050. If the display is not an AR display, the process continues directly to block 1050.

[0081] At block 1050, the process determines whether temperature is affecting color. If so, at block 1055, the environment in which the display is being shown is evaluated and the color mode is set to adjust for color drift due to temperature. The process then returns to block 1060. If temperature is not affecting the display, the process continues directly to block 1060.

[0082] At block 1060, the process determines whether smart adjustment is enabled. Smart adjustment is the adjustment of the display color mode based on the content being displayed. In one embodiment, the color mode may be enabled by a user. If smart adjustment is enabled, the process continues to block 1065. At block 1065, the content of the image frame being displayed is evaluated and an optimized color mode is selected for the frame. In one embodiment, as described above, the optimized color frame is selected to improve display quality. In one embodiment, other factors such as battery level may be considered in selecting the display mode.

[0083] At block 1070, the process determines if the system is still displaying. If so, the process returns to block 1050 to continue monitoring temperature and providing smart adjustments. In one embodiment, only smart adjustments and temperature are shown here, but the other display factors discussed above apply as well. Once the display is finished, the process ends at block 1075.

[0084] Of course, while the above figures are illustrated as flow charts, in one embodiment, the order of operations is not constrained to the order depicted, so long as the processes are not dependent on one another. Additionally, in one embodiment, the system may be implemented as an interrupt driven system, so the system does not check for occurrences, but rather sends notifications for occurrences to trigger actions.

[0085] 11 is a block diagram of one embodiment of a special purpose computer system. However, it will be apparent to one skilled in the art that other alternatives of various system architectures may also be used. Additionally, the system may have some features implemented using a distributed cloud system.

[0086] 11 includes a bus or other internal communication means 1140 for communicating information, and a processing unit 1110 coupled to the bus 1140 for processing information. The processing unit 1110 may be a Central Processing Unit (CPU), a Digital Signal Processor (DSP), a Graphics Processor (GPU), or any combination of the above elements, or another type of processing unit 1110.

[0087] The system further includes memory 1120, which in one embodiment may be a Random Access Memory (RAM) or other storage device 1120 coupled to bus 1140 for storing information and instructions executed by processor 1110. Memory 1120 may also be used for storing temporary variables or other intermediate information during execution of instructions by processing unit 1110. The system also includes a Read Only Memory (ROM) 1150 and / or static storage device 1150, which in one embodiment is coupled to bus 1140, for storing static information and instructions for processor 1110. In one embodiment, the system also includes a data storage device 1130 coupled to bus 1140 for storing information and instructions.

[0088] In some embodiments, the system may further be coupled to output device(s) 1170, such as a computer display, speakers, or other output mechanism. Input device(s) 1175 may be coupled to the bus 1160. The input device(s) 1175 may be an alphanumeric input device, such as a keyboard, cursor control device 1180, touch screen, or other input mechanism.

[0089] Another device that may optionally be coupled to the computer system 1100 is a network device 1185 for accessing other nodes in a distributed system via a network.

[0090] It should be noted that any or all of the system components and associated hardware illustrated in FIG. 11 may be used in various embodiments of the present invention.

[0091] Those skilled in the art will appreciate that a particular machine implementing the present invention may be configured in a variety of ways according to the particular implementation. The control logic or software implementing the present invention may be stored in main memory 1120, mass storage device 1130, or other storage medium accessible locally or remotely by processor 1110.

[0092] It will be apparent to one skilled in the art that the systems, methods, and processes described herein may be implemented as software stored in main memory 1120 or read-only memory 1150 and executed by processor 1110. This control logic or software may also reside on an article of manufacture including a computer readable medium in which computer readable program code is embodied and is readable by mass storage device 1130 to cause processor 1110 to operate according to the methods and teachings herein.

[0093] The present invention may also be embodied in a special purpose device that includes a subset of the computer hardware components described above, such as a head mounted display (HMD) or head up display in a vehicle or other environment. For example, such a device may include a processing unit 1110, a data storage device 1130, a bus 1140, and a memory 1120, and may not include input / output mechanisms or may include only rudimentary communication mechanisms, such as a small touch screen that allows a user to communicate with the device in a basic way. In general, the more specialized the device, the fewer elements that need to be present for the device to function. In some devices, communication with the user may be through a touch-based screen or similar mechanisms. In one embodiment, the device may not provide any direct input / output signals, but may be configured and accessed through a website or other network-based connection via network device 1185.

[0094] It will be understood by those skilled in the art that any configuration of a particular machine implemented as a computer system may be used according to the particular implementation. The control logic or software implementing the present invention may be stored on a machine-readable medium accessible locally or remotely by the processor 1110. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In one embodiment, the control logic may be implemented as a transmittable data such as an electrical, optical, acoustic, or other form of propagated signal (e.g., carrier wave, infrared signal, digital signal, etc.).

[0095] Additionally, the system may, in one embodiment, be implemented on a distributed computing system. In a distributed computing system, processing may occur on one or more remote computer systems. The system may use computer system 1100 to provide local processing and further utilize one or more remote systems for storage and / or processing. In one embodiment, the system may further utilize distributed computing. In one embodiment, computer system 1100 may represent a client and / or server computer on which the software executes. Other configurations of processing systems for performing the processes described herein may be utilized without departing from the scope of the present disclosure.

[0096] In the foregoing specification, the mode selection system has been described with reference to specific exemplary embodiments thereof. It will, however, be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims. The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A multi-mode display, comprising: a mode selector for selecting one of a plurality of modes, each of the modes having a different light configuration, one mode including a reduced color space mode; one or more light sources controlled by the mode selector, the one or more light sources being used to display content to a user using the multi-mode display; and A multi-mode display with

2. The multi-mode display of claim 1 , wherein the mode selector comprises a user interface.

3. The mode selector includes: an ambient light sensor for determining an ambient light level; The multi-mode display of claim 1 , further comprising: an environmental background analyzer for determining the light configuration based on the ambient light level.

4. The mode selector includes: a thermal sensor for determining a temperature of the display system; The multi-mode display of claim 1 , further comprising: the mode selector for selecting the reduced color space mode when the temperature is above a threshold.

5. The mode selector includes: a battery sensor for determining a power level of the battery powered display; 2. The multi-mode display of claim 1, further comprising the mode selector for selecting a reduced color space when the power level falls below a threshold.

6. The mode selector includes: A clock that determines the time; The multi-mode display of claim 1 , further comprising the mode selector for selecting a color space to preserve scotopic vision.

7. The mode selector includes:

10. The multi-mode display of claim 1, comprising a display data analyzer for determining colors of a real-world scene in which an image is to be displayed and for selecting the mode based on the colors of the real-world scene.

8. 10. The multi-mode display of claim 1, wherein a first mode is a color sequential mode that sequentially enables red, green, and blue light sources during a frame, and a second mode is an alternate mode.

9. 9. The multi-mode display of claim 8, wherein the alternative mode is a non-color sequential mode that includes all three of the red, green, and blue light sources being turned on simultaneously to produce a white image.

10. The multi-mode display of claim 8 , wherein the alternative mode is a non-color sequential mode that includes using a fourth light source.

11. The multi-mode display of claim 8 , wherein the alternative mode is a non-color sequential mode that includes using one of the red, green, and blue light sources.

12. The multi-mode display of claim 8 , wherein the alternative modes include a color-sequential mode that uses a reduced set of colors.

13. A multi-mode display as claimed in claim 1 , wherein the modes have different frame rates.

14. The multi-mode display of claim 1 , wherein the mode is selected based on the content being presented.

15. A multi-mode display as claimed in claim 1 , in which the mode is selected on a frame-by-frame basis.

16. 10. The multi-mode display of claim 1 , wherein the one or more light sources further comprise a light array including a plurality of LEDs, the light array configured to selectively illuminate a portion of the display such that a first portion of the display has a different color spectrum than another portion of the display.

17. 1. A multi-mode display for augmented reality, the multi-mode display comprising: a mode selection system for selecting a color sequence based on data from one or more sensors, the color sequence including a full spectrum display sequence and a reduced color space display sequence; an illumination source providing a plurality of light sources, the illumination source being controlled by an output of the mode selection system; a combiner that combines light from the multiple light sources; a spatial light modulator that receives the combined light from the combiner, the spatial light modulator modulating the light; and an output section for outputting the modulated light to a user; A multi-mode display with

18. The multi-mode display of claim 17 , wherein the one or more sensors comprise one or more of an ambient light sensor, a heat sensor, an augmented reality background sensor, and a battery sensor.

19. A multi-mode display as claimed in claim 17, in which the mode is selected on a frame-by-frame basis.

20. 1. A method of providing a multi-mode display, comprising the steps of: selecting a display mode from a plurality of display modes, each of the display modes having a different light configuration, one mode including a reduced color space mode; controlling an output of one or more light sources based on the selected display mode to display content to a user using the multi-mode display; A method comprising:

Images