Method for driving a projection system and projection system

Abstract

A method for driving a projection system is described. The method includes: receiving an input image; generating a target specular image and a target image from the input image; generating a phase pattern for driving a phase modulator from a target illumination profile; generating a predicted illumination profile from the target illumination profile; generating an amplitude pattern for driving an amplitude modulator from the predicted illumination profile and the target image; and providing feedback from at least one image sensor to generate at least one of the phase pattern, the predicted illumination profile, and the amplitude pattern. One of the at least one image sensor includes an addressable region and / or an active region, the addressable region being configured to provide real-time calibration data, and the active region being configured to provide periodic calibration data, with the real-time calibration data and / or periodic calibration data further used as input when generating the predicted illumination profile.

Description

[0001] This application is a divisional application of the application filed on May 7, 2019, with application number 201980041447.4, entitled "Method for Driving a Projection System and Projection System". Technical Field

[0002] This invention relates to the field of light-directing projectors, and to methods for controlling such projectors and software for implementing such methods. Background Technology

[0003] Current projectors are typically based on so-called "light valve technology." A light valve is an amplitude spatial modulator. The entire light valve is uniformly illuminated, and the light valve (LCD, DLP, or LCOS) blocks the light (e.g., redirecting the light to the bleeder), that is, blocking unwanted light. This has two main drawbacks:

[0004] 1 / For ordinary video signals, a large amount of light energy is wasted due to leakage.

[0005] 2 / Its ability to block light is not ideal, and there is still a small amount of light leakage even when the image should be black. This reduces contrast, i.e., dynamic range.

[0006] Currently, displays (including projection displays) are expected to produce higher dynamic range (HDR). This means darker black levels and higher peak brightness levels. This will achieve more black detail and a more appropriate representation of image highlights. However, it is not intended to greatly increase the average image brightness, as this would primarily force the eyes to adapt to different levels, and might even be painful, without benefiting the perceived dynamic range.

[0007] Typically, increasing peak brightness also increases black levels, which is highly undesirable because more information is encoded near black. US5978142 proposes a cascade of two optical valves. While this approach is effective in reducing light leakage in black, it also significantly impacts luminous flux efficiency because losses in the first optical valve, imaging optics, mirrors, etc., can easily reduce peak brightness by 50%.

[0008] Furthermore, in typical high dynamic range signals, the ratio between peak brightness and average brightness becomes even greater, thus even more light energy will be blocked.

[0009] A much more efficient approach for HDR projectors is as follows: only the second modulator is a light valve type, and the first modulator distributes light to the desired location. The light distributed or redirected by the first modulator is redirected light. For the same amount of illumination light input, this solution delivers both better blacks and higher peak whites.

[0010] This approach, in which the first modulator is based on a phase-modulated LCOS device, has been proposed in WO2015054797.

[0011] In the following paper by Hoskinson and Stoeber, a method for using analog MEMS devices as a first modulator is described: High-dynamic range image projection using an auxiliary MEMS mirror array, published by the Optical Society of America in 2008.

[0012] MTT Innovations proposes a method for predicting the brightness pattern generated on a second modulator via simulation, starting from the phase pattern on the first modulator, in the following paper: High Brightness HDR Projection Using Dynamic Freeform Lensing, published in ACM Transactions on Graphics, June 2016. This model further introduces smoothness using a blur kernel to account for the limited image sharpness due to the beam quality limitations of the laser source and the additional blur introduced by the diffuser at the intermediate image plane. This paper will be referred to as Damberg and Gregson (ACM, 2016) as it will be cited throughout this description.

[0013] Geometric distortions introduced by the setup between the phase modulator and amplitude modulator and by the optics can be compensated by introducing a distortion function to warp the image plane backward toward the lens plane.

[0014] In the cases of LCOS phase modulators and MEM optical steering devices, analog first modulators are used, where the response to a particular drive signal depends on many influencing factors, such as the device's manufacturing tolerances, operating temperature, aging, etc. Deformable mirrors and MEM additionally exhibit some degree of hysteresis. The effect of a new drive signal applied depends to some extent on its previous condition. For mirrors with continuous films, the position of adjacent elements also has an effect.

[0015] Applications of light steering require very high deflection angle accuracy because even small angular deviations can introduce significant errors in the brightness of a single pixel on the screen. Furthermore, if three separate channels are used for each color, this will lead to even more noticeable color errors.

[0016] Furthermore, the final result of light redirection depends not only on the redirection amount of the first modulator, but also on the spatial and angular distribution of the (laser) light incident on the first modulator. If these illumination characteristics change, the redirected light pattern will alter. In the case where a phase modulator is used as the light redirection device, variations in the laser wavelength will also affect the final result.

[0017] If multiple discrete laser sources are combined to illuminate the light redirection, the differential aging between these sources may also affect the light redirection pattern.

[0018] Furthermore, any drift in the optical path between the first and second modulators will produce incorrect assumptions about the exact location where the light-directing pattern is imaged on the second modulator.

[0019] As further described in the paper Damberg and Gregson (ACM, 2016), a forward image forming model is used to predict the illumination profile present at the second, amplitude-only modulator via simulation. Given the phase function of the free-lens algorithm, the simple model described in that paper is used to predict the light distribution on the image plane. The smoothness introduced at the diffuser in the intermediate image plane can be modeled using a blur kernel (e.g., a systematic point spread function that can be directly measured or computed via deconvolution over a known target), and the modulation pattern required by the amplitude modulator is subsequently obtained to introduce any missing spatial information and, if necessary, additional contrast.

[0020] The aforementioned paper further describes the use of a one-time calibration and characterization of the entire optical system, which is required for optimal control of the spatial light modulator.

[0021] A single calibration is unlikely to be sufficient and appropriate to compensate for all the aforementioned effects. Frequent recalibration is expected for critical applications. Further calibration at different temperature points and different laser power levels may be necessary.

[0022] Even if all light is attempted to be redirected outside the active region, the portion that ultimately remains within it is not entirely static. It has been found that some of this "undirected light" depends on the phase pattern used. A partial explanation for this behavior can be found in edge-field phenomena known in LCOS devices. Even though the center of each pixel is driven to deliver the desired phase delay, the resulting electric field at the transition between that pixel and its adjacent pixels is influenced by both pixels. Light incident on these transition regions cannot be properly redirected. However, it is difficult to predict the exact location where the light will ultimately be.

[0023] A similar problem arises when pixels must transition from phase A in the first frame to phase B in the second frame. A one-time calibration may produce the desired light steering pattern under static conditions at both phase A and phase B. During the transition period, the light may end up in an undesirable position. Furthermore, the precise transition time and what happens during it are virtually impossible to predict. One solution could be to blank the laser source during the transition period; however, this would affect the light output available for light steering.

[0024] Adaptive optics are already used in high-end telescopes, where deformable mirrors or MEMS devices are dynamically driven to compensate for aberrations imposed by the atmosphere. The mirrors dynamically correct the wavefront to deliver better images. Such devices are typically also equipped with wavefront sensors (sensing both amplitude and phase) and are driven in a closed-loop manner to compensate for effects such as hysteresis and cross-coupling of the MEMS pixels. The wavefront is adjusted in both amplitude and phase through an iterative process. However, this iterative solution is very slow and appears impractical for video applications. Again, a blanking laser will be needed until the desired results are obtained. Summary of the Invention

[0025] The object of this invention is to provide a method for driving a projector system, the projector system including a phase modulator and at least one amplitude modulator. The phase modulator can be configured to generate a specular image incident on the amplitude modulator. The projector system can include at least one image sensor configured to receive at least a portion of an illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator. The method may include the following steps:

[0026] 1) Receive input image,

[0027] 2) Generate the target highlight image, target image, and power control signal from the input image.

[0028] a) Generate a phase pattern from the target specular image to drive the phase modulator.

[0029] b) Generate a predicted illumination profile image from the target specular image.

[0030] c) Generate an amplitude pattern from the predicted illumination profile image and the target image to drive the amplitude modulator.

[0031] 3) Receive an image from an image sensor to provide feedback to at least one of the method steps 2), 2b), and 2c) for driving the projector system.

[0032] This has the advantage of improving the image or output signal. The image sensor is used to provide feedback to the controller of the projection system, thereby improving the specular highlights projected by the projection system. This has the advantage that all artifacts not considered in the target specular image (such as fixed textures, diffraction artifacts, and DC components of undirected illumination, any drift in optomechanics, phase modulators, and / or laser sources) are now incorporated into the feedback loop by means of images acquired in real time or near real time.

[0033] Standard cinema has an average brightness level of approximately 8% (information collected by Real-D). Approximately 92% of the light energy from a laser source is thus blocked.

[0034] For HDR, the goal is to significantly increase peak brightness levels to provide a realistic impression of sunlight reflections and light sources, without increasing average brightness levels. In this case, the laser source would have to become significantly more powerful, while the light valve would block an even higher percentage of the light. This is expensive and inefficient.

[0035] The light-direction approach is expected to be much more effective, and the additional costs of the light-direction modulator stage can be easily offset by savings in laser costs and power consumption, especially for high-brightness cinema projectors.

[0036] However, since the optical steering stage is essentially an analog stage, it is highly prone to errors. Systems with closed-loop actuation can compensate for these errors and achieve reliable and accurate image reproduction.

[0037] Light-directing projectors have the potential to extend the dynamic range of both black and white without requiring much additional laser power and therefore not significantly increasing costs. However, light-directing modulators are analog components (for MEMs and phase modulators), and the stability of the highlight pattern is a major concern. Distortion of the position or amplitude of the highlight pattern will produce significant image artifacts. This invention can mitigate those artifacts.

[0038] Additionally or alternatively, at least one image sensor may operate at a multiple of the frame rate of the input image to provide subframe resolution.

[0039] The advantage is that the frame acquired by the image sensor during frame n of the input image is available for any block diagram during the display of frame n, and thus, the feedback calculated or measured from the sensor image of frame n can be applied during frame n, or before the end of the frame, or in the next frame n+1.

[0040] Additionally or alternatively, the projector may include a second amplitude modulator, and the step of generating a target highlight image may include generating a target image and a power control signal from an input image that can generate a target base image.

[0041] Additionally or alternatively, one of the at least one image sensor may include an addressable region and the at least one image sensor may include an active region, and the addressable region may be configured to provide a real-time calibration pattern and the active region may be configured to provide a periodic calibration pattern, and the step of generating a predicted backlight image from a target highlight image may further include the real-time calibration pattern and the periodic calibration pattern as input.

[0042] This has the advantage of providing a more detailed image or output signal, where the prediction is based on the actual implementation.

[0043] Advantageously, the addressable area is configured to generate a calibration pattern that provides information about the background level.

[0044] Using a background level has the advantages of providing information about the beam's steering components and also compensating for any drift in the optomechanics, phase modulator, and / or laser source.

[0045] Additionally or alternatively, one of the at least one image sensor may include an active region, and the image sensor may be configured to acquire a real-time backlight image within the active region, the real-time illumination profile image including content that changes slowly between frames n and n+1, and the step of generating a target highlight image, a target image, and a power control signal from the input image for frame n+1 may further utilize input from the slowly changing content in the real-time backlight image acquired during frame n of the input image.

[0046] This has the advantage of using the actual image from the previous frame.

[0047] Advantageously, the motion detection device is configured to determine which part of the input image frame is static and which part is dynamic between two consecutive frames.

[0048] Advantageously, an apparatus is provided for determining, on a pixel or region basis, which parts of an image are static or slowly changing, and which parts of an image are dynamic.

[0049] Advantageously, the predicted illumination profile is compared with the actual illumination profile in regions or pixels where the content is static in the input image:

[0050] When the actual illumination profile is higher than the predicted illumination profile, the target image attenuates according to the ratio of the actual illumination profile to the predicted illumination profile.

[0051] - When the actual illumination profile is lower than the predicted illumination profile but still sufficient, the target image is multiplied by the same ratio of the actual illumination profile to the predicted illumination profile.

[0052] - When the actual illumination profile is lower than the predicted illumination profile and is too weak, increase the target highlight to deliver sufficient brightness, while multiplying the target image by the same ratio of the actual illumination profile to the predicted illumination profile.

[0053] This allows corrections to be applied only to static areas of an image or areas with slowly changing content.

[0054] Additionally or alternatively, input from the predicted backlight image can be used in the step of generating the target highlight image, target image, and power control signal from the input image for frame n.

[0055] Alternatively, when driving according to a multiple of the amplitude modulator's frequency, the step of generating an amplitude pattern for driving the amplitude modulator from the predicted backlight image and the target image for frame n+1 can use the actual backlight image of the previous frame n or subframe as input. Additionally, a base pattern can be generated when the target base image is received.

[0056] This has the advantage of using real signals to improve the final output.

[0057] Alternatively, a portion of the light may be processed by a phase modulator to generate a highlight image on an amplitude modulator, while another portion of the light may be uniformly distributed on the amplitude modulator to provide a base image.

[0058] This has the advantage of generating two types of lighting from the same light source.

[0059] Additionally or alternatively, at least a portion of the illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator is a scaled-down version, optionally less than 1:1.

[0060] Preferably, the image sensor is placed in the path of those optical elements that generate the highlights.

[0061] Any discrepancies between the amplitude modulator image and the camera image are preferably mitigated through image processing.

[0062] The image sensor can be arranged, for example, to receive light reflected from an inserted glass plate, rather than placing the image sensor behind a folding mirror.

[0063] When the camera is used to monitor laser safety, it can operate at a higher frame rate and sample at a lower resolution.

[0064] Switches can be arranged to provide switching up to higher resolutions to perform calibration.

[0065] In another embodiment of the invention, a method is provided for monitoring the light level provided by at least one laser source in a projector system, the at least one laser source being driven by a power control signal. The projector system may include a phase modulator and at least one amplitude modulator, the phase modulator being configured to generate a highlight image incident on the amplitude modulator. The projector system may include at least one image sensor configured to receive at least a portion of an illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator. The image sensor may include at least one of an active region and an addressable region. The method may include the following steps:

[0066] • Compare the light intensity in at least one of the active region and the addressable region with a threshold.

[0067] • When the measured light intensity is higher than the threshold, a power control signal is used to reduce the power or turn off the laser source.

[0068] This has the advantage of providing the system with a safety device that is implemented to prevent viewers from being exposed to unsafe levels of laser light.

[0069] In another embodiment of the invention, a projector is provided, comprising: a phase modulator and at least one amplitude modulator, the phase modulator being configured to generate a highlight image incident on the amplitude modulator, the projector including at least one image sensor configured to receive at least a portion of an illumination pattern substantially equivalent to an illumination pattern incident on the amplitude modulator, the phase modulator, the amplitude modulator, and the image sensor being controlled by a driver, wherein the projector may further include a feedback loop for providing the output of the image sensor to the driver.

[0070] Additionally or alternatively, a phase modulator and an amplitude modulator can be arranged in the optical path; the phase modulator directs the beam to an intermediate image, and the image sensor and amplitude modulator can receive light from the intermediate image such that the optical path between the image sensor and the intermediate image can be (e.g., substantially) optically equivalent to the optical path between the spatial light amplitude modulator and the intermediate image. Furthermore, the directed beam can be directed to active and passive regions of the image sensor. Additionally, the directed beam can provide a calibration pattern to the passive regions. Furthermore, the intermediate image can be on a diffuser or on a moving diffuser.

[0071] Alternatively, a portion of the light may be processed by a phase modulator to generate a highlight image on an amplitude modulator, while another portion of the light may be uniformly distributed on the amplitude modulator to provide a base image.

[0072] Additionally or alternatively, the projector may include a second amplitude modulator configured to generate a base pattern of the second amplitude modulator.

[0073] Alternatively or additionally, the spatial light amplitude modulator may include at least one of a reflective spatial light amplitude modulator and a transmissive spatial light amplitude modulator. Furthermore, the spatial light amplitude modulator may include one of the following: a liquid crystal device or a plurality of micromirrors.

[0074] Alternatively or concurrently, the phase modulator may include one of the following: a deformable mirror, a MEMS, or an LCoS device.

[0075] Advantageously, the image sensor is a CMOS sensor or a CCD sensor.

[0076] Such sensors can be driven at a subframe of an amplitude modulator or an image sensor.

[0077] In addition, the illumination level can be 0-50% higher than the target image brightness level.

[0078] At least a portion of the illumination pattern that is substantially equivalent to the illumination pattern incident on the amplitude modulator is a scaled-down version, for example, the scaled-down version can be less than 1:1. This has the advantage of using smaller and cheaper sensors.

[0079] Image sensors can be placed in the path of those optical elements that generate highlights.

[0080] Any discrepancies between the amplitude modulator image and the camera image can be mitigated through image processing.

[0081] The image sensor can be arranged to receive light reflected from the inserted glass plate, rather than placing the image sensor behind the folding mirror.

[0082] When the camera is used to monitor laser safety, it can operate at a higher frame rate and sample at a lower resolution.

[0083] The switches can be configured to provide switching from lower frame rates to higher resolutions to perform calibration.

[0084] In another embodiment of the invention, a computer program product including software is provided, which, when executed on one or more processing engines, is capable of performing any of the methods described above. Additionally, the computer program product can be stored on a non-transient signal storage medium. Furthermore, a controller for a projector system according to the invention is provided.

[0085] The technical effects and advantages of embodiments of the present invention, after necessary modifications in details, correspond to the corresponding embodiments of the method according to the present invention. Attached Figure Description

[0086] These and other technical aspects and advantages of the embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, in which:

[0087] Figure 1 The flowchart for generating phase patterns, amplitude, and power control signals from an input image is explained.

[0088] Figure 2 The calibration image is shown.

[0089] Figure 3 A flowchart is shown for generating a predicted illumination profile using a calibrated image.

[0090] Figure 4 An optical layout of a projector according to an embodiment of the present invention is shown.

[0091] Figure 5 A sensor image acquired during operation of a projector is shown according to an embodiment of the present invention.

[0092] Figure 6 A flowchart of a feedback loop with real-time calibration according to an embodiment of the present invention is shown.

[0093] Figure 7 A flowchart of a feedback loop with real-time calibration according to an embodiment of the present invention is shown.

[0094] Figure 8 An example of a lighting profile across image rows in a predicted lighting profile and a target image is shown according to an embodiment of the present invention.

[0095] Figure 9 and Figure 8 Same, but Figure 9 The illumination profile in the actual illumination profile is further shown.

[0096] Figure 10 These are diagrams illustrating the method applied in an embodiment of the invention.

[0097] Figure 11 A flowchart with a feedback loop is shown according to an embodiment of the present invention.

[0098] Figure 12 A flowchart with two feedback loops is shown according to an embodiment of the present invention. Detailed Implementation

[0099] The invention will be described with reference to specific embodiments and particular drawings, but is not limited thereto by the claims alone. The described drawings are illustrative only and not restrictive. In the drawings, some elements may be enlarged and not drawn to scale for illustrative purposes. When the term "comprising" is used in this specification and claims, it does not exclude other elements or steps. Furthermore, the terms first, second, third, etc., in the specification and claims are used to distinguish between similar elements and are not necessarily used to describe order or chronological sequence. It should be understood that the terms thus used are interchangeable where appropriate, and embodiments of the invention described herein can be operated in a different order than that described or explained herein.

[0100] The terms “about” or “approximately” are synonyms and are used to indicate that the value modified by the term has an intelligible range associated with it, wherein the range may be +20%, +15%, +10%, +5%, or +1%, or alternatively ±20%, ±15%, ±10%, ±5%, or ±1%. The term “substantially” is used to indicate that a result (e.g., a measurement) is close to a target value, wherein close to may mean, for example, that the result is within 80%, 90%, 95%, or 99% of the value.

[0101] definition

[0102] Target base image: In a dual-projector setup, the target base image is the image that should be reproduced by the base projector. This signal is not only used in highlighter or hybrid projector setups.

[0103] Target image: This is the final image to be reproduced by a brightening projector or a hybrid projector. In a dual-projector setup, the target image is considered as a portion of the image without a target base image.

[0104] Highlight image: A highlight image is an image produced by a phase modulator that is incident on an amplitude modulator. A highlight image can be viewed as a detailed caustic image, which increases the brightness of the final image.

[0105] Light profile image: The light profile image is equivalent to the highlight image and can optionally add base lighting in the case of a hybrid projector.

[0106] Predicted illumination profile or predicted highlight: The predicted illumination profile corresponds to the simulated illumination profile present at the second amplitude modulator.

[0107] Target highlight or target illumination profile: The ideal illumination pattern generated by the phase modulator and incident on the amplitude modulator, assuming that the phase modulator is illuminated by a fully collimated laser beam, the phase modulator is free from optical distortion and parasitic reflections, and the diffuser is placed entirely in the intermediate image plane.

[0108] Phase modulator: A phase modulator is a device that introduces a phase change onto an incident wavefront. In the context of this invention, it produces a smooth, low-detail image on an amplitude modulator. Different techniques can be used to provide a phase modulator. These techniques include microelectromechanical (MEMS) displays, which offer very fast temporal response but lower spatial resolution. Deformable mirrors, for example, used in adaptive optics, can also be used. LCD displays, comprising liquid crystal on silicon (LCoS) devices, can also be used, offering the advantages of high spatial resolution, high speed, and high pixel fill factor. Liquid crystal displays used in transmission can also be used.

[0109] Basic Projector: A basic projector is a conventional projector with uniform illumination, which includes one amplitude modulator for each color. A single amplitude modulator can be used in conjunction with a sequence of color portions of the image. Alternatively, three amplitude modulators can be used in parallel, one for each color.

[0110] Brightening Projector: A brightening projector is a projector that incorporates a dual-modulation design, combining a phase modulator and an amplitude modulator for each color.

[0111] Hybrid projectors: A hybrid projector is a projector that combines the functionality of a brightening projector and a basic projector, while using only one amplitude modulator for each color. A portion of the light is processed by a phase modulator to produce highlight illumination on the amplitude modulator, while the other portion of the light is evenly distributed on the amplitude modulator.

[0112] Dual-projector setup: A setup that combines a base projector and a highlight projector to produce a combined image. The images can be combined onto a screen, or the light paths can be combined into a single projection lens. The term dual-projector setup is used when both the base image and the highlight image are processed by separate amplitude modulators.

[0113] Tone mapping: A method for compressing or expanding the dynamic range of an input signal to make it suitable for the dynamic range of a projector.

[0114] Appendix labels and function blocks: Figure 1 , 6 7, 11, and 12 illustrate workflows exposed as linked function blocks. Table 1 below discloses the reference numerals associated with these blocks, along with textual descriptions providing additional information about each reference numeral:

[0115] Table 1

[0116]

[0117]

[0118]

[0119]

[0120] Description of Explanatory Embodiments

[0121] One aspect of the invention is a dual-modulation projection system that provides a highlight image. This dual-modulation projection system can be further combined with a base projector to form a dual projector. Alternatively, in a hybrid projector, the highlight image and the base image are combined at the amplitude modulator level. Since the dual projector is illustrated with dashed lines in the flowchart, embodiments of the invention are first described with respect to the highlight projector and its combination with the base projector. Collimated laser light is received by a phase modulator. Light leaving the phase modulator in a given channel can be combined with light from the remaining channels and relayed through a diffuser to a prism structure, such as a Phillips prism. The Phillips prism splits the light into its constituent colors, each of which is modulated by a spatial amplitude modulator (such as a DMD or LCOS) mounted on the prism, recombines within the prism, and directs to the projection lens. The phase modulator introduces a phase change into the incident wavefront. The purpose of a phase modulator is to redistribute light from the input illumination to the target illumination profile (or target highlight) to produce a higher dynamic range and improved local peak illuminance than conventional projectors. This redistributes light from dark areas to bright areas, thus creating highlights. This results in economical use of available light. The target highlight is ideally chosen to approximate the upper envelope of the intensity of the target image.

[0122] Although the phase modulator redirects the incident wavefront to redistribute the light onto the amplitude modulator, even if all the light is redirected outside the active region, a portion of the light always remains unredirected and ends up in the projected image as background. In the first approximation, this can be modeled as a DC component added to the redirected portion of the light. This DC component of the unredirected light is problematic because it reduces the projector's contrast.

[0123] The phase modulator further introduces several artifacts, such as fixed texture, diffraction artifacts, and DC components of undirected illumination.

[0124] It is recommended to incorporate a diffuser placed in the center of the image when designing a brightening projector. Preferably, this diffuser is a moving diffuser.

[0125] The main purpose of a diffuser is to introduce angular diversity for speckle removal purposes and to broaden the beam emitted from the projection lens to achieve laser safety.

[0126] The diffuser can also be used to provide a smoothing effect by spatial averaging of the image, by placing the (optionally movable) diffuser in the intermediate image plane such that the image relayed from the diffuser to the spatial amplitude modulator (such as a DMD) is not a sharp image.

[0127] Various aspects of this invention can be used with any type of projector or projector system, including those with light steering capabilities. Thus, various aspects of this invention include dual-projector systems consisting of a base projector and a brightening projector, or hybrid projectors having a base projector and a brightening projector integrated into a single projector. The brightening projector can be a dual-phase / amplitude modulation projector.

[0128] Figure 1 A flowchart is shown to illustrate the steps required to control a brightening projector (and a dual-projector or hybrid projector system) and generate phase patterns, amplitude patterns, and power control signals for the light source from an input image expressed in RGB laser primary colors. Each rectangular block corresponds to a set of operations, and parallelograms indicate input and output operations. The set of operations within each block can be executed by a computer program product. These sets of operations or algorithms can run on a projector controller, for example, with a processing engine such as a microcontroller, ASIC, or FPGA. Solid arrows indicate required interactions between blocks, while dashed arrows indicate optional interactions between blocks. Solid blocks are required for every configuration, while dashed blocks are required only for dual-projector setups.

[0129] The input to this method is an input image expressed as a linear RGB laser primary color 100, which has been derived from the color-transformed and linearized input image. However, the invention is not limited to this, and other types of input images can be provided, for example, when using different types of light sources.

[0130] Image 100 is the input to content mapping block 110, which is responsible for determining the split between target highlight 115, target image 120, and optionally, target base image 160 for dual projectors, and power control signal 125. Target highlight 115 represents the illumination pattern that should be generated by the phase modulator on the surface of the amplitude modulator, and target image 120 represents the final image after the illumination pattern has been modulated by the amplitude modulator. Target base image 160 represents the image generated by the base projector for dual projector setups.

[0131] Content mapping block 110 is also responsible for remapping content that is infeasible given a system power budget and imaging parameters. The algorithm can, for example, first verify whether input image 100 is feasible given a system power budget. Content mapping block 110 thus generates target highlights 115, target image 120, and optionally target base image 160.

[0132] Therefore, in the content mapping block, the input image 100 is processed to determine the following:

[0133] 1. Target Image 120. When the input image 100 is within the system's capabilities, the target image will be identical to the input image 100. If the input image 100 exceeds the system's capabilities (e.g., due to peak brightness exceeding its capabilities or average brightness exceeding its capabilities), the image 100 will be tone-mapped to an adjusted target image that falls within the system's capabilities.

[0134] 2. (Various) Power control signal 125. The total laser power required, which results in a signal that can be used for real-time power modulation of the RGB laser light source. The use of this signal is optional; the laser can always be operated at full power, and either all light can be distributed across the image, or some excess light can be sent outside the active image area (e.g., to a beam vent). In the case of a dual-projector setup or a hybrid projector, there will be a separate base light source that can also be modulated in real time. Using both signals, both the base illumination intensity and the intensity of the highlights can be controlled in real time to match the image requirements. This not only reduces overall power consumption but can also be used to optimize contrast.

[0135] 3. Target Highlight 115. This refers to the distribution of laser power in a low-resolution illumination pattern. A condition for this target highlight image is that it provides a brightness level exceeding that of the target image 120 for each pixel in each color. The target highlight image can be the same for all three colors (white image) or different for all three colors.

[0136] 4. Target base image 160.

[0137] To form the target image 120 (i.e., the image generated by the phase modulator at the amplitude modulator level), the target highlight 115 is used as input to the phase pattern generation algorithm block. The phase pattern can be computed using algorithms such as those described in Damberg and Gregson (ACM, 2016). However, the invention is not limited thereto, and those skilled in the art will appreciate that other methods are also applicable. The output of the phase pattern generation algorithm block 130 is a phase pattern 135, which corresponds to the driving parameters required for light redistribution by the phase modulator. Ideally, when this phase pattern is applied to the phase modulator, it produces an illumination profile pattern that is exactly the same as the target highlight 115. However, this will not be the case for all the reasons listed above.

[0138] The target highlight 115 is also used as input to the forward model processing block 140.

[0139] As described in Damberg and Gregson (ACM, 2016), the algorithm also utilizes a forward image forming model to predict the illumination profile present at the second, amplitude-only modulator via simulation. The forward image forming model corresponds to... Figure 1 The forward model block in the flowchart.

[0140] Thus, the forward model processing block 140 generates a predicted illumination profile image, which is further used to calculate the amplitude pattern and the base pattern.

[0141] Predicted illumination profile images can be generated using system calibration data to provide better predictions of actual illumination profile patterns. Predicted illumination profile images can also be generated using simulations, but many effects will not be considered because they must be modeled.

[0142] System calibration data can be captured by an external camera in a one-time calibration procedure, characterizing how small light spots are blurred due to imperfect laser beam quality and optical system parameters. The so-called point spread function (PSF) can be characterized for different colors (lasers) and at different locations. Figure 2 Examples of PSFs captured at different locations for different colors are shown. The second part of the calibration is capturing an image of so-called undirected light, which in the first approximation is a fixed illumination pattern. The predicted illumination profile image ( Figure 3 145) is the sum of a blurred version of the target highlight image (305) plus a fixed pattern (310) of undirected light, such as Figure 3 As explained in the block diagram.

[0143] The forward model block also receives a power control signal. The predicted illumination profile image is multiplied by the power control signal of the specular laser source. In the case of a hybrid projector, the base illumination can also be predicted because it is a fixed, nearly uniform illumination, so the pattern can be easily captured during calibration and added to the predicted illumination profile image after multiplying the pattern by the power control signal of the base light source.

[0144] The predicted illumination profile image 145 (which is similar to the brightness profile generated by an LED matrix with local dimming capability using LCD backlighting) is combined with the target image 120 and used as input to the amplitude pattern generation block 150 to determine the amplitude signal necessary to drive the second modulator (an amplitude modulator for brightening or mixing projectors), i.e., the amplitude pattern 155. In the case where a base projector is also present, amplitude pattern generation also produces a base pattern. If the base illumination will be perfectly uniform, the base pattern can be identical to the target base image. However, when the base illumination uniformity is not ideal, color and brightness uniformity corrections can be applied to derive the base pattern from the target base image.

[0145] Optionally, the predicted illumination profile 145 from the forward modeling block 140 is used as a feedback signal to the content mapping block 110, illustrated by arrow 146 in the flowchart. This allows verification that the brightness of the predicted illumination profile effectively exceeds the brightness of the target image, so that the target image brightness can be effectively achieved using an appropriate amplitude signal for the second modulator. In cases where this is not the case, the content mapping block can then add the target specular image and / or laser power control signal.

[0146] One aspect of the invention is the refinement of a one-time calibration using feedback (e.g., an intermittent or continuous feedback mechanism of a projector). To achieve such an intermittent or continuous feedback mechanism, an image sensor is provided, for example, integrated into or potentially placed within the optical path of the projector. The image sensor receives an image (or a portion of an image) having an illumination pattern equivalent to the illumination pattern incident on an amplitude modulator. The illumination pattern may be about or adapted to the image (such as video) to be projected. Such images or videos are projected as frames.

[0147] The image sensor can be driven at the same frequency as the amplitude modulator or the input image, or at a multiple of the frequency of the amplitude modulator or the input image. Driving the image sensor at a higher speed has the advantage of increasing the speed of the feedback loop.

[0148] The image sensor, phase modulator, and at least one amplitude modulator can be driven by the projector controller. The projector controller can also further drive the light source, i.e., a laser source.

[0149] Images from an image sensor are analyzed using image analysis software or digital image processing techniques known in the art to retrieve desired information from the images, and the information is provided as feedback to a controller configured to execute a feedback loop according to the invention.

[0150] It is possible Figure 1 The flowchart enables intermittent or continuous feedback at any level, where images of the illumination pattern acquired by the image sensor can thereby improve the projector's driving scheme at different levels, as further explained, such as any of the following:

[0151] 1. A feedback mechanism for fine-tuning the parameters of the forward prediction model, i.e., as a refinement of the forward model 140 during a one-time calibration.

[0152] 2. A near real-time feedback mechanism for correcting target brightness in the next frame for semi-static content.

[0153] 3. Real-time feedback mechanism for adjusting the drive signal of the amplitude modulator.

[0154] 4. A combination of previous methods. For example, two or more of these methods 1 to 3, or a combination of all three methods 1 to 3.

[0155] An image sensor can, for example, be integrated into projector optics behind a highly reflective dichroic folding mirror. This dichroic folding mirror can be configured to reflect 99% to 99.5% of light. The 1% or 0.5% of light leaking through the mirror will be sufficient to be forwarded to and received by the image sensor. The optical path of the image sensor preferably has characteristics as similar as possible to those of the optical path toward the second spatial amplitude modulator, or in other words, the optical path of the image sensor is optically substantially equivalent (so that it has the same (or nearly the same) optical characteristics) to the optical path of the second spatial amplitude modulator. For example, if the second spatial amplitude modulator is a reflective spatial amplitude modulator (such as a DMD device), it typically forms an angle relative to the optical axis of the incident beam. In this case, the image sensor preferably also forms a similar angle relative to the optical axis. Spatial amplitude modulators (such as DMD optical systems) typically use TIR prisms, where the prism shape is optimized to minimize path length differences caused by positioning the spatial amplitude modulator (such as the DMD) at an angle relative to the optical axis. A similar situation can be achieved by introducing a dummy prism in the optical path toward the image sensor. If the image sensor has a different size than the reflective spatial modulator (such as a DMD device), the magnification of the imaging optics toward the sensor is preferably different from the magnification of the imaging optics toward the spatial amplitude modulator (such as the DMD).

[0156] Alternatively, in order to optically attempt to replicate the optical path characteristics from the intermediate image to the second spatial amplitude modulator (such as a DMD), the image sensing optics can be designed to capture only the intermediate image and apply electronic corrections to account for geometric distortion and blurring caused by the rotation of the second spatial amplitude modulator relative to the optical axis.

[0157] Alternatively or additionally, the image from a spatial amplitude modulator (such as a DMD) can be a scaled-down version. This can be less than 1:1, as this offers the advantage of reducing the cost of the image sensor.

[0158] Alternatively or additionally, the image sensor can be placed in the path of those optical elements that generate the highlights. This can provide the advantage of better utilization of available space. In the case of adding a base illumination to the highlight illumination, this base illumination is characterized in the calibration step and is considered constant over time.

[0159] Alternatively or additionally, any discrepancies between the amplitude modulator image (e.g., a DMD image) and the camera image can be mitigated by image processing (such as, for example, image distortion, flat field correction, and / or position-dependent blurring). In this case, such image processing may increase latency, and true closed-loop drive may no longer be possible. Even so, the image sensor and image processing can still be used for calibration, static image feedback, or for laser safety detection.

[0160] Alternatively or additionally, the image sensor can receive light reflected from the inserted glass plate, instead of placing the image sensor behind the folding mirror. This offers the advantage of better utilization of available space.

[0161] Alternatively or additionally, the camera can operate at a high frame rate (e.g., 1400 Hz). For example, when the camera is used to monitor laser safety, sampling can be performed at a low resolution (e.g., 40×21). For example, whenever it is permissible not to use a projector and the amplitude modulator can be set to black (such as the time between the end of one movie and the start of another), a switch from a lower frame rate to a higher resolution can be provided to perform calibration, provided that time permits.

[0162] It is important to note that features providing fundamental optical equivalence can be provided either optically or electronically. Suboptimal optical equivalence can be corrected electronically. This requires relaying the intermediate image to both the second modulator and the sensor, so that most of the light ends up on the second modulator and only a small portion on the sensor.

[0163] The resolution of the image in the intermediate image plane (in the phase modulator plane) is very limited. The point spread function will span several hundred to several hundred pixels across the second modulator (amplitude modulator). The image sensor can therefore capture images at a resolution far lower than the original resolution of the second modulator because the low-resolution image from the sensor can be upsampled with good accuracy.

[0164] Figure 4 This is a schematic representation of a portion of the optical path of a brightening projector according to an embodiment of the present invention. A light source (such as a laser light source (not shown)) provides a beam 1 incident on a phase modulator 2. In this embodiment, the phase modulator 2 is a reflective phase modulator, but it could be a transmissive phase modulator. Illumination and sensing imaging optics 4 are provided along the optical path after diffuser 3. A mirror 5 reflects an intermediate image onto a prism structure 10 after illumination imaging optics 11. From amplitude modulator 9 through prism structure 10, the final image is projected through a projection lens (not shown). A small amount of light also passes through mirror 5 and falls on sensing and imaging optics 6, a dummy prism 8, and an image sensor 7. This small amount of light could, for example, be provided by a semi-silvered mirror 5.

[0165] In embodiments including dual projectors (i.e., a brightening and a base projector), the images generated by the two projectors are superimposed on the projection screen. In embodiments including a hybrid projector, for each color, the brightening and base beams are superimposed upstream of the amplitude modulator 9.

[0166] Various embodiments of various aspects of the present invention are thus described.

[0167] 1. Feedback mechanism for fine-tuning the parameters of the forward model 140.

[0168] The maximum steering angle from phase modulator 2 (or virtually any such MEM device) will typically be the same in both the horizontal and vertical directions. When the width of the final image is greater than its height (as is often the case in modern 16:9 aspect ratio displays or cinema formats), and if the system is designed such that light can be redirected by the phase modulator across the entire width of the intermediate image 3, but outside the active area representing the display portion of the intermediate image 3, it is possible to redirect the light beyond the top and bottom edges of the active area, for example, beyond the edges of the active area on the intermediate image 3 that represents the height of the image to be displayed.

[0169] The addressable area 17, outside the active area 18, can be used to generate calibration patterns (e.g., in the factory or during normal operation of the projector) Figure 5The upper and / or lower calibration patterns 15 and 16 are shown respectively. This will allow for intermittent or continuous refinement of the forward model 140, thereby compensating for any drift in the optomechanics, phase modulator, and / or laser source.

[0170] The level in the background of the calibration pattern can be used as an indicator of the amount of undirected light. This information can then be coupled back to the forward model 140 and used to calculate the next predicted illumination profile in cases where the content is primarily static.

[0171] Generating those calibration patterns outside the active region will require additional laser energy, but calibration can only be activated if and only if the image content does not require all available laser energy. In this case, if the laser source cannot tolerate rapid dimming, turning outside the active region may be the only solution to dissipate excess light.

[0172] In this embodiment, the optical path and mechanics must be designed to image the portion of the phase modulator corresponding to the entire addressable area onto the image sensor, onto the moving diffuser, and from the moving diffuser onto the image sensor. However, in the imaging path toward the amplitude modulator, it would be preferable to block light from the active area (e.g., using a cooled aperture) before it reaches the amplitude modulator and potentially causes unwanted heating and stray light.

[0173] The image sensor can be a panchromatic sensor (e.g., with a Bayer filter) or a monochrome sensor. In the latter case, a test pattern for calibration can be presented sequentially for each primary color.

[0174] In this embodiment, a calibration procedure using the active region 18 can be performed during projector startup or shutdown, or more generally, when an image does not need to be formed on the projection screen. This calibration is called daily calibration (but it can also be performed only when the system requires it or on a periodic basis).

[0175] Calibration data using addressable regions 15 and 16 can be executed during projection or in real time.

[0176] Figure 6 The flowchart in the document contains functional blocks – see Table 1. For example... Figure 6 As explained in the flowchart, forward model block 140 can now consider multiple calibration inputs:

[0177] - One-time calibration data captured from the projection screen using an external camera during setup. This one-time calibration data provides actual on-screen PSF data for multiple colors from multiple locations, as well as a luminance profile from undirected light, as previously described. Optionally, in a dual-projector setup or in the case of a hybrid projector setup, this one-time calibration data also provides a lighting profile of the base illumination.

[0178] - The same information is captured using the internal image sensor 7 when the projector is turned on or off. During a one-time calibration, the relationship between this signal and the signal from an external sensor is established to characterize how the optical system of the second modulator stage introduces some additional blurring and some attenuation of corner intensity into the PSF through optical vignetting. Daily automatic calibration cycles allow compensation for any drift in the intensity balance between the PSF size or position or between the RGB laser sources. When using... Figure 3 In the case of the flowchart, it would be advantageous to use the daily calibration data, which now includes appropriate compensation for the optical behavior of the second modulator, to deliver the predicted illumination profile pattern 145. Thus, in Figure 3 In the flowchart, the applied PSF 300 and fixed pattern 310 are now equipped with periodic calibration images acquired using image sensor 7.

[0179] - Information captured by image sensor 7 outside the active area (i.e., in addressable areas 16, 17) during operation (sensor image 220 outside the active area).

[0180] This can be used to verify the stability of the PSF in terms of size and position, as well as the stability of the unsteering light components. Intensity variations of the unsteering light components outside the active region are used as multiplication factors for the daily captured unsteering light components. Systematic variations in the width or position of the PSF (e.g., a systematic right shift of the PSF pattern relative to other colors for one color, or a systematically wider PSF) will result in corrections to the daily captured PSF information in the active region (e.g., by applying appropriate offsets and appropriate additional blur to the captured daily PSF data). Thus, real-time calibration data acquired using the image sensor is used to update the daily calibration data 230 acquired using image sensor 7.

[0181] - Power control signal, as described above.

[0182] As a result, the forward model now receives (near) real-time calibration data from the internal image sensor 7 and can generate a more accurate predicted illumination profile 145, which can mitigate the slow and moderate drift of system parameters.

[0183] In another embodiment of the invention, the sensor can also be used to protect the maximum light level in the highlights, ensuring that the light level in front of the projection lens remains within desired limits to avoid exposing viewers to unsafe laser levels. The tolerable highlight level will depend on the installation (such as the type of projection lens used, the location where viewers might interfere with the projected beam, etc.). Therefore, the maximum light level needs to be calibrated during projector installation. Thus, a threshold can be established, for example, during projector installation. The content mapping block will already tone-map the content to keep it below the set limits. However, in cases where the algorithm malfunctions, causing too much light to concentrate in a particular location, the sensor can provide a second protection system.

[0184] If the sensor detects that the brightness level exceeds the set limit or threshold, it will shut down or completely turn off the laser sources via a power control signal.

[0185] 2. A near real-time feedback mechanism for correcting target brightness in the next frame for semi-static content.

[0186] In a second embodiment of the invention, the image captured by the image sensor 7 in the active region (sensor image 240 within the active region) (actual illumination profile image 250) is used as input to the content mapping block 110.

[0187] For static and slowly changing content, the brightness level of the image will change slowly, and it can be assumed that the brightness levels in two consecutive frames will be substantially the same. Therefore, it would be beneficial to correct the brightness error in the target highlight of frame n (e.g., the current frame) by adapting the brightness level of the target highlight for frame n+1 (the subsequent frame). The ratio between the predicted illumination profile brightness level and the actually measured illumination profile brightness level (from the actual illumination profile image, which can be processed to be linearized and normalized) is derived and used as a multiplication factor for the static content in the next frame n+1 of the illumination profile. A new drive signal is then calculated for the first modulator to attempt to achieve the corrected target brightness level in the illumination profile. This new input to the content mapping block 110 can also be used to calculate the target image in the next frame and the power control signal for the next frame.

[0188] Motion detection mechanisms can be further used to determine which parts of an image frame are static and which are dynamic, and to decide whether to use the original or corrected image on a per-pixel or region-by-region basis. A weighted average of these two inputs can be used to obtain a soft transition rather than a hard transition between adjacent pixels.

[0189] The image captured by image sensor 7 in the active region (actual illumination profile) (sensor image 240 in the active region) and the predicted illumination profile from the previous image frame can also be used as input to the content mapping block, along with the input image of the current frame. Figure 7 As shown in the block diagram. Figure 7 The flowchart in the document contains function blocks – see Table 1.

[0190] In image segments where the content is static (the input image of the current image frame is substantially the same as or identical to that of the previous image frame), the predicted illumination profile is compared with the actual illumination profile:

[0191] - When the actual illumination profile is higher than the predicted illumination profile, the target image is attenuated by the same factor (actual / predicted).

[0192] - When the actual illumination profile is lower than the predicted illumination profile but still sufficient, the target image is multiplied by the same factor (actual / predicted).

[0193] - When the actual illumination profile is lower than the predicted illumination profile and is no longer sufficient, increase the target highlights to deliver sufficient brightness, while simultaneously multiplying the target image (by the factor actual / predicted) to compensate for the expected difference between the recently predicted illumination profile and the actual illumination profile.

[0194] When the content is moving, motion vectors can be derived to select the corresponding image portion from the actual illumination profile image and the predicted illumination profile image of the previous frame, and the same correction mechanism described above can be applied to static content.

[0195] When the content is unrelated to the previous frame, the content mapping block 110 performs the standard algorithm, ignoring both the predicted and actual lighting profile information from the previous frame.

[0196] The main problem with the sensor is that it provides information about the frame too late. The illumination profile pattern needs to exist before the image sensor can begin sensing. Information is only available at the end of the image sensor frame. Therefore, the actual illumination profile information is only available one frame later. Image sensors can operate much faster than projectors (as will be discussed later in the manual). Figure 10 (As explained), the image sensor operates at subframe rate, and the first feedback is available before the end of the frame. Information from previous frames can also be used, but only if the content is static.

[0197] 3. Real-time feedback for adjusting the drive signal of the amplitude modulator

[0198] For this third embodiment of the present invention, the following method is used:

[0199] In this embodiment, it is assumed that the target does not reach a precise brightness level, but rather an illumination level higher than the target image brightness level (e.g., 0-50% higher). This assumption provides a certain tolerance for error. It is important to note that the illumination pattern delivered by light redirection will be scattered to some extent; this is partly due to the deviation of the illumination beam from the ideal collimated beam, and this can be (intentionally) amplified by moving diffusers (4 and 11) that are slightly offset from the focal plane of the relay optics preceding the amplitude modulator. The predicted illumination profile brightness level should always exceed the target image brightness level (and fine peaks), such as... Figure 8 As shown in the diagram, the predicted illumination profile already accounts for the blurring and efficiency of the diffuser. However, the amplitude modulator can only attenuate or reduce brightness. If the illumination profile cannot provide enough light even when the amplitude modulator is driven to its maximum extent, the target image cannot be reproduced.

[0200] As previously described, image sensor 7 receives a small portion of light that is optically substantially equivalent to the illumination pattern on the second modulator (i.e., the actual illumination profile image, e.g., by means of a high-speed CMOS or CCD camera sensor). In this configuration, image sensor 7 is preferably driven at a multiple of the frame rate of the amplitude modulator. As mentioned above, image sensor 7 can be positioned behind the translucent folding mirror in the optical path at an optically equivalent location to the position of the second modulator.

[0201] The actual measured illumination pattern, obtained using the actual illumination profile image acquired by image sensor 7, is used after linearization and normalization to determine / adapt the drive signal for the second modulator (amplitude modulator). This does not necessarily mean that the second modulator can only be addressed after the illumination pattern is known. For example, in the case of a PWM-addressable second modulator (e.g., a spatial amplitude modulator such as a DMD), the PWM scheme may begin by following the illumination estimated from the forward model 140 (predicted illumination profile 145) or assuming that the target illumination level (target highlight) will perfectly match. When the actual measured illumination pattern (actual illumination profile) is known later in the frame, the residual PWM scheme can be corrected to compensate for the difference. In cases where the actual measured illumination is higher than the target illumination, the residual PWM duty cycle can be reduced to compensate for the increased illumination in the next subframe and the excess illumination delivered during the first subframe(s). If the actual measured illumination is lower than the target illumination, the residual PWM duty cycle can be increased to compensate for the reduced illumination in the next subframe and the missing brightness during the first subframe(s).

[0202] Figure 9The pixel brightness profile across pixel rows is shown, where the x-axis is the horizontal position of the pixel. The solid line 8000 indicates the final target image brightness delivered after the second modulator. The dashed line 8100 indicates the predicted illumination profile signal, while the dotted-dash line 9000 indicates the actual illumination profile signal as derived using image sensor 7.

[0203] During the first subframe(s) with a duration ti, information from image sensor 7 is not yet available. The drive signal (amplitude pattern) for the second modulator is now calculated as follows:

[0204]

[0205] During the remaining time of a frame with duration t frame–ti, the actual illumination profile, as measured by the image sensor (after normalization to the maximum white level), can be used, and correction terms are applied to adjust for errors generated during the initial subframes.

[0206] Ampl2 (amplitude 2)

[0207]

[0208] All of these calculations should be performed in linear optical space (e.g., using gamma correction or any other known transfer function).

[0209] Figure 10 An example is explained where the camera (i.e., image sensor 7) operates at three times the frame period of the video signal. The camera output signal is delayed by up to one-third of the video frame period. A corrected PWM scheme can only be calculated so that it can be applied to the next subframe if the camera output is available. During the first two subframes, the spatial amplitude modulator (such as a DMD) is driven according to the initial PWM scheme. Only in the third subframe can the corrected PWM scheme, taking into account the actual brightness level acquired by the camera in subframe 1.1, be applied.

[0210] In practice, and especially when using slow modulators (such as LCOS phase modulators) to generate beam steering, brightness can vary over the duration of a frame. If the content is dynamic, the information from the last subframe 1.3 is useless. However, for static content, it would be beneficial to use the actual brightness levels acquired by the camera across all subframes. Brightness deviations that may occur during the frame 1 process can be compensated for during the frame 2 process.

[0211] In the case of static content, the driving signals for the initial subframes can be calculated as follows:

[0212]

[0213] The generation of the drive signal toward the spatial amplitude modulator (such as a DMD) thus takes into account, for example, the illumination level predicted by the forward model 140 or according to the target illumination level (further referred to as the predicted illumination level), the actual illumination level measured by the high-speed image sensor, the current target image, and the target image from the previous frame. If the current target image is the same as the target image from the previous frame, it will start the frame by considering the ratio between the measured illumination level and the target image. If the target image is different from the previous frame, it will start the frame by considering the ratio between the predicted illumination level and the target image. Once the measured illumination level of the new frame is available, the measured illumination level data will be used, and any errors generated in the previous portion of the frame due to the difference between the predicted illumination level and the measured illumination level will be corrected.

[0214] The image sensor should independently measure the intensity of the three primary colors, and in this case, it should be a panchromatic sensor (e.g., with a Bayer filter).

[0215] like Figure 11 As explained in the flowchart, in this algorithm, the actual illumination profile image captured by the image sensor is now used as an additional input to the amplitude pattern generation block, following the predicted illumination profile signal. Figure 11 The flowchart in the document contains functional blocks - see Table 1.

[0216] To calculate the initial PWM frame, the amplitude signal is determined from the predicted illumination profile signal and the target image when the actual illumination profile signal is not yet available (available information still comes from previous frames).

[0217] To calculate the corrected PWM frame, the actual illumination profile signal is used as a basis, and a correction is applied to compensate for the difference between the actual illumination profile and the predicted illumination profile in the initial PWM frame. This latter compensation is weighted based on the relative time duration between the initial frame and the corrected frames(s).

[0218] Provided the measured illumination is high enough to deliver the target image brightness and compensate for any under-brightness caused by erroneous predictions used in previous subframes, the final result should be a perfect reproduction of the target image. However, this is contingent on the sensor measuring an accurate representation of the brightness level at the spatial amplitude modulator (such as a DMD). Furthermore, a one-time calibration of uniformity, geometry, and color sensitivity will be required here. However, this calibration is expected to be far more stable than the specular beam generation itself.

[0219] Using this method, finite drift of optomechanical systems, phase modulators, or laser sources over time and temperature can be mitigated. Furthermore, variations in undirected components and temporal effects due to the response speed of the phase modulator can be considered on a frame-by-frame basis.

[0220] In the case of a hybrid projector where both highlight and base lighting are combined on the same amplitude modulator, sensor 7 receives this same combination (as a fixed portion of the total illumination directed towards the amplitude modulator). In this case, the target image is defined as a complete image, i.e., a combination of the target highlight and the target base. Subsequently, the method for deriving the amplitude pattern generation is the same as the method for brightening the projector.

[0221] 4. A hybrid combination of two previous methods

[0222] While previous methods could address various types of drift, handle small variables in non-steering light components, and the effects of transitions between frames, they could not accommodate large drifts in steering position or intensity.

[0223] Hybrid models can be used to combine the advantages of previous methods, such as Figure 12 As explained in the flowchart. Figure 12 The flowchart in the document contains function blocks – see Table 1.

[0224] Here, slow but potentially large drift can be addressed through near real-time feedback of the forward model parameters. Furthermore, during projector shutdown, those parameters of the forward model can be further updated (e.g., on a daily basis) using a set of calibration patterns in the active region. For static content, any residual errors can be compensated for in the next image frame. Additionally, final correction (including correction for dynamic effects) can be achieved using real-time feedback of the amplitude modulator's drive signal.

[0225] A single image sensor can be used to capture both the active region and the test pattern generated outside the active region. In other embodiments, a separate image sensor can acquire different portions of the beam incident on the phase modulator, for example, an image sensor imaging information in the active region (or the pupil of the projection lens) and an image sensor imaging information outside the active region on the diffuser plane.

[0226] In other embodiments, the image sensor may also be implemented to image the beam after it has been reflected or transmitted by an amplitude modulator.

[0227] In embodiments of the invention, the projector or projection system may include a phase modulator and an amplitude modulator in each color channel. Embodiments of the invention can also be implemented using a projector or projection system that includes a color-sequentially operating projector with one phase modulator and one amplitude modulator; however, current LCOS phase modulators are not fast enough. In other embodiments of the invention, the projector or projection system may include a phase modulator for each channel and a single spatial amplitude modulator, such as a DMD amplitude modulator, operating in a color-sequential mode. In both color-sequential modes, sequential pulse modulation of the red, green, and blue lasers would then be required, which may be disadvantageous in terms of optical efficiency.

[0228] The methods according to the invention can be executed by control units, such as control units or processing devices, or any control unit (including microcontrollers) used in conjunction with embodiments of the invention, which can be as independent devices or embedded in a projector or as part of the projector's optical subsystem. The invention can use processing engines adapted to perform various functions. The processing engine preferably has processing capabilities such as those provided by one or more microprocessors, FPGAs, or central processing units (CPUs) and / or graphics processing units (GPUs), and is adapted to perform the corresponding functions by being programmed with software (i.e., one or more computer programs). References to software can encompass any type of program written in any language (via compiled or interpreted languages) that can be executed directly or indirectly by a processor. Implementations of any method of the invention can be performed by logic circuits, electronic hardware, processors, or circuits that can encompass any kind of logic or analog circuitry, integrated to any degree, and are not limited to general-purpose processors, digital signal processors, ASICs, FPGAs, discrete components, or transistor logic gates, etc.

[0229] Such control units or processing devices may have memory (such as non-transient computer-readable media, RAM and / or ROM), an operating system, optional displays (such as fixed-format displays), ports for data input devices (such as keyboards), pointing devices (such as "mouse"), serial or parallel ports for communicating with other devices, network cards for connecting to any network, and connections.

[0230] The software can be embodied in a computer program product adapted to perform the functions of any of the methods of the present invention listed below when the software is loaded onto a controller and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.). Therefore, a processing device control unit associated with any of the embodiments of the present invention can be incorporated into a computer system capable of running one or more computer applications in the form of computer software.

[0231] The methods described above with reference to embodiments of the present invention can be executed by one or more computer applications running on a computer system, the computer applications being loaded into memory and running on a system such as Windows provided by Microsoft Corporation. TM The computer system runs on or in association with operating systems such as Linux and Android. The computer system may include main memory, preferably random access memory (RAM), and may also include non-transient hard disk drives and / or removable non-transient memory and / or non-transient solid-state memory. Non-transient removable memory may be optical discs such as CD-ROMs or DVD-ROMs, or magnetic tapes, which can be read and written by a suitable reader. Removable non-transient memory may be a computer-readable medium in which computer software and / or data are stored. Non-volatile storage memory can be used to store persistent information that should not be lost even when the computer system is powered off. Applications can use the information and store it in non-volatile memory.

[0232] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0233] 1) Receive input image,

[0234] 2) Generate the target highlight image, target image, and power control signal from the input image.

[0235] a) Generate a phase pattern from the target specular image to drive the phase modulator.

[0236] b) Generate a predicted illumination profile image from the target specular image.

[0237] c) Generate an amplitude pattern from the predicted illumination profile image and the target image to drive the amplitude modulator.

[0238] 3) Receive an image from an image sensor to provide feedback to at least one of the method steps 2), 2b), and 2c) for driving the projector system.

[0239] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0240] - Generate the target highlight image, target image, power control signal, and target base image from the input image.

[0241] - Further, a predicted illumination profile image is generated from the target specular image by using a real-time calibration pattern from the addressable region of the image sensor as input and a periodic calibration pattern from the active region of the image sensor.

[0242] - Generate a calibration pattern in the addressable area.

[0243] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0244] - Generate a target highlight image, target image, and power control signal from the input image for frame n+1 by slowly changing content in a real-time illumination profile image acquired during frame n of the input image.

[0245] - Use a motion detection device, wherein the motion detection device is configured to determine which part of the input image frame is static and which part is dynamic between two consecutive frames.

[0246] - A device that uses pixels or regions to determine which parts of an image are static or slowly changing, and which parts of an image are dynamic.

[0247] - Compare the predicted illumination profile with the actual illumination profile in static regions or pixels of the input image:

[0248] When the actual illumination profile is higher than the predicted illumination profile, the target image attenuates according to the ratio of the actual illumination profile to the predicted illumination profile.

[0249] When the actual illumination profile is lower than the predicted illumination profile but still sufficient, the target image is multiplied by the same ratio of the actual illumination profile to the predicted illumination profile.

[0250] When the actual illumination profile is lower than the predicted illumination profile and is too weak, the target highlight is increased to deliver sufficient brightness, while the target image is multiplied by the same ratio of the actual illumination profile to the predicted illumination profile.

[0251] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0252] - For frame n, a target specular image, a target image, and a power control signal are generated from the input image, which are further used as input from the predicted illumination profile image.

[0253] - When driven by a multiple of the amplitude modulator, an amplitude pattern for driving the amplitude modulator is generated for frame n+1 from the predicted illumination profile image and the target image, using the actual illumination profile image of the previous frame n or subframe.

[0254] - Generate a base pattern when the target base image is received as additional input.

[0255] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0256] - A portion of the light processed by the phase modulator on the amplitude modulator is used to generate the highlight image, and another portion of the light, which is uniformly distributed on the amplitude modulator, is used to generate the base image.

[0257] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0258] - Monitor the light level provided by at least one laser source in the projector system.

[0259] - Compare the light intensity in at least one of the active region and the addressable region with a threshold.

[0260] - When the measured light intensity is higher than the threshold, a power control signal is used to reduce the power or turn off the laser source.

[0261] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0262] At least a portion of the illumination pattern that is substantially equivalent to the illumination pattern incident on the amplitude modulator may be a scaled-down version, for example, the scaled-down version may be less than 1:1.

[0263] An image sensor is used, positioned in the path of the optical elements that generate the highlights.

[0264] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0265] Any discrepancies between the amplitude modulator image and the camera image can be mitigated through image processing.

[0266] Instead of placing the image sensor behind the folding mirror, the image sensor can be arranged to receive light reflected from the inserted glass plate.

[0267] The software embodied in a computer program product is adapted to perform the following functions when the software is loaded onto one or more corresponding devices and executed on one or more processing engines (such as microprocessors, ASICs, FPGAs, etc.):

[0268] When the camera is used to monitor laser safety, it is operated at a high frame rate and the sampling is at a low resolution.

[0269] Arrange the switches to provide switching to higher resolution to perform calibration.

[0270] Any of the software described above can be implemented as a computer program product that has been compiled for use in the processing engine of any of the servers or network nodes. The computer program product can be stored on a non-transient signal storage medium, such as an optical disc (CD-ROM or DVD-ROM), digital magnetic tape, a hard disk, solid-state storage (such as USB flash memory), ROM, etc.

[0271] For illustrative purposes, specific examples of systems, methods, and apparatus have been described herein. These are merely examples. The techniques provided herein can be applied to systems other than those illustrated above. Many modifications, alterations, additions, omissions, and substitutions are possible within the practice of this invention. This invention includes variations to the described embodiments that will be apparent to those skilled in the art, including variations obtained by: replacing features, elements, and / or actions with equivalent features, elements, and / or actions; mixing and matching features, elements, and / or actions from different embodiments; combining features, elements, and / or actions from the embodiments described herein with features, elements, and / or actions from other technologies; and / or omitting combinations of features, elements, and / or actions from the described embodiments. Therefore, the appended claims and the claims introduced herein are intended to be construed as including all reasonably inferred modifications, substitutions, additions, omissions, and sub-combinations. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the entire specification.

[0272] Although the invention has been described above with reference to specific embodiments, this is done for illustrative purposes and not for limiting the invention. Those skilled in the art will understand that various modifications and different combinations of the disclosed features can be made without departing from the scope of the invention.

Claims

1. A method of driving a projector system, the projector system comprising a light source, a phase modulator, at least one amplitude modulator configured to receive at least a portion of an actual illumination profile generated by the phase modulator, and at least one image sensor configured to receive at least a portion of the actual illumination profile generated by the phase modulator, the method comprising: Receive input image, Generate a target highlight image and a target image from the input image. A phase pattern for driving the phase modulator is generated from the target illumination profile. Generate a predicted illumination profile from the target illumination profile. An amplitude pattern for driving the amplitude modulator is generated from the predicted illumination profile and the target image. Feedback is provided from the at least one image sensor to generate at least one of the phase pattern, the predicted illumination profile, and the amplitude pattern. Wherein, one of the at least one image sensor includes an addressable region and / or an active region. The addressable region is configured to provide real-time calibration data. The active region is configured to provide periodic calibration data. Generating the predicted illumination profile further includes using the real-time calibration data and / or the periodic calibration data as input.

2. The method as described in claim 1, characterized in that, Further includes: The light source is driven using a power control signal. When the light intensity measured by the at least one image sensor is higher than a threshold, the power control signal is used to reduce the power or turn off the light source.

3. The method as described in claim 1, characterized in that, Further includes: The motion detection device is used to determine the static and dynamic portions of the input image in two consecutive image frames.

4. The method as described in claim 3, characterized in that, The static and dynamic portions of the input image are determined based on pixels or regions.

5. The method as described in claim 4, characterized in that, Further includes: When the pixel or region of the input image is a static portion, the predicted illumination profile is compared with the actual illumination profile. When the actual illumination profile is higher than the predicted illumination profile, the target image attenuates according to the ratio of the actual illumination profile to the predicted illumination profile. When the actual illumination profile is lower than the predicted illumination profile but still sufficient, the target image is multiplied by the ratio of the actual illumination profile to the predicted illumination profile. When the actual illumination profile is lower than the predicted illumination profile and is no longer sufficient, the target illumination profile is increased and the target image is multiplied by the ratio of the actual illumination profile to the predicted illumination profile.

6. The method as described in claim 1, characterized in that, The target illumination profile is an illumination pattern that should be generated by the phase modulator on the surface of the amplitude modulator.

7. The method as described in claim 1, characterized in that, The addressable region is a region outside the active region, and the addressable region is configured to receive at least a portion of the actual illumination profile outside the active region.

8. The method as described in claim 1, characterized in that, The projector system further includes another amplitude modulator, and the method further includes: The target base image is generated from the at least one amplitude modulator.

9. The method as described in claim 8, characterized in that, Further includes: When the target base image is received as additional input, a base pattern is further generated. When the base illumination is substantially uniform, the base pattern is identical to the target base image. When the base illumination is non-uniform, color and brightness uniformity corrections are applied to derive the base pattern from the target base image.

10. The method as described in claim 1, characterized in that, Further includes: The portion of the actual illumination profile received by the image sensor is reduced in size.

11. The method as described in claim 1, characterized in that, The portion of the actual illumination profile received by the image sensor that is equivalent to the portion of the actual illumination profile incident on the amplitude modulator is a scaled-down version, the scaled-down version being less than 1:

1.

12. The method as described in claim 1, characterized in that, The image sensor is placed in the path of the optical element that generates the target illumination profile.

13. The method as described in claim 1, characterized in that, The image sensor is arranged to receive light reflected from the inserted glass plate.

14. The method as described in claim 1, characterized in that, The image sensor is a camera, and the method further includes: When the camera is used to monitor laser safety, the frame rate of the camera operation is increased and the sampling resolution is reduced.

15. The method as described in claim 14, characterized in that, The image sensor is provided with a switch that allows switching to a higher resolution to perform calibration.

Images