Increased angular diversity for beam steering

Abstract

Systems and methods for displaying images. One example method includes receiving image data indicative of at least one image to be displayed, dividing, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions, and dividing, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions. The method includes identifying a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness, generating, for each subframe of the plurality of subframes, a set of regional drive values, and generating, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions. The beam-steering drive values for a first subframe are angularly different than the beam-steering drive values for a second subframe.

Description

INCREASED ANGULAR DIVERSITY FOR BEAM STEERING 1. Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from European Application No. 24306246.0 filed on 24 July 2024, and U.S. Provisional Application No.63 / 690,778 filed on 4 September 2024, each of which is incorporated by reference herein in its entirety. 2. Field of the Disclosure

[0002] This application relates generally to dual modulation projection systems, and more particularly to systems and methods for generating high-contrast, high-brightness, and high- quality images. 3. Background

[0003] Some dual-modulation projection systems utilize a pre-modulator and a primary modulator, which are both spatial light modulators (SLMs), to project high-quality images (i.e. images having high-contrast, high dynamic range, etc.). The pre-modulator provides an intermediate image, which can be a rough approximation of a desired image as defined by corresponding image data, to the primary modulator. Typically, the pre-modulator is either a beam-steering device, which redirects light at various angles with respect to a surface of the pre- modulator, or an amplitude modulating device, which spatially alters an amplitude distribution of an incident lightfield. Some SLMs, such as digital micro-mirror devices (DMDs) are technically beam-steering modulators, but can be used to produce perceived amplitude modulation and are often referred to as amplitude modulators.

[0004] The primary modulator spatially modulates the intermediate image to form a final image, which is a finer representation of the desired image defined by the image data. Typically, the primary modulator is an amplitude modulator. Examples of amplitude modulators include, but are not limited to, liquid crystal on silicon (LCOS) devices, DMDs, and so on. In order to drive the primary modulator to create the final image with sufficient quality, the intermediate image must be known at a high level of detail. Thus, there are a number of problems associated with driving the pre-modulator and the primary modulator to generate acceptable quality in the final image.

[0005] A major problem associated with driving the pre-modulator is determining the required pre-modulator drive values to generate the intermediate image with sufficient quality while maintaining computational efficiency. Another problem is determining drive values that make efficient use of the lightfield incident on the pre-modulator. These problems are particularly difficult to solve in systems utilizing a beam-steering device as the pre-modulator.

[0006] A major problem associated with driving the primary modulator is simulating the intermediate image with sufficient detail to determine the required primary modulator drive values while maintaining computational efficiency. BRIEF SUMMARY OF THE DISCLOSURE

[0007] The present invention overcomes problems associated with the prior art by providing a dual modulation display systems and driving methods that produce high efficiency, high quality images using a beam-steering spatial light modulator as a premodulator to provide an incident lightfield on a primary modulator. The invention facilitates improved computational efficiency (by way of reduced computational complexity) and control in the generation of the lightfield by the beam-steering premodulator and increased accuracy in modeling the generated lightfield.

[0008] Example controllers using a “z-collapse” method is also disclosed. In one such example controller, the beam-steering drive module divides the at least one image into a plurality of regions and determines which regions of the plurality of regions contain at least one location having a brightness greater than a predetermined brightness, based at least in part on the image data. The beam-steering drive module generates a plurality of sets of regional drive values. Each set of the regional drive values corresponds to a full plane of drive values for the beam- steering SLM. Each set of the regional drive values also has values operative to generate a region of the lightfield corresponding to one of the regions of the image containing at least one location having the brightness greater than the predetermined brightness, if the set of regional drive values is used individually to drive the beam-steering SLM. The beam-steering drive module determines an amount of light required for each region of the lightfield and combines the sets of regional drive values based at least in part on the determined amounts of light to generate the beam-steering drive values. In a particular example controller, the beam-steering drive module combines the sets of regional drive values such that a portion of the beam-steering drive values corresponding to one of the sets of regional drive values is proportional to a totalbrightness of the corresponding one of the regions of the image containing at least one location having the brightness greater than the predetermined brightness. Optionally, the predetermined brightness can be zero. As another options, the number of the plurality of regions can be equal to the number of pixels in the image data.

[0009] Example controllers using a “Band-Limited-Angular-Spectrum” (BLAS) or “Fresnel” algorithm are also disclosed. For example, BLAS as described herein is a diffraction model, or propagator, that maps lightfields between phase and amplitude modulator planes. BLAS computes the Rayleigh-Sommerfeld diffraction between two planes by means of a convolution implemented in the Fourier domain with a pair of padded Fourier Frequency Transforms (FFTs) and Inverse FFTs (IFFTs). Fresnel as described herein refers to another propagator that is an approximation of the Rayleigh-Sommerfeld and can be computed with a single Fourier transform without padding. The BLAS and Fresnel algorithms may also be iterative phase-retrieval algorithms, referred to as Gerchberg-Saxton (GS) algorithms with BLAS or Fresnel operators (e.g., propagators). The GS-BLAS and GS-Fresnel algorithms described herein solve for phase drive values representing the entire image, and provide for a higher quality lightfield compared to traditional methods. The GS-BLAS and GS-Fresnel algorithms provide multiple subsolutions per instance by using different regions to simultaneously reconstruct the entire image. The subsolutions are used to create new subframes with low computational expense.

[0010] In one aspect of the present disclosure, there is provided a method for displaying images. The method includes receiving image data indicative of at least one image to be displayed, dividing, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions, and dividing, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions. The method includes identifying a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data, generating, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator, and generating, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions. The beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

[0011] In another aspect of the present disclosure, there is provided a non-transitory computer-readable medium storing instructions that, when executed by a processor of a projection system, cause the projection system to perform operations comprising receiving image data indicative of at least one image to be displayed, dividing, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions, and dividing, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions. The instructions include identifying a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data, generating, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator, and generating, for each subframe of the plurality of subframes, beam- steering drive values by combining the set of regional drive values based on the identified subset of image regions. The beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

[0012] In another aspect of the present disclosure, there is provided an apparatus for controlling a dual-modulation projection system. The apparatus includes an electronic processor coupled to a memory, the memory storing instructions performed by the electronic processor. When the electronic processor performs the instructions, the electronic processor is configured to receive image data indicative of at least one image to be displayed, divide, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions, divide, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions, identify a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data, generate, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator, and generate, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions. The beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

[0013] In this manner, various aspects of the present disclosure provide for the display of images having a high dynamic range and high resolution, and effect improvements in at least the technical fields of image projection, holography, signal processing, and the like.DESCRIPTION OF THE DRAWINGS

[0014] These and other more detailed and specific features of various embodiments are more fully disclosed in the following description, reference being had to the accompanying drawings, in which:

[0015] FIG.1 is a block diagram of an example dual-modulation projection system;

[0016] FIG.2 is a block diagram of the controller of FIG.1;

[0017] FIG.3 is a block diagram illustrating an example data flow between the modules of the controller of FIG.1 to drive the other components of the projection system of FIG.1;

[0018] FIG.4 is a flow chart of an example method for generating beam-steering drive values from image data;

[0019] FIG.5 is a diagram illustrating an example of the method of FIG.4;

[0020] FIGS.6-7 are diagrams illustrating example modulation systems;

[0021] FIG.8 is a diagram illustrating light beams targeting a center of a reconstructed image plane;

[0022] FIG.9 is a diagram illustrating parallel light beams creating a flat field on a reconstructed image plane;

[0023] FIG.10 is a diagram illustrating randomized light beams creating a flat field on a reconstructed image plane;

[0024] FIG.11A is a diagram illustrating a plurality of subframes of randomized light beams;

[0025] FIG.11B is a diagram of the integrated plurality of subframes of FIG.11A;

[0026] FIG.12 is a flow chart of another example method for generating beam-steering drive values from image data;

[0027] FIG.13A illustrates a reconstructed spot for an 8x8 pixel sized square lens on a 512x512 image grid;

[0028] FIG.13B illustrates a reconstructed spot for a 16x16 pixel sized square lens on a 512x512 image grid;

[0029] FIG.13C illustrates a reconstructed spot for a 32x32 pixel sized square lens on a 512x512 image grid;

[0030] FIG.14A illustrates a Z-Collapse solution for a flat field with a diffraction simulation;

[0031] FIG.14B illustrates a Z-Collapse solution for a flat field with a blurred lightfield;

[0032] FIGS.15A-15D illustrate different decomposed levels for a target image;

[0033] FIG.16A illustrates an example reconstructed spot for a square lens;

[0034] FIG.16B illustrates an example reconstructed spot for a circular lens;

[0035] FIG.16C illustrates an example reconstructed spot for a hexagonal lens;

[0036] FIG.17A illustrates an example phase solution for 32x32 adjacent circular lenses;

[0037] FIG.17B illustrates a diffraction simulation for the phase solution of FIG.17A;

[0038] FIG.18A illustrates an example phase solution for 32x32 stacked square 3x3 lenses;

[0039] FIG.18B illustrates a diffraction simulation for the phase solution of FIG.18A;

[0040] FIG.19A illustrates an example phase solution for 32x32 stacked square 1x3 lenses;

[0041] FIG.19B illustrates a diffraction simulation for the phase solution of FIG.19A;

[0042] FIG.20A illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with no random offset for a single subframe;

[0043] FIG.20B illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with random offset for a single subframe;

[0044] FIG.20C illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with random offset over 32 subframes;

[0045] FIG.21A illustrates an example phase solution for 32x32 adjacent circular lenses with random offset over 32 subframes;

[0046] FIG.21B illustrates a diffraction simulation for the phase solution of FIG.21A;

[0047] FIG.22A illustrates an example phase solution for 32x32 stacked square 3x3 lenses with random offset over 32 subframes;

[0048] FIG.22B illustrates a diffraction simulation for the phase solution of FIG.22A;

[0049] FIG.23A illustrates an example phase solution for 32x32 stacked square 1x3 lenses with random offset over 32 subframes;

[0050] FIG.23B illustrates a diffraction simulation for the phase solution of FIG.23A;

[0051] FIG.24 illustrates example raster scans;

[0052] FIG.25 illustrates an example random scan;

[0053] FIGS.26A-26D illustrate an example layered approach for subframe decomposition;

[0054] FIG.27 illustrates an example level approach for subframe decomposition;

[0055] FIGS.28A-28D illustrate an example random approach for subframe decomposition;

[0056] FIGS.29A-29F illustrate an example temporal Z-Collapse feedback loop;

[0057] FIG.30 is a flow chart of another example method for generating beam-steering drive values from image data;

[0058] FIG.31A is a diagram illustrating backward propagation of a wavefield between the amplitude modulator and the beam-steering modulator of FIG.1;

[0059] FIG.31B is a diagram illustrating forward propagation of a wavefield between the beam-steering modulator and the amplitude modulator of FIG.1;

[0060] FIG.32 illustrates a block diagram of a process for performing one forward propagation using the BLAS operator;

[0061] FIG.33 illustrates a block diagram of a process for performing a single forward propagation using a Fresnel transform;

[0062] FIG.34 illustrates a relationship between sampling of a beam-steering modulator of FIG.1 and reconstruction using the BLAS operator;

[0063] FIG.35 illustrates a relationship between sampling of a beam-steering modulator of FIG.1 and reconstruction using the Fresnel operator;

[0064] FIGS.36A-36C illustrate a process of a phase modulator drive encoding scheme that leverages the autoscaling property of the Fresnel transform propagator; and

[0065] FIG.37 illustrates a plot of the relationship between the computational complexity and the number of subframes for various algorithms described herein. DETAILED DESCRIPTION

[0066] This disclosure and aspects thereof can be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. The foregoing is intended solely to give a general idea of various aspects of the present disclosure, and does not limit the scope of the disclosure in any way.

[0067] In the following description, numerous details are set forth, such as optical device configurations, timings, operations, and the like, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to one skilled in the art that these specific details are merely examples and not intended to limit the scope of this application.

[0068] Moreover, while the present disclosure focuses mainly on examples in which the various circuits are used in digital projection systems, it will be understood that these are merely examples. It will further be understood that the disclosed systems and methods can be used in any device in which there is a need to project light; for example, cinema, consumer, and other commercial projection systems, heads-up displays, virtual reality displays, and the like. Disclosed systems and methods may be implemented in additional display devices, such as with an OLED display, an LCD display, a quantum dot display, or the like.

[0069] FIG.1 is a block diagram of a dual-modulation projection system 100, according to the present invention. Projection system 100 generates high quality images from image data and includes a light source 102, a beam-steering modulator 104, an amplitude modulator 106, projection optics 108, and a controller 110. Light source 102 shines a flat lightfield onto beam- steering modulator 104. Beam-steering modulator 104 selectively steers portions the lightcomprising the flat lightfield through a set of intermediate optics 112 and onto amplitude modulator 106, to form an intermediate image on the surface of amplitude modulator 106. Amplitude modulator 106 spatially modulates the intermediate image to form a final image, which is directed toward projection optics 108. Projection optics 108 is a set of lenses, prisms, and / or mirrors, which direct the final image toward a screen or other surface to be viewed by an audience.

[0070] Controller 110 controls and coordinates the other elements of projection system 100, based on image data received from a data source (not shown). Controller 110 provides control instructions to light source 102, beam-steering modulator 104, and amplitude modulator 106, based at least in part on the received image data. The control instructions include, at least, beam- steering drive values and amplitude drive values sent to beam-steering modulator 104 and amplitude modulator 106, respectively. These control instructions drive beam-steering modulator 104 and amplitude modulator 106 in order to generate the intermediate and final images. Controller 110 utilizes numerous methods and algorithms, which are discussed in detail below, to generate the drive values based on the received image data, and also to generate the amplitude drive values based on lightfield simulations.

[0071] In the example embodiment, light source 102 is an array of tunable lasers. In alternate embodiments, light source 102 can be replaced by an array of light-emitting diodes (LEDs), a dimmable bulb and appropriate optics, or any other suitable light source, including those now known or yet to be invented. Additionally, beam-steering modulator 104 and amplitude modulator 106 can be liquid crystal phase and amplitude spatial light modulators (SLMs), respectively. In alternate embodiments, beam-steering modulator 104 can be a tip-tilt mirror device, a microelectromechanical systems (MEMS) device, or any other light steering device, including those now known or yet to be invented. Amplitude modulator 106 can be a digital micro-mirror device (DMD) or any other suitable amplitude modulating device, including those now known or yet to be invented.

[0072] In the description of example embodiments beam-steering modulator 104 and amplitude modulator 106 are thus named to distinguish between an SLM that is used to steer light to create a lightfield on a primary modulator (beam-steering modulator 104) and an SLM that modulates selected portions of the lightfield to create an image for viewing (amplitude modulator 106). However, these terms are not used in a limiting sense. For example, DMDs selectively steer light along or out of an optical path, but are used as amplitude modulators by time multiplexing the amount of light steered into or out of an image to create an intermediategray level (perceived amplitude modulation). As another example, liquid crystal SLMs selectively alter the phase of light and can, therefore, be considered a phase modulating or beam steering device. However, the birefringent property of liquid crystals also results in polarization rotation, and so liquid crystal SLMs can be used with internal or external polarizers to provide amplitude modulation. Therefore, devices referred to as “amplitude modulators”, “phase modulators”, or “beam-steering modulators” are understood to include any device capable of performing the titled function, either alone or in combination with other devices.

[0073] FIG.2 shows controller 110 (e.g., an apparatus), including a data transfer interface 202, non-volatile data storage 204, one or more processing unit(s) 206 (e.g., one or more electronic, photonic, and / or quantum processors), and a working memory 208. The components of controller 110 communicate with one another via a system bus 210, which is interconnected between the components of controller 110. Data transfer interface 202 controls the transfer of data, including image data and control instructions, to and from controller 110. Non-volatile data storage 204 (e.g., a non-transitory computer-readable medium) stores data and code and retains the data and code even when controller 110 is powered down. Processing unit(s) 206 impart functionality to controller 110 by executing code stored in non-volatile data storage 204 and / or working memory 208.

[0074] Working memory 208 provides temporary storage for data and code. Some functionality of controller 110 is represented by data and code modules shown within working memory 208. The data and code modules are transferred (in whole or in part) into and out of working memory 208 from non-volatile data storage 204, as determined by the execution of code by processing unit(s) 206. The data and code modules can be implemented, for example, with any combination of hardware, software, and / or firmware.

[0075] Working memory 208 includes a control / coordination module 212, a data buffer 214, a communication module 216, system configuration settings 218, a calibration module 220, an optical database(s) 222, a beam-steering module 224, a lightfield simulation module 226, and an amplitude module 228. Control / coordination module 212 is a higher level program that provides overall coordination and control of the other functional aspects of controller 110. Data buffer 214 temporarily stores data to be utilized by the other components of controller 110. Communication module 216 facilitates communication with external devices in order to send / receive code / control instructions. System configuration settings 218 include user defined settings that partially control the operation of other components of controller 110. Calibration module 220 includes data and algorithms used to calibrate projection system 100. Opticaldatabase(s) 222 is a database including definitions and / or characteristics of a variety of optical elements that can be referenced by the other components of controller 110. Beam-steering module 224 includes data and algorithms for generating beam-steering drive values from image data. Lightfield simulation module 226 includes data and algorithms for generating a simulation of the lightfield generated by beam-steering modulator 104 from the beam-steering drive values. Amplitude module 228 includes data and algorithms for generating amplitude drive values from image data and the lightfield simulation.

[0076] FIG.3 is a block diagram illustrating an example data flow between the modules of the controller of FIG.1 to drive the other components of the projection system of FIG.1. The modules shown in FIG.3 are stored and executed within working memory 208 (FIG.2) of controller 110. First, beam-steering module 224 receives image data from data transfer interface 202. Using one (or more) of a variety of methods and / or algorithms, which will be described in greater detail below, beam-steering module 224 generates a set of drive values for driving beam- steering modulator 104 (FIG.1) from the image data. The beam steering drive values are provided to drive beam-steering modulator 104, and also provided to lightfield simulation module 226, to be utilized for generating a simulation of the resulting lightfield on amplitude modulator 106. Using one (or more) of a variety of methods and / or algorithms, lightfield simulation module 226 generates a lightfield simulation, which will be utilized for generating a set of drive values for driving amplitude modulator 106. The lightfield simulation is provided directly to amplitude module 228. Amplitude modulator uses one (or more) methods and / or algorithms to generate a set of drive values for driving amplitude modulator 106 from the lightfield simulation and the image data, and provides the amplitude drive values to amplitude modulator 106.

[0077] In the example embodiment, beam-steering module 224, lightfield simulation module 226, and amplitude module 228 each utilize relevant data from system configuration settings 218, calibration module 220, and optical database(s) 222. For example, beam-steering module 224 and lightfield simulation module 226 utilize data from optical database(s) 222 describing the characteristics of optics 112, in order to predict how optics 112 will affect the lightfield generated by beam-steering modulator 104. Beam-steering module 224 and lightfield simulation module 226 can also utilize data from system configuration settings 218 or calibration module 220 to determine where optics 112 is placed, how optics 112 affected the lightfield during calibrations, etc. Beam-steering module 224, lightfield simulation module 226, and amplitude module 228 are together able to generate beam-steering and amplitude drive values that arerequired to produce high quality images via projection system 100, utilizing data from system configuration settings 218, calibration module 220, and optical database(s) 222.

[0078] FIG.4 is a flow chart summarizing an example method 400 for generating beam- steering drive values from image data. The method 400 may be referred to as a “Z-Collapse” method 400. In the example embodiment, method 400 is performed by beam-steering drive module 224, utilizing data from other elements of controller 110. In alternate embodiments some steps of method 400 can be performed by other elements and / or system designers / users. In a first step 402, a desired image (based on the image data) is divided into a plurality of regions. Next, in a second step 404, a subset of the plurality of regions that have non-zero brightness is identified. Then, in a third step 406, a set of full-plane elements that each directs light to a corresponding one of the subset of regions is generated. In a fourth step 408, beam-steering drive values are generated by combining the set of full-plane elements, based, at least in part, on the relative brightnesses of the regions comprising the subset of regions. Finally, in a fifth step 410, drive values are generated for locations on the modulator drive that are unpopulated using a dump algorithm. For example, after combining all full plane elements in order to meet relative brightness levels of all image regions, unpopulated locations on the modulated drive may remain. A dump algorithm steers excess light off-screen by means of drive values that implement a lens or diffraction grating, such as using a Fourier filter. Method 400 will be discussed in more detail with reference to FIG.5 below.

[0079] FIG.5 is a diagram illustrating method 400. A desired image 502 is divided into a plurality of regions 504. Three of regions 504, labeled 504A, 504B, and 504C, contain areas of non-zero brightness (indicated by an image of an ellipsoid 506). Beam-steering module 224 identifies regions 504A, 504B, and 504C as having non-zero brightness by examining the image data corresponding to desired image 502, as described in step 404 of method 400. For regions 504A, 504B, and 504C, beam-steering module 224 generates full-plane drive values 508A, 508B, and 508C, respectively, as described in step 406 of method 400. Full-plane drive values 508 can be pre-calculated or generated in real time utilizing any of the other example methods for generating beam-steering drive values as discussed in this disclosure.

[0080] Each of full-plane drive values 508 would generate a spot either at the center of the corresponding region 504 on amplitude modulator 106 when used to drive the entire (full-plane) beam-steering modulator 104, or the full-plane drive values 508 would generate a spot at the center of mass of the light distribution in the corresponding region 504 on amplitude modulator 106 when used to drive the entire beam-steering modulator . For example, if full-plane drivevalues 508A are used to drive beam-steering modulator 104, the portion of ellipsoid 506 within region 504A will be the only thing displayed on amplitude modulator 106. The portion of ellipsoid within region 504A will also be displayed at the same location and to the same scale as in desired image 502. Beam-steering module 224 combines full-plane drive values 508, as described in step 408 of method 400, to generate beam-steering drive values that will drive beam-steering modulator 104 to generate a recreated image 510 on amplitude modulator 106. Recreated image 510 is as similar to desired image 502 as is possible given the capabilities of beam-steering modulator 104. In some examples, iterative methods for generating the full plane drive values are implemented, such as using the GS-BLAS or GS-Fresnel, as will be described in more detail.

[0081] When combining full-plane drive values 508, beam-steering module 224 must allocate portions of beam-steering modulator 104 based on the relative brightnesses of regions 504. Brighter ones of regions 504 require more light to be illuminated properly, so they require that more of the area of beam-steering modulator 104 direct light toward them. In the example embodiment, ellipsoid 506 has uniform brightness across its area, so the relative brightnesses of regions 504A, 504B, and 504C depend on the relative areas of ellipsoid 506 within them. Approximately one-fifth of ellipsoid 506 is contained within each of regions 504A and 504C, while the other three-fifths are contained within region 504B. Therefore, the final beam-steering drive values will comprise one-fifth of each of full-plane drive values 508A and 508C and three- fifths of full-plane drive values 508B, as shown on recreated image 510. In some implementations, an optimal transport algorithm is implemented to determine which regions 504 of the lenses should be exposed by beam steering drive values that recreate image 502. The solution to the optimal transport algorithm generates a set of beam steering drive values that minimizes some cost, such as steering angles. In other implementations, beam steering drive values may be generated using other schemes, such as a hierarchical or priority allocator algorithms where the allocation criterion prioritizes regions of higher brightness, a pseudo- random pattern, a raster method, or a random pattern.

[0082] The example embodiment illustrated by FIG.5 is simplified to provide a clear understanding of method 400. In practical embodiments, the resolution of regions 504 will be much higher, and in some cases as high as the resolution of desired image 502. In such embodiments beam-steering module 224 analyzes the image data corresponding to desired image 502 to determine which image regions 504 have non-zero brightnesses. For each of these regions 504, full-plane drive values are generated, each of the full-plane drive values being a phase representation of a lens that directs the entire incident lightfield onto a correspondingregion 504 (or pixel). When the full-plane drive values are combined, beam-steering module 224 utilizes the image data corresponding to desired image 502 to determine the relative brightnesses of the regions 504, and, therefore, the relative areas of beam-steering modulator 104 dedicated to directing light onto the regions 504, thus, automatically generating recreated image 510 with an appropriate intensity distribution. Again, an optimal transport solution is generated to allocate areas of beam-steering modulator 104 for maximum efficiency.

[0083] In some instances, a Vogel approximation method is used to provide a solution to light allocation described in method 400. The method 400 ensures the proximity of the beam- steering SLM partitions allocated to the same (or near same) PSF locations in the lightfield, in 2D space. This limits the diffraction loss potentially caused by having too many switches between partitions steering to different locations, which can look like a “block-artifact”. The method 400 also ensures locality of the beam-steering SLM partitions with respect to the target position, which limits deflection angles and, therefore, diffraction losses.

[0084] Optionally, a multi-resolution Vogel approximation method (MRVAM) can be used in method 400. The MRVAM solves the transportation (optical) problem using multiple levels of resolution. First, the problem is solved on a coarse grid (e.g., 30 x 18 partitions) generating a first approximation solution. Then, the solution is refined on a finer grid (e.g., 120 x 72), using the result of the first approximation and allowing the fine solution to span the equivalent of a 3 x 3 “coarse region.” This method facilitates solving much more complicated systems, while keeping computational complexity low.

[0085] In the z-collapse method 400, the beam-steering modulator 104 steers light to create a desired image at a plane that is not the beam-steering modulator 104. However, the resolution of the image is dependent on the etendue of the illumination. To create a high-resolution image, one or more low-etendue light sources may illuminate the beam-steering modulator 104. Etendue is a measurement of the effective spatial and angular size of a light source, and is related to terms such as mm2*sr (steradians), M2factor, or beam parameter product (BPP). For projection systems based on beam steering, the smallest focused spot size (or pixel) of the projected image is dependent on the etendue level. As the etendue of the illumination defines the “spread” of the light as it is steered (e.g., the cone angle of the light), too much spread results in a blurred image. When the cone angle is small, high resolution and high brightness is achieved.

[0086] As one example, FIG.6 illustrates an example dual-modulation system 600 that includes the beam-steering modulator 104 and the amplitude modulator 106. The dual-modulation system 600 may be, for example, a portion of the projection system 100. A low- etendue light 602 is projected onto the beam-steering modulator 104. The beam-steering modulator 104 steers the low-etendue light 602 as first light 604 onto the amplitude modulator 106. The first light 604 has a narrow cone angle (for example, less than 1°), resulting in a spot 606 having a small spot size (e.g., small pixel size) on the amplitude modulator 106. Using the low-etendue light 602, the achieved dual-modulation image 608 is essentially equivalent to desired image 610. However, due to the small cone angle (or high f-numbers) of the illumination, the light for each image point follows a reduced number of paths through optics, and spatially localized defects, such as dust or scratches, are exposed throughout the system and detectable in the achieved dual-modulation image 608. If fiber coupling between the light source 102 and the projection system is utilized, significant modal noise may be presented as a noise pattern over the entire illumination beam which changes as the fiber is moved. This creates unknown spatial and temporal error within the dual-modulation system 600. Further, the coherence of low-etendue light 602 creates diffraction artifacts which deviate from the traditional view of a projection system. for example, the edge of an aperture or a speck of dust may cause spatial ringing artifacts. Accordingly, issues are present in utilizing low-etendue light 602.

[0087] FIG.7 illustrates an example dual-modulation system 700 that includes the beam- steering modulator 104 and the amplitude modulator 106. In the example of FIG.7, a high- etendue light 702 is projected onto the beam-steering modulator 104. The beam-steering modulator 104 steers the high-etendue light 702 as first light 704 onto the amplitude modulator 106. The first light 704 has a larger cone angle (for example, greater than 1°) than compared to the first light 604 of FIG.6, resulting in a spot 706 that is larger than the spot 606 of FIG.6. As shown in the achieved dual-modulation image 708, the spot 706 is larger and dimmer than the spot 606, as the energy is more spread. Accordingly, issues are also present in implementing high-etendue light 702.

[0088] It is noted that, while FIGS.6 and 7 illustrate dual-modulation systems, this issue of high etendue light and embodiments described herein also apply to systems with a single modulator (for example, only a phase modulator) to create the final image. Additionally, while it is possible to increase etendue and conserve energy, it is difficult to reduce etendue without throwing away light and reducing the energy of the light beam.

[0089] Some embodiments described herein utilize the beam-steering modulator 104 (e.g., the phase modulator) to introduce angular diversity when creating an image. For example, thebeam-steering modulator 104 is capable of providing many phase solutions in rapid succession during a typical image frame time. Accordingly, a sequence of phase subframes may be used to reproduce an image, and each subframe may introduce angular diversity such that the issues described with respect to FIGS.6-7 are mitigated. Additionally, with respect to FIG.5, rather than generating a single phase solution with a single mapping for a given region 504 of the beam-steering modulator 104 to a location on the image plane of the recreated image 510, each sub-solution may steer from a different region 504 over time to create the same image detail. As the light is averaged over the frame, modal noise is also mitigated.

[0090] Additionally, when a single phase solution is utilized, the steered light pattern occurs at one instant in time, and interference from diffraction may create unwanted artifacts with coherent light. By spreading out parts of the pattern over time, diffracted light occurs at different instants in time and add up incoherently in intensity.

[0091] To visualize the addition of angular diversity, first consider a situation when the target image is a center dot containing all light energy. FIG.8 illustrates an output of beam- steering modulator 104. Steered light 802 is directed onto reconstructed image plane 800. The origin of every light ray in the steered light 802 may be considered one of the z-collapsed regions 504 on the beam-steering modulator 104.

[0092] In the example of FIG.8, every point on the beam-steering modulator 104 is mapped to a single point on the reconstructed image plane 800, forming a unique solution. The range of angles is maximized because additional angle could only be added by steering from beyond the beam-steering modulator 104. The intensity across the steered light 802 is also completely averaged in the intensity of the recreated dot on the reconstructed image plane 800. Beyond the reconstructed image plane 800, a cone of light 804 is formed.

[0093] For comparison, FIG.9 illustrates a flat field output of beam-steering modulator 104. In the example of FIG.9, the points on the reconstructed image plane 800 receive a collimated ray that exposes the previously-noted artifacts. Any unknown or uncalibrated non-uniformity in the illumination beam, such as from modal noise, is directly transferred in the image on the reconstructed image plane 800.

[0094] Another possible steering solution is illustrated in FIG.10, which randomizes the steering of the flat field as randomized light 1002 projected by the beam-steering modulator 104 onto the reconstructed image plane 800. However, as the level of the flat field is the diffuseillumination angle, the angles of the randomized light 1002 are still traveling in one direction at each point.

[0095] Accordingly, embodiments described herein provide a plurality of subframes with different illumination angles at each point such that the effective cone angle over the time window of period T is increased. The time window may be, for example, the integration time of the human eye for the final image, or a bit-plane time of the amplitude modulator 106. FIG.11A illustrates a plurality of subframes 1102A-1102N at time intervals t1 through tN, where the plurality of subframes 1102A-1102N form a single frame. When integrated over the entire frame, the plurality of subframes 1102A-1102N create a plurality of light cones 1104 past the reconstructed image plane 800, shown in FIG.11B. While only three light cones 1104 are illustrated in FIG.11B, a light cone 1104 emanates from every point on the reconstructed image plane 800.

[0096] By controlling the beam-steering modulator 104 to introduce angular diversity over a plurality of subframes, the light follows multiple paths through the optics for every point in the image, averaging out any artifacts resulting from dust or scratches. Additionally, for any point in the image, light originates from a variety of locations on the modulator, mitigating modal noise. Finally, the temporal nature in adjusting the illumination angles over a plurality of subframes ensures that diffraction does not cause interference in the output image.

[0097] FIG.12 is a flow chart summarizing another example z-collapse method 1200 for generating beam-steering drive values from image data. In the example embodiment, method 1200 is performed by beam-steering drive module 224, utilizing data from other elements of controller 110. In alternate embodiments some steps of method 1200 can be performed by other elements and / or system designers / users.

[0098] In a first step 1202, the beam-steering modulator 104 is divided, for each subframe, into a plurality of modulator regions. The plurality of modulator regions may be the same across different subframes or may vary across different subframes. Additionally, the plurality of modulator regions may be periodic grids of equal or different pitches, and may have any number of shapes (for example, hexagonal, square, or rectangular regions). In some instances, some of the modulator regions are aggregated to create a larger region that is the union of a selected set of modulator regions for a given subframe.

[0099] In a second step 1204, a desired image (based on the image data) is divided, for each subframe, into a plurality of image regions, each image region associated with a modulatorregion. Each image region may be associated with any number of modulator regions, ranging from zero to the total number of regions on the beam-steering modulator 104. The plurality of image regions may be the same across different subframes or may vary across different subframes. In one example, the desired image is divided into temporal sub-images that, when combined, form the desired image. For example, the desired image is decomposed into halftone components, and each halftone component is assigned a subframe. Additionally, the plurality of image regions may be periodic grids of equal or different pitches, and may have any number of shapes (for example, hexagonal, square, or rectangular regions).

[0100] In a third step 1206, a subset of the plurality of image regions that have non-zero brightness is identified. Then, in a fourth step 1208, a set of full-plane elements that form each modulator region and that each directs light to a corresponding one of the subset of image regions is generated for each subframe.

[0101] Finally, in a fifth step 1210, the beam-steering drive values are generated, for each subframe, by combining the set of full-plane elements based, at least in part, on the relative brightnesses of the image regions comprising the subset of image regions. The beam-steering drive values may be, for example, lens patterns that are panned or tilted to target the desired image location, patterns that take into account the specific 3D geometry of each modulator in the projection system 100 such that the lenses minimize aberrations, patterns determined from other algorithms such as a Gerchberg-Saxton (GS) method, methods implementing the Band-Limited- Angular-Spectrum (BLAS) operator or Fresnel operator (described below), combinations thereof and the like. Additionally, the beam-steering drive values may be parameterized in aspects beyond simple steering of the modulators. For example, each image region may be analyzed only in its location and intensity, but also analyzed in variance or hose “diffuse” the region is. Lenses may also be stored with a “focal length” parameter to create varying levels of focusing.

[0102] In some instances, the lenses described herein are virtual lenses positioned over a region of the desired image. For example, a phase representation of a convex lens can be placed over a local maximum of the aggregate of the brightness of the desired image, because the local maximum corresponds to a relatively bright region. The virtual lenses may target the center of each image region, or may target the center of mass of the light distribution in each region. The virtual lenses may be altered, combined, stacked, and overlapped in such a way as to achieve a desired light. The beam-steering drive values may be a phase representation of a lens that directs the entire incident lightfield onto a corresponding pixel or region. The lenses are generally implemented at the beam-steering modulator 104, and do not refer to any physical optical lensesincluded in the optics 112. In one implementation, generating the virtual lenses includes centering the lenses on the target region of the desired image. For a desired image where light is present in every region (e.g., no black regions), a lens is provided for every region. After the lenses are centered in every region where light is present, the lenses are combined into a single phase drive, for example by exposing only a region of each lens. The area exposed for a particular lens determines the achieved brightness of the lens. Therefore, there is a quantization provided by the virtual lenses where the number of regions in the beam-steering modulator 104 determines the achieved brightness steps.

[0103] Accordingly, the above-described subframe generation may involve dividing the beam-steering modulator 104 into tile regions, each of which groups regions of the beam- steering modulator 104 together. The tile regions may be compact or may be interleaved. Pre- computed tile patterns have a level threshold for each tile region in some pattern. The pre- computed tile patterns may be spatially pseudo-random. Additionally, the size of each tile region may correspond to the levels of intensity quantization for a given subframe. Depending on the intensity of the selected target image region, the matching phase solution plane may occupy one or more tile regions in the tile pattern. The tile pattern may be varied (e.g., randomized) over the same subframe or over multiple subframes over a period T, providing for increased angular diversity.

[0104] In some implementations, the z-collapse methods described herein include partitioning the beam-steering modulator 104 into a grid of lens elements (e.g., a grid of beam- steering drive values) and the reconstruction (or image) plane into another grid of discretized levels that represent the energy in the source image (e.g., a low-resolution image with discretized levels). The grid of lens elements (e.g., discretized regions of the beam-steering modulator 104) may be a fixed grid or a variable grid. The lens elements may have a given shape. Additionally, the image plane may be partitioned onto the same grid or a different grid than the lens elements. To partition the modulator and the image plane into grids, several parameters may be set, including the modulator (or image) grid size, the lens shape, the lens / image allocation, subframe decomposition, and a feedback loop, as described below. The lens elements are filled with the phase solution of the virtual lens targeting the spot they are allocated to.

[0105] First, with regards to the modulator / image grid size, the number of modulator / image regions is limited by the relationship between the lens and the reconstructed spot size. The diffracted spot size is inversely proportional to the size of the lens elements, as shown in FIGS. 13A-13C. FIG.13A illustrates a reconstructed spot for an 8x8 pixel sized square lens on a512x512 image grid. FIG.13B illustrates a reconstructed spot for a 16x16 pixel sized square lens on a 512x512 image grid. FIG.13C illustrates a reconstructed spot for a 32x32 pixel sized square lens on a 512x512 image grid. The reconstructed spot size is constrained by the lens size and the working distance. For example, the total number of lenses (the lens grid size multiplied by the subframes) defines how many discrete levels are achieved for a desired image. The resolution is limited by the spot size and the PSF size. Particularly, the resolution cannot be smaller than the spot size but must be large enough for a given PSF.

[0106] In some instances, display levels may be reproduced up to the number of modulator regions (or based on the grid size) when no subframes are used. The modulator grid size may be selected based on a compromise between a number of addressable levels and spot size due to larger spot sizes causes halos in the image.

[0107] In some implementations, the reconstructed spot size is smaller than the modulator / image regions. A PSF large enough to fill the voids between spots is desirable. FIG. 14A illustrates a Z-Collapse solution for a flat field with a diffraction simulation. FIG.14B illustrates a Z-Collapse solution for a flat field with a blurred lightfield. In some implementations, the image grid size, the grid position, and / or the spatial phase may be adjusted on a per-image basis. Additionally, a phase solution may contain lenses of different sizes (e.g., multi-resolution image decomposition).

[0108] For sparse images such as those generated by a “highlight projector”, approximately 32x18 (=576) levels are enough to display the image at an acceptable resolution. However, 32x18 levels is too coarse a level decomposition for full images. As many images have some energy everywhere that needs to be filled with a limited number of discrete levels, such energy may result in the Z-Collapse producing a flat field with no luminance boost. For example, FIGS. 15A-15D illustrate different decomposed levels for a target image (shown in FIG.15A). FIG. 15B illustrates a flat field produced with 32x18 lenses in a single subframe.

[0109] As one example implementation, a 2048x1080 PLM is provided at a working distance. The PLM is divided into a 32x18 grid of 64x60 lenses. The desired image is divided into a 64x36 grid of 32x30 image regions. As the spot size increases with working distance, should the working distance be halved, the number of lens elements would increase by 4x to maintain the same spot size. In this instance, if the image grid is maintained, the image is decomposed in 4x as many levels. Additionally, as the PSF reduces size linearly with distance, the PSF is increased. In another instance, the PLM is divided into a 32x36 grid of 64x30 lenses,and the desired image is divided into a 64x30 (or 128x60) grid of 32x26 (or 16x18) image regions. In yet another instance, the PLM is divided into a 64x72 grid of 32x30 lenses, and the desired image is divided into a 54x30 grid of 16x18 image regions. In such examples, the working distance may be the minimum propagation distance at which the higher diffraction orders are fully separated from the 0thorder at the reconstruction plane. At shorter working distances (for example, 1 / 2 to 1 / 8 an optical working distance), the PLM is divided into a 32x36 grid of 64x30 lenses, and the desired image is divided into a 128x60 grid of 32x26 image regions.

[0110] By introducing subframes (for example, S subframes), the number of lenses, and therefore the number of image levels, is effectively increased by a factor of S. For example, when the beam-steering modulator 104 is capable of showing greater than 500 subframes, 576 discrete levels is effectively increased to approximately 300,000 levels.

[0111] With 512 subframes, a decomposition image can be produced that matches the source image much more closely and provides a boost over a flat image. For example, FIG.15C illustrates example decomposed levels with 32x18 lenses and 512 subframes on a matching 32x18 image grid. Additionally, a finer image grid may be used as the lens grid. FIG.15D illustrates decomposed levels with 32x18 lenses and 512 subframes on a 64x36 image grid.

[0112] Reconstructed spot shape varies according to the shape of lens elements. For example, a square or rectangular lens has a strong interference pattern horizontally and vertically, but is relatively easier to implement a square or rectangular lens compared to other lens shapes, and the rectangular has a shape that aligns with modulators. An example reconstructed spot for a square lens is illustrated in FIG.16A. A circular lens has a more diffuse interference pattern compared to other lens shapes, with less strong features, but is more difficult to implement and only offers up to 90% occupancy on a modulator as the regions in-between the lenses would be steered to a light dump. An example reconstructed spot for a circular lens is illustrated in FIG.16B. A hexagonal lens offers a compromise between the rectangular and circular lenses. An example reconstructed spot for a hexagonal lens is illustrated in FIG.16C.

[0113] All lenses in a given subframe interfere with each other, including both the lenses targeting the same region (e.g., “stacked” lenses or lenses belonging to the same plane), as well as lenses targeting different regions. For example, FIGS.17A-17B, 18A-18B, and 19A-19B illustrate different examples of lens interference. FIG.17A illustrates an example phase solution for 32x32 adjacent circular lenses. FIG.17B illustrates a diffraction simulation for the phasesolution of FIG.17A. In the example of FIG.17A, the “black regions” exterior to the circular lenses have an amplitude of zero, as only the interference of the “active” portions of the lens elements are illustrated. FIG.18A illustrates an example phase solution for a 3x3 grid of 32x32 stacked square lenses. FIG.18B illustrates a diffraction simulation for the phase solution of FIG. 18A. FIG.19A illustrates an example phase solution for 32x32 stacked square 1x3 lenses. FIG. 19B illustrates a diffraction simulation for the phase solution of FIG.19A.

[0114] The adjacent circular lenses (steering straight) of FIGS.17A-17B produce a strong patterning on the image plane. The stacked adjacent square lenses of FIGS.18A-18B and 19A- 19B effectively produce a single larger lens (96x96 in FIGS.18A-18B, 36x36 in FIGS.19A- 19B), which causes the reconstructed spot to change size and shape. When using a vertical or horizontal raster allocation method, as will be described below, elongated lenses may be created. Both the patterning and the elongated lenses are undesirable. Rather, lenses that can be stacked while maintaining the shape of the reconstructed spot to be the same as that of a single lens, while the spot intensity scales with the number of lenses, is desirable.

[0115] To “decouple” the lenses, a random offset may be added to each lens individually. The random offset may be constant across the lens region, may be different from lens to lens, and may be different between subframes for a given lens. When integrating over several subframes, the resulting spot from the lenses converge to that of a single lens, as shown in FIGS. 20A-20C. FIG.20A illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with no random offset for a single subframe. FIG.20B illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with random offset for a single subframe. FIG.20C illustrates a resulting spot from a stack of 3x3 square lenses of size 32x32 with random offset over 32 subframes.

[0116] In addition, random offset causes the lenses to span various regions of the modulator stroke, therefore averaging errors out due to non-linearity. The addition of random offset to each lens corrects many undesirable behaviors, as shown in FIGS.21A-21B, 22A-22B, and 23A-23B. FIG.21A illustrates an example phase solution for 32x32 adjacent circular lenses with random offset over 32 subframes. FIG.21B illustrates a diffraction simulation for the phase solution of FIG.21A. FIG.22A illustrates an example phase solution for 32x32 stacked square 3x3 lenses with random offset over 32 subframes. FIG.22B illustrates a diffraction simulation for the phase solution of FIG.22A. FIG.23A illustrates an example phase solution for 32x32 stacked square 1x3 lenses with random offset over 32 subframes. FIG.23B illustrates a diffraction simulation for the phase solution of FIG.23A. While FIGS.20A-20C, 21A-21B, 22A-22B, and 23A-23Billustrate the impact of random offset, in some instances, the offset may be predetermined and may not be random.

[0117] Additionally, lenses and image regions may (separately) be allocated as they are processed in each subframe. Multiple modulator regions may be allocated to a single image region to add intensities in that region and increase the brightness. In some implementations, the lenses and image regions are allocated using a raster scanning method. With a raster scanning method, zones are scanned along a predetermined path. FIG.24 illustrates example raster scans S1-S8. In other implementations, the lenses and image regions are allocated using a random scanning method. With a random scanning method, regions are processed in a random order. FIG.25 illustrates an example random scan.

[0118] The image grid and modulator grid are scanned independently. Using a raster method, for each subframe, the raster scans the regions of the (low-resolution) image in the order defined by its scanning method. For every non-zero region, the raster assigns the appropriate number of modulator regions. The lenses are allocated to the respective modulator region following the modulator scanning.

[0119] In some instances, the image allocation method and the modulator allocation method are the same. In other instances, the image allocation method and the modulator allocation method are different or are performing in opposing directions. The image allocation method and the modulator allocation method may also vary from subframe to subframe (e.g., left to right for a first subframe, top to bottom for a second subframe, right to left for a third subframe, and the like).

[0120] Using the same path for both the image allocation and the modulator allocation may result in the highest efficiency compared to other methods of allocation, but the lowest diversity. Alternatively, the highest diversity is achieved when one or both of the image allocation and the modulator allocation uses a random allocation method, but at the cost of efficiency (reduced from, for example, approximately 85% to approximately 65%. In some implementations, the amount of diversity is controlled by the choice of the random distribution (for example, a uniform random distribution, a normal random distribution, and the like). In some instances, the planes may be divided up in regions to control the locality (for example, a “locally random” allocation).

[0121] The final image may be created both by interference (lenses in the same phase solution) and summed incoherently in intensity (subframes in time). Because diffractionsimulation (to simulate interference) is computationally expensive, a lightweight algorithm may be used in the intensity domain. A subframe decomposition may distribute the levels in a way that minimizes the interference and maximizes temporal intensity integration. Subframe decomposition refers to how discrete image levels are distributed over subframes. For example, a layered approach decomposes the image into S “slices” with similar levels which may approximately match the energy distribution of the image, as shown in FIGS.26A-26D. Particularly, FIG.26A illustrates a layered approach for a first subframe. FIG.26B illustrates a layered approach for a 15thsubframe. FIG.26C illustrates a layered approach for a 100thsubframe. FIG.26D illustrates the integrated levels of the layered approach after 128 subframes, and is the sum of all the subframe target sub-images. The layered approach allocates lenses to a maximum number of image regions in each subframe, producing sub-images that look approximately similar from subframe to subframe. In one instance, the layered subframe decomposition divides level-discretized low-resolution images by the number of subframes, and rounds each slice such that the slices have a discrete amount of levels. Further lens allocation and image allocation methods may also be implemented to increase diversity in the image.

[0122] In another implementation, a level approach is implemented that allocates all the levels in one region before moving onto other regions for the next subframe, according to the selected image scanning method. An example of the level approach is shown in FIG.27. The level approach results in phase solutions that target the fewest number of regions, and thus offer the highest diversity. However, when implementing the level approach, the scanning method may become visible on screen, as only parts of the image is generated at any one time.

[0123] In yet another implementation, a random approach is implemented that randomizes the levels across subframes to create subframe targets, such as halftone levels. The random approach randomly selects which levels and regions to process for each subframe. An example of the random approach is illustrated in FIGS.28A-28D. FIG.28A illustrates a random approach for a first subframe. FIG.28B illustrates a random approach for a 13th subframes. FIG.28C illustrates a random approach for a 128th(final) subframe. FIG.28D illustrates the integrated levels using the random approach on all subframes. The random approach, with a simple raster for both the image plane and modulator plane, provides a compromise between diversity and efficiency.

[0124] Subframe decomposition, together with the image allocation methods and modulator allocations methods, provide a Z-Collapse algorithm for solving for an image. In some instances, a global approach may be implemented that solves for all subframes at once. Forexample, a Vogel solver may be combined with temporal Z-Collapse. The Vogel Approximation Method finds a quasi-optimal allocation that minimizes transportation cost (for example, minimizes the amount of steering by modulators, increasing efficiency). In some implementations, the Vogel Approximation Method distributes the levels across subframes first, preserving the locality between subframes. While this increases intensity by adding the levels, diversity between subframes is reduced. The Vogel Approximation method processes both the modulator and image regions together. Vogel Approximation may be performed for each subframe, or for the entire system at once.

[0125] In some instances, a feedback loop is implemented alongside the decomposition approach such that the target image (which may be a low-resolution image) is compared to the currently achieved levels and is updated so that the final level distribution approximately matches the original image. The currently achieved levels may be computed by a lightfield simulation. The feedback may occur every subframe, or occur after several subframes repeatedly, depending on the desired granularity and desired level of complexity.

[0126] During feedback, the target image is compared to a lightfield simulation that models the levels currently achieved. The lightfield model may range from a simple model that scales the energy according to the steering distance (low complexity) to a full diffraction model that accurately simulates light propagation (high complexity). The diffraction model bridges the gap between the Z-Collapse algorithm that works with discrete levels and a realistic outcome where the energy achieved is dependent on the steering efficiency, the spot shape, the spot size, and other complex interactions between the lenses.

[0127] FIGS.29A-29F illustrate an example temporal Z-Collapse feedback loop. FIG.29A illustrates a level-discretized low-resolution target image for a current feedback cycle consisting of N subframes. The image of FIG.29A may be updated at each feedback cycle and remains the target for the next N subframes. FIG.29B illustrates the integrated requested levels for all the previous subframes the target image of FIG.29A. FIG.29C illustrates the diffraction model for the integrated reconstruction. FIG.29D illustrates a diffraction model of the achieved levels on the reconstruction grid. FIG.29E illustrates the error between the target image of FIG.29A and the image of FIG.29B. The error of FIG.29E indicates where energy is missing and where energy is in excess. The error is used to update the target image for the next cycle, shown in FIG.29F.

[0128] FIG.30 is a flow chart summarizing yet another example method 3000 for generating beam-steering drive values from image data. The method 3000 may be referred to as a “GS- BLAS” method. In the example embodiment, method 3000 is performed by beam-steering drive module 224, utilizing data from other elements of controller 110. In alternate embodiments some steps of method 3000 can be performed by other elements and / or system designers / users. In a first step 3002, a mathematical description of a wavefield corresponding to a desired image (lightfield) at an amplitude modulator is generated. The wavefield includes an amplitude distribution that is derived from the desired image (via image data describing the desired image) and assuming zero-phase (i.e. all the light waves arriving at the amplitude modulator are in phase). Next, in a second step 3004, the mathematical description of the wavefield at the amplitude modulator is multiplied by a propagation operator to generate a mathematical description of the wavefield at a beam-steering modulator (back propagated). Then, in a third step 3006, the mathematical description of the wavefield at the beam-steering modulator is filtered. Filtering the mathematical description of the wavefield includes filtering the mathematical description using a low pass filter and removing amplitude information. Low pass filtering increases / controls diffraction efficiency of generated phase drive solutions by constraining the steering angles of the solutions to relatively small steering angles. The steering angles can be constrained to any desirable predetermined angle. The amplitude information is removed, because the beam-steering SLM is capable of phase modulation only. Next, in a fourth step 3008, the filtered mathematical description of the wavefield at the beam-steering modulator is multiplied by the propagation operator to generate a mathematical description of the wavefield at the amplitude modulator (forward propagated). Then, at a decision block 3010, it is decided whether to terminate method 3000 or to continue. Method 3000 is terminated when the mathematical description of the wavefield at the amplitude modulator is sufficiently similar to the desired image. If the termination condition is not met, method 3000 returns to step 3004. If the termination condition is met, method 3000 continues to a fifth step 1312, wherein beam- steering drive values are generated, based, at least in part, on the filtered mathematical description of the wavefield at the beam-steering modulator (as generated in the latest iteration of step 3006). Method 3000 will be discussed in more detail with reference to FIGS.31A-31B below.

[0129] FIG.31A is a diagram illustrating backward propagation of a wavefield between amplitude modulator 106 and the beam-steering modulator 104. In the example of FIG.31A, assume the beam-steering modulator plane is coplanar with the amplitude modulator plane. However, in other instances, the beam-steering modulator plane may not be coplanar with theamplitude modulator plane. An emitting surface of beam-steering modulator 104 is coplanar with a first coordinate system 3102, which includes an ^^-axis 3104 and a ^^-axis 3106. A lightfield surface of amplitude modulator 106 is coplanar with a second coordinate system 3108, which is separated from first coordinate system 3102 by a distance ^^ and includes a ^^′-axis 3110and a ^^′-axis 3112. Beam-steering module 224 generates a function ^^^^^ᇱ, ^^ᇱ^, which is amathematical description of the desired lightfield at amplitude modulator 106, as described instep 3002 of method 3000. The function ^^^^^ᇱ, ^^ᇱ^ includes an amplitude distribution, which isequivalent to the intensity distribution ^^^^^ᇱ, ^^ᇱ^ of the desired image at amplitude modulator106. The function ^^^^^ᇱ, ^^ᇱ^ is also zero-phase, in this example embodiment, because generatinga random phase considerably reduces efficiency of a liquid crystal beam-steering device. In addition, the algorithm is deterministic in that it always produces the same result for a given same input image. In alternate embodiments utilizing multi-element mirror devices it is possible to use a random phase to generate the function ^^^^^ᇱ,^^ᇱ^, because there is no efficiency cost associated with steering light at oblique angles.

[0130] Next, beam-steering module 224 back-propagates the function ^^^^^ᇱ,^^ᇱ^ to generate afunction ^^^^^, ^^^, which is a mathematical description of the desired lightfield at beam-steeringmodulator 104, as described in step 3004 of method 3000. Beam-steering module 224 generates the function ^^^^^,^^^: ^^^^^,^^^ ൌ Ƒି^^ Ƒ^ ^^^^^ᇱ,^^ᇱ^^ Gିௗ^^, γ^^where Ƒ^^^^^^^^ is the Fourier transform of ^^^^^^, Ƒି^^^^^^^^^ is the inverse Fourier transform of^^^^^^, and Gିௗ^^, γ^ is the Rayleigh-Sommerfeld propagation operator in terms of angularfrequencies and at a distance െ^^. In general, the Rayleigh-Sommerfeld propagation operator generates a wavefield anywhere in space that results in / from the input wavefield. In the example embodiment, the propagation operator generates the wavefield at beam-steering modulator 104 that will result in the desired image at amplitude modulator 106.

[0131] It should be noted that other propagation operators can be used based on the particular application of method 3000. The Rayleigh-Sommerfeld diffraction model is useful for applications that require a high degree of precision, because the model makes no simplifying assumptions and calculates every term of the diffraction equation. Other models, such as the Fresnel diffraction model and the Fraunhofer diffraction model make simplifying assumptions to eliminate higher order terms that do not significantly contribute to the diffraction equation in particular systems. The Fresnel diffraction model may be realized by evaluation of an analytical integral, by convolution in the Angular Spectrum (assuming the Fresnel kernel) or by evaluationusing a Fourier Transform substitution in the integral (referred to as a Fresnel Transform). For example, the Fresnel diffraction model calculates only the first and second order terms and is useful for situations where beam-steering modulator 104 and amplitude modulator 106 are relatively close together. Conversely, the Fraunhofer diffraction model calculates only the first order term and is useful for the situation where beam-steering modulator 104 and amplitude modulator 106 are relatively far apart.

[0132] Beam-steering module 224 filters the function ^^^^^,^^^, as described in step 3006 of method 3000. Because beam-steering modulator 104 can only modulate phase and not amplitude, the wavefield at beam-steering modulator 104 must have a constant amplitude. Therefore, beam-steering module 224 sets the amplitude distribution to a constant value (1 in the example embodiment) to generate the phase function at the beam-steering modulator: ^^^^^^,^^^ ൌ ^^^^^^^ொ^ఝ^where ^^ is the imaginary unit and ^^^^^^ is the phase distribution. The phase distribution includes the angular spectrum of the wavefield at each spatial location of the beam-steering modulator 104, which is a distribution of the various phase shifts (expressed in angular units, such as radians) that beam-steering modulator 104 introduces in the wavefield at each spatial location (or modulation cells, such as in the case of a piston mirror of a PLM). The Fourier transform of ^^^^^,^^^ yields the angular spectrum of the wavefield at the beam steering modulator.

[0133] FIG.31B is a diagram illustrating the forward propagation of the phase function ^^^^^^,^^^ from beam-steering modulator 104 to amplitude modulator 106. First, the phase is quantized to the bit-depth of beam-steering modulator 104 to account for errors that occur from being unable to replicate the phase to arbitrary precision. Additionally, the angular spectrum is filtered to contain only angular frequencies smaller than a specified threshold ^^ ^ ௧ ൌ ଶே∆, where ^^ is the phase step count (FIG.7) and ∆ is the pixel pitch of beam-steeringThe wavefield at beam-steering modulator 104 is constrained by the following low-pass filter: ^^^ଶ ^ ^^ଶ ^ ^^௧where ^^, ^^ ∈ ^െ^^ ^^௫, ^^^^௫^ are ^^- and ^^- directions, respectively,and ^^^^௫is the maximum angular frequency achievable by beam-steering modulator 104.Beam-steering module 224 generates the filtered spectrum Ƒ ^^^^^^^,^^^^^^^^, γ^ by multiplying thelow-pass filter into the angular spectrum of the wavefield at beam-steering modulator 104.Then, beam-steering module 224 generates a new wavefield at amplitude modulator 106, as described in step 3008 of method 3000, as follows: ^^^^^^′,^^′^ ൌ Ƒି^^ Ƒ^ ^^^^^^,^^^^ ^^^^^, ^^^ Gௗ^^, γ^^

[0134] The wavefieldimage that models both thenoise introduced by quantization and the phase-only nature of beam-steering modulator 104. Next, beam-steering module 224 replaces the amplitudes of the wavefield ^^^^^^′,^^′^ with thesquare root of the original intensity distribution ^^^^^ᇱ, ^^ᇱ^ and utilizes ^^^^^^′,^^′^ to begin the nextiteration of method 3000. In some instances, the beam-steering module 224 replaces theamplitudes of the wavefield ^^^^^^′,^^′^with the amplitude distribution of the desired lightfield^^^^^′,^^′^at the amplitude modulator 106.

[0135] Compensation for relative tilt between the beam-steering modulator and the amplitude modulator can also be provided. For example, a rotation in the angular spectrum can provide such compensation. See, for example, S. De Nicola, A. Finizio, G. Pierattini, D. Alfieri, P. Ferraro, “Reconstruction of digital holograms on tilted planes”, Proc. SPIE 6311, Optical Information Systems IV, 63110K (2006 / 08 / 30); doi: 10.1117 / 12.683859; http: / / dx.doi.org / 10.1117 / 12.683859, which is incorporated herein by reference. Then, the propagation operation, filtering, and angular rotation can all be performed in the same angular spectrum. As a result, the tilt compensation can be performed at no additional complexity cost.

[0136] Beam-steering module 224 terminates method 3000 when a particular predetermined termination condition is met. In the example embodiment, method 3000 is terminated when the total power of the wavefield at beam-steering modulator 104 does not increase past a predefined threshold between consecutive iterations. At every iteration ^^, the total power is calculated over the ROIs of the image (every pixel in the reconstructed wavefield where the corresponding pixel in the original image isn’t black). The power is calculated as follows: ^^^^^,^^^ ൌ ^1 ^^^^^,^^^ ് 0^^^^^^^^

[0137] Beam-steering when the following condition is met: ห^^^ െ ^^^ି^ห^ ^^௧where ^^௧is the relative power increase between iterations and is chosen empirically. After the termination condition is met, it may be desirable to continue to perform iterations without filtering, as follows: ^^^^^^′,^^′^ ൌ Ƒି^^ Ƒ^ ^^^^^^,^^^^ Gௗ^^, γ^^

[0138] At this point, each successive iteration will add improvements to the quality of the reconstructed image at the expense of an efficiency penalty. Once the unfiltered iterations areterminated, beam-steering module 224 extracts the phase of the final version of ^^^^^^, ^^^ andquantizes it to the bit-depth of beam-steering modulator 104. Beam-steering module 224 then utilizes the resulting phase distribution to generate beam-steering drive values for driving beam- steering modulator 104, as described in step 3012 of method 3000.

[0139] Several variations of method 3000 can also be used to generate beam-steering drive values, based on particular efficiency needs, image quality requirements, etc. For example, thelow pass filter ^^^^^, ^^^ can be trivially adjusted to generate a solution that does not steer light atspecified angles, as follows: ì1 ^^^ଶ ^ ^^ଶ ^ ^^௧భ ^^^^^ேwhere 0 ^ ^^^ ^ ⋯ ^ ^^ே ^ ^^௧on the particular needs of theapplication. Such a variation can be useful for systems where it is desirable to block DC- frequency light, such as reflections from the surface of beam-steering modulator 104. In addition, other types of filters including, but not limited to, Gaussian filters, sinc filters, and so on.

[0140] Another variation includes altering a filter ^^ெ^^^, ^^^ after successive iterations. Forexample, the initial filter ^^^^^^, ^^^ can be set as follows:1 ^^^ଶ ^ ^^ଶ ^ ^^^^^^^^^, ^^^ ^where ^^^is chosen to maximize the steering modulator 104. After each iteration, the efficiency constraint is progressively relaxed. This allows beam-steering module 224 to test the reconstructed image for visual quality metrics after each iteration and to terminate method 3000 when the image quality is deemed acceptable as compared to predefinedquality requirements. As another variation, the filter cut-off threshold can be derived from computing an F(efficiency) = quality estimator curve for a given input image (prior to running the iteration algorithm) and selecting the best compromise between efficiency and quality.

[0141] Yet another variation includes utilizing a pre-calculated phase drive instead of zero-phase for the initial phase distribution of the function ^^^^^ᇱ, ^^ᇱ^. For example, another of themethods described in this disclosure can be used to generate a low-resolution phase-drive, which is then forward propagated, according to step 3008 of method 3000. The resulting propagatedfield can then be used as the function ^^^^^ᇱ, ^^ᇱ^ for step 3004 of method 3000. Additionally, thefilter threshold ^^௧can be chosen so that the diffraction efficiency of the resulting solution is no worse than the initial, low-resolution phase drive.

[0142] As previously mentioned, the method 3000 is repeated until the mathematical description of the wavefield at the amplitude modulator is sufficiently similar to the desired image. The method 3000 may be updated to compute for multiple subframes. As the method 3000 may start from a random (or pseudo-random) seed, angular diversity is generated across multiple solutions with different seeds.

[0143] One method for computing multiple subframes includes the generation of a virtual diffuser in the illumination field (arriving at the modulator) and / or the reconstruction field when performing the method 3000. By using a virtual diffuser, the method 3000 may only be performed once for the frame rather than being performed for every subframe. The subframes are generated by applying the virtual diffuser pattern to the solution. In some instances, applying the virtual diffuser pattern allows for a reduced number of diffractive propagation operations.

[0144] Another method includes reducing or eliminating the iterative nature of the method 3000 (e.g., removing block 3010) and instead expanding the temporal extent of the solution space. For example, in one instance, the method 3000 may be performed ten times to generate an acceptable solution. Instead, to increase angular diversity and reduce computational complexity, the method 3000 is performed a single time and ten subframes are generated with similar computational complexity.

[0145] Yet another method includes applying a filter, such as a high-pass filter, to the initial seed of the method 3000 to increase angular diversity. For example, the seed initially provided to the method 3000 contains a random selection of steering angles that seeds the final steering solution. A filter may be applied such that the initial seed is biased toward the desired high steering angles.

[0146] A further method includes utilizing a pre-computed optimized phase to seed the method 3000. The optimized phase is generated by pre-performing the method 3000 to generate “base” light fields. As these solutions are pre-computed, they are performed with many iterations and the high-quality result is stored in memory. Examples of “base” light fields may include flat fields at different APL levels, or spatial distributions of light that are used as an initial prediction for different target images.

[0147] Another example provides an encoding scheme that implements the properties of a Fresnel diffraction transform, allowing the computing of many subframe solutions (e.g., driving schemes for the beam-steering modulator 104) per frame to yield increased steering diversity. Increased steering diversity may be at the expense of a lower reconstruction quality per subframe that, when integrated together, still provide frame reconstructions at an acceptable quality for use in the projection system 100.

[0148] FIG.32 illustrates a block diagram of a process for performing one forward propagation of the BLAS propagator. In the example of FIG.32, the beam-steering modulator 104 has a resolution of NxM pixels. A 2x padding 3204 of the phase modulation samples 3202 with zeros may be performed to ensure enough spectral bandwidth for a FFT convolution. The zero-padded phase modulation samples is convolved with a 2Nx2M BLAS kernel 3206. The FFT convolution window 3207 then has a size of 2Nx2M to compute a single forward (or backward) propagation of the BLAS propagator. The convolution generates residual data 3210 that includes NxM reconstruction samples 3208. In some implementations, the GS-BLAS algorithm performs 2 propagations per iteration and 3 iterations are performed in total. Since the beam-steering modulator 104 may have a 2K resolution and may subframes are required per video frame to achieve high steering diversity, the GS-BLAS algorithm requires a large processing bandwidth that may be achieved using parallel GPU computing with multiple GPUs. To reduce the computing cost, a Fresnel Transform can be used instead of the BLAS propagator.

[0149] FIG.33 illustrates a block diagram of a process for performing a single forward propagation using a Fresnel transform. As opposed to FIG.32, the Fresnel transform does not perform zero-padding, nor does the Fresnel transform perform a FFT convolution. Rather, the Fresnel transform is computed with a single FFT and a pointwise multiplication. For example, the NxM phase modulator samples 3302 are multiplied with NxM Fresnel FFT operator 3304 at NxM FFT multiplication 3305 to generate NxM reconstructed samples 3306. However, even with one NxM FFT per propagation, the total bandwidth to compute a subframe solution may still be greater than desired if many subframes are to be computed in real-time. To decrease thecomputational complexity without decreasing the number of iterations, the FFT size may be reduced in a propagation.

[0150] FIG.34 illustrates a relationship between sampling of the beam-steering modulator 104 and reconstruction using the BLAS propagator. The BLAS propagator converts NxM phase modulator samples 3402 to NxM reconstruction samples 3404. Both the phase modulator samples 3402 and the reconstruction samples 3404 have a pixel pitch of Δx, Δy. In practice, the sampling pitch corresponds to the physical dimensions of a micromirror (or any other type of phase modulation element, such as liquid-crystal cells in LCOS devices) of the beam-steering modulator 104, and the NxM samples to the resolution of the beam-steering modulator 104. A BLAS propagation of a field sampled at the modulation plane to any plane along the propagation axis (z in the example of FIG.34) yields a reconstructed field with the same sample pitch as the modulation field. For example, modulator aperture 3406 sampled along the propagation axis yields reconstruction window 3408 with the same pitch. Similarly, a smaller modulator aperture 3410 with N / 2 x M / 2 samples sampled along the propagation axis yields a smaller N / 2 x M / 2 reconstruction window 3412. Therefore, reducing the number of samples of the modulation field while maintaining the sampling pitch yields a reconstructed field the same size of the modulation field.

[0151] The relationship between sampling of the beam-steering modulator 104 and reconstruction in Fresnel propagation is illustrated in FIG.35. In Fresnel propagation, the sampling pitch of the reconstructed field is not the same as the sampling pitch of the modulation field. Rather, the sampling pitch of the reconstructed field is dependent on the propagation distance z, the number of samples, and the sample pitch of the modulation field. This dependency is referred to as the Fresnel transform reconstruction autoscaling property. For the purpose of this example, the propagation distance z is selected such that both the BLAS and Fresnel transform reconstructions are of the same physical size (Δx*N, Δy*M). As shown in FIG.35, reducing the number of samples of the modulation field yields a reconstructed field with the same number of samples but a larger sample pitch. This larger sample pitch results in a reconstruction that is approximately the same size as the modulation field. For example, NxM phase modulator samples 3502 with pixel-pitch Δx, Δy, after Fresnel propagation, becomes NxM reconstruction samples 3504 with pitch Δu, Δv. An NxM modulator aperture 3506 with NxM samples and pitch Δx, Δy sampled along the propagation axis yields NxM reconstruction window 3508 with pitch Δu, Δv. An NxM modulator aperture 3510 with N / 2 x M / 2 samples and pitchΔx, Δy sampled along the propagation axis yields N / 2 x M / 2 reconstruction window 3512 with pitch Δu, Δv.

[0152] FIGS.36A-36C illustrate a process of a phase modulator drive encoding scheme that leverages the autoscaling property of the Fresnel transform. In FIG.36A, the beam-steering modulator 104 is subdivided into K=4 smaller submodulator regions 3600 (each having a size of N / 2 x M / 2 and a pitch of Δx, Δy) and for each a phase subsolution is computed using iterative algorithms previously described herein. In instances where the beam-steering module 104 is subdivided into K=4 smaller submodulator regions 3600, the Fresnel transform may be referred to as GS-Fresnel-SS2x (see FIG.37). As shown in FIG.36B, each phase subsolution reconstructs the full extent of the image, and are not partial sections of the image. The K computed phase subsolutions are assembled in K different phase subframes, each making the full resolution of the beam-steering modulator 104, such that each subsolution does not occupy the same location, shown in FIG.36C. The phase of each subsolution is corrected by a tilt factor making the K individual reconstructions in a subframe align into a common reconstruction window. Therefore, if, for example, 24 subframes are to be displayed per frame and a modulator subdivision of K=4 is performed, then 24 different subframes are generated by only computing 24 smaller N / 2 x M / 2 phase subsolutions using the Fresnel iterative algorithm.

[0153] In some instances, in a coherent or partially-coherent system, the superpositions of K reconstructions from K subsolutions laid spatially in a subframe phase solution may introduce interference effects into the final reconstructed image. The interference effects may not impact the average brightness across the entire reconstruction, but could randomly raise or lower the brightness of individual image features.

[0154] In one example implementation to reduce interference effects, the tilt factor that is applied to each K subsolution may be randomized in a subframe phase solution such that each subsolution is shifted off-center by a sub-pixel amount. The shift changes the phase delays from each K subsolution, which randomizes the interference effects in a per-subframe basis and temporally averages the subframes to the desired image levels.

[0155] In another example implementation to reduce interference effects, random phase delay may be added to an entire subframe subsolution, which changes how the subsolution interferes with reconstructions of other subsolutions. In such an example, the number of different subframes generated from computing 24 smaller N / 2 x M / 2 phase subsolutions can go higher than 24 subframes, as adding a random phase offset to a phase subsolution creates adifferent phase subsolution that reconstructs a variation of the same subframe image but with different interference effects (such as noise). Such effects may be averaged out temporally.

[0156] In yet another example implementation to reduce interference effects, a random diffuser state of a virtual diffuser may be added to a subframe phase subsolution. The random diffuser state may also change how the reconstruction of the phase subsolution interferes with the reconstruction of other subsolutions.

[0157] Embodiments described herein are not limited to only using Fresnel transforms, and other propagators with autoscaling properties may be implemented in place of Fresnel transforms.

[0158] FIG.37 illustrates a plot showing the relationship between the computational complexity and the number of subframes for the GS-GLAS, the GS-Fresnel, and the GS-Fresnel- SS2x algorithms. In the example of FIG.37, each algorithm assumes 5 iterations per subframe and a beam-steering modulator 104 having a resolution of 2K.

[0159] The above description provides projection systems with increased angular diversity at an image reconstruction plane. Systems, methods, and devices in accordance with the present disclosure may take any one or more of the following configurations.

[0160] (1) A method for displaying images, the method comprising: receiving image data indicative of at least one image to be displayed; dividing, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions; dividing, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions; identifying a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data; generating, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator; and generating, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions, wherein the beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

[0161] (2) The method according to (1), wherein the beam-steering drive values are associated with a phase representation of a virtual lens.

[0162] (3) The method according to any one of (1) to (2), further comprising: partitioning the full plane drive values into a grid of beam-steering drive values and partitioning the plurality of image regions into a grid of discretized levels representing energy of the at least one image to be displayed.

[0163] (4) The method according to any one of (1) to (3), further comprising: generating the at least one image using the beam-steering drive values, wherein the at least one image includes a plurality of image display levels, and wherein a number of the plurality of image display levels is based on a number of the plurality of subframes.

[0164] (5) The method according to any one of (1) to (4), further comprising: adding, for each subframe of the plurality of subframes, random offset to the beam-steering drive values.

[0165] (6) The method according to any one of (1) to (5), further comprising: randomly allocating, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions.

[0166] (7) The method according to (6), wherein the random allocation is different for each subframe of the plurality of subframes.

[0167] (8) The method according to any one of (1) to (7), further comprising: allocating, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions using a raster scanning method.

[0168] (9) The method according to any one of (1) to (8), further comprising: computing a first image generated by the beam-steering drive values using a lightfield simulation; comparing the at least one image to the first image; and adjusting the beam-steering drive values based on the comparison.

[0169] (10) The method according to any one of (1) to (9), further comprising: allocating the beam-steering drive values to a maximum number of image regions for each subframe of the plurality of subframes, thereby generating a plurality of sub-images that are approximately equal in each of the plurality of subframes.

[0170] (11) A non-transitory computer-readable medium storing instructions that, when executed by a processor of a projection system, cause the projection system to perform operations comprising the method according to any one of (1) to (10).

[0171] (12) An apparatus for controlling a dual-modulation projection system, the apparatus comprising: an electronic processor coupled to a memory, the memory storing instructions preformedprocessor, wherein, when the electronic processor performs the instructions, the electronic processor is configured to: receive image data indicative of at least one image to be displayed; divide, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions; divide, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions; identify a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data; generate, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator; and generate, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions, wherein the beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

[0172] (13) The apparatus according to (12), wherein the beam-steering drive values are associated with a phase representation of a virtual lens.

[0173] (14) The apparatus according to any one of (12) to (13), wherein the electronic processor is configured to: partition the full plane drive values into a grid of beam-steering drive values and partition the plurality of image regions into a first grid of discretized levels and a second grid of discretized levels, the first grid and the second grid representing energy of the at least one image to be displayed.

[0174] (15) The apparatus according to any one of (12) to (14), wherein the electronic processor is configured to: generate the at least one image using the beam-steering drive values, wherein the at least one image includes a plurality of image display levels, and wherein a number of the plurality of image display levels is based on a number of the plurality of subframes.

[0175] (16) The apparatus according to any one of (12) to (15), wherein the electronic processor is configured to: add, for each subframe of the plurality of subframes, random offset to the beam-steering drive values.

[0176] (17) The apparatus according to any one of (12) to (16), wherein the electronic processor is configured to: randomly allocate, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions.

[0177] (18) The apparatus according to any one of (12) to (17), wherein the electronic processor is configured to: allocate, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions using a raster scanning method.

[0178] (19) The apparatus according to any one of (12) to (18), wherein the electronic processor is configured to: compute a first image generated by the beam-steering drive values using a lightfield simulation; compare the at least one image to the first image; and adjust the beam-steering drive values based on the comparison.

[0179] (20) The apparatus according to any one of (12) to (19), wherein the electronic processor is configured to: allocate the beam-steering drive values to a maximum number of image regions for each subframe of the plurality of subframes, thereby generating a plurality of sub-images that are approximately equal in each of the plurality of subframes.

[0180] (21) The apparatus according to any one of (12) to (20), wherein the beam-steering modulator has a resolution of 2048x1080, the beam-steering modulator is divided into a 32x36 grid of 64x30 modulator regions, and wherein the at least one image is divided into a 64x30 grid of 32x26 image regions.

[0181] (22) The apparatus according to any one of (12) to (20), wherein the beam-steering modulator has a resolution of 2048x1080, the beam-steering modulator is divided into a 32x36 grid of 64x30 modulator regions, and wherein the at least one image is divided into a 128x60 grid of 32x26 image regions.

[0182] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims.

[0183] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined withreference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0184] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0185] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

CLAIMS What is claimed is:

1. A method for displaying images, the method comprising: receiving image data indicative of at least one image to be displayed; dividing, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions; dividing, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions; identifying a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data; generating, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam- steering modulator; and generating, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions, wherein the beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

2. The method of claim 1, wherein the beam-steering drive values are associated with a phase representation of a virtual lens.

3. The method of claim 1 or 2, further comprising: partitioning the full plane drive values into a grid of beam-steering drive values and partitioning the plurality of image regions into a grid of discretized levels representing energy of the at least one image to be displayed.

4. The method of any one of claims 1 to 3, further comprising: generating the at least one image using the beam-steering drive values, wherein the at least one image includes a plurality of image display levels, and wherein a number of the plurality of image display levels is based on a number of the plurality of subframes.

5. The method of any one of claims 1 to 4, further comprising:adding, for each subframe of the plurality of subframes, random offset to the beam- steering drive values.

6. The method of any one of claims 1 to 5, further comprising: randomly allocating, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions.

7. The method of claim 6, wherein the random allocation is different for each subframe of the plurality of subframes.

8. The method of any one of claims 1 to 7, further comprising: allocating, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions using a raster scanning method.

9. The method of any one of claims 1 to 8, further comprising: computing a first image generated by the beam-steering drive values using a lightfield simulation; comparing the at least one image to the first image; and adjusting the beam-steering drive values based on the comparison.

10. The method of any one of claims 1 to 9, further comprising: allocating the beam-steering drive values to a maximum number of image regions for each subframe of the plurality of subframes, thereby generating a plurality of sub-images that are approximately equal in each of the plurality of subframes.

11. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a projection system, cause the projection system to perform operations comprising the method according to any one of claims 1 to 10.

12. An apparatus for controlling a dual-modulation projection system, the apparatus comprising: an electronic processor coupled to a memory, the memory storing instructions performed by the electronic processor, wherein, when the electronic processor performs the instructions, the electronic processor is configured to: receive image data indicative of at least one image to be displayed;divide, for each subframe of a plurality of subframes, a beam-steering modulator into a plurality of modulator regions; divide, for each subframe of the plurality of subframes, the at least one image into a plurality of image regions; identify a subset of image regions that contain at least one location having a brightness greater than a predetermined brightness based at least in part on the image data; generate, for each subframe of the plurality of subframes, a set of regional drive values, each set of the regional drive values corresponding to a full plane of drive values for the beam-steering modulator; and generate, for each subframe of the plurality of subframes, beam-steering drive values by combining the set of regional drive values based on the identified subset of image regions, wherein the beam-steering drive values for a first subframe of the plurality of subframes are angularly different than the beam-steering drive values for a second subframe of the plurality of subframes.

13. The apparatus of claim 12, wherein the beam-steering drive values are associated with a phase representation of a virtual lens.

14. The apparatus of claim 12 or 13, wherein the electronic processor is configured to: partition the full plane drive values into a grid of beam-steering drive values and partition the plurality of image regions into a first grid of discretized levels and a second grid of discretized levels, the first grid and the second grid representing energy of the at least one image to be displayed.

15. The apparatus of any one of claims 12 to 14, wherein the electronic processor is configured to: generate the at least one image using the beam-steering drive values, wherein the at least one image includes a plurality of image display levels, and wherein a number of the plurality of image display levels is based on a number of the plurality of subframes.

16. The apparatus of any one of claims 12 to 15, wherein the electronic processor is configured to: add, for each subframe of the plurality of subframes, random offset to the beam-steering drive values.

17. The apparatus of any one of claims 12 to 16, wherein the electronic processor is configured to: randomly allocate, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions.

18. The apparatus of any one of claims 12 to 17, wherein the electronic processor is configured to: allocate, for each subframe of the plurality of subframes, the set of regional drive values to the identified subset of image regions using a raster scanning method.

19. The apparatus of any one of claims 12 to 18, wherein the electronic processor is configured to: compute a first image generated by the beam-steering drive values using a lightfield simulation; compare the at least one image to the first image; and adjust the beam-steering drive values based on the comparison.

20. The apparatus of any one of claims 12 to 19, wherein the electronic processor is configured to: allocate the beam-steering drive values to a maximum number of image regions for each subframe of the plurality of subframes, thereby generating a plurality of sub-images that are approximately equal in each of the plurality of subframes.

21. The apparatus of any one of claims 12 to 20, wherein the beam-steering modulator has a resolution of 2048x1080, the beam-steering modulator is divided into a 32x36 grid of 64x30 modulator regions, and wherein the at least one image is divided into a 64x30 grid of 32x26 image regions.

22. The apparatus of any one of claims 12 to 20, wherein the beam-steering modulator has a resolution of 2048x1080, the beam-steering modulator is divided into a 32x36 grid of 64x30 modulator regions, and wherein the at least one image is divided into a 128x60 grid of 32x26 image regions.

Images