Method and apparatus for displaying video data and image data

Abstract

Methods and apparatus for displaying video data and image data are disclosed. A method for controlling a phase modulator to display an image defined by image data, the method comprising determining a proximal operator of an objective function based on the image data; transforming the proximal operator into a frequency space; evaluating the transformed proximal operator in the frequency space to obtain a phase function in the frequency space, and inverse transforming the phase function to obtain a dephasing function, the dephasing function relating a phase of the phase modulator to a two-dimensional position.

Description

[0001] This application was filed on May 15, 2020, with application number 202010413872.6, and is an invention.

[0002] This is a second-generation divisional application of a first-generation divisional application entitled "Method and Apparatus for Displaying Video Data and Image Data". The aforementioned first-generation divisional application is a divisional application of Chinese patent application filed on January 10, 2017, with application number 201580037846.5, entitled "Efficient Dynamic High-Contrast Lens Action Applied to Imaging, Illumination, and Projection". The international filing date of the parent application of the aforementioned first-generation divisional application is June 3, 2015, with international application number PCT / CA2015 / 050515.

[0003] Cross-reference to related applications

[0004] This application claims priority to U.S. Patent Application No. 62 / 007341, filed June 3, 2014, and U.S. Patent Application No. 62 / 118945, filed February 20, 2015. For U.S. purposes, this application claims the benefit of U.S. Patent Application No. 62 / 007341, filed June 3, 2014, entitled "DYNAMIC FREEFORM LENSING WITH APPLICATIONS TOHIGH DYNAMIC RANGE PROJECTION," and U.S. Patent Application No. 62 / 118945, filed February 20, 2015, entitled "EFFICIENT, NUMERICAL APPROACHES FOR HIGH CONTRAST FREEFORM LENSING," filed February 20, 2015, and both are incorporated herein by reference for various purposes. Technical Field

[0005] This invention relates to generating desired light patterns. In some embodiments, the desired light pattern corresponds to an image specified by image data. Specific embodiments provide methods for controlling freeform lenses, such as phase-shifting light modulators, variable mirrors, etc., to achieve desired light distribution. Other embodiments provide projectors for projecting light. Background Technology

[0006] Both light efficiency and dynamic range are major concerns in commercial projector design. Even though most images only require a small amount of locally very bright highlights above their average picture level to appear realistic [Rempel et al., 2011], high contrast and peak brightness are crucial for higher perceived image quality (luminance, chromaticity) [Rempel et al., 2009]. On the other hand, the optical system should be efficient to minimize power consumption and simplify thermal management. This latter issue makes achieving very high peak brightness by increasing the power of the projector light source impractical.

[0007] Amplitude spatial light modulators (or SLMs) are typically used to create hues and colors in an image by selectively blocking light through pixels. Because the blocked light is absorbed, such SLMs tend to be optically inefficient.

[0008] HDR (High Dynamic Range) image projection can be achieved by providing two or more stages of light modulators (Hoskinson et al.). Many light modulators (e.g., LCD panels) generate the desired light field through subtraction (i.e., by absorbing unwanted light). Some efforts have been made to create the desired light field by redistributing light. However, many available light redistribution techniques have significant drawbacks. For example, some require lasers, which can result in laser spots. Some are computationally very intensive. Some require very high spatial frequency control of the light, which places demands on the light modulator and also leads to artifacts caused by light diffraction.

[0009] Freeform lenses can be designed to produce specific caustic images under predefined illumination conditions [Finckh et al., 2010; Papa et al., 2011; Schwartzburg et al., 2014; Yue et al., 2014], and the freeform lens can be an aspherical, asymmetric lens. A caustic image is a redistribution or “redistribution” of light incident on a freeform lens [Hoskinson et al., 2010], and the computer graphics approach to designing such freeform lenses is called target-based caustics. Designing freeform lenses to achieve a specific desired image can be computationally intensive.

[0010] Freeform lenses can be applied to general lighting applications (e.g., [Minano et al., 2009]), and more specifically, to target-based caustics (Berry 2006; Hullin et al., 2013). Some methods for designing freeform lenses employ discrete optimization approaches that act on a pixelated version of the problem (e.g., [Papas et al., 2011; Papas et al., 2012; Papas et al., 2012]). Other optimizations for continuous surfaces do not exhibit a clear pixelated structure (e.g., [Finckh et al., 2010; Kiser et al., 2013; Pauly and Kiser 2012; Schwartzburg et al., 2014; Yue et al., 2014]).

[0011] Holographic image formation models (e.g., [Lesem et al. 1969]) are suitable for creating digital holograms [Haugen et al. 1983]. Holographic projection systems have been proposed for research and special applications [Buckley 2008]. Many of these systems use diffraction patterns (or holograms) addressed on a phase SLM combined with coherent light (laser) for image generation. While efficient in principle for image formation, the challenge in holography for projectors lies in achieving sufficiently good image quality. The limited diffraction efficiency achievable with binary phase modulators [Buckley 2008] and the requirement for Fourier lenses often result in bright DC spots in the effective image area or reduced contrast of the entire image due to increased black levels (in cases where DC spots are expanded). Holographic projection typically requires coherent light.

[0012] The inventors have recognized the need for more efficient methods to design freeform lenses to achieve desired light patterns. In particular, they have determined that sufficiently efficient design methods can be applied to provide real-time or near-real-time generation of dynamic freeform lenses. For example, such dynamic freeform lenses can deliver video content or dynamically changing light effects. Summary of the Invention

[0013] This invention provides a method for controlling the freeform lens effect of light provided by a spatial light modulator. The light can be projected and / or further modulated. Another aspect of the invention provides apparatus, such as projectors, displays, lighting systems, and components thereof, for implementing the methods described herein.

[0014] Dynamic freeform lenses can be applied to optical projection systems. Such systems can advantageously be light-efficient, providing high (local) peak brightness and high contrast (high dynamic range, HDR). Some implementations employ dynamic freeform lenses implemented on a phase-only SLM. The phase-only SLM can be combined with a conventional light-blocking SLM (e.g., a reflective LCD in a cascaded modulation method). When controlled as described herein, the phase modulator can produce a smooth yet highly detailed "caustic" image. If desired, such a caustic image can be further modulated by an amplitude modulator. Compared to conventional projectors, this method can provide both higher dynamic range and / or improved (local) peak brightness.

[0015] This application specifically describes:

[0016] • Lighting system and projector, in which (near)collimated light is used to illuminate the phase modulator, and

[0017] The phase pattern addressed on the phase modulator forms the desired light field or image with or without other optical elements;

[0018] • A Fourier domain optimization method for generating freeform lens configurations that can dynamically steer light at high frame rates using phase modulators;

[0019] • Real-time freeform surface lens effect algorithm and its application in lighting systems, projectors and video / image processing systems;

[0020] • A dual-modulation projector design that combines a phase modulator and an amplitude modulator for image generation and can operate using both broadband light and monochromatic light (such as laser).

[0021] The exemplary freeform lens optimization method is based on a first-order (paraxial) approximation, which is applicable to long focal lengths and widely used in optics. In this linear model, the local deflection of light is proportional to the gradient of the phase modulation function, while the intensity is proportional to the Laplacian operator. The phase modulation function can be solved, for example, using an optimization method on the lens surface rather than the image plane, to obtain a very simple implementation that directly optimizes the shape or phase function of the refractive lens without additional steps. This method can be solved very efficiently in the Fourier domain. In some implementations, the algorithm is sufficiently efficient for on-the-fly computation of freeform lens effect configurations used to reproduce video sequences.

[0022] One exemplary aspect provides a dual-modulation projector design in which a spatial light modulator that affects only the phase of the illumination is combined with a spatial light modulator that affects its amplitude (intensity). The phase-only modulator curves the wavefront of the light reflected from it and acts as a pre-modulator for the conventional amplitude modulator. This approach works with both white light and laser illumination to generate coarse image representations without significant energy loss.

[0023] Dual-modulation HDR projectors utilize a freeform lens optimization approach to deliver energy-efficient high dynamic range and high-intensity projection. This approach enables illumination using white light (or other broadband light) as well as coherent lasers. The use of broadband light significantly improves image quality by eliminating laser speckle and averaging other diffraction artifacts. Real-time implementation of high-resolution freeform lenses allows for the use of techniques such as video processing. Dual-modulation HDR projectors can be constructed entirely from currently available robust components.

[0024] In some implementations, the phase modulator creates a smooth but still fairly detailed "caustic" image on the amplitude modulator. Since the caustic image merely redistributes or "redistributes" the light, this method produces both higher dynamic range and improved (local) peak brightness compared to conventional projectors that use a single amplitude modulator to modulate the light.

[0025] Some implementations employ linear models, in which the local deflection of light is proportional to the gradient of the phase modulation function, and the intensity is proportional to the Laplace operator.

[0026] Some implementations combine the application of this model with the parameterization of optimization problems in the lens surface rather than in the image plane to obtain a very simple implementation that directly optimizes the shape or phase function of the refractive lens without any additional steps. Although the objective function is non-convex due to image distortion, operator convergence can usually be achieved within a few iterations.

[0027] The techniques described herein are applied, for example, to the control of dynamic freeform lenses in high-efficiency, high (local) peak brightness, and high-contrast (high dynamic range, HDR) projection systems.

[0028] Some aspects of this invention provide algorithms that can be applied to efficiently determine the phase pattern of a phase modulator to create a desired light distribution in the image plane. In some embodiments, a (near) one-to-one relationship is established between the phase at a location in the lens plane and the corresponding region in the image plane. This is in contrast to the diverging or converging rays or beams required by conventional holographic methods.

[0029] Other aspects and exemplary embodiments are shown in the accompanying drawings and / or described in the following description. Attached Figure Description

[0030] The accompanying drawings illustrate non-limiting exemplary embodiments of the present invention.

[0031] Figure 1 This is a schematic diagram of an exemplary geometry for image formation. Phase modulation occurs in the lens plane, which is positioned at a distance of focal length f from the image plane. This results in the curvature of the wavefront, which is represented by the phase function p(x).

[0032] Figure 2 This is a schematic diagram illustrating the intensity change caused by the distortion of the differential area dx.

[0033] Figure 3 This is a schematic diagram showing the geometry of refraction in a freeform lens defined by a height field h(x).

[0034] Figure 4 The stages in the algorithm for the lens effect of freeform surfaces are shown.

[0035] Figure 5C , Figure 5D and Figure 5E An example of a refractive lens manufactured using the method described herein is shown.

[0036] Figure 5A and Figure 5B A phase-only spatial light modulator is shown for using white light to drive a projector display. The same setup can also be used with laser illumination. This approach is particularly useful in energy-efficient dual-modulation HDR projectors. The image on the right shows a refractive lens designed using the same freeform lens algorithm for target-based caustics. For photographic purposes, both results are displayed on a backlit screen instead of a front-lit one, making the displayed "Lena" image appear mirrored.

[0037] Figure 6A and Figure 6B These are photographs of a prototype implementation. The layout of the narrowband dynamic lens effect testing setup includes a HeNe laser source, a beam expander, a linear polarizing filter and a folded mirror, a phase-only SLM, and a projection screen at a distance of 50 mm from the SLM. The SLM phase pattern used to generate the freeform lens (in this case, the Siggraph logo) is also displayed on a laptop screen for visualization. Note that Fresnel-like phase distortion is used to achieve a large phase transition. Bottom: The white light configuration bypasses the laser module and includes a white LED, collimating optics and a linear polarizing filter, a phase-only SLM, and a projection screen at a distance of 50 mm from the SLM. In this setup, the SLM is calibrated for a center wavelength of 550 nm.

[0038] Figure 7This is a system diagram of an example high-brightness HDR projector: light from an extended and collimated laser beam is reflected by a phase-only modulator. The phase delay per pixel is analogous to the height field of a dynamic lens calculated using an algorithm as described herein. The effective focal plane of this freeform lens lies in-plane with an off-the-shelf reflective projection head consisting of a polarizing beamsplitter, an LCoS microdisplay, and projection lenses. Light from the darker parts of the image can be used to create high-brightness features while simultaneously reducing black levels.

[0039] Figure 8 This is a system diagram of an example high-brightness HDR projector including an intermediate image plane, where light from the phase phase is further shaped, for example, by adding a light-shaping diffuser: light from an expanded and collimated laser beam is reflected by a phase-only modulator. The amount of phase delay per pixel is analogous to the height field of a dynamic lens calculated using an algorithm as described herein. The effective focal plane of this freeform lens lies in-plane with the intermediate image plane, which is relayed to an off-the-shelf reflective projection head via relay optics, the off-the-shelf reflecting projection head comprising a polarizing beamsplitter, an LCoS microdisplay, and a projection lens. Light from the darker parts of the image can be used to create high-brightness features while simultaneously reducing black levels.

[0040] Figure 9 The comparison of the simulation and capture results is shown row by row from top to bottom. Phase pattern: Phase pattern calculated by Algorithm 1. Simulation: Huygens-Fresnel simulation of the predicted image. Direct: Diffraction shown without a diffuser sheet. artifacts A photograph of the actual image. Diffuser: By adding a thin-film diffuser, artifacts such as diffraction fringes are almost completely reduced. Standard: A standard photograph, using amplitude-modulated projection with a single amplitude modulator, shows improved black levels and low contrast. Recommendation ( HDR Our lens effect method redistributes light from dark areas to bright areas, resulting in improved black levels and increased highlight intensity. The last two lines appear slightly distorted due to the camera's off-angle position, which was necessary due to short-projection, a closed screen, and a baffle to effectively block ambient light in order to capture the system's black levels.

[0041] Figure 10A , Figure 10B and Figure 10C :and Figure 8 The positions A through C, from left to right, are: A: Phase pattern present at the phase-only LCoS modulator; B: Direct image produced by a lens in the intermediate image plane (before the diffuser); and C: Intensity distribution present at the amplitude LCoS modulator after passing through the thin-film light-forming diffuser.

[0042] Figure 11A , Figure 11B and Figure 11C An exemplary high dynamic range projection system based on dual modulation is illustrated. The first stage modulates the phase of the source illumination to form a coarse intermediate image. This is followed by an amplitude modulation stage to form the final image. Because the light is redistributed rather than blocked, the use of phase modulation results in greater contrast and darker black levels compared to conventional projection.

[0043] Figure 12A The geometry for an image formation model is shown, in which phase modulation p(x) occurs in the lens plane, and the resulting deflection creates a caustic image on the image plane at a distance f. Figure 12B The results show that the local intensity on the image plane is related to the variation of the differential surface area between the lens surface and the corresponding patch on the image plane.

[0044] Figure 13A , Figure 13B and Figure 13C By mirroring the input image, pure Neumann boundary conditions at image edges can be achieved while preserving the Toeplitz matrix structure. This prevents distortion of image boundaries. Simulation results were obtained using LuxRenderTM.

[0045] Figure 14A , Figure 14B , Figure 14C and Figure 14D LuxRender ray tracing simulation: The smoothness parameter α is not conducive to achieving high brightness but poor image quality in strong caustics in images.

[0046] Figure 15 The layout of a simple, exemplary dynamic lens effect test setup is used with broadband light. A beam from a light source such as a white LED and collimating optics (a modified flash lamp), along with a linear polarization filter (set up for the proper use of a phase modulator), is reflected by an SLM operating in phase-only mode and arrives at a small projection screen 50 mm away from the SLM. In this setup, the SLM is calibrated for a center wavelength of 550 nm. Due to light engine power limitations, although this setup demonstrates that phase modulation works with broadband light, it is insufficient to drive a dual-modulation setup (the reduced intensity also introduces camera capture noise in the inlay). This paves the way for future broadband-illuminated phase + amplitude dual-modulation setups. For example, such a setup could use an industry-standard xenon bulb, a cost-effective blue laser + phosphor light source, or an LED as the light source.

[0047] Figure 16The setup for a single modulation test of a laser includes a light source (yellow box, 532nm DPSS laser and laser controller), beam spreading and collimating optics (orange box), a reflective phase SLM (blue), various folding mirrors, and a simple projection lens to relay the image from and between the image planes onto a projection screen (green). The phase pattern displayed on the computer screen is linearly correlated with the desired phase delay in the optical path to form the image. The phase pattern is phase-distorted at multiples of a wavelength and can be directly addressed onto the microdisplay SLM.

[0048] Figure 17 A simplified system diagram of an exemplary high-brightness HDR projector: Light from an extended and collimated laser beam is reflected by a phase-only modulator. The phase delay per pixel is analogous to the height field of a dynamic lens calculated using our algorithm. The effective focal plane of this freeform lens is in-plane with an off-the-shelf reflective projection head consisting of a polarizing beam splitter, an LCoS microdisplay, and projection lenses. Light from the darker parts of the image can be used to create high-brightness features while simultaneously reducing black levels.

[0049] Figure 18 The comparison of the simulation and captured results is shown row by row from top to bottom. Phase Pattern: The phase pattern calculated by Algorithm 4.1. Simulation: Huygens-Fresnel simulation of the predicted image. Direct: A photograph showing the actual image without diffraction artifacts. Diffuser: Artifacts such as diffraction fringes are almost completely reduced by adding a thin-film diffuser. Standard: A standard photograph; amplitude-modulated projection using a single amplitude modulator shows improved black levels and low contrast. Recommended (HDR): Light is redistributed from dark areas to bright areas using our lensing effect method, resulting in improved black levels and increased luminance intensity. The last two rows appear slightly distorted due to the camera's off-angle position, which was necessary due to short-throw projection and a closed screen and baffle to effectively block ambient light in order to capture the system's black levels.

[0050] Figure 19A and Figure 19B Photographs of the prototype projector in LDR comparison mode (left image) and HDR mode (right image). Left: Light redistribution is activated, resulting in increased peak brightness and reduced black levels. Right: LDR projector compared using the same hardware. In LDR mode, a flat phase distribution results in a uniform illumination distribution at the amplitude attenuator (second SLM). Each image is vertically split to show a long exposure time (where dark level details are visible) on the left and a short exposure time (where details in highlights are visible) on the right. Both exposures produce the same projected image on the screen. Detailed Implementation

[0051] Specific details are set forth in the following description to provide a more thorough understanding of the invention. However, the invention may be practiced without these details. In other instances, well-known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.

[0052] Freeform surface lens effect

[0053] Some implementations provide novel methods for determining the lens shape or phase function that provides the desired light field when illuminated. The output of this method can be used to control a phase modulator, variable lens, or variable mirror to obtain the desired light field.

[0054] In some embodiments shown, a phase-only SLM is used as a programmable freeform surface lens. The lens can be illuminated with broadband light (e.g., white light). This eliminates speckle, while the spatial smoothness of the lens modulation pattern reduces diffraction artifacts. Any remaining diffraction is averaged by the broadband characteristics of the illumination, resulting in only a small amount of blurring that can be modeled and compensated for in a dual-modulation setup.

[0055] Some implementations optimize the phase function or equivalently the lens shape directly without requiring a subsequent integration step. This is facilitated by parameterization that expresses the optimization problem directly in the lens surface rather than the image plane. This results in simpler equations for the freeform lens optimization problem compared to methods described in the literature.

[0056] Phase modulation image formation

[0057] This application relates in part to a method for displaying a desired light pattern using a modulator that does not absorb much light but moves the modulator around within the image plane. In this way, light can be redistributed from dark image areas to bright image areas. For example, the modulator can be controlled to provide moving bright spots. An example of a modulator suitable for this application is an LCoS SLM operating in a phase-only manner. The SLM can have a suitable resolution such as 1, 25, or greater megapixels. Control of the SLM can be achieved by optimizing a continuous phase function that represents the desired curvature of the light wavefront as light passes through the SLM.

[0058] Devices and methods, implemented in various ways, enable the use of broadband light (e.g., from lamps, LEDs, or arrays of lasers with different wavelengths) as well as monochromatic lasers. Phase modulation arrays, such as liquid crystal-based SLMs operating in a phase-only configuration, are applied as programmable freeform lenses. Broadband illumination can help eliminate screen speckle, while the spatial smoothness of the lens modulation pattern reduces other artifacts such as diffraction. Any residual diffraction effects in the image plane can be averaged by the broadband characteristics of the illumination, resulting in only a small amount of blurring that can be easily modeled and compensated for by setting one or more additional modulators.

[0059] A method for directly optimizing a phase function (i.e., the shape of the wavefront in the lens plane) or an equivalent lens shape without requiring a subsequent integration step involves parameterization that allows the optimization problem to be expressed directly in the lens plane rather than in the image plane.

[0060] To obtain an image forming model for phase modulation display, consider Figure 1 The illustrated geometric configuration involves placing the lens surface and the image plane (e.g., a screen) parallel to each other at a focal length f. Collimated light is incident from the normal direction at the lens surface. A phase modulator (or lens) in the lens surface deforms the phase of the light, resulting in a curvilinear phase function p(x) corresponding to a local deflection of the light ray. In a related embodiment, a variable reflector is provided in the lens surface.

[0061] The effect of phase delay introduced by the smoothed phase function can be related to the equivalent physical refractive lens under the paraxial approximation, which can be derived using geometric optics or according to the Hyugens principle. The paraxial approximation is applicable when sinθ≈θ. For projection systems with |θ|≤12° (in this example, the full range corresponds to redirecting light from one side of the image to the other), the error of the paraxial approximation is less than 1%. This is advantageous for direct optimization of the phase surface.

[0062] Using the simple paraxial approximation sinφ≈φ, which is effective for small deflection angles, it can be shown that the geometric displacement in the image plane is proportional to the gradient of the phase function.

[0063] Regarding the paraxial approximation sinφ≈φ that is effective for small deflection angles, it is obtained in 2D.

[0064]

[0065] In 3D, this leads to the following equation for the mapping between a point x on the lens surface and its corresponding point u on the image plane:

[0066]

[0067] Intensity Modulation

[0068] Using the above mapping, we can derive the intensity variation associated with this deformation. Let dx be the differential area on the lens surface, and let du = m(x)·dx be the differential area of ​​the corresponding region on the image plane, where m(.) is the spatial magnification factor. Then, the intensity on the image plane is given as...

[0069]

[0070] Here, i0 is the intensity of the collimated light incident on the lens surface. For simplicity, we will assume i0 = 1 below. This corresponds to uniform illumination of the lens surface.

[0071] The magnification factor m(.) can be expressed as the derivative of the mapping between the lens plane and the image plane (see also...). Figure 2 ):

[0072]

[0073] This yields the following expression for the intensity distribution on the image plane:

[0074]

[0075] In other words, the magnification m and therefore the intensity i(u) on the image plane can be calculated directly from the Laplace operator of the scalar phase function on the lens surface.

[0076] Optimization problem

[0077] Although the image formation pattern according to Equation 5 can be directly transformed into an optimization problem, we find that better convergence can be achieved by first linearizing the equation using the first-order Taylor approximation method. This yields...

[0078]

[0079] The left side can be interpreted as a distorted image on the lens surface, where the target intensity i(u) in the image plane is twisted backward using the distortion u(x) produced by the given phase function p(x).

[0080] Based on this image formation model, the following optimization problem can be constructed to determine the phase function p(x) of a given target image i(u):

[0081]

[0082] Among them, i p It is a distorted image The target intensity i(u) in the image plane is twisted backward onto the lens surface using the distortion u(x) generated by the given phase function p(x).

[0083] This optimization problem can be solved by iterating between updating the phase function and updating the warped image, as shown in the following exemplary algorithm 0:

[0084]

[0085] After directly discretizing i(.) and p(.) into pixels, the phase update corresponds to solving a linear least squares problem using the discrete Laplacian operator as the system matrix. This positive semi-definite system can be solved using any of several different algorithms, including conjugate gradient (CG), BICGSTAB, and quasi-minimum residual (QMR). Such algorithms can be executed programmatically. Image warping corresponds to simple texture mapping operations, which can be efficiently implemented on a GPU (Graphics Processing Unit).

[0086] exist Figure 4 The convergence behavior of the algorithm is shown in the figure. Figure 4 The algorithm is illustrated in six iterations. As the phase function p(k) converges toward the solution, the target image i is gradually deformed by backward warping onto the lens planes (i~k). The algorithm uses the undeformed target image to optimize the initial phase function. Using this phase function, the target image on the lens plane is updated by backward warping the target on the image plane. As the phase function converges, this process increasingly warps the target image on the modulator plane. Although the backward warping step implies a non-convex objective function, empirical evidence shows that convergence is achieved in only a relatively small number of iterations (5-10). The overall processing time can be further accelerated by first processing the lower image resolution and upsampling the results.

[0087] Solution in Fourier domain

[0088] The convergence speed of this algorithm can be further improved by understanding that its computational cost is primarily due to solutions to large-scale biharmonic problems. For example, the Krylov subspace method (QMR) can be employed; however, convergence is typically slow due to the difficulty in finding efficient preprocessors and the scale of the system. Algorithms for efficient solutions to biharmonic systems are an ongoing research topic, including, for example, preprocessing methods [Silvester and...].

[2004] , multi-grid methods [Zhao 2004], and operator splitting schemes [Tang and Christov 2006]. Scaled to the millions of degrees of freedom required for real-time imaging problems is extremely challenging.

[0089] The nearest-neighbor operator-based alternative approach enables the representation of the problem in the Fourier domain, and thus allows for efficient problem-solving using highly parallelizable Fast Fourier Transform libraries. This alternative approach allows for the acquisition of solutions in real-time or near real-time using commodity, low-cost data processors.

[0090] For example, as described in [Ng et al., 1999], mirror-filling of the input image makes the image filled by ▽ 4 The discretization of the system results in a periodic boundary condition with pure Newman boundary conditions at the nominal image edges. This is in Figure 3 As shown in the figure. This modification makes the product ▽ in the objective function (Equation 7) 4 p is represented as a convolution via the Fourier convolution theorem, which enables the use of faster Fourier domain solvers.

[0091] For periodic boundary conditions, this problem can be solved very efficiently in Fourier space using the nearest neighbor operator. The nearest neighbor method based on sparse optimization allows for regularization without compromising the system's structure.

[0092] For any convex function F(z), the nearest neighbor operator prox γF (As defined in Equation 8) It functions like a single step in trust region optimization, where the value of z decreases F but is not too far from the input variable q:

[0093]

[0094] For least squares objective The resulting nearest neighbor operator is shown in Equation 9.

[0095] prox γF (q)=(γ+A T A) -1 (γq+A T b) (9)

[0096] Because the nearest neighbor operator contains a strictly convex regularization term, the entire operator is a strictly convex function even if F is only weakly convex. This property of the nearest neighbor operator helps in designing algorithms with fast convergence. Direct fixed-point optimization algorithms, such as the nearest-point method [Parikh and Boyd 2013], utilize this by applying the nearest neighbor operator (i.e., z) to the target. k+1 =prox γF (z k The optimization of strictly convex or weakly convex functions is achieved by repeatedly evaluating F until it converges to a minimum value. Since the approximate regularization term can also be represented as a Toeplitz matrix (a simple identity matrix), the approximate regularization term does not break the loop structure of the problem, nor does it change the solution by performing unnecessary regularization.

[0097] By representing the forward Fourier transform and the backward Fourier transform as F() & F() respectively -1 (), through the complex conjugate of * and pointwise multiplication and division, for the Toeplitz matrix A, the nearest neighbor operator of Equation 7 can be re-expressed in the Fourier domain as Equation 10.

[0098]

[0099] A constant α ≥ 0 has been added to regularize the solver by making the solution satisfy low curvature. This is consistent with applying a formally modified... The corresponding solution to the adverse consequences is given by equation 7, such as...

[0100] As shown in Equation 11.

[0101]

[0102] The effect of parameter α is to provide a smoother solution compared to solutions that can be found in other ways. This helps prevent the method from producing unwanted caustics, attempting to achieve very bright highlights in darker areas at the expense of image quality. Figure 13 shows the effect of the α parameter on the simulation.

[0103] By limiting and And q = p k (x), the above problem can be solved iteratively in Fourier space using Algorithm 1. This change enables a nonlinear solution to be computed in each iteration using a forward / backward Fourier transform, an image warping, and some minor component forms. As shown in the figure, Equation 11 is a nonlinear variant of the Common Nearest Neighbor algorithm, i.e., the nearest point method, which is used to make the solution obtained by evaluating: p k+1 ←prox γF (p k ) by recursively calling prox γF A fixed-point algorithm to minimize any convex F.

[0104]

[0105] The reformulation of the algorithm results in it being several orders of magnitude faster than the QMR solver described above when executed on a CPU using an FFT-based solver. If the QMR solver takes 20 minutes or more to compute per frame, the Fourier version of Algorithm 1 takes approximately 0.6 seconds on a Core i5 desktop at the same resolution (256×128), representing a speedup of approximately 2000x. The transformation to the Fourier domain solution also makes it easier to implement operations that can run in parallel on one or more GPUs. The algorithm has been implemented in both C++ and CUDA using CUFFT for both forward and inverse Fourier transforms [NVIDIA]. When run on a GeForce 770 GPU, the CUDA & CUFFT version of the code achieves speedups nearly 150x faster than the single-threaded CPU version and approximately 300,000x faster than the native CPU version implemented using QMR. The algorithm described in this paper is the first freeform lens method known to the inventors capable of real-time operation, see Table 1. This contrasts with methods that produce satisfactory results, such as [Schwartzburg et al., 2014], but is about five orders of magnitude slower than our GPU algorithm. This currently prevents its use in systems capable of real-time projection.

[0106] algorithm resolution runtime CPU 256×128 600ms GPU 256×128 4ms GPU 480×270 14ms GPU 960×540 52ms GPU 1920×1080 212ms

[0107] Table 1: Running time for various resolution inputs using Algorithm 1 in 10 iterations

[0108] This algorithm is well-suited for hardware implementation on devices such as GPUs, FPGAs, or ASICs due to its use of highly parallel FFTs and component-based operations. Algorithm 1 is run for a fixed number of iterations (typically 10). Convergence to a solution is fast, requiring fewer than 10 iterations; however, for hardware implementations, computation time independent of frame content is highly desirable. The choice of the smoothing factor α can be somewhat content-dependent.

[0109] Simulation results

[0110] Using the equivalence between physical lenses and phase functions allows for the generation of solid lens models for testing via geometric optics simulations (using Blender+LuxRender). While these models may not satisfy the paraxial approximation, they are well-suited for rapid qualitative comparisons because thickness effects tend to manifest as low spatial frequency distortions. Examples are shown in Figures 12 and 13, illustrating the effect of mirror filling and the choice of α, respectively. It is important to note that because the prototype conforms well to the paraxial approximation, these distortions do not affect the prototype projector results.

[0111] When higher physical accuracy is required, Huygens-Fresnel simulations can be applied, which approximate (complex) incident illumination as a superposition of (complex) point sources. While the increased cost of simulations limits the resolution to levels where diffraction effects from discrete pixels need to be addressed, simulation results show... Figure 18 , Figure 19A and Figure 19B The results are consistent with experimental observations (see, for example, the caustics on Marilyn's nose in the "simulated" and "direct" images). Similarly, spots from the laser source are not modeled.

[0112] Based on these results, the following conclusions can be drawn: Phase modulation is performed as expected, and the main limitations to image quality are diffraction artifacts and speckles.

[0113] Static refractive lens

[0114] The phase function p(x) can be used directly to drive digital phase modulation displays (see below). However, alternatively, a refractive lens surface can be created using a transparent material, and then the phase function can be converted into a geometric model of the lens shape.

[0115] It is possible to model a lens shape that is flat on one side and has a freeform height field h(x) on the other side (see [link]). Figure 3 In the (x, z) plane, the deflection angle φ is related to the incident angle at the height field (√). i The angle of departure (θ0) is related to the angle of exit, as shown below.

[0116]

[0117] A similar relationship exists in the (y, z) plane.

[0118] Furthermore, the refractive index of the lens material is n. Using Snell's law and again the paraxial approximation, we can obtain...

[0119]

[0120] Using Equations 12 and 13, and The lens shape can be exported as

[0121]

[0122] Where h0 is the base thickness of the lens.

[0123] The height h(x) is a linear function of the phase. The refractive index n is simply a scalar multiplier of the phase function p(.). Since p itself is approximately linear at the focal length f, it can be seen that uniform scaling of the height field and uniform change of the refractive index simply act as a refocusing of the lens. This also shows that the exemplary optimization process presented above can be equivalently adjusted to directly optimize h(.) instead of p(.). The above formula is preferred only when, for example, a spatial phase modulator is used in a video projector where control is sought.

[0124] Figure 5B , Figure 5C , Figure 5D and Figure 5E Some exemplary 3D-printed refractive lenses are shown. Figure 5C The lens itself is shown, as well as Figure 5D and Figure 5E The caustics produced by the lens are shown (Lena image and Siggraph logo). Due to the resolution limitations of 3D printers, the lens size was optimized for large feature sizes, resulting in a short focal length.

[0125] Figure 5B , Figure 5C , Figure 5D and Figure 5E The results of target-based caustics using a freeform refractive lens generated using our method are shown. (VeroClear) TM The material was used to mount the lens on the Objet Connex 260 rapid prototyping machine. Figure 5C (As shown) 3D printing. Then, the lens is thoroughly cleaned, and the licensed side is manually polished using fine-grit sandpaper and polishing compound. This type of 3D printer has a layer thickness of 42μm, which limits the feature sizes that can be easily created.

[0126] As mentioned above, the model can be rescaled to achieve different focal lengths. To accommodate the resolution limitations of the manufacturing method, a very short focal length f is chosen (approximately 1″ for the Siggraph logo and approximately 5″ for the Lena image). While these scales test the limits of the paraxial approximation used in the derivation of the image forming model, the image quality remains quite good. Improving image quality and reducing feature size simultaneously through better manufacturing methods such as injection molding, high-precision milling, or even detailed manual polishing of 3D-printed surfaces makes far-field projection feasible.

[0127] Dynamic lens effect

[0128] To apply the concept of freeform lenses in projection displays, spatial light modulators capable of manipulating the shape of the wavefront of reflected or transmitted light can be used. Several different techniques can be employed for this purpose.

[0129] Several adaptive optics devices are inherently well-suited for real-time video capabilities. Such devices include display-based microelectromechanical systems (MEMS) such as analog 2D arrays of mirrors fabricated by [Hoskinson et al., 2012], or deformable mirrors used in wavefront sensing and correction applications. Continuous deformable mirrors [Menn et al., 2007] appear to be a particularly attractive option because they eliminate diffraction caused by regular pixel structures. While functional mirrors with numerous 4096 actuators have been reported, the spatial resolution of these MEMS-based devices remains several orders of magnitude lower than that of existing digital microdisplays conventionally used in digital projectors. This makes their use at this point less attractive in dual-modulation setups.

[0130] Some implementations advantageously utilize wavefront modulators based on liquid crystal display (LCD) technology. LCDs are typically configured as amplitude (intensity) modulators by sandwiching an LCD-based wavefront modulator between two linear polarization filters. However, when operating without a second polarizer, the LCD-based wavefront modulator delays (modulates) the phase of the transmitted light differently depending on the rotational state of the liquid crystal in each pixel. The electric field across the cell gap of each pixel controls the amount of phase delay. In principle, such a standard display is sufficient to achieve a dynamic lensing effect. However, there are also dedicated, commercially available microdisplays optimized to a) maximize the amount of phase delay (on the order of 2π and larger) and b) minimize the amount of polarization variation. Therefore, the pixel values ​​of this type of SLM directly correspond to the phase function p(.) derived above. The larger phase delay results in a steeper gradient on the lens surface due to the need for thicker cell gaps, but at the cost of faster switching speeds. If the phase change in the SLM does not affect the polarization state (“phase-only”), this makes it possible to combine the display with other optoelectronic components further along the optical path, particularly conventional amplitude SLMs for dual modulation purposes. For more information on this topic, see [Robinson et al., 2005].

[0131] The exemplary prototype implementation uses a reflective liquid crystal on silicon (LCoS) chip distributed by [HOLOEYE]. This chip has a spatial resolution of 1920 × 1080 discrete pixels at a pixel pitch of 6.4 μm and can be updated at up to 60 Hz. Access to a lookup table allows for modulation of the modulator to be calibrated for different operating wavelengths. Compared to other technologies, the display achieves a fill factor of up to 93% and a reflectivity of 75%. The phase delay is calibrated between 0 and 2π, equal to one wavelength of light. This is sufficient to produce a freeform lens with a long focal length. For shorter focal lengths, a more strongly curved wavefront is required, which creates a larger value for p(.). This problem can be addressed by phase warping, i.e., using only the fractional part of p(.) to drive the SLM. This results in a Fresnel lens-like pattern.

[0132] Two testbeds were constructed. The first prototype contained a phase SLM without a second amplitude modulator and could be reconfigured between two types of light sources: a red 632.8nm HeNe laser and a white LED. This prototype enabled the independent testing of the freeform surface lensing effect method and the evaluation of light source type-based artifacts such as diffraction. The second prototype was a fully dual-modulation projector using a green 532nm diode-pumped solid-state (DPSS) laser as the light source.

[0133] Due to its good beam quality and low power, a laser-based system using a HeNe gas laser was first implemented, making it safe for experiments (Figure 6, top). This setup enables the confirmation and analysis of the desired diffraction pattern.

[0134] Compared to diffraction-based projection methods [Slinger et al., 2005], a significant advantage of refraction-based methods is the reduced requirements for the light source. Diffraction patterns used in 2D holographic projection systems ideally require spatially and temporally coherent light for image formation; our method enables the use of partially collimated broadband light for light redirection. This is advantageous because even recent laser-based projection systems require widening to reduce artifacts such as screen spot contrast and observer metamerism.

[0135] The demonstration uses a single white broadband LED as a prototype light source. In this example, the LED has a short-wavelength light-emitting die (blue) and a conversion phosphor (green-yellow). See Figure 6, below.

[0136] A new image formation method was also applied to a laser-based system using a 532nm DPSS laser. Figure 16 Compared to the LED method, the optical power of the laser source (500mW) is sufficient to relay and amplify the resulting light intensity distribution onto a larger projection screen for evaluation.

[0137] For example, through wavefront simulation ( Figure 18 As anticipated and subsequently confirmed in the second line, the use of single-frequency lasers resulted in artifacts including noticeable screen spot contrast due to interference and diffraction “stripes”. Figure 18 (3rd line). As previously mentioned, these artifacts can be reduced to below the perceptible visibility threshold by using, for example, a set of lasers or broadband light sources such as LEDs and lamps with different center wavelengths

[2015] . A similar image “smoothing” effect can be achieved by averaging the image spatially or temporally using, for example, a diffuser or a commercially available series of deformable mirrors that introduce small angular variations in a pseudo-random manner at high speeds. This is particularly useful when limited to, for example, using narrowband light sources in a test setup. For ease of implementation, a thin-film diffuser is chosen to be placed in the intermediate image plane after the phase SLM. It can be ( Figure 8 (See the fourth line) for a photo showing the distribution of "purification" intensity.

[0138] The first prototype of a high-brightness, high-dynamic-range projection system was also demonstrated, in which images are formed based on a dynamic lens effect method and a conventional LCoS-based amplitude modulation display is used to provide additional sharpness and contrast.

[0139] At a high level, the optical path of a traditional projection system involves a high-intensity light source and some form of beam shaping, such as beam spreading, collimation and homogenization, color separation, and reconstructive optics. At the heart of the projector, a small SLM attenuates the amplitude of light per pixel. Our prototype retains this architecture but replaces the uniform illumination module with laser illumination and a phased SLM. Figure 7 The lens effect system is inserted between the light source and the existing SLM, and forms an approximate light distribution on the intermediate image plane that coincides with the SLM plane.

[0140] The freeform lens effect method redistributes light from dark image regions to bright image regions, thereby increasing both contrast and local peak brightness, which is known to have a significant impact on visual realism [Rempel et al., 2011].

[0141] Initially, a coarse forward image forming model for phase-based SLMs was used to predict the illumination distribution present at the second amplitude-only modulator. Given a phase function from a freeform lens effect algorithm, a simple model from Equations 2 and 4 was used to predict the light distribution on the image plane. The amount of smoothness introduced at the diffuser at the intermediate image plane was approximated using a blur kernel, and the desired modulation pattern for the amplitude modulator was obtained to introduce any missing spatial information and additional contrast if needed. It should be noted that characterization and careful calibration of the entire optical system are required to optimally drive the SLM. For this work, no significant effort was made other than careful spatial registration of the two images (of the illumination distribution caused by amplitude modulation and phase delay on the SLM) and calibration to linear increments of light intensity.

[0142] Similar to the case of flat-panel HDR displays [Seetzen et al., 2004], a forward image forming model for phase-only SLMs can be used to predict the “backlight” illumination of the second amplitude-only modulator. The modulation pattern for the amplitude modulator can be obtained by dividing the HDR target image by the “backlight” pattern.

[0143] Figure 18 The simulation options and experimental results of our method are shown. Figure 18 The first row (“Phase Pattern”) shows the phase pattern applied to the phase modulator calculated by Algorithm 4.1, where black corresponds to no phase delay and white corresponds to a delay of 2π. These patterns illustrate how a phase pattern with a maximum phase delay greater than 2π can be distorted to the maximum phase delay of the modulator, resulting in a Fresnel lens-like pattern.

[0144] Figure 18 The second line ('Simulation') shows a simulation of the phase pattern using the Huygens-Fresnel principle. Unlike geometric optics simulations such as path tracing, these simulations are able to capture many diffraction artifacts. The third line ('Direct') shows a photograph of a prototype using only the phase modulation shown due to noise caused by the laser spot and diffraction artifacts. Figure 18 The fourth row (“Diffuse”) introduces a diffuser sheet, which can almost completely remove these artifacts; the photos in this row use the same camera settings as the “Direct” row.

[0145] Phase pattern: The phase pattern calculated using Algorithm 1.

[0146] Simulation: Huygens-Fresnel simulation of the predicted image.

[0147] Direct: A photograph showing the actual image of diffraction artifacts without a diffuser.

[0148] Diffuser: By adding a thin film diffuser, artifacts such as diffraction fringes are almost completely reduced.

[0149] Standard: A standard photograph, using a single amplitude modulator for amplitude-modulated projection, shows an improved black level and low contrast.

[0150] Recommendation (HDR): Our lensing technique redistributes light from dark areas to bright areas, resulting in improved black levels and increased brightness intensity. The last two lines appear slightly distorted due to the camera's off-angle position, which was necessary due to short-projection, a closed screen, and a baffle to effectively block ambient light in order to capture the system's black levels.

[0151] exist Figure 18 The fifth line (“Standard”) shows a photograph of our dual-modulation projector operating using only the amplitude modulator. This is achieved by disabling light redistribution by providing a constant-value phase function. The result is a typical single-stage projector with leaky light polluting black levels and low overall contrast due to the inefficient use of high brightness intensity limited by the available power.

[0152] Finally, Figure 18 The last line (“Recommendation (HDR)”) shows photographs using our proposed phase + amplitude dual modulation method. These photographs were captured using the same camera setup as the “Standard” results (line 5) and demonstrate that our method not only restores better black levels but also, as expected, increases the brightness of highlights by redistributing light from dark areas of the image to brighter areas. This allows for better utilization of available power, enabling high dynamic range projection with significantly reduced power consumption compared to dual amplitude modulation methods.

[0153] Figure 5A (Left) shows the Lena image reproduced on the white light version of this device. As expected, broadband illumination averages out most of the diffraction artifacts, producing only a relatively small spatial blur, very similar to the backlight blur in the original dual-modulation work by Seetzen et al.

[2004] . This blur can be easily calibrated and can be compensated for in the dual-modulation setup.

[0154] The results from our dual modulation setup are shown in Figure 9 And in Figure 10. Figure 9 The effect of the freeform surface lens effect method is shown only, with the amplitude SLM set to a constant value. As in the HeNe laser setup, although less noticeable in the text due to the larger focal length, a range of diffraction artifacts can be identified, and the use of phase distortion is reduced. Figure 10 shows the results of the practical dual-modulation method. The second modulator stage has increased contrast and significantly increased detail, but some high-frequency artifacts cannot be eliminated.

[0155] The following references provide background information and are incorporated herein by reference.

[0156] ·BERRY, M. 2006. Oriental magic mirrors and the Laplacian image. European Physical Journal, 27, 1, 109.

[0157] • BIMBER, O. and IWAI, D. 2008. Superimposing dynamic range. ACM Transactions on Graphics. 27, 5, 150.

[0158] ·BLACKHAM, G. and NEALE, A., 1998. Image display apparatus, March 18.

[0159] European patent application. EPI9,970,306,624.

[0160] ·BUCKLEY, E. 2008.70.2: Invited paper: holographic laserprojection technology. In Proc. SID, Volume. 39, 1074-1079.

[0161] ·DAMBERG, G., SEETZEN, H., WARD, G., HEIDRICH, W. and WHITEHEAD, L. 2007.3.2: High dynamic range projection systems. In Proc. SID, Vol. 38, Wiley Online Library, 4-7.

[0162] ·DAMBERG, G., SEETZEN, H., WARD, G., KANG, M., LONGHURST, P., HEIDRICH, W. and WHITEHEAD, L. 2007. High dynamic range projector.

[0163] Siggraph is an emerging technology.

[0164] · FINCKH, M., DAMMERTZ, H. and LENSCH, HP 2010. Geometry construction from caustic images. In Proc. ECCV, 464-477.

[0165] ·HAUGEN,PR,BARTELT,H. and CASE,SK1983. Image formation by multifacetholograms. Applied Optics 22,18,2822-2829.

[0166] ·HOLOEYE.Photonics corporation.URLhttp: / / www.holoeye.com.

[0167] ·HOSKINSON, R., STOEBER, B., HEIDRICH, W. and FELS, S. 2010. Light reallocation for high contrast projection using an analog micromirror array.

[0168] ACM Transactions on Graphics (TOG) 29, 6, 165.

[0169] ·HOSKINSON, R., HAMPL, S. and STOEBER, B. 2012. Arrays of large-area, tip / tilt micromirrors for use in a high-contrast projector. Sensors and Actuators A: Physics 173, 1, 172-179.

[0170] ·HULLIN, MB, IHRKE, I., HEIDRICH, W., WEYRICH, T., DAMBERG, G., AND FUCHS, M. 2013. State of the art in computational fabrication and display of material appearance. Eurographies Annual Meeting (STAR).

[0171] ·KISER, T., EIGENSATZ, M., NGUYEN, MM, BOMPAS, P., ANDPAULY, M. 2013. Architectural caustics controlling light with geometry.

[0172] Advanced Architectural Geometry 2012. Springer Publishing, 91-106.

[0173] ·[Kusakabe et al. 2009]Kusakabe,Y.,Kanazawa,M.,Nojiri,Y.,Furuya,M.

[0174] and Yoshimura, M. 2009. A high-dynamic-range and high-resolution projector with dual modulation, Vol. 7241, 72410Q-72410Q-11.

[0175] • LESEM, L., HIRSCH, P., and JORDAN, J. 1969. The kinoform: a new wavefront reconstruction device. IBM Journal of Research and Development 13, 2, 150-155. • MENN, S., CORNELISSEN, SA, and BIERDEN, PA 2007. Advances in mems deformable mirror technology for laser beam shaping. In Optical Devices

[0176] +International Society for Applications, Optics and Photonics 66630M-66630M.

[0177] • MINANO, JC, BENITEZ, P. and SANTAMARIA, A. 2009. Freeform optics for illumination. Optical Review 16, 2, 99-102.

[0178] [Ng et al. 1999] Ng, MK, Chan, RH and Tang, W.-C. 1999. A fast algorithm for deblurring models with neumann boundary conditions. SIAM Journal of Scientific Computing 21, 3, 851-866.

[0179] ·[NVIDIA]NVIDIA,C.Programming guide,cusparse,cublas,and cufftlibraryuser guides.{Online}.

[0180] • PAPAS, M., JAROSZ, W., JAKOB, W., RUSINKIEWICZ, S., MATUSIK, W., and WEYRICH, T. 2011. Goal-based caustics. Computer Graphics Forum 30, 2, 503-511.

[0181] • PAPAS, M., HOUIT, T., NOWROUZEZAHRAI, D., GROSS, M. and JAROSZ, W. 2012. The magic lens: refractive steganography. ACM Transactions on Graphics (TOG) 31, 6, 186.

[0182] ·[Parikh and Boyd 2013]Parikh, N. and Boyd, S. 2013. Proximal algorithms.

[0183] The basis and trends of optimization 1,3,123-231.

[0184] ·PAULY,M.,AND KISER,T.2012.Caustic art.Tech.rep.

[0185] ·REMPEL, A., HEIDRICH, W., LI, H. and MANTIUK, R. 2009. Videoviewingpreferences for hdr displays under varying ambient illumination.

[0186] Proc.APGV, 45-52.

[0187] · REMPEL, A., HEIDRICH, W. and MANTIUK, R. 2011. The role of contrast in theperceived depth of monocular imagery. Proc. APGV, 115.

[0188] • ROBINSON, MD, SHARP, G. and CHEN, J. 2005. Polarization engineering for LCD projection, Vol. 4. John Willie International Publishing Co.

[0189] ·SCHWARTZBURG,Y.,TESTUZ,R.,T AGLIAS ACCHI,A. and PAULY,M.

[0190] 2014. High-contrast computational caustic design. Proc. Siggraph (ACM). (Published).

[0191] ·SEETZEN, H., HEIDRICH, W., STUERZLINGER, W., WARD, G., WHITEHEAD, L., TRENTACOSTE, M., GHOSH and VOROZCOVS, A.

[0192] 2004. High dynamic range display systems. Proceedings of the ACM SIGGRAPH (August), 760-768.

[0193] ·SEETZEN,H.2009.High dynamic range display and projection systems.

[0194] Doctoral dissertation from the University of British Columbia.

[0195] ·[Silvester and Mihajlovi A2004]Silvester, DJ and Mihajlovi c',MD2004.

[0196] A black-box multigrid preconditioner for the biharmonic equation. BIT Numerical Mathematics 44, 1, 151-163.

[0197] ·SLINGER, C., CAMERON, C and STANLEY, M. 2005.

[0198] Computer-generated holography as a generic display technology. IEEE Computers 38, 8, 46-53.

[0199] · [Tang and Christov 2006] Tang, XH and Christov, CI 2006. An operatorsplitting scheme for biharmonic equation with accelerated convergence. Proceedings of the Fifth International Symposium of the Journal of Large Scale Science, Springer Publishing Company, Berlin, Heidelberg, LSSC'05, 387-394.

[0200] ·YUE, Y., IWASAKI, K., CHEN, B.-Y., DOBASHI, Y. and NISHITA, T. 2012.

[0201] Pixel art with refracted light by rearrangeable sticks. Computer Graphics Forum 31, 2pt3, 575-582.

[0202] ·YUE, Y., IWASAKI, K., CHEN, B.-Y., DOBASHI, Y. and NISHITA, T. 2014.

[0203] Poisson-based continuous surface generation for goal-based caustics. ACM Transactions on Graphics (published).

[0204] [Zhao 2004] Zhao, J. 2004. Convergence of v-cycle and f-cycle multigrid methods for the biharmonic problem using the morley element. Numerical Analysis of Electronic Transactions 17, 112-132.

[0205] It should be understood that some implementations provide one or more of the following:

[0206] A novel algorithm (“Target-Based Caustics”) for optimizing freeform lenses is significantly simpler than some existing algorithms. This algorithm can be applied to control the projection of light in real-time or near real-time.

[0207] Some implementations operate directly in phase space and can therefore be implemented as iterative methods that can generate modulation patterns not only for phase modulators but also for conventional refractive lenses without additional steps such as Poisson integration.

[0208] A novel dual-modulation projector design combines a phase modulator and an amplitude modulator for image generation and is capable of operating using white (incoherent) light. To the best of our knowledge, the methods and apparatus described herein can also be applied to generate static light fields, for example, for architectural lighting and / or vehicle lighting.

[0209] • Directly optimize the modulation phase of light without compromising between data terms and surface integrability.

[0210] • This is made feasible by finding parameterizations that allow the optimization problem to be represented in the modulator / lens plane rather than the image plane.

[0211] Our derivation relies on small-angle image formation (paraxial approximation), which is established in the optical community.

[0212] Explanation of terms Unless the context explicitly requires otherwise, throughout the specification and claims:

[0213] • "Including" and "containing" should be interpreted as including, not as exclusive or exhaustive; that is, in meaning, "including but not limited to".

[0214] • “Connection,” “coupled,” or any variation thereof means any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements can be physical, logical, or a combination thereof;

[0215] • When used to describe this instruction manual, the words "this article," "above," "below," and similar terms shall be used.

[0216] This should refer to the entire specification and not any particular part thereof;

[0217] For a list of two or more items, "or" encompasses all of the following interpretations of the word:

[0218] Any item in the list, all items in the list, and any combination of items in the list;

[0219] The singular forms “a,” “an,” and “the” also include the meaning of any suitable plural form. Terms indicating direction, such as “vertical,” “lateral,” “horizontal,” “upward,” “downward,” “forward,” “backward,” “inward,” “outward,” “vertical,” “lateral,” “left,” “right,” “front,” “backward,” “top,” “bottom,” “above,” and “below,” are used in this specification and the appended claims (if applicable) according to the specific orientation of the device described and shown. Various alternative orientations can be assumed for the subject matter described herein. Therefore, these directional terms are not strictly limited and should not be interpreted narrowly.

[0220] Embodiments of the present invention can be implemented using specially designed hardware, configurable hardware, or programmable data processors. The programmable data processor is configured by providing software (optionally including "firmware") that is executable on a data processor, a dedicated computer, or a data processor specially programmed, configured, or constructed to perform one or more steps of the methods detailed herein and / or combinations of two or more of these. Examples of specially designed hardware include logic circuits, application-specific integrated circuits ("ASICs"), large-scale integrated circuits ("LSIs"), very large-scale integrated circuits ("VLSIs"), and the like. Examples of configurable hardware include one or more programmable logic devices such as programmable array logic ("PALs"), programmable logic arrays ("PLAs"), and field-programmable gate arrays ("FPGAs"). Examples of programmable data processors include microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors, math coprocessors, general-purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors in the control circuitry of a device can implement the methods described herein by executing software instructions in a program memory accessible to the processor.

[0221] Processing can be centralized or distributed. In the case of distributed processing, information, including software and / or data, can be stored centrally or distributed. Such information can be exchanged between different functional units via communication networks such as local area networks (LANs), wide area networks (WANs), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channels.

[0222] For example, while processes or blocks are presented in a given order, alternative examples may execute routines with steps or employ systems with blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or subcombinations. Each of these processes or blocks can be implemented in a variety of different ways. Furthermore, although processes or blocks are sometimes shown to be executed serially, they may alternatively be executed in parallel, or may be executed at different times.

[0223] Furthermore, although the elements are sometimes shown to be performed sequentially, they may alternatively be performed simultaneously or in a different order. Therefore, the appended claims are intended to be interpreted as including all such variations within their intended scope.

[0224] Software and other modules may reside on servers, workstations, personal computers, tablet computers, image data encoders, image data decoders, PDAs, color grading tools, video projectors, audiovisual receivers, displays (e.g., televisions), digital cinema projectors, media players, and other devices suitable for the purposes described herein. Those skilled in the art will understand that aspects of the system can be implemented using other communication, data processing, or computer system configurations, including Internet devices, handheld devices (including personal digital assistants (PDAs)), wearable computers, cellular or mobile phones of all types, multiprocessor systems, microprocessor-based or programmable consumer electronics (e.g., video projectors, audiovisual receivers, displays such as televisions), set-top boxes, network PCs, microcomputers, mainframe computers, etc.

[0225] This invention can also be provided in the form of a program product. The program product may include any non-transitory medium carrying a set of computer-readable instructions that, when executed by a data processor, cause the data processor to perform the method of this invention. The program product according to the invention can be any of a variety of forms. The program product may include, for example, non-transitory media such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD-ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hard-wired or pre-programmed chips (e.g., EEPROM semiconductor chips), nanotechnology memories, etc. Computer-readable signals on the program product may optionally be compressed or encrypted.

[0226] In some embodiments, the invention can be implemented in software. For clarity, "software" includes any instructions that execute on a processor and may include (but is not limited to) firmware, resident software, microcode, etc. As known to those skilled in the art, both the processing hardware and software can be centralized or distributed (or a combination thereof), either wholly or partially. For example, the software and other modules can be accessed via local memory, via a network, via a browser or other applications in a distributed computing environment, or via other means suitable for the purposes described above. In some embodiments, image data is processed by the processor executing software instructions to obtain control signals for a phase modulator. In some embodiments, the software can be executed in real time (other embodiments are also possible).

[0227] In the case of the components mentioned above (e.g., software modules, processors, components, devices, circuits, etc.), unless otherwise stated, references to such components (including references to “devices”) should be interpreted as including any component that performs the function of the described component (i.e., functionally equivalent), including components that are structurally different from those that perform the functions of the disclosed structures in the exemplary embodiments of the present invention.

[0228] For illustrative purposes, specific examples of systems, methods, and apparatuses are described herein. These are merely examples. The techniques provided herein can be applied to systems other than the exemplary systems described above. In practice of this invention, many substitutions, modifications, additions, omissions, and replacements are possible. This invention includes variations of the described embodiments that will be obvious to those skilled in the art, including variations obtained by: replacing features, elements, and / or actions with equivalent features, elements, and / or actions; mixing and matching features, elements, and / or actions from different embodiments; combining features, elements, and / or actions from embodiments described herein with features, elements, and / or actions from other technologies; and / or omitting combinations of features, elements, and / or actions from the described embodiments.

[0229] Therefore, the appended claims and subsequently introduced claims are intended to be interpreted to include all such modifications, substitutions, additions, omissions, and sub-combinations that can be reasonably inferred. The scope of the claims should not be limited to the preferred embodiments set forth in the embodiments, but should be given the broadest interpretation consistent with the description as a whole.

[0230] Regarding the implementation methods including the above embodiments, the following technical solutions are also disclosed:

[0231] Solution 1. A method for controlling a phase modulator to display a target light pattern defined by image data, the method comprising:

[0232] A distorted image is initialized based on the image data, the distorted image being distorted from the target light pattern by a distortion corresponding to a phase function p(x), the phase function p(x) representing a phase shift applied by the phase modulator to a region in the lens surface;

[0233] The phase function and the warped image are improved by performing multiple iterations, wherein each of the multiple iterations includes the following steps:

[0234] The phase function is updated by performing an optimization to obtain the updated phase function, wherein the updated phase function reduces the difference measurement between the distorted image and the reciprocal of the magnification provided by the phase function at a point in the distorted image; and

[0235] The target light pattern is distorted onto the lens surface using the distortion u(x) generated by the updated phase function p(x) to obtain an updated distorted image.

[0236] Option 2. According to the method of Option 1, wherein the difference measurement includes the sum of squares of the differences between the pixels of the distorted image and the reciprocal of the magnification at the point in the distorted image.

[0237] Option 3. The method according to Option 1 or 2, wherein updating the phase function includes:

[0238] Calculate the pixels of the distorted image and The difference between corresponding values.

[0239] Option 4. The method according to Option 1 or 2, wherein updating the phase function includes:

[0240] Calculate the pixels of the distorted image and The difference between corresponding values.

[0241] Option 5. The method according to any one of Options 1 to 3, wherein the step of updating the phase function includes solving a linear least squares problem.

[0242] Scheme 6. The method according to Scheme 4, wherein the least squares problem comprises a system matrix with a discrete Laplace operator.

[0243] Solution 7. According to the method described in Solution 1, the step of updating the phase function includes solving:

[0244]

[0245] Scheme 8. The method according to any one of Schemes 4 to 6, wherein performing the optimization includes applying an algorithm consisting of a group of selected conjugate gradients (CG), BICGSTAB, and quasi-minimum residuals (QMR).

[0246] Option 9. The method according to any one of Options 1 to 7, wherein the step of backward twisting the target intensity in the image plane onto the lens surface includes performing a texture mapping operation.

[0247] Option 10. The method according to Option 8, wherein the texture mapping operation is implemented on a graphics processing unit.

[0248] Option 11. The method according to Option 8 or 9, wherein the step of backward twisting the target intensity in the image plane onto the lens surface includes calculating:

[0249]

[0250] Option 12. The method according to any one of Options 1 to 11, comprising: modeling blur in an image at the image plane, and generating control values ​​for an amplitude modulator that tend to at least partially compensate for the blur.

[0251] Option 13. The method according to any one of Options 1 to 12, comprising: displaying the target light pattern by controlling the phase modulator according to the phase function and illuminating the phase modulator with light.

[0252] Option 14. The method according to Option 13, wherein the light is broadband light.

[0253] Option 15. The method according to Option 14, wherein the broadband light is white light.

[0254] Option 16. The method according to Option 13, wherein the light is monochromatic.

[0255] Option 17. The method according to Option 13 or 16, wherein the light is a laser.

[0256] Option 18. The method according to any one of Options 13 to 17, wherein the light is collimated.

[0257] Option 19. The method according to Option 18, wherein the light is incident on the phase modulator in a direction perpendicular to the lens surface.

[0258] Option 20. The method according to any one of Options 1 to 19, wherein the light pattern comprises one or more light spots.

[0259] Option 21. The method according to Option 20, comprising: controlling the phase function applied to the phase modulator to move the one or more light spots.

[0260] Option 22. The method according to Option 20 or 21, wherein the one or more light spots have an intensity exceeding the maximum uniform illumination intensity at the image plane.

[0261] Option 23. The method according to any one of Options 1 to 22, wherein the resolution of the phase modulator is at least 1 megapixel.

[0262] Option 24. The method according to Option 23, wherein the phase modulator comprises at least 5 megapixels.

[0263] Option 25. The method according to any one of Options 1 to 24, wherein the light pattern occupies an image region in the image plane, and light rays from any point on the phase modulator, pointing to any point on the boundary of the image region, form an angle θ with the normal of the phase modulator, where |θ| ≤ 12°.

[0264] Option 26. The method according to any one of Options 1 to 24, wherein the numerical aperture of a point in the lens surface keeps the paraxial approximation within 1%.

[0265] Solution 27. The method according to any one of Solutions 1 to 26, wherein initializing the distorted image comprises: setting the distorted image to be the same as the target light pattern.

[0266] Option 28. The method according to any one of Options 1 to 27, wherein the phase modulator comprises a liquid crystal phase modulator.

[0267] Option 29. The method according to Option 28, wherein the phase modulator includes an LCoS device.

[0268] Option 30. The method according to any one of Options 1 to 27, wherein the phase modulator includes a variable mirror.

[0269] Option 31. The method according to any one of Options 1 to 30, wherein the image data comprises video data having a frame rate of at least 20 frames per second.

[0270] Solution 32. The method according to Solution 31, wherein the video data provides a different target light pattern for each frame, and the method includes: calculating a different phase function for each frame.

[0271] Option 33. The method according to Option 32 includes: calculating the different phase functions in real time.

[0272] Option 34. The method according to any one of Options 1 to 33, wherein the improved phase function and the distorted image are performed in 10 or fewer iterations.

[0273] Scheme 35. The method according to any one of Schemes 1 to 34, wherein the improved phase function and the distorted image are performed in a fixed number of iterations.

[0274] Option 36. The method according to any one of Options 1 to 35, comprising: performing one or more steps of improving the phase function and the distorted image in parallel in one or more graphics processor units.

[0275] Scheme 37. The method according to any one of Schemes 1 to 35, comprising: performing at least some steps of improving the phase function and the distorted image in the frequency domain.

[0276] Option 38. The method according to Option 37, comprising: generating an optimization function; performing a Fourier transform on the distorted image; using the Fourier transform of the distorted image to generate the phase function in the frequency domain; and performing an inverse Fourier transform on the phase function.

[0277] Option 39. The method according to Option 38, comprising: performing the Fourier transform in hardware configured to perform the Fourier transform.

[0278] Option 40. The method according to any one of Options 37 to 39, comprising: expanding the image data to have periodic boundary conditions before performing the steps in the frequency domain.

[0279] Option 41. The method according to Option 40, wherein extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

[0280] Scheme 42. The method according to any one of Schemes 1 to 41, comprising: generating a control signal for a spatial light modulator to correct the intensity of light modulated by the phase modulator.

[0281] Scheme 43. The method according to any one of Schemes 1 to 42, comprising: performing one or more iterations at a first spatial resolution, and upsampling the updated phase function obtained through one or more iterations.

[0282] Option 44. The method according to Option 43, comprising: after upsampling the updated phase function, performing one or more additional iterations in the iteration at a second resolution higher than the first resolution.

[0283] Option 45. The method according to any one of Options 1 to 44, wherein the image data includes video data, the target light pattern is defined for one frame of the image data, and different target light patterns are defined in the image data for other frames of the image data.

[0284] Solution 46. An apparatus for controlling a phase modulator to display a target light pattern defined by image data, the apparatus comprising a data processor communicating with the phase modulator, the data processor being configured to:

[0285] Receive the image data as input;

[0286] A distorted image is initialized based on the image data, the distorted image being distorted from the target light pattern by a distortion corresponding to a phase function p(x), the phase function p(x) representing a phase shift applied by the phase modulator to a region in the lens surface;

[0287] The phase function and the warped image are improved by performing multiple iterations, wherein each of the multiple iterations includes the following steps:

[0288] The phase function is updated by performing an optimization to obtain the updated phase function, wherein the updated phase function reduces the difference measurement between the distorted image and the reciprocal of the magnification provided by the phase function at a point in the distorted image; and

[0289] The target light pattern is distorted onto the lens surface using a distortion u(x) generated by the updated phase function p(x) to obtain an updated distorted image; and

[0290] Control signals for the phase modulator are generated based on the improved phase function.

[0291] Option 47. The device according to Option 46, wherein the difference measurement comprises: the sum of squares of the differences between the pixels of the distorted image and the reciprocal of the magnification at a point in the distorted image.

[0292] Option 48. The device according to Option 46 or 47, wherein the step of updating the phase function includes: the data processor calculating the pixel and phase of the distorted image. The difference between corresponding values.

[0293] Option 49. The device according to Option 46 or 47, wherein the step of updating the phase function includes: the data processor calculating the pixel and phase of the distorted image. The difference between corresponding values.

[0294] Option 50. The device according to any one of Options 46 to 48, wherein the step of updating the phase function comprises: solving a linear least squares problem by the data processor.

[0295] Option 51. The device according to Option 49, wherein the least squares problem comprises a system matrix having a discrete Laplace operator.

[0296] Option 52. The device according to Option 46, wherein the step of updating the phase function includes solving by the data processor:

[0297]

[0298] Option 53. The device according to any one of Options 49 to 51, wherein performing the optimization comprises the data processor applying an algorithm selected from the group consisting of conjugate gradient (CG), BICGSTAB, and quasi-minimum residual (QMR).

[0299] Option 54. The device according to any one of Options 46 to 52, wherein the step of backward twisting the target intensity in the image plane onto the lens surface includes performing a texture mapping operation.

[0300] Option 55. The device according to Option 53 includes a graphics processing unit, wherein the texture mapping operation is implemented on the graphics processing unit.

[0301] Option 56. The device according to Option 53 or 54, wherein the step of rearwardly twisting the target intensity in the image plane onto the lens surface includes calculation by the data processor:

[0302]

[0303] Option 57. The device according to any one of Options 46 to 56, wherein the data processor is configured to: model blur in an image at the image plane and generate control values ​​for the amplitude modulator that tend to at least partially compensate for the blur.

[0304] Option 58. The device according to any one of Options 46 to 57, comprising the phase modulator and a light source for projecting light onto the phase modulator, wherein the data processor is configured to generate the target light pattern by controlling the phase modulator according to the phase function and controlling the light source to illuminate the phase modulator with light.

[0305] Option 59. The device according to Option 58, wherein the light is broadband light.

[0306] Option 60. The device according to Option 59, wherein the broadband light is white light.

[0307] Option 61. The device according to Option 58, wherein the light is monochromatic.

[0308] Option 62. The device according to Option 59 or 61, wherein the light is a laser.

[0309] Option 63. The device according to any one of Options 58 to 62, wherein the light is collimated.

[0310] Option 64. The device according to Option 63, wherein the light source is configured to project light incident on the phase modulator in a direction perpendicular to the lens surface.

[0311] Option 65. The device according to any one of Options 58 to 64, wherein the phase modulator has a resolution of at least 1 megapixel.

[0312] Option 66. The device according to Option 65, wherein the phase modulator has a resolution of at least 5 megapixels.

[0313] Option 67. The device according to any one of Options 58 to 66, wherein the target light pattern occupies an image region in the image plane, and light rays from any point on the phase modulator, pointing to any point on the boundary of the image region, form an angle θ with the normal of the phase modulator, where |θ| ≤ 12°.

[0314] Option 68. The device according to any one of Options 58 to 67, wherein the phase modulator comprises a liquid crystal phase modulator.

[0315] Option 69. The device according to Option 68, wherein the phase modulator includes an LCoS device.

[0316] Option 70. The device according to any one of Options 58 to 67, wherein the phase modulator includes a variable mirror.

[0317] Option 71. The device according to any one of Options 46 to 70, wherein the target light pattern comprises one or more light spots.

[0318] Option 72. The device according to Option 71, wherein the data processor is configured to control a phase function applied to the phase modulator to move the one or more light spots.

[0319] Option 73. The device according to Option 71 or 72, wherein the one or more light spots have an intensity exceeding the maximum uniform illumination intensity at the image plane.

[0320] Option 74. The device according to any one of Options 46 to 73, wherein the numerical aperture of a point in the lens surface keeps the paraxial approximation within 1%.

[0321] Option 75. The device according to any one of Options 46 to 74, wherein the data processor is configured to initialize the distorted image by: the data processor being configured to set the distorted image to be identical to the target light pattern.

[0322] Option 76. The device according to any one of Options 46 to 75, wherein the image data comprises video data having a frame rate of at least 20 frames per second.

[0323] Solution 77. The device according to Solution 76, wherein the video data provides a different target light pattern for each frame, and the data processor is configured to calculate a different phase function for each frame.

[0324] Option 78. The device according to Option 77, wherein the data processor is configured to: calculate the different phase functions in real time.

[0325] Option 79. The device according to any one of options 46 to 78, wherein the data processor is configured to improve the phase function and the distorted image in 10 or fewer iterations.

[0326] Option 80. The device according to any one of Options 46 to 78, wherein the data processor is configured to improve the phase function and the distorted image by a fixed number of iterations.

[0327] Option 81. The device according to any one of Options 46 to 78, comprising one or more graphics processor units, wherein the data processor is configured to perform one or more steps of improving the phase function and the distorted image in parallel in the one or more graphics processor units.

[0328] Option 82. The device according to any one of Options 46 to 80, wherein the data processor is configured to perform at least some steps in the frequency domain to improve the phase function and the distorted image.

[0329] Option 83. The device according to Option 82, wherein the data processor is configured to: perform a Fourier transform on the distorted image; use the Fourier transform of the distorted image to generate the phase function in the frequency domain; and perform an inverse Fourier transform on the phase function.

[0330] Option 84. The device according to Option 83 includes a hardware Fourier transform device, wherein the data processor is configured to control the Fourier transform device to perform a Fourier transform.

[0331] Option 85. The device according to any one of Options 46 to 84, comprising a spatial light modulator, wherein the data processor is configured to apply a control signal to the spatial light modulator to correct the intensity of light modulated by the phase modulator.

[0332] Option 86. The device according to any one of Options 82 to 84, wherein the data processor is configured to: expand the image data to have periodic boundary conditions before performing the step in the frequency domain.

[0333] Solution 87. The device according to Solution 86, wherein extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

[0334] Option 88. The apparatus according to any one of Options 46 to 87, comprising: generating a control signal for a spatial light modulator to correct the intensity of light modulated by the phase modulator.

[0335] Option 89. The device according to any one of Options 46 to 88, wherein the data processor is configured to: perform one or more iterations at a first spatial resolution, and upsample the updated phase function obtained through one or more iterations.

[0336] Option 90. The device according to Option 89, wherein the data processor is configured to: after upsampling the updated phase function, perform one or more additional iterations in the iteration at a second resolution higher than the first resolution.

[0337] Option 91. The device according to any one of Options 46 to 90, wherein the image data includes video data, the target light pattern is defined for one frame of the image data, and different target light patterns are defined in the image data for other frames of the image data.

[0338] Option 92. A method for generating control values ​​for a phase modulator from image data defining a target light pattern, the method comprising:

[0339] A mapping is established between points in the light pattern and corresponding points on the phase modulator;

[0340] Using the mapping, a phase function p is derived by mapping the target light pattern onto the coordinate space of the phase modulator, the phase function p including the control value; and

[0341] The mapped target light pattern is processed in the coordinate space of the phase modulator.

[0342] Scheme 93. The method according to Scheme 92, wherein processing the mapped target light pattern includes: optimizing the experimental phase function based on a comparison of the intensity of the mapped target light pattern at a point on the phase modulator with the corresponding optical properties of the phase function in the vicinity of the point.

[0343] Option 94. The method according to Option 93, wherein the corresponding optical property includes magnification.

[0344] Scheme 95. The method according to any one of Schemes 93 and 94, comprising: determining the optical property based on the Laplace operator of the phase function at the corresponding point.

[0345] Option 96. The method according to Option 95, comprising: using a discrete Laplace operator to determine the Laplace operator of the phase function.

[0346] Solution 97. A method for displaying video data, the video data specifying video frames to be displayed at a frame rate, the method comprising:

[0347] The video data is processed in real time to obtain a sequence of phase modulator control signals at the frame rate.

[0348] The phase modulator control signal is applied to the illuminated two-dimensional spatial phase modulator, and

[0349] The phase-modulated light is guided to the viewing area.

[0350] Option 98. The method according to Option 97, comprising: further amplitude modulating the phase-modulated light.

[0351] Scheme 99. The method according to Scheme 98, wherein further amplitude modulation of the phase-modulated light includes: controlling a spatial light modulator in the path of the phase-modulated light.

[0352] Option 100. The method according to Option 99, comprising: calculating blur in phase-modulated light, and controlling the spatial light modulator to reduce the blur.

[0353] Solution 101. The method according to any one of Solutions 97 to 100, wherein processing the video data includes:

[0354] A mapping is established between points in the light pattern and corresponding points on the light modulator;

[0355] Using the mapping, a phase function p is derived by mapping the target light pattern onto the coordinate space of the phase modulator, the phase function p including the control value; and

[0356] The mapped target light pattern is processed in the coordinate space of the phase modulator.

[0357] Scheme 102. The method according to Scheme 101, wherein processing the mapped target light pattern includes: optimizing the experimental phase function based on a comparison of the intensity of the mapped target light pattern at a point on the phase modulator with the corresponding optical properties of the phase function in the vicinity of the point.

[0358] Option 103. The method according to Option 102, wherein the corresponding optical property includes magnification.

[0359] Option 104. The method according to Option 102 or 103, comprising: determining the optical property based on the Laplace operator of the phase function at the corresponding point.

[0360] Option 105. The method according to Option 104, comprising: using a discrete Laplace operator to determine the Laplace operator of the phase function.

[0361] Solution 106. The method according to any one of Solutions 97 to 102, wherein the processing of the video data is performed in the frequency domain.

[0362] Solution 107. The method according to Solution 106, wherein processing the video data includes: generating an optimization function; performing a Fourier transform on the optimization function; generating a phase function in the frequency domain; and performing an inverse Fourier transform on the phase function.

[0363] Option 108. The method according to Option 107, comprising: performing a Fourier transform in hardware configured to perform a Fourier transform.

[0364] Option 109. The method according to any one of Options 92 to 96 and 101 to 108, wherein the phase modulator has a maximum phase delay, and the method comprises: subtracting a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0365] Solution 110. A method for controlling a phase modulator to display an image defined by image data, the method comprising:

[0366] The objective function is determined based on the image data;

[0367] Transform the objective function into the frequency space;

[0368] Minimize the transformed objective function in the frequency space to obtain the phase function in the frequency space; and

[0369] The phase function is inversely transformed to obtain a dephase function, which correlates the phase of the phase modulator with its position in two dimensions.

[0370] Scheme 111. The method according to Scheme 110, wherein transforming the objective function includes: calculating the Fourier transform of the objective function.

[0371] Solution 112. The method according to Solution 111, comprising: expanding the image data to have periodic boundary conditions before transformation, and making the objective function based on the expanded image data.

[0372] Solution 113. The method according to Solution 112, wherein extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

[0373] Scheme 114. The method according to any one of Schemes 110 to 113, wherein the objective function is a least squares objective function.

[0374] Option 115. The method according to any one of Options 110 to 114, wherein the objective function includes a cost for deviating from the input variable.

[0375] Scheme 116. The method according to any one of Schemes 110 to 115, wherein the method is performed iteratively, and in each of the multiple iterations, the input variable of the objective function is the solution phase function of the previous iteration.

[0376] Solution 117. The method according to Solution 116, comprising: caching the Fourier transform of the solution phase function of the previous iteration, and applying the cached Fourier transform of the solution phase function in the current iteration.

[0377] Scheme 118. The method according to any one of Schemes 110 to 117, wherein the objective function comprises a nearest neighbor operator given by:

[0378] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0379] Option 119. The method according to Option 118, wherein evaluating the transformed objective function includes: determining

[0380]

[0381] Option 120. The method according to Option 119, wherein,

[0382] Option 121. The method according to any one of Options 119 and 120, wherein,

[0383] Scheme 122. The method according to any one of Schemes 119 to 121, wherein α>0 is a regularization parameter.

[0384] Option 123. The method according to any one of Options 110 to 122, wherein the method comprises: initializing the phase surface as a constant value.

[0385] Option 124. The method according to any one of Options 110 to 123, wherein the evaluation of the transformed objective function is performed in parallel for different points.

[0386] Option 125. The method according to Option 124, wherein the evaluation is performed in the graphics processing unit.

[0387] Solution 126. The method according to any one of Solutions 110 to 125, comprising: displaying an image by controlling the pixels of the phase modulator to simultaneously illuminate the phase modulator according to the dephase function.

[0388] Option 127. The method according to Option 126, comprising: using collimated light to uniformly illuminate the phase modulator.

[0389] Option 128. The method according to any one of Options 126 to 127, wherein the phase modulator comprises a liquid crystal pixel array, and the method comprises: setting a control signal to the pixel according to the dephase function.

[0390] Option 129. The method according to Option 128, wherein the phase modulator is an LCoS phase modulator.

[0391] Option 130. The method according to Option 110, wherein the phase modulator is a deformable mirror.

[0392] Option 131. The method according to any one of Options 126 to 130, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0393] Scheme 132. The method according to any one of Schemes 110 to 131, wherein the phase modulator has a maximum phase delay, and the method comprises: subtracting a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0394] Solution 133. A method for controlling a phase modulator to display an image defined by image data, the method comprising:

[0395] The fixed-point iteration is determined based on the image data;

[0396] The fixed point is iteratively transformed into the frequency space;

[0397] The fixed-point iteration is evaluated in the frequency space to obtain the phase function in the frequency space; and

[0398] The phase function is inversely transformed to obtain a dephase function, which correlates the phase of the phase modulator with its position in two dimensions.

[0399] Scheme 134. The method according to Scheme 133, wherein transforming the fixed-point iteration includes: calculating the Fourier transform of the fixed-point iteration.

[0400] Option 135. The method according to Option 134, comprising: expanding the image data to have periodic boundary conditions prior to the transformation, and making the fixed point iteration based on the expanded image data.

[0401] Solution 136. The method according to Solution 135, wherein extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

[0402] Scheme 137. The method according to any one of Schemes 133 to 136, wherein the fixed-point iteration is least-squares fixed-point iteration.

[0403] Scheme 138. The method according to any one of Schemes 133 to 137, wherein the fixed-point iteration includes a cost for deviating from the input variable.

[0404] Scheme 139. The method according to any one of Schemes 133 to 138, wherein the method is performed iteratively, and in each of the multiple iterations, the input variable of the fixed-point iteration is the solution phase function of the previous iteration.

[0405] Option 140. The method according to Option 139, comprising: caching the Fourier transform of the solution phase function of the previous iteration, and applying the cached Fourier transform of the solution phase function in the current iteration.

[0406] Scheme 141. The method according to any one of Schemes 133 to 140, wherein the fixed-point iteration includes a nearest-neighbor operator given by the following formula:

[0407] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0408] Option 142. The method according to Option 141, wherein evaluating the transformed fixed-point iteration includes: determining

[0409]

[0410] Option 143. The method according to Option 142, wherein,

[0411] Option 144. The method according to any one of Options 142 and 143, wherein,

[0412] Scheme 145. The method according to any one of Schemes 143 to 144, wherein α>0 is a regularization parameter.

[0413] Scheme 146. The method according to any one of Schemes 133 to 145, wherein the method comprises: initializing the phase surface as a constant value.

[0414] Scheme 147. The method according to any one of Schemes 133 to 146, wherein the evaluation of the transformed fixed-point iteration is performed in parallel for different points.

[0415] Option 148. The method according to Option 147, wherein the evaluation is performed in the graphics processing unit.

[0416] Option 149. The method according to any one of Options 133 to 148, comprising: displaying an image by controlling the pixels of the phase modulator to simultaneously illuminate the phase modulator according to the dephase function.

[0417] Option 150. The method according to Option 17, comprising: uniformly illuminating the phase modulator with collimated light.

[0418] Option 151. The method according to any one of Options 149 to 150, wherein the phase modulator comprises a liquid crystal pixel array, and the method comprises: setting a control signal to the pixel according to the dephase function.

[0419] Option 152. The method according to Option 151, wherein the phase modulator is an LCoS phase modulator.

[0420] Option 153. The method according to Option 151, wherein the phase modulator is a deformable mirror.

[0421] Scheme 154. The method according to any one of Schemes 149 to 153, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0422] Option 155. The method according to any one of Options 133 to 154, wherein the phase modulator has a maximum phase delay, and the method comprises: subtracting a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0423] Solution 156. An apparatus for generating control values ​​for a phase modulator from image data defining a target light pattern, the apparatus comprising a data processor communicating with the phase modulator, the data processor being configured to:

[0424] A mapping is established between points in the light pattern and corresponding points on the phase modulator;

[0425] Using the mapping, a phase function p is derived by mapping the target light pattern onto the coordinate space of the phase modulator, the phase function p including the control value; and

[0426] The mapped target light pattern is processed in the coordinate space of the phase modulator.

[0427] Solution 157. The apparatus according to Solution 156, wherein the data processor is configured to process the mapped target light pattern, including the data processor being configured to: optimize the experimental phase function based on a comparison of the intensity of the mapped target light pattern at a point on the phase modulator with the corresponding optical properties of the phase function in the vicinity of the point.

[0428] Option 158. The device according to Option 157, wherein the corresponding optical property includes magnification.

[0429] Option 159. The device according to any one of Options 157 and 158, wherein the data processor is configured to determine the optical properties based on the Laplace operator of the phase function at the corresponding point.

[0430] Option 160. The device according to Option 159, wherein the data processor is configured to: use a discrete Laplace operator to determine the Laplace operator of the phase function.

[0431] Solution 161. An apparatus for displaying video data, the video data specifying video frames displayed at a frame rate, the apparatus comprising a data processor configured to:

[0432] The video data is processed in real time to obtain a sequence of phase modulator control signals at the frame rate;

[0433] The phase modulator control signal is applied to the illuminated two-dimensional spatial phase modulator; and

[0434] The spatial phase modulator is controlled to guide the obtained phase-modulated light to the viewing area.

[0435] Option 162. The device according to Option 161, wherein the data processor is configured to control a spatial light modulator in the path of the phase-modulated light to amplitude modulate the phase-modulated light.

[0436] Solution 163. The apparatus according to Solution 162, wherein the data processor is configured to: calculate blur in phase-modulated light and control the spatial light modulator to reduce the blur.

[0437] Option 164. The device according to any one of Options 161 to 163, wherein the data processor is configured to process the video data, including the data processor being configured to:

[0438] A mapping is established between points in the light pattern and corresponding points on the light modulator;

[0439] Using the mapping, a phase function p is derived by mapping the target light pattern onto the coordinate space of the phase modulator, the phase function p including the control value; and

[0440] The mapped target light pattern is processed in the coordinate space of the phase modulator.

[0441] Option 165. The device according to Option 164, wherein the data processor is configured to process the mapped target light pattern, including the data processor being configured to: optimize the experimental phase function based on a comparison of the intensity of the mapped target light pattern at a point on the phase modulator with the corresponding optical properties of the phase function in the vicinity of the point.

[0442] Option 166. The device according to Option 165, wherein the corresponding optical property includes magnification.

[0443] Option 167. The device according to Option 165 or 166, wherein the data processor is configured to determine the optical properties based on the Laplace operator of the phase function at the corresponding point.

[0444] Solution 168. The device according to Solution 167, wherein the data processor is configured to: use a discrete Laplace operator to determine the Laplace operator of the phase function.

[0445] Option 169. The device according to any one of Options 161 to 165, wherein the data processor is configured to process the video data in the frequency domain.

[0446] Option 170. The device according to Option 169, wherein the data processor is configured to process the video data, including that the data processor is configured to:

[0447] Generate an optimization function;

[0448] A phase function is generated in the frequency domain by performing a Fourier transform on the optimized function; and

[0449] Perform an inverse Fourier transform on the phase function in the frequency domain.

[0450] Option 171. The device according to Option 170 includes hardware configured to perform a Fourier transform, and the data processor is configured to control the hardware to perform the Fourier transform.

[0451] Option 172. The device according to any one of Options 161 to 165 and 169 to 171, wherein the phase modulator has a maximum phase delay, and the data processor is configured to subtract a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0452] Solution 173. An apparatus for controlling a phase modulator to display an image defined by image data, the method comprising:

[0453] The objective function is determined based on the image data;

[0454] Transform the objective function into the frequency space;

[0455] Minimize the transformed objective function in the frequency space to obtain the phase function in the frequency space; and

[0456] The phase function is inversely transformed to obtain a dephase function, which correlates the phase of the phase modulator with its position in two dimensions.

[0457] Option 174. The device according to Option 173, wherein the data processor is configured to transform the objective function by: the data processor being configured to compute a Fourier transform of the objective function.

[0458] Solution 175. The apparatus according to Solution 174, wherein the data processor is configured to: expand the image data to have periodic boundary conditions prior to the transformation, and make the objective function based on the expanded image data.

[0459] Solution 176. The device according to Solution 175, wherein the data processor is configured to extend the image data by: the data processor being configured to form a mirror image of the image data across each boundary of the image data.

[0460] Scheme 177. The device according to any one of Schemes 173 to 176, wherein the objective function is a least-squares objective function.

[0461] Scheme 178. The device according to any one of Schemes 173 to 177, wherein the objective function includes costs for deviating from the input variables.

[0462] Scheme 179. The apparatus according to any one of Schemes 173 to 178, wherein the data processor is configured to: iteratively determine the objective function; transform the objective function and evaluate the transformed objective function; and in each of the plurality of iterations, the input variables of the objective function are the solution phase functions of the previous iteration.

[0463] Option 180. The device according to Option 179, wherein the data processor is configured to: cache the Fourier transform of the solution phase function of the previous iteration, and apply the cached Fourier transform of the solution phase function in the current iteration.

[0464] Option 181. The device according to any one of Options 173 to 180, wherein the objective function comprises a nearest neighbor operator given by:

[0465] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0466] Option 182. The device according to Option 181, wherein the data processor is configured to evaluate the transformed objective function comprising: the data processor being configured to determine

[0467]

[0468] Option 183. The device according to Option 182, wherein,

[0469] Option 184. The device according to Option 182 or 183, wherein,

[0470] Scheme 185. The device according to any one of Schemes 182 to 184, wherein α>0 is a regularization parameter.

[0471] Option 186. The device according to any one of options 173 to 185, wherein the data processor is configured to initialize the phase surface as a constant value.

[0472] Option 187. The device according to any one of options 173 to 186, wherein the data processor is configured to evaluate the transformed objective function in parallel for different points.

[0473] Option 188. The device according to Option 187, wherein the data processor includes a graphics processing unit, and the graphics processing unit evaluates the transformed objective function.

[0474] Option 189. The device according to any one of Options 173 to 188, comprising the phase modulator and a light source for projecting light onto the phase modulator, wherein the data processor is configured to control the phase modulator to display an image by controlling the pixels of the phase modulator according to the dephase function while controlling the light source to illuminate the phase modulator.

[0475] Option 190. The device according to Option 189, wherein the light source is configured to uniformly illuminate the phase modulator using collimated light.

[0476] Option 191. The device according to any one of Options 189 to 190, wherein the phase modulator comprises a liquid crystal pixel array, and the data processor is configured to set a control signal to the pixel according to the dephase function.

[0477] Option 192. The device according to Option 191, wherein the phase modulator is an LCoS phase modulator.

[0478] Option 193. The device according to Option 19, wherein the phase modulator is a deformable mirror.

[0479] Option 194. The device according to any one of Options 189 to 193, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0480] Scheme 195. The device according to any one of Schemes 173 to 194, wherein the phase modulator has a maximum phase delay, and the processor is configured to subtract a multiple of 2π from any phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0481] Solution 196. An apparatus for controlling a phase modulator to display an image defined by image data, the apparatus comprising a data processor communicating with the phase modulator, the data processor being configured to:

[0482] The fixed-point iteration is determined based on the image data;

[0483] The fixed point is iteratively transformed into the frequency space;

[0484] The fixed-point iteration is evaluated in the frequency space to obtain the phase function in the frequency space; and

[0485] The phase function is inversely transformed to obtain a dephase function, which correlates the phase of the phase modulator with its position in two dimensions.

[0486] Solution 197. The device according to Solution 196, wherein the data processor is configured to transform the fixed-point iteration by: the data processor being configured to compute a Fourier transform of the fixed-point iteration.

[0487] Solution 198. The apparatus according to Solution 197, wherein the data processor is configured to: expand the image data to have periodic boundary conditions prior to transformation, and make the fixed point iteration based on the expanded image data.

[0488] Solution 199. The device according to Solution 198, wherein the data processor is configured to extend the image data by: the data processor being configured to form a mirror image of the image data across each boundary of the image data.

[0489] Scheme 200. The device according to any one of Schemes 196 to 199, wherein the fixed-point iteration is a least-squares fixed-point iteration.

[0490] Solution 201. The device according to any one of Solutions 196 to 200, wherein the fixed-point iteration includes a cost for deviating from the input variable.

[0491] Option 202. The device according to any one of options 196 to 201, wherein the data processor is configured to: iteratively determine the fixed-point iteration; transform the fixed-point iteration and evaluate the transformed fixed-point iteration; and in each of the plurality of iterations, the input variable of the fixed-point iteration is the solution phase function of the previous iteration.

[0492] Solution 203. The device according to Solution 202, wherein the data processor is configured to: cache the Fourier transform of the solution phase function of the previous iteration, and apply the cached Fourier transform of the solution phase function in the current iteration.

[0493] Scheme 204. The device according to any one of Schemes 196 to 203, wherein the fixed-point iteration includes a nearest-neighbor operator given by the following formula:

[0494] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0495] Solution 205. The device according to Solution 204, wherein the data processor is configured to evaluate the transformed fixed-point iteration comprising: the data processor being configured to determine

[0496]

[0497] Solution 206. The device according to Solution 205, wherein,

[0498] Solution 207. The device according to Solution 205 or 206, wherein,

[0499] Scheme 208. The device according to any one of Schemes 205 to 207, wherein α>0 is a regularization parameter.

[0500] Solution 209. The device according to any one of Solutions 196 to 208, wherein the data processor is configured to initialize the phase surface as a constant value.

[0501] Option 210. The device according to any one of options 196 to 209, wherein the data processor is configured to: evaluate the transformed fixed-point iteration in parallel for different points.

[0502] Option 211. The device according to Option 210, wherein the data processor includes a graphics processing unit, and the graphics processing unit evaluates the transformed fixed-point iteration.

[0503] Solution 212. The device according to any one of Solutions 196 to 211, comprising the phase modulator and a light source for projecting light onto the phase modulator, wherein the data processor is configured to control the phase modulator to display an image by controlling the pixels of the phase modulator according to the dephase function while controlling the light source to illuminate the phase modulator.

[0504] Option 213. The device according to Option 212, wherein the light source is configured to uniformly illuminate the phase modulator using collimated light.

[0505] Option 214. The device according to any one of Options 212 to 213, wherein the phase modulator comprises a liquid crystal pixel array, and the data processor is configured to set a control signal to the pixel according to the dephase function.

[0506] Option 215. The device according to Option 214, wherein the phase modulator is an LCoS phase modulator.

[0507] Option 216. The device according to Option 214, wherein the phase modulator is a deformable mirror.

[0508] Option 217. The device according to any one of Options 212 to 216, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0509] Scheme 218. The device according to any one of Schemes 1 to 22, wherein the phase modulator has a maximum phase delay, and the processor is configured to subtract a multiple of 2π from any phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0510] Option 219. A method for controlling a phase modulator to display an image defined by image data, the method comprising:

[0511] The nearest neighbor operator of the objective function is determined based on the image data;

[0512] Transform the nearest neighbor operator into the frequency space;

[0513] The transformed nearest neighbor operator is evaluated in the frequency space to obtain a phase function in the frequency space, and the phase function is inversely transformed to obtain a dephase function that associates the phase of the phase modulator with a two-dimensional position.

[0514] Scheme 220. The method according to Scheme 219, wherein transforming the nearest neighbor operator includes: calculating the Fourier transform of the nearest neighbor operator.

[0515] Solution 221. The method according to Solution 220, comprising: expanding the image data to have periodic boundary conditions before transformation, and making the nearest neighbor operator based on the expanded image data.

[0516] Solution 222. The method according to Solution 221, wherein extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

[0517] Scheme 223. The method according to any one of Schemes 219 to 222, wherein the objective function is a least squares objective function.

[0518] Scheme 224. The method according to any one of Schemes 219 to 223, wherein the nearest neighbor operator includes a cost for deviating from the input variable.

[0519] Scheme 225. The method according to any one of schemes 219 to 224, wherein the method is performed iteratively, and in each of the multiple iterations, the input variable of the nearest neighbor operator is the solution phase function of the previous iteration.

[0520] Option 226. The method according to Option 225, comprising: caching the Fourier transform of the solution phase function of the previous iteration, and applying the cached Fourier transform of the solution phase function in the current iteration.

[0521] Scheme 227. The method according to any one of Schemes 219 to 226, wherein the nearest neighbor operator is given by the following formula:

[0522] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0523] Option 228. The method according to Option 227, wherein evaluating the transformed nearest neighbor operator includes: determining

[0524]

[0525] Option 229. The method according to Option 226, wherein,

[0526] Option 230. The method according to any one of Options 228 and 229, wherein,

[0527] Scheme 231. The method according to any one of Schemes 228 to 230, wherein α>0 is a regularization parameter.

[0528] Scheme 232. The method according to any one of Schemes 219 to 231, wherein the method comprises: initializing the phase surface as a constant value.

[0529] Scheme 233. The method according to any one of Schemes 219 to 232, wherein the evaluation of the transformed nearest neighbor operator is performed in parallel for different points.

[0530] Option 234. The method according to Option 233, wherein the evaluation is performed in the graphics processing unit.

[0531] Solution 235. The method according to any one of Solutions 219 to 234, comprising: displaying an image by controlling the pixels of the phase modulator to simultaneously illuminate the phase modulator according to the dephase function.

[0532] Option 236. The method according to Option 235, comprising: uniformly illuminating the phase modulator with collimated light.

[0533] Option 237. The method according to any one of Options 235 to 236, wherein the phase modulator comprises a liquid crystal pixel array, and the method comprises: setting a control signal to the pixel according to the dephase function.

[0534] Option 238. The method according to Option 237, wherein the phase modulator is an LCoS phase modulator.

[0535] Option 239. The method according to Option 237, wherein the phase modulator is a deformable mirror.

[0536] Option 240. The method according to any one of Options 235 to 239, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0537] Option 241. The method according to any one of Options 219 to 240, wherein the phase modulator has a maximum phase delay, and the method includes: subtracting a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0538] Solution 242. An apparatus for controlling a phase modulator to display an image defined by image data, the apparatus comprising a data processor communicating with the phase modulator, the data processor being configured to:

[0539] The nearest neighbor operator of the objective function is determined based on the image data;

[0540] Transform the nearest neighbor operator into the frequency space;

[0541] The transformed nearest neighbor operator is evaluated in the frequency space to obtain a phase function in the frequency space, and the phase function is inversely transformed to obtain a dephase function that associates the phase of the phase modulator with a two-dimensional position.

[0542] Solution 243. The device according to Solution 242, wherein the data processor is configured to transform the nearest neighbor operator by: the data processor being configured to compute the Fourier transform of the nearest neighbor operator.

[0543] Solution 244. The device according to Solution 243, wherein the data processor is configured to: expand the image data to have periodic boundary conditions before transformation, and make the nearest neighbor operator based on the expanded image data.

[0544] Solution 245. The device according to Solution 244, wherein the data processor is configured to extend the image data by: the data processor being configured to form a mirror image of the image data across each boundary of the image data.

[0545] Scheme 246. The device according to any one of Schemes 242 to 245, wherein the objective function is a least-squares objective function.

[0546] Option 247. The device according to any one of Options 242 to 246, wherein the nearest neighbor operator includes a cost for deviating from the input variable.

[0547] Scheme 248. The apparatus according to any one of Schemes 242 to 247, wherein the data processor is configured to: iteratively determine the nearest neighbor operator; transform the nearest neighbor operator and evaluate the transformed nearest neighbor operator; and in each of the plurality of iterations, the input variables of the nearest neighbor operator are the solution phase function of the previous iteration.

[0548] Option 249. The device according to Option 248, wherein the data processor is configured to: cache the Fourier transform of the solution phase function of the previous iteration, and apply the cached Fourier transform of the solution phase function in the current iteration.

[0549] Option 250. The device according to any one of Options 242 to 249, wherein the nearest neighbor operator is given by the following formula:

[0550] prox γF (q)=(γ+A T A) -1 (γq+A T b).

[0551] Option 251. The device according to Option 250, wherein the data processor is configured to evaluate the transformed nearest neighbor operator comprising: the data processor being configured to determine

[0552]

[0553] Option 252. The device according to Option 251, wherein,

[0554] Option 253. The device according to Option 251 or 252, wherein,

[0555] Scheme 254. The device according to any one of Schemes 251 to 253, wherein α>0 is a regularization parameter.

[0556] Option 255. The device according to any one of options 242 to 254, wherein the data processor is configured to initialize the phase surface as a constant value.

[0557] Option 256. The device according to any one of options 242 to 255, wherein the data processor is configured to evaluate the transformed nearest neighbor operator in parallel for different points.

[0558] Option 257. The device according to Option 256, wherein the data processor includes a graphics processing unit, and the graphics processing unit evaluates a transformed nearest neighbor operator.

[0559] Option 258. The device according to any one of Options 242 to 257, comprising the phase modulator and a light source for projecting light onto the phase modulator, wherein the data processor is configured to control the phase modulator to display an image by controlling the pixels of the phase modulator according to the dephase function while controlling the light source to illuminate the phase modulator.

[0560] Option 259. The device according to Option 258, wherein the light source is configured to uniformly illuminate the phase modulator using collimated light.

[0561] Option 260. The device according to any one of Options 258 to 259, wherein the phase modulator comprises a liquid crystal pixel array, and the data processor is configured to set a control signal to the pixel according to the dephase function.

[0562] Option 261. The device according to Option 260, wherein the phase modulator is an LCoS phase modulator.

[0563] Option 262. The device according to Option 260, wherein the phase modulator is a deformable mirror.

[0564] Option 263. The device according to any one of Options 258 to 262, wherein the maximum numerical aperture of the point on the phase modulator is 0.21 or less.

[0565] Scheme 264. The device according to any one of Schemes 242 to 264, wherein the phase modulator has a maximum phase delay, and the processor is configured to subtract a multiple of 2π from any phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

[0566] Solution 265. A computer-readable medium comprising computer-readable software instructions configured to cause a data processor to perform the method described in any one of the above method solutions.

[0567] Option 266. An apparatus having any novel and inventive feature, combination of features, or recombination of features described herein.

[0568] Option 267. A method having any novel and inventive steps, actions, combinations of steps and / or actions, or recombinations of steps and / or actions as described herein.

Claims

1. A method for controlling a phase modulator to display an image defined by image data, the method comprising: The nearest neighbor operator of the objective function is determined based on the image data; Transform the nearest neighbor operator into the frequency space; The transformed nearest neighbor operator is evaluated in the frequency space to obtain a phase function in the frequency space, and the phase function is inversely transformed to obtain a dephase function that associates the phase of the phase modulator with a two-dimensional position.

2. The method according to claim 1, wherein, Transforming the nearest neighbor operator includes: calculating the Fourier transform of the nearest neighbor operator.

3. The method according to claim 2, comprising: Before the transformation, the image data is expanded to have periodic boundary conditions, and the nearest neighbor operator is made based on the expanded image data.

4. The method according to claim 3, wherein, Extending the image data includes: forming a mirror image of the image data across each boundary of the image data.

5. The method according to claim 1, wherein, The objective function is a least squares objective function.

6. The method according to any one of claims 1 to 5, wherein, The nearest neighbor operator includes the cost for deviating from the input variable.

7. The method according to any one of claims 1 to 5, wherein, The method is performed iteratively, and in each of the multiple iterations, the input variable of the nearest neighbor operator is the solution phase function of the previous iteration.

8. The method of claim 7, comprising: The Fourier transform of the solution phase function from the previous iteration is cached, and the cached Fourier transform of the solution phase function is applied in the current iteration.

9. The method according to any one of claims 1 to 5, wherein, The nearest neighbor operator is given by the following formula: prox γF (q)=(γ+A T A) -1 (γq+A T b) Where F represents the convex function F(z), q is the input variable, and A represents the Toeplitz matrix.

10. The method according to claim 9, wherein, Evaluating the transformed nearest neighbor operator includes: determining 11. The method according to claim 9, wherein, 12. The method according to claim 10, wherein, Where i p (x) is a distorted image, where x refers to a point on the lens surface.

13. The method according to claim 10, wherein, α>0 is the regularization parameter.

14. The method according to any one of claims 1 to 5, wherein, The method includes: initializing the phase surface as a constant value.

15. The method according to any one of claims 1 to 5, wherein, The transformed nearest neighbor operator is evaluated in parallel for different points.

16. The method according to claim 15, wherein, The evaluation is performed in the graphics processing unit.

17. The method according to any one of claims 1 to 5, comprising: An image is displayed by controlling the pixels of the phase modulator to simultaneously illuminate the phase modulator according to the dephase function.

18. The method of claim 17, comprising: The phase modulator is uniformly illuminated with collimated light.

19. The method of claim 17, wherein, The phase modulator includes a liquid crystal pixel array, and the method includes setting a control signal for the pixel according to the dephase function.

20. The method according to claim 19, wherein, The phase modulator is an LCoS phase modulator.

21. The method according to claim 19, wherein, The phase modulator is a deformable mirror.

22. The method according to claim 19, wherein, The maximum numerical aperture of the point on the phase modulator is 0.21 or smaller.

23. The method according to any one of claims 1 to 5, wherein, The phase modulator has a maximum phase delay, and the method includes subtracting a multiple of 2π from the phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

24. An apparatus for controlling a phase modulator to display an image defined by image data, the apparatus comprising a data processor in communication with the phase modulator, the data processor being configured to: The nearest neighbor operator of the objective function is determined based on the image data; Transform the nearest neighbor operator into the frequency space; The transformed nearest neighbor operator is evaluated in the frequency space to obtain a phase function in the frequency space, and the phase function is inversely transformed to obtain a dephase function that associates the phase of the phase modulator with a two-dimensional position.

25. The device according to claim 24, wherein, The data processor is configured to transform the nearest neighbor operator by: the data processor being configured to compute the Fourier transform of the nearest neighbor operator.

26. The device according to claim 25, wherein, The data processor is configured to: expand the image data to have periodic boundary conditions before transformation, and make the nearest neighbor operator based on the expanded image data.

27. The device according to claim 26, wherein, The data processor is configured to extend the image data by: the data processor being configured to form a mirror image of the image data across each boundary of the image data.

28. The device according to claim 25, wherein, The objective function is a least squares objective function.

29. The device according to any one of claims 24 to 28, wherein, The nearest neighbor operator includes the cost for deviating from the input variable.

30. The device according to any one of claims 24 to 28, wherein, The data processor is configured to: iteratively determine the nearest neighbor operator; transform the nearest neighbor operator; and evaluate the transformed nearest neighbor operator; Furthermore, in each of the multiple iterations, the input variable of the nearest neighbor operator is the solution phase function of the previous iteration.

31. The device according to claim 30, wherein, The data processor is configured to cache the Fourier transform of the solution phase function from the previous iteration and apply the cached Fourier transform of the solution phase function in the current iteration.

32. The device according to any one of claims 24 to 28, wherein, The nearest neighbor operator is given by the following formula: prox γF (q)=(γ+A T A) -1 (γq+A T b), Where F represents the convex function F(z), q is the input variable, and A represents the Toeplitz matrix.

33. The device according to claim 32, wherein, The data processor is configured to evaluate the transformed nearest neighbor operator, including: the data processor is configured to determine...

34. The device according to claim 33, wherein, 35. The device according to claim 33, wherein, Where i p (x) is a distorted image, where x refers to a point on the lens surface.

36. The device according to claim 33, wherein, α>0 is the regularization parameter.

37. The device according to any one of claims 24 to 28, wherein, The data processor is configured to initialize the phase surface as a constant value.

38. The device according to any one of claims 24 to 28, wherein, The data processor is configured to evaluate the transformed nearest neighbor operator in parallel for different points.

39. The device according to claim 38, wherein, The data processor includes a graphics processing unit, which evaluates the transformed nearest neighbor operator.

40. The device according to any one of claims 24 to 28, comprising the phase modulator and a light source for projecting light onto the phase modulator, wherein the data processor is configured to control the phase modulator to display an image by controlling the pixels of the phase modulator according to the dephase function while controlling the light source to illuminate the phase modulator.

41. The device according to claim 40, wherein, The light source is configured to uniformly illuminate the phase modulator using collimated light.

42. The device according to claim 40, wherein, The phase modulator includes a liquid crystal pixel array, and the data processor is configured to set control signals for the pixels according to the dephase function.

43. The device according to claim 42, wherein, The phase modulator is an LCoS phase modulator.

44. The device according to claim 42, wherein, The phase modulator is a deformable mirror.

45. The device according to any one of claims 24 to 28, wherein, The maximum numerical aperture of the point on the phase modulator is 0.21 or smaller.

46. ​​The device according to any one of claims 24 to 28, wherein, The phase modulator has a maximum phase delay, and the processor is configured to subtract a multiple of 2π from any phase shift of the phase function that exceeds the maximum phase delay of the phase modulator.

Images