Image processing

Abstract

A method for performing a complex Fourier transform on a complex data set corresponding to an image is disclosed. The method includes receiving the complex data set and performing a first 1D complex Fourier transform on the complex data set in Cartesian form; converting the complex data set to polar form and compressing the complex data set in polar form; performing a row-column transform on the complex data set; decompressing the complex data set and converting the complex data set back to Cartesian form; and performing a second 1D Fourier transform on the complex data set in Cartesian form, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform. A corresponding system and iterative computation for computer-generated holograms are also disclosed.

Description

Technical Field

[0001] The present disclosure relates to image processing. More specifically, the present disclosure relates to efficient implementation of two-dimensional Fourier transforms of complex image data, and data compression of complex-valued image data. In particular, although not exclusively, the present disclosure relates to processing of image data for holographic displays. Background Art

[0002] Light scattered from an object contains both amplitude and phase information. This amplitude and phase information can be captured, for example, on a photographic plate, using well-known interference techniques to form a holographic recording or "hologram" consisting of interference fringes. The hologram can be reconstructed by illuminating it with appropriate light to form a two- or three-dimensional holographic reconstruction or playback image representing the original object.

[0003] Computer-generated holography numerically simulates interference processes. Computer-generated holograms (CGHs) can be computed using techniques based on mathematical transformations, such as the Fresnel transform or the Fourier transform. These types of holograms are referred to as Fresnel or Fourier holograms. A Fourier hologram can be considered a Fourier domain representation of an object or a frequency domain representation of an object. CGHs can also be computed using coherent ray tracing or point cloud techniques, for example.

[0004] The CGH may be encoded on a spatial light modulator "SLM", which is arranged to modulate the amplitude and / or phase of incident light. For example, light modulation may be achieved using electrically addressable liquid crystals, optically addressable liquid crystals, or micromirrors.

[0005] An SLM may comprise a plurality of individually addressable pixels, which may also be referred to as cells or elements. The light modulation scheme may be binary, multi-level or continuous. Alternatively, the device may be continuous (i.e. comprising no pixels), and the light modulation may therefore be continuous across the device. An SLM may be reflective, meaning that the modulated light is output from the SLM in reflection. An SLM may also be transmissive, meaning that the modulated light output from the SLM is in transmission.

[0006] The described techniques can be used to provide holographic projectors for imaging. Such projectors have applications in heads-up displays (HUDs) and head-mounted displays (HMDs), including near-eye devices, for example. Summary of the Invention

[0007] Aspects of the present disclosure are defined in the accompanying independent claims.

[0008] In a first aspect, an image processing system is provided, which is arranged to perform a Fourier transform on a first complex data set representing (i.e. corresponding to) a two-dimensional image. The first complex data set comprises a two-dimensional array of complex values ​​in Cartesian form. The values ​​of the two-dimensional array represent (i.e. correspond to) pixels of the two-dimensional image. For example, each value in the two-dimensional array of the complex data set comprises a pixel value. Each value is represented in a first digital format having a first digit. The two-dimensional array defines a first linear array, such as a row, representing the values ​​of pixels along one dimension, and a second linear array, such as a column, representing the values ​​of pixels along another dimension. Of course, it will be understood that the references to rows or columns used herein are for convenience, and the assignment between linear arrays and rows / columns and between rows / columns and the various dimensions of the image is arbitrary, and the terms row and column can be used interchangeably. In addition, a row of the array can correspond to a column of pixels of the image, a column of the array can correspond to a row of pixels of the image, and vice versa.

[0009] In some arrangements, an image processing system includes one or more processors configured to perform a Fourier transform of a first complex data set corresponding to an image, and a tangible, non-transitory, computer-readable medium storing instructions executable by the one or more processors. The instructions, when executed, cause the image processing system to perform functions including: performing a 1D Fourier transform on each first linear array of values ​​of the first complex data set in Cartesian form; converting each value of each Fourier transformed first linear array into polar form; converting each value in the polar form into a second digital format having a second number of digits, wherein the second number of digits is less than the first number of digits; storing the values ​​in the second digital format in a first linear array of memory via the first linear array; reading the values ​​from the second linear array of memory via the second linear array; converting each value read from the memory back to the first digital format; converting each value converted back to the first digital format back to Cartesian form; performing a 1D Fourier transform on each second linear array of values ​​converted back to Cartesian form, wherein the second linear array of Fourier transformed values ​​is used to form a second complex data set; and forming image data using the second complex data set. The system may also include a controller configured to output the image data for display.

[0010] In another arrangement, the system includes a first Fourier transform engine configured to receive a first complex data set (a first linear array) via a first linear array and perform a 1D Fourier transform on each of the first linear arrays. The first transform engine is configured to receive the complex data set processed by the first Fourier transform engine, convert each value in polar form, and convert each value in the polar form into a second digital format having a second number of digits, wherein the second number of digits is smaller than the first number of digits. A row-column transform engine is configured to receive the complex data set processed by the first transform engine, store the values ​​in the second digital format in a memory (a first linear array) via the first linear array, and read the values ​​from the memory (a second linear array) via a second linear array. In effect, the arrays are transposed. A second transform engine is configured to receive the complex data set processed by the second linear array of the row-column transform engine via the second linear array, convert each value back to the first digital format, and convert each value converted back to the first digital format back into Cartesian form. A second Fourier transform engine is configured to receive the complex data set processed by the second linear array of the second transform engine via the second linear array and perform a 1D Fourier transform on each of the second linear arrays. The system is further arranged to form image data for display using data values ​​of the complex data set processed by the second Fourier transform engine.For example, the system may comprise an output interface arranged to output the image data for display.

[0011] Advantageously, by reducing the number of bits required to store each complex data value in the row-column transformed data set, certain memory bottlenecks are alleviated. While most engines in the system can process the data stream through linear arrays, the row-column (or transpose) engine must receive and store all of the first linear array before reading out the second linear array. Therefore, reducing the number of bits per value at this stage is particularly effective because it reduces the length of time data is retained at the row-column transform memory bottleneck, thereby increasing processing speed. Furthermore, the memory capacity required for the row-column transform is also reduced. Furthermore, the present disclosure provides an efficient way to reduce the number of bits per value by operating on polar coordinates to reduce the number of bits per value, thereby mitigating quality loss and ensuring that image quality is maintained. This is because phase information is better preserved when represented with a reduced number of bits in a polar coordinate representation. Therefore, this way of transforming data is particularly suitable for algorithms where important information (perhaps the most important image-forming information) is actually encoded in the phase, such as algorithms that generate image holograms or similar image processing algorithms. In contrast, compression in Cartesian coordinates leads to larger errors in the phase component—particularly when a large number of low-amplitude values ​​are generated—as is the case with the Fourier transform-based processes disclosed herein. In some respects, it can be said that the improved methods disclosed herein prioritize phase information and are therefore advantageous in situations such as fully complex holography, where errors in the phase component are found to be more destructive than errors in the amplitude component. Once the values ​​have been stored (buffered) in preparation for the row-to-column transform, they can be converted back into a more expensive digital format, more suitable for Fourier transform calculations, to avoid the propagation of rounding errors. In addition, the speed of the row-to-column transform is increased, for example by writing smaller data words for each value (e.g., 16 bits each for phase and amplitude). In a 64-bit computing architecture, two values ​​can be written for each 64-bit word.

[0012] The image processing system of the first aspect receives and processes image data for display. For example, the system may receive image data from an image source comprising a two-dimensional array of complex values ​​in Cartesian form representing pixels of a two-dimensional input image. The system may process the input image data to form image data comprising a two-dimensional array of complex values ​​in Cartesian form representing pixels of a two-dimensional output image for display. It should be understood that the complex array of output image data is displayed on individual pixels of a display device. Image processing includes performing a two-dimensional Fourier transform of the input image data, which is commonly used in image processing techniques. As described herein, such image processing techniques include methods and algorithms for forming a computer-generated hologram corresponding to the input image. Therefore, in some embodiments, a display device encoded with the output image data displays an output image corresponding to the processed input image. In other embodiments, a display device encoded with the output image data displays a hologram corresponding to the input image. Therefore, as used herein, the term "image data" includes data that, when encoded on pixels of a display device, causes the display device to display an image or a hologram of an image.

[0013] In some embodiments, the image processing system of the first aspect can be used as a building block to implement image processing for computationally generating holograms for display and the holographic reconstructions disclosed herein. For example, such a system can include two data processing systems as described above, functionally linked together with an intermediate data processor that manipulates the amplitude of the complex values ​​of the output of the first data processing system to generate inputs for the second data processing system. In some embodiments, the second data processing system can be connected back to the first data processing system in a similar or identical manner to implement iterative computations. It should be understood that the first and second data processing systems (and corresponding engines) are effectively the same system, such that the outputs of the data processing systems are manipulated as described above and then fed back into the inputs. While the first and second data processing systems are not the same system and are not identical, they can still share one or more engines. Furthermore, the engines within each data processing system can correspond to separate instances (i.e., two Fourier transform engines, two transform engines, etc.), or one or more engines can effectively be a single engine (i.e., the first Fourier transform engine is also the second engine, the first transform engine is also the second engine, etc.), with values ​​routed between the engines accordingly. Similarly, each data processing system can use the same or different high-order digital format and the same or different low-order digital format.

[0014] It should be understood that a variety of standard or custom number formats can be used. For example, a complex value can be stored as two double floating point numbers (128 bits), one for each of the complex number / imaginary part or phase / angle, depending on the form of the complex number, and similarly two single floating point numbers (64 bits) or two half floating point numbers (32 bits). Custom number formats, for example, assuming a fixed decimal point and fixed / no exponent are equally possible, such as encoding each complex value to 128, 64, or 32 bits in this way. In such custom number formats, the number of bits assigned to the real / imaginary part can be the same for each of the real and imaginary parts, or different numbers of bits can be assigned to each component of the complex value. In any case, the present disclosure includes converting a floating point (or other decimal or binary number) from a format with a first digit to a format with a second digit, such as by scaling, reallocating the bit value to a decimal value, rounding, deleting, and adding one or more of the least significant bits, etc.

[0015] In some embodiments, converting the polar coordinate value into the second digital format may include converting each amplitude value from x1 bits to y1 bits, where x1>y1, and converting each phase value from x2 bits to y2 bits, where x2>y2. The number of bits x1 and x2 may be the same and / or y1 and y2 may be the same, such as x and y. In a specific example, x1=x2=32 bits (64 bits per data value) and y1=y2=8 bits (16 bits per data value). Converting each value in polar coordinate form into the second digital format may include multiplying each amplitude value by a scaling factor and / or may include setting any amplitude value above the maximum allowable value to the maximum allowable value. For example, the thresholding may be performed after scaling. Thresholding may include removing the least significant bit from the amplitude value and / or rounding. The maximum allowable value may be the maximum amplitude value that can be represented by y1 bits. In some embodiments, converting each value in polar coordinate form into the second digital format includes quantizing the phase value of the polar coordinate value using a lower magnitude than the first digital format. For example, this may involve removing the least significant bit from the phase value or rounding. Of course, other techniques for reducing the number of bits required to store each complex value may also be used.

[0016] In some embodiments, converting each value in Cartesian form back to the first digital format includes converting each amplitude value from y1 bits back to x1 bits and converting each phase value from y2 bits back to x2 bits. For example, this may involve multiplying each amplitude value by the inverse of a scale factor.

[0017] In some embodiments, the memory of the row-column conversion engine is static random access memory (SRAM), which advantageously has faster access times than dynamic random access memory (DRAM). Reducing the required memory by reducing the number of bits per complex value helps to (partially) offset the higher cost per bit of SRAM.

[0018] In some embodiments, a first Fourier transform engine is configured to process a first linear array while the first transform engine processes another first linear array, wherein a second Fourier transform engine is configured to process a second linear array while the second transform engine processes another second linear array, or both. For example, the first and optional second Fourier transform engines can receive complex values ​​as a stream of values. These values ​​can be processed over a serial bus.

[0019] In some embodiments, the system includes an output interface arranged to output image data for display. For example, the controller may include an output interface of the system arranged to output a hologram representing an input image to a holographic display device, as described herein. The hologram may be a pure phase hologram or a full complex hologram comprising phase and amplitude components. The input interface may be arranged to receive a sequence of images from an image source (e.g., a camera) in real time, and the output interface may be arranged to output a sequence of holograms representing the corresponding images (or corresponding processed images) in real time. As a result, the images are processed in real time upon reception, i.e., with low latency, such that holographic video frames can be generated from online image video frames at typical video frame rates (e.g., 24fps). In some embodiments, the holographic display device includes a spatial light modulator arranged to display a hologram and a light source arranged to illuminate the spatial light modulator to form a holographic reconstruction. In some embodiments, the image processing system and the holographic display may comprise a single integrated device.

[0020] In a second aspect, an image processing method is disclosed for performing a complex Fourier transform on a first complex data set corresponding to an image. The first complex data set includes a two-dimensional array of complex values ​​in Cartesian form, each complex value represented in a first digital format having a first digit. The values ​​of the two-dimensional array represent (i.e., correspond to) pixels of the two-dimensional image. The two-dimensional array defines a first linear array of values ​​representing pixels along one dimension and a second linear array of values ​​representing pixels along another dimension, such as rows and columns as described above.

[0021] The method includes the following steps: performing a 1D Fourier transform on each first linear array in a first complex data set; converting each value of each Fourier transformed first linear array into polar form; converting each value in the polar form into a second digital format having a second number of digits, wherein the second number of digits is less than the first number of digits; storing the values ​​in the second digital format in a first linear array in a memory via the first linear array; reading the values ​​from the second linear array in the memory via the second linear array; converting each value read from the memory back to the first digital format; converting each value converted back to the first digital format back to Cartesian form; and performing a 1D Fourier transform on each second linear array of values ​​converted back to Cartesian form. The method also includes using the Fourier transformed second linear array of values ​​to form image data for display. For example, the Fourier transformed second linear array of values ​​can be used to form a second complex data set, and the method can further include using the second complex data set to form image data.

[0022] Similarly, as described above, for this system, the second linear array of Fourier transforms can be used as the first linear array in the first Fourier transform step, after amplitude manipulation as described above (and described in more detail below), in an algorithm or similar image processing technique for generating an image hologram. As previously mentioned, the digital format between the two paths can be the same or different, and for an iterative process, the steps can be repeated over the first Fourier transform step. In addition, all paths can use the same memory for row-column transforms, or, for example, separate such memories can be used for odd and even paths.

[0023] Of course, the features of the embodiments of the first aspect described above are also applicable to the second aspect and other aspects disclosed below.

[0024] In a third aspect, an image processing system is configured to process a complex data set corresponding to an image. The first complex data set includes complex values ​​in Cartesian form, each complex value represented in a first digital format having a first digit. The complex values ​​represent (i.e., correspond to) pixels of the image. The system includes a conversion engine configured to receive the complex data set, convert each value in polar form, and convert each value in the polar form into a second digital format having a second digit, wherein the second digit is smaller than the first digit.

[0025] In some embodiments, the complex values ​​of the data set are arranged in a two-dimensional array that defines a first linear array of values ​​along one dimension and a second linear array of values ​​along another dimension. In some embodiments, the data processing system further comprises a row-column transform engine arranged to receive the complex data set processed by the first transform engine, store the values ​​in the second digital format in a first linear array of memory via the first linear array, and read the values ​​from the second linear array of memory via the second linear array.

[0026] In some embodiments, the system includes a Fourier transform engine arranged to receive a complex data set of the first linear arrays through the first linear arrays and perform a 1D Fourier transform on each of the first linear arrays, and the transform engine is arranged to receive the complex data set from the Fourier transform engine one linear array at a time, each linear array being Fourier transformed by the Fourier transform engine.

[0027] In a fourth aspect, a method for processing an image is disclosed for processing a first complex data set corresponding to an image. The complex data set includes complex values ​​in Cartesian form, each complex value represented in a first digital format having a first digit. The complex values ​​represent (i.e., correspond to) pixels of the image. The method includes converting each value in polar form, and converting each value in the polar form into a second digital format having a second digit, wherein the second digit is smaller than the first digit.

[0028] In some embodiments, the complex values ​​of the data set are arranged in a two-dimensional array that defines a first linear array representing values ​​of pixels along one dimension and a second linear array representing values ​​of pixels along another dimension. In some embodiments, the method includes storing the values ​​in the second digital format in a memory first linear array via the first linear array and reading the values ​​from the memory second linear array via the second linear array.

[0029] In a fifth aspect, an image processing system is arranged to perform a complex Fourier transform on a complex data set corresponding to an image. The system comprises:

[0030] a first Fourier transform engine arranged to receive a complex data set and to perform a first 1D complex Fourier transform of the complex data set in Cartesian form;

[0031] a first data conversion engine arranged to receive the complex data set processed by the first Fourier transform engine, convert the complex data set into polar form, and compress the complex data set in polar form;

[0032] a row-column transformation engine arranged to receive the complex data set processed by the first data conversion engine and perform a row-column transformation of the complex data set;

[0033] a second data conversion engine arranged to receive the complex data set processed by the row-column transform engine, decompress the complex data set and convert the complex data set back into Cartesian form; and

[0034] a second Fourier transform engine arranged to receive the complex data set processed by the second data conversion engine and perform a second 1D complex Fourier transform of the complex data in Cartesian form, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform. The system is further arranged to use the data values ​​of the complex data set processed by the second Fourier transform engine to form image data for display.

[0035] In a sixth aspect, an image processing method is disclosed for performing a complex Fourier transform on a complex data set corresponding to an image. The method comprises the following steps:

[0036] receiving a complex data set and performing a first 1D complex Fourier transform of the complex data set in Cartesian form;

[0037] Convert complex data sets into polar coordinate form and compress complex data sets in polar coordinate form;

[0038] Perform row-column transformations on complex datasets;

[0039] Unpack the complex dataset and convert the complex dataset back to Cartesian form;

[0040] A second 1D Fourier transform is performed on the Cartesian complex data set, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform. The method further includes using the Fourier transformed complex data set to form image data for display.

[0041] Compression of image data may be performed as described above (by converting to a different digital format to reduce the number of bits per pixel value) or in any other suitable manner.

[0042] In a seventh aspect, a system for compressing a complex data set representing an image is disclosed, the system comprising a data conversion engine arranged to receive the complex data set, convert the complex data set into a polar form, and compress the complex data set in the polar form.

[0043] In an eighth aspect, a method of compressing a complex data set representing an image is provided, the method comprising converting the complex data set into a polar form, and compressing the complex data set in the polar form.

[0044] The term "hologram" is used to refer to a record that contains amplitude and / or phase information about an object. The term "holographic reconstruction" is used to refer to an optical reconstruction of an object formed by illuminating a hologram. The term "playback plane" is used herein to refer to the plane in space in which the holographic reconstruction is fully formed. The term "playback field" is used herein to refer to a sub-region of the playback plane that can receive spatially modulated light from a spatial light modulator. The terms "image" and "image region" refer to the region of the playback field that is illuminated by the light that forms the holographic reconstruction. In embodiments, an "image" may include discrete points referred to as "image pixels."

[0045] The terms "encoding," "writing," or "addressing" are used to describe the process of providing a plurality of control values ​​to a plurality of pixels of the SLM, which control values ​​respectively determine the modulation level of each pixel. It can be said that the pixels of the SLM are configured to "display" a light modulation profile in response to receiving the plurality of control values.

[0046] It has been discovered that a holographic reconstruction of acceptable quality can be formed from a "hologram" that contains only phase information related to the original object. Such a holographic recording may be referred to as a pure phase hologram. However, any reference to a pure phase hologram is by way of example only. The present disclosure is equally applicable to forming a holographic reconstruction using both amplitude and phase information related to the original object. In some embodiments, this is achieved by complex modulation using a so-called fully complex hologram that contains both amplitude and phase information related to the original object. Because the value (gray level) assigned to each pixel of the hologram has an amplitude and a phase component, such a hologram may be referred to as a fully complex hologram. The value (gray level) assigned to each pixel may be represented as a complex number having an amplitude and a phase component. In some embodiments, a fully complex computer-generated hologram is calculated.

[0047] Reference may be made to the phase value, phase component, phase information, or simply phase of a pixel of a computer-generated hologram or spatial light modulator as shorthand for "phase delay". That is, any phase value described is actually a number (e.g., in the range of 0 to 2π) that represents the amount of phase delay provided by that pixel. For example, a pixel of a spatial light modulator described as having a phase value of π / 2 will cause the phase of the received light to change by π / 2 radians. In some embodiments, each pixel of the spatial light modulator can operate at one of a plurality of possible modulation values ​​(e.g., phase delay values). The term "grayscale" may be used to refer to a plurality of available modulation levels. For example, the term "grayscale" may be used for convenience to refer to a plurality of available phase levels in a pure phase modulator, even though the different phase levels do not provide different shades of gray. The term "grayscale" may also be used for convenience to refer to a plurality of available complex modulation levels in a complex modulator.

[0048] Although different aspects, embodiments, and groups of aspects and embodiments may be disclosed separately in this disclosure, any feature of any embodiment or group of embodiments may be combined with any other feature or combination of features of any embodiment or group of embodiments. That is, all possible combinations and permutations of the features disclosed in this disclosure are contemplated and are hereby disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Certain embodiments are described, by way of example only, with reference to the following drawings:

[0050] Figure 1 is a schematic diagram showing a reflective SLM producing a holographic reconstruction on a screen;

[0051] Figure 2A A first iteration of an example Gerchberg-Saxton type algorithm is shown;

[0052] Figure 2B The second and subsequent iterations of an example Gerchberg-Saxton type algorithm are shown;

[0053] Figure 2C An alternative second and subsequent iteration of an example Gerchberg-Saxton type algorithm is shown;

[0054] Figure 3 is a schematic diagram of a reflective LCOS SLM;

[0055] Figures 4A to 4D The 2D Fourier transform is shown;

[0056] Figure 5 An image processing system arranged to Fourier transform and convert image data is shown;

[0057] Figure 6 shows an image processing system arranged to implement a 2D Fourier transform;

[0058] Figure 7 shows an image processing system arranged to implement an inverse 2D Fourier transform;

[0059] Figure 8 An image processing system arranged to carry out iterative calculations suitable for computer generation of holograms is shown;

[0060] Figure 9 Methods of Fourier transforming and converting image data are shown, implementing 2D Fourier transforms and iterative calculations suitable for computer generation of holograms; and

[0061] Figure 10 A computing device is shown.

[0062] The same reference numbers will be used throughout the drawings to refer to the same or like parts. DETAILED DESCRIPTION

[0063] The present invention is not limited to the embodiments described below, but extends to the full scope of the appended claims. That is, the present invention can be implemented in different forms and should not be construed as limited to the described embodiments, which are set forth for illustrative purposes.

[0064] Unless otherwise stated, terms in the singular may include plural forms.

[0065] A structure described as being formed on / under or above / below another structure should be construed to include the case where the structures are in contact with each other and the case where a third structure is provided therebetween.

[0066] When describing a temporal relationship, for example, when the chronological order of events is described as "after," "subsequently," "next," "before," etc., the present disclosure should be considered to include both sequential and non-sequential events unless otherwise specified. For example, the description should be considered to include non-sequential situations unless words such as "only," "immediately," or "directly" are used.

[0067] Although the terms "first," "second," etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish between the various elements. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the appended claims.

[0068] The features of different embodiments may be coupled or combined with each other in part or in whole, and may interoperate with each other in different ways. Some embodiments may be performed independently of each other, or may be performed together in an interdependent relationship.

[0069] Optical configuration

[0070] Figure 1 An embodiment is shown in which a computer-generated hologram is encoded on a single spatial light modulator. The computer-generated hologram is a Fourier transform of the object being reconstructed. Thus, the hologram can be said to be a Fourier, frequency, or spectral domain representation of the object. In this embodiment, the spatial light modulator is a reflective liquid crystal on silicon (LCOS) device. The hologram is encoded on the spatial light modulator, and a holographic reconstruction is formed at a playback field, such as a light-receiving surface like a screen or diffuser.

[0071] A light source 110, such as a laser or laser diode, is arranged to illuminate the SLM 140 via a collimating lens 111. The collimating lens causes a substantially planar wavefront of light to be incident on the SLM. Figure 1In embodiments, the direction of the wavefront is off-normal (e.g., two or three degrees off-normal to the plane of the transparent layer). However, in other embodiments, a substantially planar wavefront is provided at normal incidence, and a beam splitter is arranged to separate the input and output optical paths. Figure 1 In the embodiment shown, the arrangement is such that light from the light source reflects from the mirrored back surface of the SLM and interacts with the light modulating layer to form an exit wavefront 112. The exit wavefront 112 is applied to an optical device comprising a Fourier transform lens 120, whose focus is located on a screen 125. More specifically, the Fourier transform lens 120 receives the modulated light beam from the SLM 140 and performs a frequency-to-space transform to produce a holographic reconstruction on the screen 125.

[0072] It's important to note that in this type of holography, every pixel of the hologram contributes to the entire reconstruction. There is no one-to-one correlation between a specific point on the playback field (or image pixel) and a specific light modulation element (or hologram pixel). In other words, the modulated light leaving the light modulation layer is distributed across the entire playback field.

[0073] In these embodiments, the position of the holographic reconstruction in space is determined by the power (focus) of the Fourier transform lens. Figure 1 In the illustrated embodiment, the Fourier transform lens is a physical lens. That is, the Fourier transform lens is an optical Fourier transform lens, and the Fourier transform is performed optically. Any lens can function as a Fourier transform lens, but the accuracy of the Fourier transform it performs will be limited by the lens's performance. Those skilled in the art understand how to use lenses to perform optical Fourier transforms.

[0074] Hologram computing

[0075] In some embodiments, the computer-generated hologram is a Fourier transform hologram, or simply a Fourier hologram or a Fourier-based hologram, in which the image is reconstructed in the far field by utilizing the Fourier transform properties of a positive lens (a physical lens or a software lens implemented by a corresponding phase delay distribution displayed on the SLM). The Fourier hologram is calculated by Fourier transforming the desired light field in the playback plane back to the lens plane. The computer-generated Fourier hologram can be calculated using the Fourier transform.

[0076] Algorithms such as the Gerchberg-Saxton algorithm can be used to calculate Fourier transform holograms. Furthermore, the Gerchberg-Saxton algorithm can be used to calculate a hologram in the Fourier domain (i.e., a Fourier transform hologram) from pure amplitude information in the spatial domain (e.g., a photograph). Phase information associated with the object is effectively "retrieved" from the pure amplitude information in the spatial domain. In some embodiments, computer-generated holograms are calculated from pure amplitude information using the Gerchberg-Saxton algorithm or a variation thereof.

[0077] The Gerchberg-Saxton algorithm considers that when the intensity cross section I of the beam in planes A and B is known, A (x,y) and I B (x,y) and I A (x,y) and I B (x,y) is related by a single Fourier transform. For a given intensity cross section, the phase distribution in planes A and B is approximated as Ψ A (x,y) and Ψ B (x,y). The Gerchberg-Saxton algorithm finds a solution to this problem by following an iterative process. More specifically, the Gerchberg-Saxton algorithm iteratively applies spatial and spectral constraints while repeatedly transferring representations of I between the spatial domain and the Fourier (spectral or frequency) domain. A (x,y) and I B A data set (amplitude and phase) of (x, y) is provided. A corresponding computer-generated hologram in the spectral domain is obtained by at least one iteration of the algorithm. The algorithm is convergent and arranged to produce a hologram representing the input image. The hologram can be a pure amplitude hologram, a pure phase hologram, or a full complex hologram.

[0078] In some embodiments, the pure phase hologram is calculated using an algorithm based on the Gerchberg-Saxton algorithm, such as described in British Patent Nos. 2,498,170 or 2,501,112, the entire contents of which are incorporated herein by reference. However, the embodiments disclosed herein describe the calculation of pure phase holograms by way of example only. In these embodiments, the Gerchberg-Saxton algorithm retrieves the phase information Ψ[u,v] of the Fourier transform of a data set, which produces known amplitude information T[x,y], where the amplitude information T[x,y] represents the target image (e.g., a photograph). Since amplitude and phase are inherently combined in the Fourier transform, the transformed amplitude and phase contain useful information about the accuracy of the calculated data set. Therefore, the algorithm can be used iteratively with feedback of the amplitude and phase information. However, in these embodiments, the pure phase information Ψ[u,v] is used as a hologram to form a holographic representation of the target image on the image plane. The hologram is a data set (e.g., a 2D array) of phase values.

[0079] In other embodiments, an algorithm based on the Gerchberg-Saxton algorithm is used to calculate a full complex hologram. A full complex hologram is a hologram having an amplitude component and a phase component. A hologram is a data set (e.g., a 2D array) comprising an array of complex data values, where each complex data value comprises an amplitude component and a phase component.

[0080] In some embodiments, the algorithm processes complex data and the Fourier transform is a complex Fourier transform. The complex data can be viewed as comprising (i) a real component and an imaginary component, referred to as Cartesian form, or (ii) an amplitude component and a phase component, referred to as polar form. In some embodiments, the two components of the complex data are treated differently at various stages of the algorithm.

[0081] Figure 2A A first iteration of an algorithm for calculating a pure phase hologram according to some embodiments is shown. The input to the algorithm is an input image 210 comprising a 2D array of pixels or data values, wherein each pixel or data value is an amplitude or magnitude value. That is, each pixel or data value of the input image 210 does not have a phase component. Therefore, the input image 210 can be regarded as a pure amplitude or pure amplitude or pure intensity distribution. An example of such an input image 210 is a frame of a photograph or a video comprising a time sequence of frames. The first iteration of the algorithm begins with a data formation step 202A, which comprises assigning a random phase value to each pixel of the input image using a random phase distribution (or random phase seed) 230 to form a starting complex data set, wherein each data element of the data set comprises an amplitude and a phase. It can be said that the starting complex data set represents the input image in the spatial domain.

[0082] First processing block 250 receives the starting complex data set and performs a complex Fourier transform to form a Fourier transformed complex data set. Second processing block 253 receives the Fourier transformed complex data set and outputs hologram 280A. In some embodiments, hologram 280A is a pure phase hologram. In these embodiments, second processing block 253 quantizes each phase value and sets each amplitude value to unity to form hologram 280A. Each phase value is quantized according to a phase level that can be represented at a pixel of the spatial light modulator that will be used to "display" the pure phase hologram. For example, if each pixel of the spatial light modulator provides 256 different phase levels, each phase value of the hologram is quantized to one of the 256 possible phase levels. Hologram 280A is a pure phase Fourier hologram representing the input image. In other embodiments, hologram 280A is a fully complex hologram comprising an array of complex data values ​​(each comprising an amplitude component and a phase component) derived from a received Fourier transformed complex data set. In some embodiments, second processing block 253 constrains each complex data value to one of a plurality of permissible complex modulation levels to form hologram 280A. The constraining step may include setting each complex data value to the closest permissible complex modulation level in the complex plane. Hologram 280A can be said to represent the input image in the spectral domain, the Fourier domain, or the frequency domain. In some embodiments, the algorithm stops at this point.

[0083] However, in other embodiments, the algorithm continues as Figure 2A In other words, following Figure 2A The steps indicated by dashed arrows are optional (ie, not essential for all embodiments).

[0084] The third processing block 256 receives the modified complex data set from the second processing block 253 and performs an inverse Fourier transform to form an inverse Fourier transformed complex data set. The inverse Fourier transformed complex data set can be said to represent the input image in the spatial domain.

[0085] Fourth processing block 259 receives the inverse Fourier transformed complex data set and extracts the distribution of amplitude values ​​211A and the distribution of phase values ​​213A. Optionally, fourth processing block 259 evaluates the distribution of amplitude values ​​211A. Specifically, fourth processing block 259 may compare the distribution of amplitude values ​​211A of the inverse Fourier transformed complex data set with input image 510, which itself is a distribution of amplitude values. If the difference between the distribution of amplitude values ​​211A and input image 210 is sufficiently small, fourth processing block 259 may determine that hologram 280A is acceptable. In other words, if the difference between the distribution of amplitude values ​​211A and input image 210 is sufficiently small, fourth processing block 259 may determine that hologram 280A is a sufficiently accurate representation of input image 210. In some embodiments, the distribution of phase values ​​213A of the inverse Fourier transformed complex data set is ignored for comparison purposes. It will be appreciated that any number of different methods may be employed to compare the distribution of amplitude values ​​211A to the input image 210, and the present disclosure is not limited to any particular method. In some embodiments, a mean square error is calculated, and if the mean square error is less than a threshold, the hologram 280A is deemed acceptable. If the fourth processing block 259 determines that the hologram 280A is unacceptable, further iterations of the algorithm may be performed. However, this comparison step is not required, and in other embodiments, the number of iterations of the algorithm performed is predetermined, preset, or user-defined.

[0086] Figure 2B 2 represents the second iteration of the algorithm and any further iterations of the algorithm. The distribution of phase values ​​213A from the previous iteration is fed back through the processing blocks of the algorithm. The distribution of amplitude values ​​211A is rejected in favor of the distribution of amplitude values ​​of the input image 210. In the first iteration, the data forming step 202A forms a first complex data set by combining the distribution of amplitude values ​​of the input image 210 with the random phase distribution 230. However, in the second and subsequent iterations, the data forming step 202B includes forming a complex data set by combining (i) the distribution of phase values ​​213A from the previous iteration of the algorithm with (ii) the distribution of amplitude values ​​of the input image 210.

[0087] Then, with reference to Figure 2A The same method as described is handled by Figure 2BThe complex data set formed in step 202B is used to form the second iterative hologram 280B. Therefore, the description of the process will not be repeated here. When the second iterative hologram 280B has been calculated, the algorithm can stop. However, any number of further iterations of the algorithm can be performed. It will be understood that the third processing block 256 is only required when the fourth processing block 259 is required or further iterations are required. The output hologram 280B generally gets better with each iteration. However, in practice, a point is often reached where no measurable improvement can be observed, or the positive benefits of performing further iterations are offset by the negative effects of the additional processing time. Therefore, the algorithm is described as iterative and convergent.

[0088] Figure 2C 213A for the previous iteration is fed back through a processing block of the algorithm. The distribution of amplitude values ​​211A is rejected in favor of an alternative distribution of amplitude values. In this alternative embodiment, the alternative distribution of amplitude values ​​is derived from the distribution of amplitude values ​​211 for the previous iteration. Specifically, processing block 258 subtracts the distribution of amplitude values ​​of input image 210 from the distribution of amplitude values ​​211 for the previous iteration, scales the difference by a gain factor α, and subtracts the scaled difference from input image 210. This is expressed mathematically by the following equation, where the subscript text and number represent the number of iterations:

[0089] R n+1 [x,y]=F'{exp(iψ n [u,v])}

[0090] ψ n [u,v]=∠F{η·exp(i∠R n [x,y])}

[0091] η=T[x,y]-α(|R n [x,y]|-T[x,y])

[0092] in:

[0093] F' is the inverse Fourier transform;

[0094] F is the forward Fourier transform;

[0095] R[x,y] is the complex data set output by the third processing block 256;

[0096] T[x,y] is the input or target image;

[0097] ∠ is the phase component;

[0098] Ψ is the pure phase hologram 280B;

[0099] η is the new distribution of amplitude values ​​211B; and

[0100] α is the gain factor.

[0101] The gain factor α may be fixed or variable. In some embodiments, the gain factor α is determined based on the size and rate of the input target image data. In some embodiments, the gain factor α depends on the number of iterations. In some embodiments, the gain factor α is only a function of the number of iterations.

[0102] In all other respects, Figure 2C Examples and Figure 2A and Figure 2B It can be said that the pure phase hologram Ψ(u,v) comprises a phase distribution in the frequency domain or in the Fourier domain.

[0103] In some embodiments, the Fourier transform is computationally performed by including the lens data in the holographic data. That is, the hologram includes data representing the lens as well as data representing the object. In these embodiments, the Fourier transform is omitted. Figure 1 A physical Fourier transform lens 120 is shown. In the field of computer-generated holography, it is known how to calculate holographic data representing a lens. Holographic data representing a lens can be referred to as a software lens. For example, a purely phase-based holographic lens can be formed by calculating the phase delay at each point of the lens due to its refractive index and spatially varying optical path length. For example, the optical path length at the center of a convex lens is greater than that at the edge of the lens. A purely amplitude-based holographic lens can be formed using a Fresnel zone plate. It is also known in the field of computer-generated holography how to combine holographic data representing a lens with holographic data representing an object, thereby performing a Fourier transform without requiring a physical Fourier lens. In some embodiments, the lens data is combined with the holographic data via simple vector addition. In some embodiments, a physical lens is used in conjunction with a software lens to perform the Fourier transform. Alternatively, in other embodiments, the Fourier transform lens is omitted entirely, allowing holographic reconstruction to occur in the far field. In further embodiments, the hologram can include grating data—data arranged to perform a grating function, such as beam steering. Similarly, in the field of computer-generated holography, it is known how to calculate such holographic data and combine it with holographic data representing an object. For example, a phase-only holographic grating can be formed by simulating the phase delay induced at each point on the surface of a blazed grating. A pure amplitude holographic grating can simply be superimposed on a pure amplitude hologram representing an object to provide angular steering of the amplitude-only hologram.

[0104] In some embodiments, a real-time engine is provided that uses an algorithm to receive image data and calculate a hologram in real time. In some embodiments, the image data is a video comprising a sequence of image frames. In other embodiments, the hologram is pre-calculated, stored in computer memory, and called up as needed for display on the SLM. That is, in some embodiments, a library of pre-determined holograms is provided.

[0105] The embodiments relate to Fourier holography and Gerchberg-Saxton type algorithms by way of example only. The present disclosure is equally applicable to Fresnel holography and holograms computed by other techniques such as techniques based on point cloud methods.

[0106] Optical Modulation

[0107] Computer-generated holograms can be displayed using spatial light modulators. If the hologram is a pure phase hologram, a spatial light modulator that modulates the phase is required. If the hologram is a fully complex hologram, a spatial light modulator that modulates both phase and amplitude can be used, or a first spatial light modulator that modulates the phase and a second spatial light modulator that modulates the amplitude can be used.

[0108] In some embodiments, the light modulation elements (i.e., pixels) of the spatial light modulator are cells comprising liquid crystals. That is, in some embodiments, the spatial light modulator is a liquid crystal device in which the optically active component is liquid crystal. Each liquid crystal cell is configured to selectively provide multiple light modulation levels. That is, each liquid crystal cell is configured to operate at a light modulation level selected from a plurality of possible light modulation levels at any time. Each liquid crystal cell can be dynamically reconfigured to a light modulation level that is different from the plurality of light modulation levels. In some embodiments, the spatial light modulator is a reflective liquid crystal on silicon (LCOS) spatial light modulator, but the present disclosure is not limited to this type of spatial light modulator.

[0109] LCOS devices provide a dense array of light modulating elements or pixels within a small aperture (e.g., a few centimeters wide). The pixels are typically on the order of 10 microns or smaller, which results in a diffraction angle of a few degrees, meaning the optical system can be compact. It is much easier to fully illuminate the small aperture of an LCOS SLM than the large apertures of other liquid crystal devices. LCOS devices are typically reflective, which means that the circuitry that drives the LCOS SLM pixels can be buried under the reflective surface. The result is a higher aperture ratio. In other words, the pixels are tightly packed, which means there are almost no dead spaces between pixels. This is advantageous because it reduces optical noise in the playback field. LCOS SLMs use a silicon backplane, which has the advantage that the pixels are optically flat. This is particularly important for phase modulation devices.

[0110] The following is just an example, Figure 3A suitable LCOS SLM is described below. An LCOS device is formed using a single crystal silicon substrate 302. It has a 2D array of square planar aluminum electrodes 301, separated by gaps 301a, arranged on the upper surface of the substrate. Each electrode 301 can be addressed by a circuit 302a embedded in the substrate 302. Each electrode forms its own plane mirror. An orientation layer 303 is disposed on the electrode array, and a liquid crystal layer 304 is disposed on the orientation layer 303. A second orientation layer 305 is disposed on the liquid crystal layer 304, and a planar transparent layer 306, such as glass, is disposed on the second orientation layer 305. A single transparent electrode 307, such as made of ITO, is disposed between the transparent layer 306 and the second orientation layer 305.

[0111] Each square electrode 301, together with the covered area of ​​transparent electrode 307 and the intervening liquid crystal material, defines a controllable phase-modulating element 308, commonly referred to as a pixel. Taking into account the space 301a between pixels, the effective pixel area, or fill factor, is the percentage of the total pixel that is optically active. By controlling the voltage applied to each electrode 301 relative to transparent electrode 307, the properties of the liquid crystal material of each phase-modulating element can be varied, thereby providing variable retardation to light incident thereon. The effect is to provide pure phase modulation of the wavefront, i.e., no amplitude effects occur.

[0112] The described LCOS SLM outputs spatially modulated light in a reflective manner. A reflective LCOS SLM has the advantage that the signal lines, grating lines and transistors are located below the mirror surface, which results in a high fill factor (typically greater than 90%) and high resolution. Another advantage of using a reflective LCOS spatial light modulator is that the thickness of the liquid crystal layer can be half that required when using a transmissive device. This greatly increases the switching speed of the liquid crystal (a key advantage for projecting moving video images). However, the teachings of the present disclosure can be equally implemented using a transmissive LCOS SLM.

[0113] Implementation of 2D Fourier Transform

[0114] refer to Figure 4A -D, a two-dimensional (2D) Fourier transform of image data representing a 2D image can be decomposed into two one-dimensional (1D) Fourier transforms, one along each dimension. Figure 4A A 2D array 410 of complex data values ​​representing an image 405 is shown, for example, with magnitudes corresponding to pixel values ​​and random phases, as described above. As indicated by arrow 415, a Fourier transform 425 is calculated for each row of the array 410 to calculate the corresponding row 420 of the row Fourier transform array 430 (see FIG. Figure 4B and 4C ). It can be said that the array 430 is Fourier transformed row by row. The array 430 is then Fourier transformed column by column, as Figure 4C4, to provide an array 440 of data values ​​representing a 2D Fourier transform 445 of the array 410 representing the image 405 (see Figure 4D ).

[0115] refer to Figure 5 An image data processing system, particularly adapted for performing a two-dimensional (2D) Fourier transform of a 2D image, comprises a first Fourier transform engine 552 arranged to: receive a first data stream 501 representing a first array of first complex data values, wherein each first complex data value is a real value formed of x bits and an imaginary value formed of x bits, i.e., the data values ​​are in Cartesian form; perform a first complex Fourier transform on the first array of first complex data values ​​to form a second array of second complex data values, wherein each second complex data value is a real value formed of x bits and an imaginary value formed of x bits; and output a second data stream 502 representing the second array of second complex data values. For example, x=32 bits.

[0116] Specifically, the first array represents Figure 4A The input image is shown in array 410 in which the amplitude and random phase of the complex data value corresponding to the image intensity (pixel value) in the input image are assigned to each data value. For example, these data assignments can be made in polar form and then the data values ​​are converted to Cartesian form. The array can be represented by the complex data value A ij It is expressed as follows:

[0117]

[0118] The corresponding data stream (first data stream 501) received by the first Fourier transform engine 552 can be represented as follows, where the Fourier transform engine 552 receives data values ​​row by row:

[0119] [A 11 A 12 A 13 … A 1n A 21 A 22 A 23 … A 2n A m1 A m2 A m3 … A mn ]

[0120] The Fourier transform engine 552 processes the data values ​​one row at a time as follows, where FFT stands for Fast Fourier Transform, although it will be appreciated that any Fourier transform implementation may be used:

[0121]

[0122]

[0123]

[0124] The Fourier transform engine 552 outputs each transform row one by one, thereby creating another data stream (second data stream 502), which corresponds to the Fourier transform of one of the dimensions of the array, such as Figure 4C the rows of the array shown in array 430 in

[0125] [A′ 11 A′ 12 A′ 13 … A′ 1n A′ 21 A′ 22 A′ 23 … A′ 2n A′ m1 A′ m2 A′ m3 … A′ mn

[0126] The first data conversion engine 554 is arranged to: receive the second data stream 502; convert each second complex data value into polar coordinate form, which has an amplitude value formed by x bits and a phase value formed by x bits; compress each amplitude value and each phase value to form a third array of third complex data values, such as which has an amplitude value formed by y bits and a phase value formed by y bits, where y < x; and output a third data stream 503 representing the third array of third complex data values. For example, x = 32 bits and y = 8 bits.

[0127] The combined operation of conversion and compression can be expressed as follows:

[0128]

[0129] The third data stream 503 can be expressed as follows:

[0130] [a′ 11 a′ 12 a′ 13 … a′ 1n a′[[ID=*59]] 21 a′ 22 a′ 23 … a′ 2n a′ m1 a′ m2 a′ m3 … a′ mn

[0131] ​​It should be understood that compression of the data values ​​representing the image may include (or even comprise) converting each data value in polar coordinate form to a digital format using fewer bits, such as converting each of the amplitude and phase to a digital format using fewer bits. For example, the original digital format may use a double (64-bit) or single (32-bit) floating point number for each phase and amplitude, and the engine 554 may convert each number to a half (16-bit) or quarter (8-bit) floating point number in any manner, such as by removing a desired number of least significant bits from the fractional portion of the floating point number. The digital formats (original and compressed) are of course not limited to this, and the numbers may be represented in other formats, such as binary or integer formats. For example, in one possible format, the phase 2π corresponds to the maximum representable binary / integer value for the phase portion, where values ​​between 0 and 2π are evenly spaced on the available levels between 0 and this number, and the amplitude is similarly represented using a threshold amplitude (maximum representable amplitude) corresponding to the maximum representable value and intermediate values ​​between zero and this amplitude, which are evenly spaced on the available levels between them. In a specific example, the original digital format represents each phase and amplitude in 1024 levels (32 bits each, for a total of 64 bits), and the compressed digital format represents each phase and amplitude in 64 levels (8 bits each, for a total of 16 bits).

[0132] In either case, the compression phase can include reallocating values ​​to available levels between 0 and 2π in the low-order representation. For amplitude values ​​(not necessarily limited to fixed intervals), the values ​​can be limited to fixed intervals based on knowledge of the amplitudes that are likely to occur. For example, the amplitude values ​​can be scaled to lower values ​​so that they fit better within the available range and assigned to the corresponding available representation levels, while still exceeding the maximum representable number of amplitude values ​​set (thresholded) to that number. The scaling factor used to scale down (multiply) the values ​​can be set based on knowledge of typical data sets as a compromise, so that when the available levels represent an acceptable precision, no more than an acceptable number of amplitudes can be thresholded (the levels are not numerically spaced larger / coarser than desired).

[0133] It should also be understood that while the conversion and compression of image data are described as separate operations, this is done for convenience in describing the first data conversion engine 554 and encompasses embodiments in which separate operations are present and those in which no separate operations are present. For example, each data value may be read by the processor into a working register (other internal representation of the processor), the data value may be converted to polar form, and written out to memory in a lower bit format for further processing, which may be described from the outside of the processor as a single operation (although various machine-level operations of course occur within the processor to achieve this).

[0134] The row-column transform engine 556 is arranged to: receive a third data stream 503; write a third array of third complex data values ​​to a memory; and read out a fourth data stream 504 representing a fourth array of fourth complex data values, wherein the fourth array of fourth complex data values ​​is a row-column transform or transpose of the third array of third complex data values.

[0135] Specifically, the conversion engine 556 receives the third data stream 503 generated by the first data conversion engine 554 row by row and writes the data values ​​row by row to the memory. Once all of the third data stream 503 (i.e., all rows of the array) have been written to the memory, the data values ​​are read out of the memory column by column. The resulting data stream (fourth data stream 504) can be represented as follows:

[0136] [a′ 11 a′ 21 a′ 31 … a′ m1 a′ 12 a′ 22 a′ 32 … a′ m2 a′ 1n a′ 2n a′ 3n … a′ mn ]

[0137] The corresponding (row Fourier transformed, compressed and transposed) array can be represented as follows:

[0138]

[0139] The above array may form image data representing an output image, or may be used as input image data for further processing iterations as described below.

[0140] Thus, in some embodiments, an image data processing system is provided that provides a row-by-row Fourier transform array in transposed form in preparation for column-by-column processing, whereby it is of course understood that rows and columns are referred to herein as examples of respective linear arrays in respective dimensions, and accordingly, processing can commence column-by-column and the data processing system provides the array row-by-row. In other words, the term "row" is used only to refer to a linear array spanning one dimension (row or column, horizontal or vertical, etc.), and the term "column" is used only to refer to a linear array spanning another dimension in a 2D array.

[0141] refer to Figure 6 , the second data conversion engine 558 is arranged to: receive the fourth data stream 504; decompress each fourth complex data value; convert each fourth complex data value into a fifth complex data value, which includes a real value and an imaginary value, i.e., convert into Cartesian form; and output a fifth data stream 505 representing a fifth array of fifth complex data values.

[0142] The operation of the second data transformation engine 558 can be conceptualized as follows:

[0143]

[0144] Specifically, the decompression of the image data may include the inverse operations of the above-mentioned image data compression, such as requantizing the phase values ​​to a larger number of available levels (e.g. padding with additional zeros and adding empty levels between the levels of the third / fourth data streams 503, 504), and, depending on the circumstances, scaling the amplitude values ​​with the inverse of the scale factor and requantizing, or converting back to, for example, a dual format.

[0145] It should be understood that the row-column transformation engine may be implemented as a memory that receives the third data stream and sequentially writes it to the memory by the first data conversion engine 554. In some embodiments, the engine has the function of sending a signal to the second data conversion engine 558 to notify that all data streams 503 have been written (which may alternatively be implemented in the first data conversion engine 554). In such an embodiment, the signal triggers the second data conversion engine 558 to read the data from the memory. 11 Starting with every nth data value, that is, (ignoring any offset and specific addressing scheme) sequentially reading memory addresses 1+(k-1)·n, where k={1; 2; 3; 4; ...; m·n}. Of course, in other embodiments, this addressing / reading scheme is implemented by a row-column conversion engine.

[0146] The second Fourier transform engine 560 is arranged to: receive the fifth data stream 505; perform a second 1D complex Fourier transform on the fifth array of fifth complex data values ​​to form a sixth array of sixth complex data values, wherein each sixth complex data value is a real value formed by x bits and an imaginary value formed by x bits; and output a sixth data stream 506 representing the sixth array of sixth complex data values, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform, i.e., in terms of rows and columns, the columns are transformed rather than the rows.

[0147] The operation of the second Fourier transform engine 560 can be expressed as follows:

[0148]

[0149]

[0150]

[0151]

[0152] Specifically, the second Fourier transform engine 560 receives the data values ​​in the fifth data stream 505 column by column, transforms one column at a time, and outputs the sixth data stream 506 column by column.

[0153] The sixth array of data values ​​is the transpose of the 2D Fourier transform of the first array (e.g., the input image) and can be represented as follows:

[0154]

[0155] The corresponding sixth data stream can be expressed as follows:

[0156] [A″ 11 A″ 21 … A″ m1 A″ 12 A″ 22 … A″ m2 A″ 1n A″ 2n … A″ mn ]

[0157] Thus, in some embodiments, an image data processing system is provided that provides a 2D Fourier transform that is decomposed into a row-wise and column-wise Fourier transform of a 2D array. Since the first array comprises image data representing an input image, as described above and as Figure 4A As shown, the sixth data stream can be viewed as a computer-generated hologram (after one or more iterations of the data processing system description generally described above and with specific reference to the following). In particular, the phase value of each isolated complex data value can be viewed as a computer-generated, pure phase hologram of the input image. Thus, the sixth data stream corresponding to the computer-generated hologram can be output for display by a holographic display device for holographic reconstruction as described herein. Thus, in some embodiments, the data processing system forms an image processing system for generating a hologram of an image represented by a 2D array. The values ​​of the two-dimensional sixth array of data values ​​(corresponding to the sixth data stream) represent (i.e., correspond to) pixels of the two-dimensional hologram corresponding to the two-dimensional input image.

[0158] refer to Figure 7 , an image data processing system implementing a 2D inverse Fourier transform includes a first inverse Fourier transform engine 564, which is arranged to: receive a seventh data stream 507; perform a first 1D complex inverse Fourier transform on the seventh array of seventh complex numbers to form an eighth array of eighth complex data values, wherein each eighth complex data value is a real value formed by x bits and an imaginary value formed by x bits; and output an eighth data stream 508 representing the eighth array of eighth complex data values.

[0159] As described below, the seventh data stream 507 may be derived, for example, from the sixth data stream 506 .

[0160] The third data conversion engine 566 is arranged to: receive the eighth data stream 508; convert each eighth complex data value into an amplitude value formed by x bits and a phase value formed by x bits; compress each amplitude value and each phase value to form a ninth array of ninth complex data values; and output a ninth data stream 509 representing the ninth array of ninth complex data values.

[0161] In some embodiments, the inverse Fourier transform image data processing system is similar to the above reference Figure 5 and 6 The depicted data processing systems share a row-column transform engine 556, such as the memory discussed above. In this case, the row-column transform engine 556 is further configured to: receive a ninth data stream 509; write a ninth array of ninth complex data values ​​to the memory; and read out a tenth data stream 510 representing a tenth array of tenth complex data values, wherein the tenth array of tenth complex data values ​​is a row-column transform of the ninth array of ninth complex data values. In other embodiments, each data processing system has its own row-column transform engine.

[0162] The fourth data conversion engine 568 is arranged to: receive the tenth data stream 510; decompress each tenth complex data value; convert each tenth complex data value into an eleventh complex data value comprising real and imaginary values; and output an eleventh data stream 511 representing an eleventh array of the eleventh complex data values.

[0163] The second inverse Fourier transform engine 570 is arranged to: receive an eleventh data stream 511; perform a second 1D complex inverse Fourier transform on the eleventh array of eleventh complex data values ​​to form a twelfth array of twelfth complex data values, wherein each twelfth complex data value is a real value formed by x bits and an imaginary value formed by x bits; and output a twelfth data stream 512 representing the twelfth array of twelfth complex data values, wherein the second 1D complex inverse Fourier transform is orthogonal to the first 1D complex inverse Fourier transform.

[0164] Readers skilled in the relevant art will appreciate that an inverse Fourier transform is simply a Fourier transform applied to an already Fourier transformed data set. For example, if a Fourier transform transforms an image into a Fourier representation, an inverse transform transforms the transformed data set back into an image representation by again Fourier transforming the transformed data set. Therefore, the detailed descriptions above for engines 552, 554, 558, and 560 apply equally to engines 564, 566, 568, and 570, respectively. In some embodiments, the corresponding engines may be configured in the same manner (differently in other embodiments). In fact, the output of the second Fourier transform engine 560 (sixth data stream 506) can be fed directly into the input of the first Fourier transform engine 552 as the first data stream 501, or by inserting a row-column transform (transpose), since the operations for the forward and inverse transforms are identical. In this case, engines 564, 566, 568, and 570 are effectively replaced by engines 552, 554, 558, and 560, respectively, thereby reusing their respective computational resources. Likewise, when they perform the same operation on the received data value streams, the Fourier transform engines 552 and 560 can be replaced by a single Fourier transform engine by routing the data value streams accordingly. In other words, in some embodiments, the first, second, third, and / or fourth Fourier transform engines are a single Fourier transform engine, although they can each be a different independent engine.

[0165] refer to Figure 8 In some embodiments, the above-described image data processing systems are combined for image processing to implement iterative calculation of computer-generated holograms as described above. Therefore, the sixth data stream 506 calculated by the first iteration is not output as hologram data for display, but is converted through an inverse Fourier transform and used as input image data for the next iteration. The first amplitude engine 562 is arranged to receive the sixth data stream 506 and output a seventh data stream 507, and the third Fourier transform engine 564 is arranged to receive the seventh data stream from the first amplitude engine 562. As described above, the first amplitude engine 562 is arranged to replace the amplitudes of the complex data values ​​in the sixth data stream 506 before the inverse transform occurs, for example by setting all amplitudes to unit "1".

[0166] The second amplitude engine 572 is arranged to receive the twelfth data stream 512 and output a thirteenth data stream 513, which replaces the first data stream 501 for the next iteration of the computer generation algorithm of the hologram, as described above. The first Fourier transform engine 552 is arranged to receive the thirteenth data stream 513 from the second amplitude engine 572. The second amplitude engine 572 is arranged to perform one or two functions.

[0167] On the one hand, in some embodiments, the second amplitude engine 572 is arranged to compare the amplitudes of the complex values ​​in the twelfth data stream 512 with the corresponding amplitudes or pixel values ​​of the input image, as described above, and terminate the iterative algorithm if a goodness-of-fit criterion is met as described above. In other embodiments, the stopping criterion is based on, for example, the number of iterations, rather than a goodness-of-fit criterion. In any case, if the stopping criterion is met, the algorithm terminates and returns a computer-generated hologram based on the sixth data stream 506 (and the corresponding sixth array, possibly transposed as needed), for example, returning the corresponding phase value array as a pure phase hologram or the complex value array as a full complex hologram. Therefore, the sixth data stream / array is stored in memory until the stopping criterion for the current iteration has been evaluated (in the case of the criterion of the number of iterations, this may occur once the sixth data stream 506 has been fully calculated).

[0168] If further iteration is required, i.e., the stopping criteria are not met, then the second amplitude engine 572 replaces the amplitude of the data values ​​in the twelfth data stream 512, as described above, for example, by replacing the amplitude with a corresponding one of the input image or a combination of the amplitude and a corresponding one of the input image, as described above.

[0169] Reference Figure 9 The corresponding image processing method is described. The method can be implemented using the above system or in any other way. It should be understood that the above reference Figures 5 to 8 The description of the features of the described system components also applies to the corresponding method steps, regardless of how they are implemented. Therefore, the disclosure of the following method steps includes the corresponding detailed disclosure made above.

[0170] At step 902, the method is initialized by initializing counter variables m=1 and n=1, indexing a 2D NxM array of complex data values ​​in Cartesian form to be processed and the array itself, for example based on an image for which a computer-generated hologram is to be calculated as described above, although the method is not limited thereto and can be used for other image processing techniques. It should be understood that in embodiments where data values ​​are streamed column by column / row by row as appropriate, no explicit counter is required, and processing can be driven by streaming data, either counting data values, or relying on a marker in the data stream indicating the end of each row, column, and / or array.

[0171] At step 904, the nth row is Fourier transformed, and the result is converted at step 906, first into polar form and then into a low-order digital format / representation. At step 908, the nth row is stored in this form in memory, and step 910 checks whether the nth row is the last row of the array. If n is not incremented (or in a streaming implementation, the next M value is received from the stream), steps 904 to 910 are repeated until the last row is processed. In some embodiments, the compressed data set is simply provided for further processing, and the process ends at this stage of outputting or storing the resulting array, such as Figure 9 As shown in breakpoint A in .

[0172] In other embodiments, the method proceeds to step 916 where, for example, the first column of data values ​​is read from memory and then converted back at step 918, first to the original digital format or representation (or at least one requiring more bits than the low order one) and then from polar coordinates back to Cartesian form. A Fourier transform is then performed on the column of data values ​​at step 920. In some embodiments where the method provides a 2D Fourier transform without further processing as described below, the method proceeds to step 916 where the first column of data values ​​is read from memory and then converted back to Cartesian form at step 918. Figure 9 The loop loops back to breakpoint B in to repeat steps 916 to 920 until the array has been processed column by column and the method terminates, outputs, or stores the resulting array.

[0173] In other embodiments providing an iterative process for computing a computer-generated hologram, once the column has been transformed at step 920, the amplitude of the data value is replaced, for example, with a single "1," as described above, and the resulting column is Fourier transformed at step 924, converted at step 926, and stored at step 928, consistent with the description of steps 904 to 908, with rows replaced by columns. At step 930, a check is made to see if all columns have been processed. If not, the process increments m at step 932 (unless implicit in a streaming implementation as described above). In some embodiments, steps 916 to 920 and 922 to 928 can be interleaved such that while column m is being processed and stored at steps 922 to 928, the next column, m+1, is read and processed at steps 916 to 920, provided the next column exists.

[0174] If it has been determined at step 930 that all columns have been processed, the process proceeds to step 934 to reset n (unless not required in a streaming implementation - see above). At step 936, for example, the first row is read from memory. It will be appreciated that step 908 writes the row to memory as sequential data values, and step 916 reads out each Mth value in sequence to read out the column sequence, which is then written as a column sequence at step 928, and each Nth value is read out to read out the row sequence again at step 936, etc., with sequential rows being written at step 948 described below to loop back, and reading out the column sequence restarting at step 916 (skipping M) to end the loop. Thus, with reference to Figure 9 References to rows and columns refer to the rows and columns of the original array, which are transformed in each step of the method.

[0175] At step 938, the row read at step 936 is converted back as described above for step 918, and the converted row is Fourier transformed at step 940. At step 942, the amplitude of the row is replaced based on the image to be represented, as described above, and the next iteration begins with a Fourier transform at step 944, conversion at step 946 as described above for step 906, and storage in memory at step 948. Step 950 checks to see if all rows have been processed. If not, n is incremented at step 952, unless this is implicit in the streaming implementation as described above, and the method loops back to step 936 to read the next row. On the other hand, if it is determined at step 950 that all rows have been processed, the method loops back to step 914 to reset m (unless this is not necessary in the streaming implementation—see above) and the method continues at step 916. Thus, in some embodiments, the method implements an iterative algorithm for computing a computer-generated hologram, wherein steps 902 to 942 correspond to a first iteration of the algorithm and repetitions of steps 944 to 942 through steps 914 to 940 correspond to subsequent iterations.

[0176] In some embodiments, the stopping criterion is defined based on a set number of iterations. In these embodiments, once the number of iterations has been reached and all columns have been processed in the last iteration, the method stops at point B. The data values ​​at this point, i.e., the Fourier transform at step 920, are used to form the hologram and are therefore stored for this purpose for all columns at least in the last iteration (i.e., the transform from the last segment of steps 902 or 914). For example, as described above, the phase of the data values ​​can be used as a pure phase hologram, and the corresponding array can be transposed if necessary.

[0177] In some embodiments, the stopping criterion is defined based on the goodness of fit of the amplitude distribution of the data values ​​of the array compared to the amplitude of the original input image, as described in detail above. A measure of goodness of fit is determined at breakpoint C, and if it is determined at this point that the algorithm has sufficiently converged and is to be terminated, the data values ​​of the columns transformed at step 922 since the last step 902 or 914 are used to construct the hologram as described above. To this end, it may be convenient to simply maintain a storage of columns of data values ​​generated by step 920 for all columns in the array, with each column being overwritten by its corresponding column for the next iteration, or in any other suitable manner.

[0178] Thus, when the stopping criterion is met, the method can output a hologram representing the input image for display. For example, the method can output the hologram to a holographic display device as described herein. The hologram can be a pure phase hologram or a full complex hologram including phase and amplitude components. The method can receive a sequence of input images from an image source (e.g., a camera) in real time and output a sequence of holograms representing the corresponding images in real time. As a result, the method can process images as they are received in real time, i.e., at low latency. Due to the compression of the complex image data as described herein, it is possible to reduce waiting time. In addition, by performing compression of the polar coordinate representation of the complex image data, errors in the image data are reduced. In some embodiments, the method can display each hologram on a holographic display device, such as a spatial light modulator arranged to display the hologram, and illuminate the spatial light modulator with light to form a holographic reconstruction or a holographic playback image.

[0179] According to the described methods, as described herein, transformations of large, complex data sets corresponding to image data necessary to generate corresponding holograms or processed images as described herein can be performed with reduced computing resources, such as reduced storage capacity and processing power. Furthermore, computer-generated holograms or processed images, as described herein, can be calculated with increased speed and accuracy.

[0180] Additional Features

[0181] By way of example only, the embodiments relate to electrically activated LCOS spatial light modulators. The teachings of the present disclosure may also be implemented on any spatial light modulator capable of displaying computer-generated holograms according to the present disclosure, such as any electrically activated SLM, optically activated SLM, digital micromirror device, or microelectromechanical device.

[0182] In some embodiments, the light source is a laser. In some embodiments, the detector is a photodetector. In some embodiments, the light receiving surface is a screen or a diffuser. The holographic projection system of the present disclosure can be used to provide an improved head-up display (HUD) or head-mounted display. In some embodiments, a vehicle is provided, which includes a holographic projection system installed in the vehicle to provide a HUD. The vehicle can be a motor vehicle, such as a car, truck, van, delivery truck, motorcycle, train, airplane, ship, or boat.

[0183] The quality of holographic reconstruction can be affected by the so-called zero-order problem, which is a consequence of the diffraction properties of the pixelated spatial light modulator used. This zero-order light can be considered "noise," consisting of, for example, specular reflections and other unwanted light from the spatial light modulator (SLM).

[0184] In the example of Fourier holography, this "noise" is focused at the focal point of the Fourier lens, resulting in a bright spot at the center of the holographic reconstruction. The zero-order light may be simply blocked, however this means replacing the bright spot with a dark spot. Some embodiments include an angle-selective filter to remove only the zero-order collimated light. Embodiments also include methods for managing the zero-order as described in European Patent 2,030,072, which is incorporated herein by reference in its entirety.

[0185] In some embodiments, the size of the hologram (the number of pixels in each direction) is equal to the size of the spatial light modulator, so that the hologram fills the spatial light modulator. That is, the hologram uses all the pixels of the spatial light modulator. In other embodiments, the size of the hologram is smaller than the size of the spatial light modulator. In some of these other embodiments, a portion of the hologram (i.e., a continuous subset of the pixels of the hologram) is repeated in unused pixels. This technique may be referred to as "tiling," in which the surface area of ​​the spatial light modulator is divided into multiple "tiles," each of which represents at least a subset of the hologram. Therefore, the size of each tile is smaller than the size of the spatial light modulator.

[0186] The size of the holographic playback field (i.e., the physical or spatial extent of the holographic reconstruction) is determined by the pixel pitch of the spatial light modulator (i.e., the distance between adjacent light modulating elements or pixels of the spatial light modulator). The smallest feature that can be formed in the playback field can be referred to as a "resolution element," "image point," or "image pixel." Typically, each pixel of the spatial light modulator has a quadrilateral shape. The Fourier transform of the quadrilateral aperture is a Sinker function, and therefore each image pixel is a Sinker function. More specifically, the spatial intensity distribution of each image pixel in the playback field is a Sinker function. Each Sinker function can be considered to include a primary diffraction order with peak intensity and a series of higher diffraction orders of decreasing intensity extending radially outward from the primary. The size of each Sinker function (i.e., the physical or spatial extent of each Sinker function) is determined by the size of the spatial light modulator (i.e., the physical or spatial extent of the aperture formed by the light modulating elements or the array of pixels of the spatial light modulator). Specifically, the larger the aperture formed by the array of light modulating pixels, the smaller the image pixel. Small image pixels are generally desirable.

[0187] In some embodiments, a "tiling" technique is implemented to improve image quality. Specifically, some embodiments implement tiling techniques to minimize the size of image pixels while maximizing the amount of signal content that goes into the holographic reconstruction.

[0188] However, if a coherent light source, such as a laser, is used, the resulting holographic reconstruction still suffers from "speckle." Speckle is well known and results from the interference of light scattered by optically rough surfaces. It would be desirable to reduce this speckle. Embodiments include elements and techniques for reducing perceived speckle in holographic reconstruction.

[0189] In some embodiments, the holographic pattern written to the spatial light modulator comprises at least one complete tile (ie, a complete hologram) and at least a portion of a tile (ie, a contiguous subset of pixels of the hologram).

[0190] The holographic reconstruction is generated within the zeroth diffraction order across the window defined by the spatial light modulator. Preferably, the first and subsequent orders are shifted far enough so as not to overlap with the image and so that they can be blocked using a spatial filter.

[0191] In embodiments, the holographic reconstruction is in color. In the examples disclosed herein, three different color light sources and three corresponding spatial light modulators are used to provide a composite color. These examples may be referred to as spatially separated color (SSC). In variations encompassed by the present disclosure, different holograms for each color are displayed on different regions of the same SLM and then combined to form a composite color image. However, those skilled in the art will appreciate that at least some of the apparatus and methods of the present disclosure are equally applicable to other methods of providing composite color holographic images.

[0192] One such approach is known as frame sequential color, or FSC. In an example FSC system, three lasers (red, green, and blue) are used, and each laser fires continuously on a single SLM to produce each frame of the video. The colors cycle through (red, green, blue, red, green, blue, and so on) quickly enough that a human observer sees a multicolored image from the combination of the images formed by the three lasers. Thus, each hologram is a specific color. For example, in a video at 25 frames per second, the first frame would be produced by firing the red laser for 1 / 75 of a second, then the green laser for 1 / 75 of a second, and finally the blue laser for 1 / 75 of a second. The next frame is then produced, starting with the red laser, and so on.

[0193] The advantage of the FSC method is that the entire SLM is used for each color. This means that the quality of the three-color images produced is not compromised, because all pixels on the SLM are used for each color image. However, the disadvantage of the FSC method is that the overall image produced will be less bright than the corresponding image produced by the SSC method, about three times dimmer, because each laser is only used one-third of the time. This disadvantage could be addressed by overdriving the lasers or using more powerful lasers, but this would require using more power, would involve higher costs, and would make the system less compact.

[0194] The advantage of the SSC method is that the image is brighter because all three lasers are firing simultaneously. However, if it is necessary to use only one SLM due to space limitations, the surface area of ​​the SLM can be divided into three parts, effectively equivalent to three independent SLMs. The disadvantage of this is that the quality of each single color image is reduced due to the reduction in the SLM surface area available for each monochrome image. The quality of the multi-color image is therefore correspondingly reduced. The reduction in the available SLM surface area means that fewer pixels can be used on the SLM, thereby reducing the image quality. Due to the reduced resolution, the image quality is reduced. The embodiments make use of the improved SSC technology disclosed in British Patent 2,496,108, which is incorporated herein by reference in its entirety.

[0195] While the examples describe illuminating the SLM with visible light, those skilled in the art will appreciate that the light source and SLM can also be used to direct infrared or ultraviolet light, such as disclosed herein. For example, those skilled in the art will be aware of techniques for converting infrared and ultraviolet light into visible light for the purpose of providing information to a user. For example, the present disclosure extends to the use of phosphors and / or quantum dot technology for this purpose.

[0196] Some embodiments describe 2D holographic reconstructions by way of example only. In other embodiments, the holographic reconstruction is a 3D holographic reconstruction. That is, in some embodiments, each computer-generated hologram forms a 3D holographic reconstruction.

[0197] computing devices

[0198] Figure 10 A block diagram of an embodiment of a computing device 1000 is shown, wherein a group of instructions can be executed for making the computing device perform any one or more methods discussed herein. In an alternative embodiment, the computing device can be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The computing device can operate as a server or client in a client-server network environment, or as a peer-to-peer machine in a peer-to-peer (or distributed) network environment. The computing device can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network device, a server, a network router, a switch, or a bridge, or any machine that can perform a group of instructions (sequential or other) specifying the action to be taken by the machine. In addition, although only a single computing device is shown, the term "computing device" should also be understood to include any set of machines (e.g., computers) that individually or jointly execute a group (or multiple groups) of instructions to perform any one or more methods discussed herein.

[0199] The example computing device 1000 includes a processing device 1002, a main memory 1004 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc., a static memory 1006 (e.g., flash memory, static random access memory (SRAM)), etc.), and a secondary memory (e.g., a data storage device 1018), which communicate with each other via a bus 1030.

[0200] Processing device 1002 represents one or more general-purpose processors, such as microprocessors, central processing units, and the like. More specifically, processing device 1002 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing another instruction set, or a processor implementing a combination of instruction sets. Processing device 1002 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, and the like. Processing device 1002 is configured to execute processing logic (instructions 1022) for performing the operations and steps discussed herein.

[0201] The computing device 1000 may also include a network interface device 1008. The computing device 1000 may also include a video display unit 1010 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1012 (e.g., a keyboard or a touch screen), a cursor control device 1014 (e.g., a mouse or a touch screen), and an audio device 1016 (e.g., a speaker).

[0202] The data storage device 1018 may include one or more machine-readable storage media (or, more specifically, one or more non-transitory computer-readable storage media) 1028 on which is stored one or more sets of instructions 1022 embodying any one or more of the methodologies or functions described herein. During execution of the instructions by the computer system 1000, the instructions 1022 may also reside, completely or at least partially, in the main memory 1004 and / or the processing device 1002, which also constitute computer-readable storage media.

[0203] The various methods described above can be implemented by a computer program. The computer program may include computer code, which is arranged to instruct a computer to perform one or more functions of the various methods described above. The computer program and / or code for performing this method can be provided to a device such as a computer on one or more computer-readable media or more generally on a computer program product. The computer-readable medium can be temporary or non-temporary. The one or more computer-readable media can be, for example, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, or propagation media for data transmission, for example, for downloading code via the Internet. Alternatively, the one or more computer-readable media can take the form of one or more physical computer-readable media, such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk and optical disk, such as CD-ROM, CD-R / W or DVD.

[0204] In one embodiment, the modules, components, and other features described herein may be implemented as discrete components, or integrated into the functionality of hardware components, such as ASICS, FPGAs, DSPs, or similar devices.

[0205] A "hardware component" is a tangible (e.g., non-transitory) physical component (e.g., a collection of one or more processors) that is capable of performing specific operations and that may be configured or arranged in a specific physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be or include a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations.

[0206] Thus, the phrase "hardware component" should be understood to include a tangible entity that can be physically constructed, permanently configured (eg, hardwired), or temporarily configured (eg, programmed) to operate in a particular manner or perform the particular operations described herein.

[0207] Furthermore, modules and components may be implemented as firmware or functional circuitry within a hardware device. Furthermore, modules and components may be implemented in any combination of hardware devices and software components, or may be implemented solely in software (e.g., code stored or otherwise included in a machine-readable medium or transmission medium).

[0208] This description makes reference to data processing engines such as Fourier transform engines, conversion engines, transform engines, etc. It should be understood that these engines may be implemented in corresponding dedicated hardware components, shared between hardware components or shared hardware components, such as implemented in corresponding modules, implemented in software, such as firmware, middleware, etc., or implemented in hardware.

[0209] Unless otherwise stated, it will be apparent from the following discussion that throughout the description, discussions using terms such as "receive," "determine," "compare," "enable," "maintain," "identify," "transform," "convert," "store," "read," etc., refer to actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage, transmission, or display devices.

[0210] The methods and processes described herein may be embodied on a computer-readable medium. The term "computer-readable medium" includes media arranged to store data temporarily or permanently, such as random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. The term "computer-readable medium" should also be taken to include any tangible, non-transitory medium or combination of media that can store instructions for execution by a machine, such that when the instructions are executed by one or more processors, the machine performs, in whole or in part, any one or more of the methods described herein.

[0211] The term "computer-readable medium" also encompasses cloud-based storage systems. The term "computer-readable medium" includes, but is not limited to, one or more tangible and non-transitory data storage repositories (e.g., data volumes) in the example form of solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof. In some example embodiments, instructions for execution may be conveyed by a carrier medium. Examples of such carrier media include transient media (e.g., a propagated signal conveying the instructions).

[0212] Additional Disclosed Aspects and Embodiments

[0213] The following aspects and embodiments are disclosed:

[0214] 1. A system arranged to perform a complex Fourier transform on a complex data set corresponding to an image, wherein the system comprises:

[0215] a first Fourier transform engine arranged to receive a complex data set and to perform a first 1D complex Fourier transform of the complex data set in Cartesian form;

[0216] a first data conversion engine arranged to receive the complex data set processed by the first Fourier transform engine, convert the complex data set into polar form, and compress the complex data set in polar form;

[0217] a row-column transformation engine arranged to receive the complex data set processed by the first data conversion engine and perform a row-column transformation of the complex data set;

[0218] a second data conversion engine arranged to receive the complex data set processed by the row-column transform engine, decompress the complex data set and convert the complex data set back into Cartesian form; and

[0219] A second Fourier transform engine is arranged to receive the complex data set processed by the second data conversion engine and perform a second 1D complex Fourier transform of the complex data in Cartesian form, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform.

[0220] 2. A system as set forth in item 1, wherein the first data conversion engine is arranged to compress the complex data set in polar form by multiplying each amplitude value by a scaling factor.

[0221] 3. A system as set forth in any preceding clause, wherein the first data conversion engine is arranged to compress the complex data set in polar form by converting each amplitude value and each phase value of the complex data set from x bits and y bits, where x>y

[0222] 4. A system as set forth in item 3, wherein the first data conversion engine is arranged to compress the complex data set in polar form by thresholding any amplitude values ​​above a maximum allowed value to the maximum allowed value.

[0223] 5. The system as set forth in item 4, wherein the maximum allowed value is a maximum allowed value having y bits.

[0224] 6. A system as in any preceding clause, wherein the first data conversion engine is arranged to compress the complex data set by quantizing phase values.

[0225] 7. A system as set forth in any preceding clause, wherein the second data conversion engine is arranged to decompress the complex data set by converting each amplitude value and each phase value back from y bits to x bits and multiplying each amplitude value by the inverse of a scale factor.

[0226] 8. A system as in any preceding item, wherein the row-column transform engine is arranged to perform the row-column transform by writing the received complex data set to a memory and reading the complex data set from the memory in an order different from the order in which it was received.

[0227] 9. The system of item 8, wherein the memory is static random access memory.

[0228] 10. A system arranged to perform a complex inverse Fourier transform on a complex data set corresponding to an image, wherein the system comprises:

[0229] a first inverse Fourier transform engine arranged to receive the complex data set and perform a first 1D complex inverse Fourier transform of the complex data set in Cartesian form;

[0230] a third data conversion engine arranged to receive the complex data set processed by the first inverse Fourier transform engine, convert the complex data set into polar form, and compress the complex data set in polar form;

[0231] a row-column transformation engine arranged to receive the complex data set processed by the first data conversion engine and perform a row-column transformation of the complex data set;

[0232] a fourth data conversion engine arranged to receive the complex data set processed by the row-column transform engine, decompress the complex data set and convert the complex data set back into Cartesian form; and

[0233] A second inverse Fourier transform engine is arranged to receive the complex data set processed by the second data conversion engine and perform a second 1D complex inverse Fourier transform of the complex data in Cartesian form, wherein the second 1D complex inverse Fourier transform is orthogonal to the first 1D complex inverse Fourier transform.

[0234] 11. The system as set forth in item 10, wherein the third data conversion engine is arranged to compress the complex data set in polar form by multiplying each amplitude value by a scaling factor.

[0235] 12. The system as set forth in item 10 or 11, wherein the third data conversion engine is arranged to compress the complex data set in polar form by converting each amplitude value and each phase value of the complex data set from x bits and y bits, where x>y

[0236] 13. The system as set forth in item 12, wherein the third data conversion engine is arranged to compress the complex data set in polar form by thresholding any amplitude values ​​above a maximum allowed value to the maximum allowed value.

[0237] 14. The system of item 13, wherein the maximum allowed value is the maximum allowed value of the y position.

[0238] 15. The system of any one of clauses 10 to 14, wherein the third data conversion engine is arranged to compress the complex data set by quantizing phase values.

[0239] 16. A system as listed in any one of items 10 to 15, wherein the fourth data conversion engine is configured to decompress the complex data set by converting each amplitude value and each phase value from y bits back to x bits and then multiplying each amplitude value by the inverse of the scaling factor.

[0240] 17. A system as described in any of items 10 to 16, wherein the row-column transformation engine is further arranged to write the received complex data set to a memory and read the complex data set from the memory in an order different from the order in which it was received in order to perform the row-column transformation.

[0241] 18. The system of item 17, wherein the memory is static random access memory.

[0242] 19. A system arranged to process a complex data set corresponding to an image, the system comprising:

[0243] The data processing system of any one of items 1 to 9;

[0244] The data processing system of any one of items 10 to 18; and

[0245] A data processor arranged to: receive a complex data set processed by the second inverse Fourier transform engine and replace amplitude values ​​to form a complex data set received by the first inverse Fourier transform engine; and / or receive a complex data set processed by the second inverse Fourier transform engine and replace amplitude values ​​to form a complex data set received by the first Fourier transform engine.

[0246] 20. A system as in any preceding item, wherein the first 1D complex Fourier transform is orthogonal to the first 1D complex inverse Fourier transform.

[0247] 21. A system as in any preceding item, wherein the complex data set is processed by the system as a 1D data stream.

[0248] 22. A system as in any preceding item, wherein complex data is processed by the system over a serial bus.

[0249] 23. A system as in any preceding item, further comprising a controller arranged to output a hologram representing an image in the spectral domain.

[0250] 24. The system of item 22, wherein the hologram is a pure phase hologram or a full complex hologram comprising phase and amplitude components.

[0251] 25. A system as set forth in item 22 or 23, wherein the controller is arranged to receive the sequence of images in real time and to output in real time a sequence of holograms representing the corresponding images in the spectral domain.

[0252] 26. The system of any one of items 22 to 24, further comprising a spatial light modulator arranged to display the hologram and a light source arranged to illuminate the spatial light modulator.

[0253] 27. A method of performing a complex Fourier transform on a complex data set corresponding to an image, the method comprising the following ordered steps:

[0254] receiving a complex data set and performing a first 1D complex Fourier transform of the complex data set in Cartesian form;

[0255] Convert complex data sets into polar coordinate form and compress complex data sets in polar coordinate form;

[0256] Perform row-column transformations on complex datasets;

[0257] Unpack the complex dataset and convert the complex dataset back to Cartesian form;

[0258] A second 1D Fourier transform of the Cartesian complex data set is performed, wherein the second 1D complex Fourier transform is orthogonal to the first 1D complex Fourier transform.

[0259] 28. A method of performing a complex inverse Fourier transform on a complex data set corresponding to an image, the method comprising the following ordered steps:

[0260] receiving a complex data set and performing a first 1D complex inverse Fourier transform of the complex data set in Cartesian form;

[0261] Convert complex data sets into polar coordinate form and compress complex data sets in polar coordinate form;

[0262] Perform row-column transformations on complex datasets;

[0263] Unpack the complex dataset and convert the complex dataset back to Cartesian form;

[0264] A second 1D inverse Fourier transform of the complex data set is performed in Cartesian form, wherein the second 1D complex inverse Fourier transform is orthogonal to the first 1D complex inverse Fourier transform.

[0265] 29. The method of items 26 and 27, further comprising:

[0266] replacing the amplitude values ​​of the complex data set produced by the method of item 26 to form the complex data set received according to item 27; and / or

[0267] The amplitude values ​​of the complex data set produced by the method of item 27 are replaced to form the complex data set received according to item 26.

[0268] It should be understood that the above description is intended to be illustrative and not limiting. After reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Although the present disclosure has been described with reference to specific example implementations, it should be appreciated that the present disclosure is not limited to the described implementations, but may be implemented through modification and alteration within the spirit and scope of the appended claims.

[0269] The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. An image processing system comprising: one or more processors arranged to perform a Fourier transform of a first complex data set corresponding to an image, wherein the first complex data set comprises a two-dimensional array of complex values ​​in Cartesian form, each complex value being represented in a first digital format having a first digit, the values ​​of the two-dimensional array representing pixels of the image, and the two-dimensional array defining a first linear array representing values ​​of the pixels along one dimension and a second linear array representing values ​​of the pixels along another dimension; A tangible, non-transitory computer-readable medium storing instructions executable by one or more processors, wherein the instructions, when executed, cause an image processing system to perform functions including: performing a 1D Fourier transform on each first linear array of values ​​of the first complex data set in Cartesian form; converting each value of each Fourier transformed first linear array into polar form; converting each value in polar form to a second numeric format having a second digit, wherein the second digit is smaller than the first digit; storing the value in the second digital format in the memory first linear array via the first linear array; reading a value from a second linear array of memory via the second linear array; converting each value read from memory back to the first digital format; Convert each value converted back to the first number format back to Cartesian form; performing a ID Fourier transform on each second linear array of values ​​converted back into Cartesian form, wherein the Fourier transformed second linear array of values ​​is used to form a second complex data set; and forming image data using the second complex data set; as well as A controller is arranged to output image data for display.

2. The image processing system according to claim 1, wherein The functionality also includes: The magnitude values ​​of the Fourier transformed second linear array of values ​​are replaced to form a second complex data set, wherein the first linear array in the second complex data set corresponds to the second linear array in the first complex data set.

3. The image processing system according to claim 1, wherein: Converting each value in the polar form to the second digital format includes multiplying each magnitude by a scaling factor.

4. The image processing system according to claim 1, wherein: Converting the polar form values ​​to the second digital format includes converting each magnitude value from x1 bits to y1 bits, where x1>y1, and converting each phase value from x2 bits to y2 bits, where x2>y2.

5. The image processing system according to claim 4, wherein: Converting each value in polar form to the second digital format includes setting any magnitude value above a maximum allowed value to the maximum allowed value, wherein the maximum allowed value is optionally a maximum magnitude value representable using y1 bits. The image processing system according to claim 1 , wherein: Converting each value in the polar form to the second digital format includes quantizing a phase value of the value in the polar form at a lower magnitude than the first digital format.

7. The image processing system according to claim 3, wherein: Converting each value in Cartesian form back to the first digital format includes converting each amplitude value from y1 bits back to x1 bits and converting each phase value from y2 bits back to x2 bits, and multiplying each amplitude value by the inverse of the scale factor.

8. The image processing system according to claim 1, wherein The functionality also includes: processing one first linear array while processing another first linear array; and One second linear array is processed while another second linear array is processed.

9. The image processing system according to claim 1, wherein: The functionality also includes: A first complex data set is received as a stream.

10. The image processing system according to claim 1, wherein: The functionality also includes: performing a Fourier transform of a second complex data set, wherein the second complex data set comprises a two-dimensional array of complex values ​​in Cartesian form, each complex value represented in a third numeric format having a third number of digits, the two-dimensional array defining a first linear array of values ​​along one dimension and a second linear array of values ​​along another dimension; performing a 1D Fourier transform on each first linear array of the second complex data set; Convert each value into polar form; converting each value in polar form to a fourth numeric format having a fourth digit, the fourth digit being less than the third digit; storing the value in the fourth digital format in another memory first linear array via the first linear array; reading a value from a second linear array of memory via the second linear array; converting each value back to the third number format and converting each value converted back to the first number format back to Cartesian form; performing a ID Fourier transform on each second linear array of the second complex data set; and The amplitude values ​​are replaced to form a corresponding one of the received second and first complex data sets, wherein the first linear array in the second complex data set corresponds to the second linear array in the first complex data set.

11. The image processing system according to claim 1, wherein: Outputting the image data for display comprises outputting a hologram representing the image data for display, wherein, optionally, the hologram is a pure phase hologram or a full complex hologram comprising phase and amplitude components.

12. The image processing system according to claim 11, wherein: The controller is arranged to receive a sequence of images in real time and to output a sequence of holograms representing the corresponding images in real time.

13. The image processing system according to claim 11, further comprising: A holographic display device arranged to display a hologram, wherein, optionally, the holographic display device comprises a spatial light modulator arranged to display the hologram and a light source arranged to illuminate the spatial light modulator.

14. An image processing method, comprising: performing a 1D Fourier transform on each first linear array in a first complex data set, wherein the first complex data set corresponds to an image, wherein the first complex data set comprises a two-dimensional array of complex values ​​in Cartesian form, each complex value represented in a first digital format having a first digit, the values ​​of the two-dimensional array represent pixels of the image, and the two-dimensional array defines a first linear array representing values ​​of the pixels along one dimension and a second linear array representing values ​​of the pixels along another dimension; performing a 1D Fourier transform on each first linear array of values ​​of the first complex data set in Cartesian form; converting each value of each Fourier transformed first linear array into polar form; converting each value in polar form to a second numeric format having a second digit, wherein the second digit is smaller than the first digit; storing the value in the second digital format in the memory first linear array via the first linear array; reading a value from a second linear array of memory via the second linear array; converting each value read from memory back to the first digital format; Convert each value converted back to the first number format back to Cartesian form; performing a ID Fourier transform on each second linear array of values ​​converted back to Cartesian form, wherein the Fourier transformed second linear array of values ​​is used to form a second complex data set; forming image data using the second complex data set; as well as Output image data for display.

15. The image processing method according to claim 14, comprising: replacing the magnitude of each of the values ​​of the Fourier transformed second linear array to form a corresponding second linear array of additional values; performing a ID Fourier transform on each second linear array of additional values; converting each additional value of each Fourier transformed second linear array into an additional value in polar coordinate form; converting each additional data value in the polar coordinate form into an additional value in an additional second numeric format having an additional second digit, wherein the additional second digit is smaller than the first digit; storing, via the second linear array, additional values ​​in the additional second digital format in the memory second linear array; reading another value from the memory first linear array through the first linear array; converting each additional value read from the memory back to the first digital format; converting each additional value converted back to the first number format back to Cartesian form; as well as A ID Fourier transform is performed on each first linear array of further values ​​converted back into Cartesian form.

16. The image processing method according to claim 14, wherein: Converting each value in polar form to a second numeric format includes one or more of the following: Multiply each magnitude by the scale factor; converting each amplitude value from x1 bits to y1 bits, where x1>y1, and converting each phase value from x2 bits to y2 bits, where x2>y2, wherein, optionally, any amplitude value above a maximum allowed value is set to the maximum allowed value, optionally, wherein the maximum allowed value is the maximum amplitude value representable in y1 bits; and The phase value of the value in polar form is quantized using a lower order than the first digital format.

17. The image processing method according to claim 14, comprising Fourier transforming one first linear array while transforming another first linear array, Fourier transforming one second linear array while transforming another second linear array, or both.

18. The image processing method according to claim 14, wherein: The image data output for display includes: A hologram representing the image data is output, wherein the hologram is optionally a pure phase hologram or a full complex hologram comprising phase and amplitude components.

19. The image processing method according to claim 18, further comprising: An image sequence is received in real time, and a hologram sequence representing the corresponding images is output in real time.

20. The image processing method according to claim 18, further comprising: The hologram is displayed on a holographic display device and, optionally, on and illuminates a spatial light modulator.

Images