Display device and system

Abstract

An image processing engine and method for forming a hologram of a target image for projection using data streams. A kernel is used to subsample an input or primary image and then a secondary image output is used to generate a hologram of the target image. The kernel subsampling technique using multiple two or more data streams provides efficiency improvements, including reduced data storage requirements and increased processing speed.

Description

Technical Field

[0001] The present disclosure relates to an image processor and an image processing method for holographic projection. The present disclosure further relates to a holographic projector, a holographic projection system, a method for holographically projecting an image, and a method for holographically projecting a video image. Some embodiments relate to a head-up display and a light detection and ranging system. Background Art

[0002] Light scattered from an object contains amplitude and phase information. This amplitude and phase information can be captured, for example, on a photographic plate by well-known interference techniques to form a holographic recording or "hologram" comprising interference fringes. The hologram can be reconstructed by illumination with appropriate light to form a two-dimensional or three-dimensional holographic reconstruction or playback image representing the original object.

[0003] Computer generated holography can numerically simulate interference processes. Computer generated holograms can be calculated by techniques based on mathematical transformations such as Fresnel or Fourier transforms. These types of holograms may be referred to as Fresnel / Fourier transform holograms or simply Fresnel / Fourier holograms. Fourier holograms may be considered as Fourier domain / plane representations of objects or frequency domain / plane representations of objects. Computer generated holograms may also be calculated, for example, by coherent ray tracing or point cloud techniques.

[0004] The computer generated hologram may be encoded on a spatial light modulator arranged to modulate the amplitude and / or phase of the incident light. For example, the light modulation may be achieved using electrically addressable liquid crystals, optically addressable liquid crystals or micromirrors.

[0005] A spatial light modulator typically includes a plurality of individually addressable pixels, which may also be referred to as cells or elements. The light modulation scheme may be binary, multi-level, or continuous. Alternatively, the device may be continuous (i.e., not including pixels), so that the light modulation may be continuous across the device. A spatial light modulator may be reflective, meaning that the light is modulated to reflect the output. A spatial light modulator may also be transmissive, meaning that the light is modulated to transmit the output.

[0006] The systems described herein can be used to provide holographic projectors. For example, such projectors have been applied to head-up displays "HUDs" and head-mounted displays "HMDs", including near-eye devices.

[0007] In devices that use coherent light, such as holographic projectors, moving diffusers can be used to improve image quality. Summary of the invention

[0008] Aspects of the disclosure are defined in the accompanying independent claims.

[0009] An image processing engine and method of forming a hologram of a target image for projection using data streams are disclosed. In particular, as described in more detail below, a kernel can be used to subsample an input or primary image, which can be an enlarged and / or modified version of a target image, and a secondary image can be output and used to generate a hologram of the target image. As described herein, techniques for subsampling a kernel using multiple two or more data streams provide efficiency improvements, including reduced data storage requirements (e.g., memory and / or buffer capacity) and increased processing speed.

[0010] Conventionally, a large area of ​​data storage is required to store the entire primary image data for kernel subsampling. Random memory access is required to access the primary image data for all pixels required for each intended kernel operation. Furthermore, video speed processing requires that at least part of the image data be cached, in particular at least the data entries (image pixel values) required to start processing, and at each processing stage thereafter. Therefore, the required cache capacity depends on the size of the input data (primary image), not on the size of the output data (secondary image).

[0011] For holographic image projection, in order to obtain the required image resolution (holographic reconstruction), the target image for projection can be "enlarged" to form a source image with an increased number of pixels. In addition, the source image can be modified to form an intermediate image, such as a so-called "warped image", to correct image distortions (distortions) caused by the optical components of the optical playback system of the projector, as is known in the art. The enlarged source image or an intermediate image derived therefrom can be used as a main image for kernel subsampling. Therefore, the size (resolution / number of pixels) of the main image and therefore the amount of data can be several times the size of the secondary image. As a result, a large amount of data storage and caching capabilities are required for the kernel to subsample the main image to produce the secondary image.

[0012] According to the present disclosure, data streaming is used to reduce data storage and cache capacity for sub-sampling a master image using a kernel to determine an output image for hologram calculation.

[0013] A first data stream of pixel values ​​of a primary image can be synchronized with a second data stream of kernel values ​​of a kernel such that each pixel value is paired with a corresponding kernel value to perform a kernel operation at a plurality of kernel sampling locations. For each kernel sampling location, the kernel value is repeated in the second data stream. Thus, there is a one-to-many correlation between the pixel values ​​of the primary image in the first data stream and the kernel values ​​of the second data stream. Additionally, each row of kernel values ​​of the kernel in the second data stream is paired with a plurality of rows of image pixels of the primary image in the first data stream.

[0014] In some embodiments, the data flow engine is configured to perform a data flow process. In particular, the data flow engine can form a first data stream by reading image pixel values ​​of the main image row by row. For example, the pixel values ​​can be read pixel by pixel in a raster scan order. At the same time, the data flow engine can use the following steps to form a second data stream: (i) repeatedly reading the kernel value of the first row of the kernel multiple times; (ii) repeatedly reading the kernel value of the next row of the kernel multiple times; (iii) iteratively repeating step (ii) (m-2) times; (iv) returning to step (i), and (v) when there are no more pixel values ​​in the first data stream, stopping steps (i) to (iv). A clock counter can be used to synchronize the values ​​in the first and second data streams.

[0015] In some embodiments, the image processing engine is configured to perform a kernel operation and a buffering process. In particular, the kernel operation is performed at a plurality of kernel sampling positions. For each row of pixels of the main image, the kernel operation processes a synchronous pair of values ​​of the first and second data streams associated with the kernel sampling position to derive a corresponding cumulative (partial) pixel value to be output to the buffer. The buffering process is performed by storing the cumulative (partial) pixel values ​​in consecutive positions in the buffer. The cumulative (partial) pixel values ​​output to the buffer from processing a row of pixels of the main image can form a third data stream, which is provided as feedback for processing the next row of pixels of the main image to derive updated cumulative (partial) pixel values. The feedback process can be repeated row by row until the last row of pixels of the main image has been processed for the same plurality of kernel sampling positions (i.e., kernel sampling positions in the same row or row). Then, the accumulated pixel value output to the buffer is the full or complete pixel value of a row of the secondary image. Therefore, a pixel row of the secondary image can be generated and output row by row. Advantageously, the pixel row of the secondary image can be streamed to the hologram engine in real time to calculate the hologram corresponding to the secondary image. Thus, the hologram calculation may be started before all pixels of the secondary image are derived.

[0016] In some examples, a kernel having m rows and n columns moves in a raster scan path with a stride of n pixels in the x direction and a stride of m pixels in the y direction. Thus, the kernel window subsamples a contiguous array of m×n pixels of the primary image. In these examples, the first data stream can be formed by reading pixel values ​​of the primary image pixel by pixel in a raster scan order. This simplifies the data stream processing process for forming the first data stream.

[0017] In an embodiment, data streaming can allow the determination of a secondary image from a primary image and the determination of a corresponding hologram to be performed simultaneously, thereby increasing processing speed. Specifically, the pixel values ​​determined for the secondary image can be streamed to the hologram engine in real time, as further described below. The increased processing speed enables a greater number of image holograms to be displayed in the video stream (i.e., using a greater number of secondary images or subframes). By displaying more images within the integration time of the human eye, the quality of the holographic reconstruction (holographic image) seen by the viewer is improved.

[0018] In some embodiments, multiple secondary images can be determined by subsampling the same source image using different subsampling schemes. For example, multiple different kernels and / or kernel sampling positions can be used to determine multiple different secondary images. The multiple different secondary images can be used to generate corresponding multiple holograms for projecting a target image. For example, multiple different holograms can be displayed sequentially on a spatial light modulator within the integration time of the human eye, and the spatial light modulator can be illuminated to form a series of holographic reconstructions on a playback plane for projection and viewing. It has been found that the display of multiple different holograms representing the same target image for projection can result in improved image quality seen by a viewer.

[0019] The term "hologram" is used to refer to a record containing amplitude information or phase information or some combination thereof about an object. The term "holographic reconstruction" is used to refer to the optical reconstruction of an object formed by illuminating a hologram. The system disclosed herein is described as a "holographic projector" because the holographic reconstruction is a real image and is spatially separated from the hologram. The term "playback field" is used to refer to the 2D area in which the holographic reconstruction is formed and fully focused. If the hologram is displayed on a spatial light modulator including pixels, the playback field will be repeated in the form of multiple diffraction orders, each of which is a copy of the zero-order playback field. The zero-order playback field usually corresponds to the preferred or primary playback field because it is the brightest playback field. Unless otherwise explicitly stated, the term "playback field" should be considered to refer to the zero-order playback field. The term "playback plane" is used to refer to a plane in space containing all playback fields. The terms "image", "playback image" and "image area" refer to the area of ​​the playback field illuminated by the light of the holographic reconstruction. In some embodiments, an "image" may include discrete spots of light, which may be referred to as "image spots" or, for convenience only, as "image pixels."

[0020] The terms "encoding", "writing" or "addressing" are used to describe the process of providing a plurality of pixels of the SLM with a corresponding plurality of control values ​​that respectively determine the modulation level of each pixel. It can be said that the pixels of the SLM are configured to "display" a light modulation distribution in response to receiving the plurality of control values. Thus, the SLM can be said to "display" a hologram, and a hologram can be considered to be an array of light modulation values ​​or levels.

[0021] It has been found that a holographic reconstruction of acceptable quality can be formed from a "hologram" that contains only phase information related to the Fourier transform of the original object. Such a holographic recording may be referred to as a phase-only hologram. The embodiments relate to phase-only holograms, but the disclosure is equally applicable to amplitude-only holograms.

[0022] The present disclosure is also equally applicable to forming a holographic reconstruction using amplitude and phase information related to the Fourier transform of the original object. In some embodiments, this is achieved by using complex modulation of a so-called fully complex hologram that contains amplitude and phase information related to the original object. Because the value (gray level) assigned to each pixel of the hologram has an amplitude and a phase component, such a hologram may be referred to as a fully complex hologram. The value (gray level) assigned to each pixel can be represented as a complex number with an amplitude and a phase component. In some embodiments, a fully complex computer-generated hologram is calculated.

[0023] Reference may be made to the phase value, phase component, phase information, or simply phase, of a pixel of a computer-generated hologram or spatial light modulator as shorthand for "phase delay". That is, any phase value described is actually a number (e.g., in the range of 0 to 2p) representing the amount of phase delay provided by that pixel. For example, a pixel of a spatial light modulator described as having a phase value of p / 2 will delay the phase of the received light by p / 2 radians. In some embodiments, each pixel of the spatial light modulator may be operated at one of a plurality of possible modulation values ​​(e.g., phase delay values). The term "grayscale" may be used to refer to a plurality of available modulation levels. For example, the term "grayscale" may be used for convenience to refer to a plurality of available phase levels in a phase modulator alone, even if the different phase levels do not provide different shades of gray. For convenience, the term "grayscale" may also be used to refer to a plurality of available complex modulation levels in a complex modulator.

[0024] Thus, a hologram comprises an array of gray levels, i.e. an array of light modulation values, such as phase delay values ​​or an array of complex modulation values. A hologram is also considered to be a diffraction pattern, since it is a pattern that causes diffraction when displayed on a spatial light modulator and illuminated with light of a wavelength relative to (usually less than) the pixel pitch of the spatial light modulator. Reference is made herein to combining a hologram with other diffraction patterns, such as a diffraction pattern used as a lens or a grating. For example, a diffraction pattern used as a grating may be combined with a hologram to translate the playback field on the playback plane, or a diffraction pattern used as a lens may be combined with a hologram to focus the holographic reconstruction on the playback plane in the near field.

[0025] The term "target image" is used herein to refer to a desired image for projection. That is, the target image is the image that the holographic system needs to project onto the holographic playback plane. The target image can be a still image or an image (or image frame) in a sequence of images (e.g., a video rate sequence of images).

[0026] The term "source image" is used herein to refer to an image derived from a target image. The source image may be the same as the target image, or the source image may be a high-resolution or enlarged version of the target image. In particular, the source image may be an enlarged version of the target image so as to increase its resolution (in terms of the number of pixels). That is, the source image may include more pixels than the target image. Any enlargement technique may be employed. In some embodiments, as described in the detailed description, the enlargement includes repeating the pixel values ​​of the target image. In these embodiments, the computing engine for enlarging the target image may use a simple mapping scheme to represent the repetition.

[0027] In addition, the source image may be modified, for example to account for distortion caused by the optical components of the holographic projector. In this case, the source image is an "intermediate image" derived from the source image. In the description of the embodiments, the term "intermediate image" is used herein to refer to an image derived from the source image, for example according to a warp map.

[0028] The term "primary image" is used herein to refer to an image that is subsampled as described herein. A primary image can be (1) a source image, or (2) an intermediate image derived from a source image.

[0029] The term "sub-image" is used herein to refer to an image derived from a primary image. As described herein, multiple sub-images can be derived from a single primary image. Each sub-image is formed by subsampling (also known as "undersampling") the primary image. Each sub-image contains fewer pixels than the source image. Each pixel value of the sub-image can optionally be calculated from multiple pixel values ​​(e.g., groups or arrays of pixels) of the primary image using a weighting technique as described in the detailed description. It is noteworthy that the magnification process used to form the source image from the target image is different from the subsampling technique used to form each sub-image from the primary image. The sub-images are all different from the primary image, but optionally, they can have the same number of pixels. A hologram corresponding to each sub-image is calculated.

[0030] The term "output image" is also used herein to refer to the "secondary image" derived by subsampling the primary image as it is output from the image processing engine to the holographic engine to compute the hologram using the appropriate algorithms described herein. Unless otherwise specified, the terms "target image", "primary image", "source image", "intermediate image" and "secondary / output image" are used herein (as a shorthand) to refer to image data including pixel values ​​(or the like) representing the respective images.

[0031] Although different embodiments and embodiment groups may be disclosed separately in the following detailed description, any feature of any embodiment or embodiment group may be combined with any other feature or combination of features of any embodiment or embodiment group. That is, all possible combinations and permutations of the features disclosed in this disclosure are contemplated. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Specific embodiments are described by way of example only with reference to the following drawings:

[0033] Figure 1 is a schematic diagram showing a reflective SLM producing a holographic reconstruction on a screen;

[0034] Figure 2A A first iteration of an example Gerchberg-Saxton type algorithm is shown;

[0035] Figure 2B The second and subsequent iterations of an example Gerchberg-Saxton type algorithm are shown;

[0036] Figure 2C An alternative second and subsequent iteration of an example Gerchberg-Saxton type algorithm is shown;

[0037] Figure 3 is a schematic diagram of a reflective LCOS SLM;

[0038] Figure 4A and 4B shows an example technique for subsampling a primary image using a 4×4 kernel operating at two consecutive sampling positions to derive an output image according to an embodiment;

[0039] Figure 5A Shown for Figure 4A and 4B The common core of the technology, Figure 5B An example kernel with kernel values ​​or weights is shown;

[0040] Figure 6 schematically illustrates a method for data streaming pixel data of a main image for subsampling using a kernel according to an embodiment;

[0041] Figure 7 1 shows an example of an input data stream of image pixel data for a first row of a main image and an output data stream of values ​​provided to a buffer according to an embodiment. Figure 6 The kernel subsampling process;

[0042] Figure 8 Schematically shows a Figure 7how the output data stream of values ​​shown in are used as partial pixel values ​​which are provided as feedback to the kernel subsampling process;

[0043] Fig. 9 1 shows a kernel subsampling process performed on an input data stream of image pixel data for a second row of a main image and an output data stream of values ​​provided to a buffer, the input data stream receiving the image pixel data for a second row of a main image and the output data stream providing the values ​​to the buffer according to an embodiment. Figure 7 The output data value of the process is used as feedback;

[0044] Fig.10 shows a kernel subsampling process performed on an input data stream of image pixel data of a last row of a primary image and an output stream of all (or complete) pixel values ​​corresponding to a secondary image, the input data stream receiving as feedback the output data values ​​of a process performed on pixel values ​​of a previous row of the primary image, according to an embodiment;

[0045] Fig.11 shows a flow chart of a data streaming process for a kernel subsampling process according to an embodiment, wherein a data stream of pixel values ​​is synchronized with a data stream of kernel values;

[0046] Fig.12 A flow chart showing a kernel operation and buffering process according to an embodiment, comprising input values ​​to a kernel subsampling process row by row through a data stream of pixel values ​​and kernel values, wherein the output values ​​of the subsampling process for each row are buffered as partial pixel values ​​and provided as feedback to the subsampling process for the next row; and

[0047] Fig.13 is a schematic diagram showing a holographic projector according to an embodiment.

[0048] The same reference numbers will be used throughout the drawings to refer to the same or like parts. DETAILED DESCRIPTION

[0049] The present invention is not limited to the embodiments described below, but extends to the full scope of the appended claims. That is, the present invention can be implemented in different forms and should not be construed as limited to the described embodiments, which are set forth for illustrative purposes.

[0050] Unless otherwise specified, terms in the singular may include plural forms.

[0051] A structure described as being formed on / under or above / below another structure should be construed to include the case where the structures are in contact with each other and the case where a third structure is provided therebetween.

[0052] When describing a temporal relationship, for example, when the temporal order of events is described as "after," "subsequently," "next," "before," etc., the present disclosure should be considered to include both sequential and non-sequential events unless otherwise specified. For example, the description should be considered to include non-sequential situations unless words such as "only," "immediately," or "directly" are used.

[0053] Although the terms "first", "second", etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish between various elements. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the appended claims.

[0054] The features of different embodiments may be coupled or combined with each other in part or in whole, and may interoperate with each other in different ways. Some embodiments may be performed independently of each other, or may be performed together in an interdependent relationship.

[0055] Optical Configuration

[0056] Figure 1 An embodiment is shown in which a computer generated hologram is encoded on a single spatial light modulator. The computer generated hologram is a Fourier transform of the object used for reconstruction. It can therefore be said that the hologram is a Fourier domain or frequency domain or spectral domain representation of the object. In this embodiment, the spatial light modulator is a reflective liquid crystal on silicon "LCOS" device. The hologram is encoded on the spatial light modulator and a holographic reconstruction is formed at a playback field, such as a light receiving surface such as a screen or diffuser.

[0057] A light source 110, such as a laser or laser diode, is arranged to illuminate the SLM 140 via a collimating lens 111. The collimating lens causes a substantially planar wavefront of light to be incident on the SLM. Figure 1 In some embodiments, the direction of the wavefront is off-normal (e.g., two or three degrees from being truly perpendicular to the plane of the transparent layer). However, in other embodiments, a substantially planar wavefront is provided at normal incidence, and a beam splitter arrangement is used to separate the input and output optical paths. Figure 1 In the embodiment shown, the arrangement is such that light from the light source reflects from the mirrored rear surface of the SLM and interacts with the light modulating layer to form an exit wavefront 112. The exit wavefront 112 is applied to an optical device including a Fourier transform lens 120, the focus of which is located on a screen 125. More specifically, the Fourier transform lens 120 receives the modulated light beam from the SLM 140 and performs a frequency-space transform to produce a holographic reconstruction on the screen 125.

[0058] It is important to note that in this type of hologram, every pixel of the hologram contributes to the entire reconstruction. There is no one-to-one correlation between a specific point on the playback field (or image pixel) and a specific light modulation element (or hologram pixel). In other words, the modulated light leaving the light modulation layer is distributed over the entire playback field.

[0059] In these embodiments, the position of the holographic reconstruction in space is determined by the refractive (focusing) power of the Fourier transform lens. Figure 1 In the illustrated embodiment, the Fourier transform lens is a physical lens. That is, the Fourier transform lens is an optical Fourier transform lens, and the Fourier transform is performed optically. Any lens can act as a Fourier transform lens, but the performance of the lens will limit the accuracy of the Fourier transform it performs. The skilled person understands how to use a lens to perform an optical Fourier transform.

[0060] Hologram computing

[0061] In some embodiments, the computer generated hologram is a Fourier transform hologram, or simply a Fourier hologram or a Fourier based hologram, where the image is reconstructed in the far field by utilizing the Fourier transform properties of a positive lens. The Fourier hologram is calculated by Fourier transforming the desired light field in the playback plane back to the lens plane. The computer generated Fourier hologram can be calculated using the Fourier transform.

[0062] The Fourier transform hologram may be calculated using an algorithm such as the Gerchberg-Saxton algorithm. In addition, the Gerchberg-Saxton algorithm may be used to calculate a hologram in the Fourier domain (i.e., a Fourier transform hologram) from amplitude-only information in the spatial domain (e.g., a photograph). Phase information about the object is effectively "obtained" from amplitude-only information in the spatial domain. In some embodiments, a computer-generated hologram is calculated from amplitude-only information using the Gerchberg-Saxton algorithm or a variation thereof.

[0063] The Gerchberg-Saxton algorithm considers that when the intensity cross-section I of the beam in planes A and B is known, A (x,y) and I B (x,y) and I A (x,y) and I B (x,y) is related by a single Fourier transform. For a given intensity cross section, the phase distribution in planes A and B is approximated as Ψ A (x,y) and Ψ B(x, y). The Gerchberg-Saxton algorithm finds a solution to this problem by following an iterative process. More specifically, the Gerchberg-Saxton algorithm iteratively applies spatial and spectral constraints while repeatedly transferring representations of I between the spatial domain and the Fourier (spectral or frequency) domain. A (x,y) and I B A data set (amplitude and phase) of (x,y). A corresponding computer generated hologram in the spectral domain is obtained by at least one iteration of the algorithm. The algorithm is convergent and arranged to produce a hologram representing the input image. The hologram can be an amplitude-only hologram, a phase-only hologram, or a full complex hologram.

[0064] In some embodiments, a phase-only hologram is calculated using an algorithm based on the Gerchberg-Saxton algorithm, such as described in UK Patents 2498170 or 2501112, the entire contents of which are incorporated herein by reference. However, the embodiments disclosed herein describe calculating a phase-only hologram by way of example only. In these embodiments, the Gerchberg-Saxton algorithm retrieves the phase information Ψ[u,v] of the Fourier transform of a data set, which produces known amplitude information T[x,y], where the amplitude information T[x,y] represents a target image (e.g., a photograph). Since amplitude and phase are inherently combined in the Fourier transform, the transformed amplitude and phase contain useful information about the accuracy of the calculated data set. Therefore, the algorithm can be used iteratively with feedback of the amplitude and phase information. However, in these embodiments, only the phase information Ψ[u,v] is used as a hologram to form a holographic representation of the target image on the image plane. The hologram is a data set (e.g., a 2D array) of phase values.

[0065] In other embodiments, an algorithm based on the Gerchberg-Saxton algorithm is used to calculate a full complex hologram. A full complex hologram is a hologram having an amplitude component and a phase component. A hologram is a data set (e.g., a 2D array) comprising an array of complex data values, where each complex data value comprises an amplitude component and a phase component.

[0066] In some embodiments, the algorithm processes complex data and the Fourier transform is a complex Fourier transform. The complex data can be considered to include (i) a real component and an imaginary component, or (ii) an amplitude component and a phase component. In some embodiments, the two components of the complex data are processed differently at various stages of the algorithm.

[0067] Figure 2AThe first iteration of an algorithm for calculating a phase-only hologram according to some embodiments is shown. The input to the algorithm is an input image 210 comprising a 2D array of pixels or data values, each of which is an amplitude or amplitude value. That is, each pixel or data value of the input image 210 does not have a phase component. Therefore, the input image 210 can be regarded as an amplitude-only or amplitude-only or intensity-only distribution. An example of such an input image 210 is a frame of a photograph or a video comprising a time sequence of frames. The first iteration of the algorithm begins with a data formation step 202A, which includes assigning a random phase value to each pixel of the input image using a random phase distribution (or random phase seed) 230 to form a starting complex data set, wherein each data element of the data set includes an amplitude and a phase. It can be said that the starting complex data set represents the input image in the spatial domain.

[0068] The first processing block 250 receives the starting complex data set and performs a complex Fourier transform to form a Fourier transformed complex data set. The second processing block 253 receives the Fourier transformed complex data set and outputs a hologram 280A. In some embodiments, the hologram 280A is a phase-only hologram. In these embodiments, the second processing block 253 quantizes each phase value and sets each amplitude value to 1 in order to form the hologram 280A. Each phase value is quantized according to the phase level that can be represented on the pixel of the spatial light modulator, which will be used to "display" the phase-only hologram. For example, if each pixel of the spatial light modulator provides 256 different phase levels, each phase value of the hologram is quantized to one phase level among the 256 possible phase levels. The hologram 280A is a phase-only Fourier hologram representing the input image. In other embodiments, the hologram 280A is a fully complex hologram comprising an array of complex data values ​​(each comprising an amplitude component and a phase component) derived from a received Fourier transformed complex data set. In some embodiments, the second processing block 253 constrains each complex data value to one of a plurality of allowable complex modulation levels to form the hologram 280A. The constraining step may include setting each complex data value to the closest allowable complex modulation level in the complex plane. It can be said that the hologram 280A represents the input image in the spectral or Fourier or frequency domain. In some embodiments, the algorithm stops at this point.

[0069] However, in other embodiments, the algorithm continues as Figure 2A In other words, following Figure 2A The steps indicated by dashed arrows are optional (ie, not essential to all embodiments).

[0070] The third processing block 256 receives the modified complex data set from the second processing block 253 and performs an inverse Fourier transform to form an inverse Fourier transformed complex data set. The inverse Fourier transformed complex data set can be said to represent the input image in the spatial domain.

[0071] The fourth processing block 259 receives the inverse Fourier transformed complex data set and extracts the distribution of amplitude values ​​211A and the distribution of phase values ​​213A. Optionally, the fourth processing block 259 evaluates the distribution of amplitude values ​​211A. Specifically, the fourth processing block 259 can compare the distribution of amplitude values ​​211A of the inverse Fourier transformed complex data set with the input image 510, which itself is of course a distribution of amplitude values. If the difference between the distribution of amplitude values ​​211A and the input image 210 is small enough, the fourth processing block 259 can determine that the hologram 280A is acceptable. That is, if the difference between the distribution of amplitude values ​​211A and the input image 210 is small enough, the fourth processing block 259 can determine that the hologram 280A is a sufficiently accurate representation of the input image 210. In some embodiments, for comparison purposes, the distribution of phase values ​​213A of the inverse Fourier transformed complex data set is ignored. It will be appreciated that any number of different methods may be employed to compare the distribution of amplitude values ​​211A and the input image 210, and the present disclosure is not limited to any particular method. In some embodiments, a mean square error is calculated, and if the mean square error is less than a threshold, the hologram 280A is deemed acceptable. If the fourth processing block 259 determines that the hologram 280A is unacceptable, further iterations of the algorithm may be performed. However, this comparison step is not required, and in other embodiments, the number of iterations of the algorithm performed is predetermined or preset or user defined.

[0072] Figure 2B 202B. The second iteration of the algorithm and any further iterations of the algorithm are represented. The distribution of phase values ​​213A of the previous iteration is fed back through the processing blocks of the algorithm. The distribution of amplitude values ​​211A is rejected in favor of the distribution of amplitude values ​​of the input image 210. In the first iteration, the data forming step 202A forms a first complex data set by combining the distribution of amplitude values ​​of the input image 210 with the random phase distribution 230. However, in the second and subsequent iterations, the data forming step 202B includes forming a complex data set by combining (i) the distribution of phase values ​​213A from the previous iteration of the algorithm with (ii) the distribution of amplitude values ​​of the input image 210.

[0073] Then, refer to Figure 2A The same method as described is handled by Figure 2BThe complex data set formed in the data forming step 202B is used to form a second iterative hologram 280B. Therefore, the description of the process is not repeated here. When the second iterative hologram 280B has been calculated, the algorithm can stop. However, any number of further iterations of the algorithm can be performed. It will be understood that the third processing block 256 is only required when the fourth processing block 259 is required or further iterations are required. The output hologram 280B generally gets better with each iteration. However, in practice, a point is often reached where no measurable improvement can be observed, or the positive benefits of performing further iterations are offset by the negative effects of the additional processing time. Therefore, the algorithm is described as iterative and convergent.

[0074] Figure 2C 213A of the previous iteration is fed back through a processing block of the algorithm. The distribution of amplitude values ​​211A is rejected in favor of an alternative distribution of amplitude values. In this alternative embodiment, the alternative distribution of amplitude values ​​is derived from the distribution of amplitude values ​​211 of the previous iteration. Specifically, processing block 258 subtracts the distribution of amplitude values ​​of input image 210 from the distribution of amplitude values ​​211 of the previous iteration, scales the difference by a gain factor α, and subtracts the scaled difference from input image 210. This is expressed mathematically by the following equation, where the subscript text and number represent the number of iterations:

[0075] R n+1 [x,y]=F'{exp(iψ n [u,v])}

[0076] ψ n [u,v]=∠F{η·exp(i∠R n [x,y])}

[0077] η=T[x,y]-a(|R n [x,y]|-T[x,y])

[0078] in:

[0079] F' is the inverse Fourier transform;

[0080] F is the forward Fourier transform;

[0081] R[x,y] is the complex data set output by the third processing block 256;

[0082] T[x,y] is the input or target image;

[0083] ∠ is the phase component;

[0084] Ψ is the phase-only hologram 280B;

[0085] η is the new distribution of amplitude values ​​211B; and

[0086] α is the gain factor.

[0087] The gain factor α may be fixed or variable. In some embodiments, the gain factor α is determined based on the size and rate of the input target image data. In some embodiments, the gain factor α depends on the number of iterations. In some embodiments, the gain factor α is only a function of the number of iterations.

[0088] In all other respects, Figure 2C Examples and Figure 2A and Figure 2B It can be said that only the phase hologram Ψ(u,v) comprises the phase distribution in the frequency domain or the Fourier domain.

[0089] In some embodiments, a Fourier transform is performed using a spatial light modulator. Specifically, the hologram data is combined with second data that provides optical power. That is, the data written to the spatial light modulation includes hologram data representing the object and lens data representing the lens. When displayed on the spatial light modulator and illuminated with light, the lens data simulates a physical lens, that is, it focuses light in the same way as a corresponding physical optical element. Therefore, the lens data provides light or focusing power. In these embodiments, the hologram data can be omitted. Figure 1A physical Fourier transform lens 120 of FIG. 120 . It is known how to calculate data representing a lens. The data representing a lens may be referred to as a software lens. For example, a phase-only lens may be formed by calculating the phase delay caused by each point of the lens due to its refractive index and the spatially varying optical path length. For example, the optical path length at the center of a convex lens is greater than the optical path length at the edge of the lens. An amplitude-only lens may be formed by a Fresnel zone plate. In the field of computer-generated holography, it is also known how to combine data representing a lens with a hologram so that a Fourier transform of the hologram may be performed without the need for a physical Fourier lens. In some embodiments, the lensing data is combined with the hologram by a simple addition, such as a simple vector addition. In some embodiments, a physical lens is used in combination with a software lens to perform a Fourier transform. Alternatively, in other embodiments, the Fourier transform lens is omitted entirely, so that the holographic reconstruction occurs in the far field. In further embodiments, the hologram may be combined with grating data in the same manner, i.e., data arranged to perform a grating function, such as image steering. Again, it is known in the art how to calculate such data. For example, a phase-only grating may be formed by modeling the phase delay caused by each point on the surface of the blazed grating. An amplitude-only grating may simply be superimposed with an amplitude-only hologram to provide angular steering for holographic reconstruction. The second data providing lensing and / or steering may be referred to as a light processing function or light processing pattern to distinguish it from the hologram data which may be referred to as an image forming function or image forming pattern.

[0090] In some embodiments, the Fourier transform is performed jointly by a physical Fourier transform lens and a software lens. That is, some of the optical power that contributes to the Fourier transform is provided by the software lens, while the remaining optical power that contributes to the Fourier transform is provided by one or more physical optical devices.

[0091] In some embodiments, a real-time engine is provided that is arranged to receive image data and calculate a hologram in real time using an algorithm. In some embodiments, the image data is a video comprising a sequence of image frames. In other embodiments, the hologram is pre-calculated, stored in a computer memory and called up as needed for display on the SLM. That is, in some embodiments, a repository of predetermined holograms is provided.

[0092] The embodiments relate to Fourier holography and Gerchberg-Saxton type algorithms only by way of example. The present disclosure is equally applicable to Fresnel holography and Fresnel holograms that can be calculated by similar methods. The present disclosure is also applicable to holograms calculated by other techniques such as techniques based on point cloud methods.

[0093] Light Modulation

[0094] A spatial light modulator may be used to display a diffraction pattern including a computer generated hologram. If the hologram is a phase-only hologram, a spatial light modulator that modulates the phase is required. If the hologram is a full complex hologram, a spatial light modulator that modulates both phase and amplitude may be used, or a first spatial light modulator that modulates the phase and a second spatial light modulator that modulates the amplitude may be used.

[0095] In some embodiments, the light modulation elements (i.e., pixels) of the spatial light modulator are cells comprising liquid crystals. That is, in some embodiments, the spatial light modulator is a liquid crystal device in which the optically active component is a liquid crystal. Each liquid crystal cell is configured to selectively provide multiple light modulation levels. That is, each liquid crystal cell is configured to operate at a light modulation level selected from a plurality of possible light modulation levels at any time. Each liquid crystal cell can be dynamically reconfigured to a light modulation level different from the plurality of light modulation levels. In some embodiments, the spatial light modulator is a reflective liquid crystal on silicon (LCOS) spatial light modulator, but the present disclosure is not limited to this type of spatial light modulator.

[0096] LCOS devices provide a dense array of light modulating elements or pixels within a small aperture (e.g., a few centimeters wide). Pixels are typically about 10 microns or smaller, which results in a diffraction angle of a few degrees, meaning that the optical system can be compact. It is much easier to fully illuminate the small aperture of an LCOS SLM than the large apertures of other liquid crystal devices. LCOS devices are typically reflective, which means that the circuitry that drives the LCOS SLM pixels can be buried under the reflective surface. The result is a higher aperture ratio. In other words, the pixels are tightly packed, which means there are almost no dead spaces between pixels. This is advantageous because it reduces optical noise in the playback field. LCOS SLMs use a silicon backplane, which has the advantage that the pixels are optically flat. This is particularly important for phase modulation devices.

[0097] The following is just an example, Figure 3 A suitable LCOS SLM is described below. An LCOS device is formed using a single crystal silicon substrate 302. It has a 2D array of square planar aluminum electrodes 301, separated by gaps 301a, arranged on the upper surface of the substrate. Each electrode 301 can be addressed by a circuit 302a buried in the substrate 302. Each electrode forms a respective plane mirror. An orientation layer 303 is disposed on the electrode array, and a liquid crystal layer 304 is disposed on the orientation layer 303. A second orientation layer 305 is disposed on a planar transparent layer 306, for example made of glass. A single transparent electrode 307, for example made of ITO, is disposed between the transparent layer 306 and the second orientation layer 305.

[0098] Each square electrode 301 together with the covered area of ​​the transparent electrode 307 and the intervening liquid crystal material defines a controllable phase modulation element 308, commonly referred to as a pixel. The effective pixel area or fill factor is the percentage of the total pixel that is optically active, taking into account the space between the pixels 301a. By controlling the voltage applied to each electrode 301 relative to the transparent electrode 307, the properties of the liquid crystal material of the individual phase modulation elements can be varied to provide a variable retardation to light incident thereon. The effect is to provide only phase modulation to the wavefront, i.e. no amplitude effect occurs.

[0099] The described LCOS SLM outputs spatially modulated light in a reflective manner. A reflective LCOS SLM has the advantage that the signal lines, grating lines and transistors are located below the mirror, which results in a high fill factor (typically greater than 90%) and high resolution. Another advantage of using a reflective LCOS spatial light modulator is that the thickness of the liquid crystal layer can be half that required when using a transmissive device. This greatly increases the switching speed of the liquid crystal (a key advantage for projecting moving video images). However, the teachings of the present disclosure can be equally implemented using a transmissive LCOS SLM.

[0100] Subsampling using kernels

[0101] Figure 4A , 4B Shows the use of Figure 5A and 5B The "kernel" technique shown is used to derive secondary images from a primary image. The kernel can be viewed as a moving "sampling window" or "virtual aperture". The kernel operates on the group of image pixels that fall within the sampling window to derive a single output (subsampled) pixel value representing it. Thus, the kernel is used to "undersample" or "subsample" the pixels of a high-resolution primary image (e.g., a source image or an intermediate image) to derive one or more secondary images (i.e., output images), whereby each secondary image contains fewer pixels than the primary image.

[0102] Figure 5A A generic kernel comprising 4×4 kernel values ​​(also called “kernel pixels” or “weights”) is shown, and Figure 5B An example kernel comprising 4×4 kernel values ​​is shown. Each kernel value defines a weighting factor or weight for the pixel value of the primary image at a corresponding position within the sampling window. Figure 5A As shown, the kernel value is represented by W x,y , where x and y are the corresponding coordinates of the kernel values ​​within the 4×4 array that forms the kernel. Figure 5BIn the example kernel, higher weights (i.e., kernel value = 3) are assigned to pixel values ​​at the center of the sampling window, while lower weights (i.e., kernel value = 1) are assigned to pixel values ​​at the edges and corners of the sampling window. In the illustrated embodiment, at each sampling location, the kernel operates by: (i) multiplying each pixel value of the 4×4 pixel array of the host image that falls within the sampling window by its corresponding kernel value or weight, and (ii) determining a normalized (i.e., unweighted) average of the weighted pixel values ​​(e.g., an average average calculated by dividing the sum of the weighted pixel values ​​by the sum of the kernel values ​​(i.e., the total kernel weight) to derive a single output value representing the 4×4 pixel array of the host image. In the illustrated example, the total kernel weight = 24. Thus, step (ii) can be performed by adding the weighted pixel values ​​of the pixels within the sampling window and dividing by 24.

[0103] Figure 5B A simple example kernel is shown that defines kernel weights such that pixel values ​​of inner sampled pixels (i.e., pixels at the center of the sampling window) of a 4×4 pixel array of a host image have higher weights than pixel values ​​of outer sampled pixels of the host image. As will be appreciated by those skilled in the art, there can be many variations in kernel weight values, depending on application requirements. Additionally, any kernel shape and size (arrangement, aspect ratio, and number of kernel values / weights) corresponding to a sampling window or virtual aperture can be selected according to application requirements. For example, kernel weights can be selected to obtain optimal antialiasing results.

[0104] According to an example technique, a 4×4 kernel is incrementally moved over a primary image to a series of sampling locations (i.e., the locations of a sampling window over the primary image or "sampling window locations") so as to subsample a series of consecutive (i.e., adjacent and non-overlapping) 4×4 pixel arrays of the primary image. A plurality of sampling locations are defined such that substantially all pixels of the primary image are subsampled to derive output values ​​corresponding to subsampled pixel values ​​of the entire primary image. In an embodiment, the sampling locations are spaced at regular pixel intervals or "strides" along the x and y directions over the primary image. The kernel can be said to incrementally traverse the primary image along the x and y directions at pixel intervals or strides.

[0105] Figure 4A and 4B Two consecutive sampling locations of the kernel on the main image are shown in . In particular, Figure 4A The initial (or first) sampling position or "starting position" is shown in . The sampling position can be defined as the pixel coordinate of the main image corresponding to the upper left corner of the kernel. Figure 4AIn the example shown, the first sampling position is at pixel coordinate (0,0) of the main image. The kernel moves incrementally from left to right on the main image according to the stride. In the example technique, the stride is 4 pixels in the x direction and 4 pixels in the y direction, so that the sampling windows formed at each sampling position are continuous and therefore do not overlap with each other. Therefore, the second sampling position is at pixel coordinate (0,4) of the main image, corresponding to a stride distance of 4 pixels in the x direction, as shown in FIG. Figure 4B As shown. As will be appreciated by those skilled in the art, subsequent sampling positions will be at pixel coordinates (0,8), (0,12), and so on, until it reaches the last pixel value in the first 4 rows of the primary image. As the kernel moves in the raster scan path, the kernel returns to the sampling position at pixel coordinate (4,0), which corresponds to a stride distance of 4 pixels in the y direction. The kernel then continues to traverse the primary image in strides of 4 pixels in the x direction and 4 pixels in the y direction to subsample consecutive 4×4 pixel arrays of the primary image using consecutive sampling windows until all pixels of the primary image have been subsampled. The output value determined at each sampling position is provided as a pixel value of the subsampled output image. Therefore, the output secondary image is an undersampled or subsampled version of the primary image with a reduced number of pixels.

[0106] As will be appreciated by those skilled in the art, in the above example, since the 4×4 array of 16 pixels of the primary image is represented by 1 (one) pixel of the secondary image, the number of pixels of the primary image is reduced to 1 / 16 in the secondary image. Therefore, the number of pixels in the primary image has a higher resolution than the desired resolution of the image (holographic reconstruction) projected by the holographic projector. For example, the primary image may have a minimum of 2× the desired resolution, such as 4× or 8× the desired resolution. In this way, even though the resolution is reduced compared to the high-resolution primary image, the hologram calculated from the secondary image forms a holographic reconstruction at the desired resolution. In some embodiments, the target image is "oversampled" or "enlarged" to form the primary image to achieve the desired resolution of the image (holographic reconstruction).

[0107] Subsampling the main image using the kernel to export can be changed by changing one or more of the following: Figure 4A and 4B Sampling scheme of secondary images shown: series of sampling positions; step distance in x and / or y direction and kernel size and / or weight. Thus, multiple secondary images can be determined from the same primary image. Holograms corresponding to the multiple secondary images can be calculated and then displayed on the spatial light modulator as subframes of the image frame (target image) corresponding to the primary image.

[0108] For example, as referenced above Figure 4A and 4B As stated, Figure 5BThe kernel shown in can be used to derive a first image using a series of sampling positions with a stride of 4 pixels in the x direction and 4 pixels in the y direction. The same kernel can be used to derive a second image using a series of sampling positions with the same stride of 4 pixels in the x and y directions but with a different initial (or first) sampling position or "starting position", such as coordinates (1,1). In other examples, the stride in the x and / or y direction can be varied.

[0109] In some other examples, you can use Figure 5B The kernel shown in derives the first image using a series of sampling positions with a stride of 4 pixels in the x direction and 4 pixels in the y direction, as shown in the reference above. Figure 4A and 4B A different kernel, such as a 4×4 kernel with different kernel values ​​or weights, may be used to derive a second image using the same series of sampling positions. In other examples, the size of the kernel and the kernel values ​​or weights may be varied (e.g., a 5×5 kernel with similar or different weights).

[0110] Any suitable combination of the above example sampling schemes can be used to derive multiple secondary images from the same primary image. As will be appreciated by those skilled in the art, the total number of pixels of each secondary image derived from the same primary image can vary. In particular, the number of pixels of each secondary image corresponds to the number of sampling positions used in the respective subsampling scheme used. Therefore, a scheme with fewer sampling positions will result in a secondary image with fewer pixels. In some examples, each secondary image determined from the same primary image has the same number of pixels.

[0111] Thus, multiple sub-images representing an image for projection can be generated by subsampling the primary image using a sampling scheme including a kernel of a 4×4 array kernel value. Each sub-image contains fewer pixels than the primary image. A hologram is determined for each of the multiple sub-images, and each hologram is then displayed on a display device to form a holographic reconstruction corresponding to each sub-image on a playback plane. In an embodiment, each of the multiple holograms corresponding to the sub-image is displayed on the display device in turn within the integration time of the human eye, so that its holographic reconstruction on the playback plane looks like a single high-quality reconstruction of the target image.

[0112] Data Streaming

[0113] As described above, in order to subsample the main image using the kernel, it is necessary to store pixel data corresponding to the two-dimensional array pixels of the main image, and buffer the pixel data entries (i.e., pixel values) of the main image required for the kernel operation at a specific sampling position (e.g., 4×4 pixel array data) at least at a corresponding time point. Therefore, the system requires a large amount of data storage and / or capacity to accommodate the pixel data of the relatively high-resolution main image.

[0114] Therefore, the present disclosure proposes a novel scheme to achieve sub-sampling of a primary image by a kernel at a series of sampling positions to determine a secondary image. The novel scheme allows for reduced storage and / or buffer capacity requirements. In some embodiments, the processing speed for determining a hologram corresponding to a secondary image is increased.

[0115] The novel scheme is based on data streaming. In particular, the scheme involves forming a first data stream of pixel values ​​of a main image and a second data stream of kernel values ​​of a kernel. The data streaming is performed in sequence row by row (i.e. pixel by pixel of the image / kernel). Specifically, the pixel values ​​of the first row of pixels of the main image are first streamed, then the pixel values ​​of the second row of pixels of the main image, and so on, until the pixel values ​​of the last row of pixels of the main image, at which stage the first data stream ends. In addition, the kernel values ​​of the first row of the kernel are first streamed, then the kernel values ​​of the second row of the kernel, and so on, until the kernel values ​​of the last row of the kernel; then, the data streaming sequence is iteratively repeated by streaming the kernel values ​​of the first row of the kernel, then the pixel values ​​of the second row of the kernel, and so on. The number of pixel values ​​of the first data stream corresponds to the number of kernel values ​​of the second data stream. Specifically, each pixel value of the first data stream is paired with the corresponding kernel value of the second data stream so that the corresponding weighted pixel value can be determined. This is achieved by synchronizing the pixel values ​​of the pixels of the main image in the first data stream with the corresponding kernel values ​​of the kernel in the second data stream. As will be appreciated by those skilled in the art, in the example shown, the kernel traverses the primary image from left to right (i.e., in the x-direction) as the kernel moves in a raster scan path. Thus, for a particular row of pixels of the primary image, the same row of kernel values ​​is used to weight the corresponding pixel values, and the row of kernel values ​​is repeated for each sampling position. By sequentially streaming the pixel values ​​of the pixels of the primary image and the corresponding kernel values ​​or weights of the kernel row by row, processing can be performed on a single data stream of pixel values ​​corresponding to a one-dimensional array of pixels, rather than the pixel values ​​of a two-dimensional array of pixels as in the prior art. Figure 6 This process is schematically illustrated as a first data stream forming pixel values ​​and a second data stream of kernel values ​​or weights which are input into a so-called "baseline streaming frame kernel" (or simply "frame kernel") to perform the kernel operation, as described below.

[0116] Fig.11is a flow chart of a method 1100 illustrating a data flow for subsampling a primary image using a kernel according to some embodiments. The method 1100 receives an input image corresponding to a primary image, including pixel data of a pixel array of the primary image for subsampling to determine a secondary image as described herein. The method 1100 also receives a kernel including an array of kernel values ​​or weights as described herein. In the example shown, the kernel includes an array of M rows and N columns of kernel values, thereby forming a sampling window of an M×N pixel array. In addition, a series of sampling positions for the kernel are defined. In the example shown, the stride in the x direction is N pixels and the stride in the y direction is M pixels, so that the series of sampling positions subsamples a continuous array of M×N pixels of the primary image, as described above in Figure 4A and 4B This has the advantage that each pixel value of the main image is subsampled only once, which means that the pixel data of the main image can be streamed in raster scan order, as described below. As will be appreciated by those skilled in the art, the stride in each of the x and y directions can be varied to vary the range of sampling positions as required by the application. For example, the stride may be N / 2 in the x direction and M / 2 in the y direction.

[0117] Fig.11 The method 1100 forms a first data stream of pixel values ​​and a second data stream of kernel values, wherein the first and second data streams are formed simultaneously such that each pixel value in the first data stream is synchronized in time and therefore paired with a corresponding kernel value in the second data stream.

[0118] According to an embodiment, the method starts at step 1105 in response to receiving a main image and kernel information for subsampling the main image using a kernel.

[0119] Step 1106 sets the row counter of the image to 0 (zero), and step 1108 sets the column counter of the image to 0 (zero). Step 1110 reads the pixel value of the main image at the row and column position indicated by the image row and column counters (i.e., the pixel value at coordinate (0,0)), which corresponds to the initial sampling position. Step 1112 increments the image column counter by 1. The pixel value read in step 1110 is output at "A" as the first (next) pixel value of the first data stream. In addition, step 1156 sets the row counter of the kernel to 0 (zero), and step 1158 sets the column counter of the kernel to 0 (zero). Step 1160 reads the kernel value of the kernel at the row and column position indicated by the kernel row and column counters (i.e., the kernel value at coordinate (0,0)), and step 1162 increments the kernel column counter by 1. The kernel value read in step 1160 is output at "A" as the first (next) kernel value of the second data stream. Steps 1106, 1108, and 1110 are performed simultaneously with steps 1156, 1158, and 1160, and the output of data values ​​to the first and second data streams at "A" are synchronized. In particular, the pixel values ​​and the corresponding kernel values ​​are output at "A" simultaneously (i.e., on the same clock cycle). Thus, the pixel values ​​in the first data stream and the corresponding kernel values ​​in the second data stream are paired for kernel processing, as further described below.

[0120] Step 1120 determines for the current iteration whether the pixel value read at step 1110 is the last pixel value in the current row of the main image. Since the method receives the main image as input, the size of the image in terms of the number of rows and columns of pixels is known. Therefore, if the value set in the image column counter at step 1112 is greater than the number of columns of pixels, the pixel value read in the iteration is the last pixel in the image row and the next pixel should be read from the next image row. If step 1120 determines that the pixel value read at step 1110 is not the last pixel value in the current image row, the method returns to step 1110, which reads the next pixel value in the current row (i.e., at the row and column position indicated by the image row and column counters). The method then continues in a loop through steps 1110, 1112, and 1120 until step 1120 determines that the pixel value read at step 1110 is the last pixel value in the current row. When step 1120 determines that the pixel value is the last pixel value in the current row, the method proceeds to step 1130. Additionally, a definitive indication (eg, an "end of image line" signal) may be output at "C".

[0121] Step 1130 determines whether the current image row is the last row of pixels of the main image. In particular, if the current value set in the image row counter is equal to the total number of rows of pixels of the main image, the current image row is the last row of pixels. If step 1130 determines that the current image row is not the last row of pixels of the main image, the method proceeds to step 1140, which increments the image row counter by 1. The method then returns to step 1108, which (re)sets the image column counter to 0. The method then continues with steps 1110 to 1120 by streaming the pixels of the next image row until step 1130 determines that the current image row is the last row of pixels of the main image. When step 1130 determines that the current image row is the last pixel of the main image, the method ends at step 1195. At the same time, a determined indication (e.g., an "end of image" signal) may be output at "E". Therefore, by Fig.11 The first data stream formed by method 1100 includes a stream of pixel values ​​of pixels of a main image read pixel by pixel and row by row in a raster scan order.

[0122] Step 1170 determines whether the kernel value read at step 1160 is the last kernel value in the current row of the kernel for the current iteration. Since the method receives the kernel as input, its size is known in terms of the number of rows M and columns N of kernel values ​​or weights. Therefore, if the value set in the kernel column counter at step 1162 is greater than the number of columns N of kernel values, the previous kernel value is the last kernel value in the row and the next kernel value should be read from the next row of the kernel. If step 1170 determines that the kernel value read at step 1160 is not the last kernel value in the current kernel row, the method returns to step 1160, which reads the next kernel value in the current row (i.e., at the row and column position indicated by the kernel row and column counters). The method then continues in a loop through 1160, 1162, and 1170 until step 1170 determines that the kernel value read at step 1160 is the last kernel value in the current row. When step 1170 determines that the kernel value is the last kernel value in the current row, the method proceeds to step 1180. Furthermore, a certain indication (eg, a "kernel line end" signal) may be output at "B".

[0123] Step 1180 determines for the current iteration whether the pixel value read at step 1110 is the last pixel value in the current row of the main image. Step 1180 may be performed by determining whether an "end of image row" signal is output from step 1120 at "C", or may be a separate or joint operation with respect to step 1120. If step 1180 determines that the pixel value is not the last pixel value in the current row of the main image, the method returns to step 1158, which (re)sets the kernel counter column to 0. As will be appreciated by one skilled in the art, this corresponds to moving to the next sampling position by the stride in the x-direction of the kernel shift, which in the example shown is equal to the number of kernel columns N. The method then continues through steps 1160 to 1180 by sequentially repeating the reading of kernel values ​​in the current row until step 1180 determines that the end of the image row of the main image has been reached (e.g., by receiving an "end of image row" signal). When step 1180 determines that the pixel value read at step 1110 is the last pixel value in the current row of the main image, the method proceeds to step 1185.

[0124] Step 1185 determines whether the current kernel row is the last row of kernel values ​​for the kernel. In particular, if the current value set in the kernel row counter is equal to the number of kernel rows M, then the current row is the last row of kernel values. If step 1185 determines that the current kernel row is not the last row of kernel values, the method proceeds to step 1190, which increments the kernel row counter by 1. The method then returns to step 1158, which (re)sets the kernel column counter to 0. The method then continues with steps 1160 to 1180 by sequentially repeating the streaming of kernel values ​​for the next kernel row until 1185 determines that the current kernel row is the last row of kernel values. When step 1185 determines that the current kernel row is the last row of kernel values, the method returns to step 1156, which (re)sets the kernel row counter to 0. Those skilled in the art will appreciate that this corresponds to moving to the next sampling position by the stride of the kernel shift in the y direction, which in the example shown is equal to the number of kernel rows M. At the same time, a determined indication (e.g., a "kernel end" signal) may be output at "D". Thus, by Fig.11 The second data stream formed by the method 1100 includes a stream of kernel values ​​or weights of the kernel that are read sequentially on a row-by-row basis (and repeated for each sampling position). In particular, the kernel values ​​of the second data stream are provided so that the kernel value at each position / time point in the second data stream corresponds to and is paired / synchronized with the pixel value of the pixel of the primary image in the first data stream.

[0125] As will be appreciated by those skilled in the art, Fig.11 The method of may be modified in various ways to derive synchronized first and second data streams for kernel subsampling of a main image. For example, Fig.11 The method provides a stream of pixel values ​​of the main image in a raster scan order because the example kernel process has a stride in the x direction that is the same as the number of columns in the kernel (and a stride in the y direction that is the same as the number of rows in the kernel) so as to sample a continuous array of pixels of the main image. However, in other examples, the stride in the x direction may be less than or greater than the number of columns in the kernel so as to sample overlapping or separated array pixels of the main image in the x direction, respectively. Therefore, in the case where the stride in the x direction is less than the number of columns in the kernel, each successive sampling window will overlap one or more pixels with the previous sampling window. In this case, the overlapping pixel values ​​can be repeated in the order of the pixel values ​​read from each row of the main image to form a first data stream. In contrast, if the stride in the x direction is greater than the number of columns in the kernel, each successive sampling window will be separated from the previous sampling window by one or more pixels in the x direction. In this case, the pixel values ​​in the space between the sampling windows can be omitted from the sequence of pixel values ​​read from each row of the main image to form a first data stream. In these examples, similar modifications may be required to modify the process of forming the second data stream of kernel values ​​so that the correct corresponding kernel values ​​are synchronized with the pixel values ​​based on the position of the pixel values ​​within each moving sampling window. Similarly, in other examples, the stride in the y direction can be less than or greater than the number of rows in the kernel so as to sample overlapping or separated array pixels of the main image in the y direction respectively. In these examples, the pixel values ​​of one or more pixel rows of the main image can be repeated or omitted in the first data stream, and the corresponding kernel values ​​can be synchronized in the second data stream accordingly.

[0126] Kernel Operations and Buffering

[0127] As described herein, the process of subsampling a primary image to determine a secondary image may be performed by kernel operations at a series of sampling positions. According to the present disclosure, the kernel-based subsampling process is performed using data streaming to reduce storage and / or buffer capacity requirements.

[0128] Figure 7 An example of a kernel subsampling process performed on a first data stream of pixel values ​​of a first row of a main image using corresponding kernel values ​​or weights of a second data stream is shown, wherein the first and second data streams are formed as described above. Figure 7 Further shown is the output data flow of accumulated values ​​(so called "partial pixel values") provided to a buffer. Figure 7 It can be called the first iteration because it performs the kernel operation on a row of pixels of the main image using the corresponding kernel values ​​or weights of the first row of the kernel. Since the pixel values ​​of the main image are processed row by row, the processing of each row of image pixels can be considered as an iteration.

[0129] exist Figure 7In the example shown, the kernel is a 4×4 array of kernel values, and the kernel is iteratively moved to a series of sampling positions so that consecutive 4×4 array pixels of the main image are contained within the sampling window. In particular, the stride in the x direction is 4 pixels and the stride in the y direction is 4 pixels. For ease of illustration, the main image has 20 pixels in a row (i.e., 20 columns of pixels) consecutively. As will be appreciated by those skilled in the art, in practice, the number of pixels of the main image is much larger.

[0130] like Figure 7 As shown, the first data stream 90 includes an ordered sequence of 20 pixel values ​​T, S, R, Q, P, O, N, M, L, K, J, I, H, G, F, E, D, C, B and A for pixels of a first row (row 0) of a main image at data input positions 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 and 0, respectively. In addition, the second data stream 92 includes an ordered sequence of 20 kernel values, wherein the kernel value at each data input position in the second data stream corresponds to the pixel value at the same data input position in the first data stream. As will be appreciated by those skilled in the art, the kernel values ​​used to sample the pixels of the first row of image pixels correspond to the first row of kernel values ​​of the kernel. As shown in FIG. Figure 5A In the universal kernel of , each kernel value is represented by the symbol W x,y represents the weights of the kernel, where (x, y) corresponds to the kernel coordinates of the kernel values ​​or weights.

[0131] The basic kernel processing 94 receives each pixel value of the first data stream 90 and the corresponding weight W of the synchronization of the second data stream 92 x,y , and determine the corresponding weighted pixel value. In particular, the weighted pixel value is the product of the pixel value and the corresponding weight. The basic kernel process 94 sequentially adds together the weighted pixel values ​​for each kernel sampling position to determine the corresponding (weighted) partial pixel value. Thus, the weighted pixel value for the first four pixel values ​​of the first row of pixels in the first data stream corresponding to the first sampling position (kernel position 0) is T*W 0,0 ,S*W 1,0 ,R*W 2,0 and Q*W 3,0 The sum of the four weighted pixel values ​​at the first sampling position is represented by the value 1, which is written to the first position of the output buffer 96 (buffer position 0). Figure 7As shown, the basic kernel process then sequentially determines the weighted pixel value of each group of four pixel values ​​for the first row of pixels in the first data stream, corresponding to the second to fifth sampling positions (kernel positions 1 to 4), respectively. The basic kernel process 64 also determines the sum of each of the four weighted pixel values, represented as II, III, IV and V, and writes them sequentially to the second to fifth positions (buffer positions 1 to 4) of the output buffer, respectively. In some embodiments, the basic kernel process 64 can implement a "multiply accumulate" or MAC process. For each pair of pixels and kernel values ​​received from the first and second data streams, the process multiplies the pixel value and the weight together to determine the weighted pixel value, and for the same kernel sampling position, accumulates the weighted pixel value by adding the weighted pixel value to the sum of the previously determined weighted pixel values. Therefore, the cumulative sum based on the four weighted pixel values ​​(corresponding to the number of columns of the kernel and therefore the "stride") is the partial (sub-sampled and weighted) pixel value corresponding to the kernel sampling position. These partial pixel values ​​are stored in the output buffer 96 and form a third data stream, which is provided as feedback to the kernel basic process 94. Figure 8 The feedback of partial pixel values ​​from a third data stream 98 output from processing a first row of pixels of a main image and sequentially stored in an output buffer 96 for each sampling position to a basic kernel process 94 is schematically shown. The feedback of the third data stream 98 is provided simultaneously with the first and second data streams 90, 92 so that the partial pixel values ​​are synchronized with the pixel values ​​of the next row of pixels of the main image in the first data stream 90 and the corresponding kernel values ​​in the second data stream 92.

[0132] Fig. 9 Shows Figure 7 The second iteration of the kernel subsampling process is shown in Figure 2. Fig. 9 In the embodiment of the present invention, the process is performed on the first data stream of image pixel values ​​of the second row of the main image using the corresponding kernel values ​​of the second row of the kernel of the second data stream. Fig. 9 Further shown is the use of a third data stream of partial pixel values ​​which are output in the first iteration by subsampling the pixels of the first row of the main image and provided as feedback from the output buffer. Finally, Fig. 9 An output data stream of updated partial pixel values ​​is shown being provided sequentially to an output buffer.

[0133] like Fig. 9As shown, the first data stream 90 includes an ordered sequence of 20 pixel values ​​T, Σ, Ρ, Θ, Π, Ο, N, M, Λ, K, θ, Ι, H, Γ, Φ, E, Δ, X, B and A for pixels of the second row (row 1) of the main image at data input positions 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 and 0, respectively. In addition, the second data stream 92 includes an ordered sequence of 20 kernel values, wherein the kernel value at each data input position in the second data stream corresponds to the pixel value at the same data input position in the first data stream. As will be understood by those skilled in the art, the kernel value corresponds to the next (second) row of the kernel.

[0134] The basic kernel processing 94 receives each pixel value of the second pixel row of the main image of the first data stream 90 and the corresponding weight W of the synchronization of the second data stream 92 x,y , and determine the corresponding weighted pixel value. The basic kernel process 94 adds together the weighted pixel values ​​for each kernel sampling position to further determine the corresponding (weighted) partial pixel value. Therefore, the weighted pixel value for the first four pixel values ​​of the second row of pixels in the first data stream corresponding to the first sampling position (kernel position 0) is T*W 0,1 ,Σ*W 1,1 ,P*W 2,1 and Θ*W 3,1 The sum of the four weighted pixel values ​​at the first sampling position is added to the first portion of pixel values ​​of the third data stream 98 corresponding to the cumulative sum of the weighted pixel values ​​at the same sampling position. The updated cumulative sum of the weighted pixel values ​​is represented by the value N, which is written to the first position (buffer position 0) of the output buffer 96. Fig. 9 As shown, the basic kernel processing then determines the cumulative sum of the weighted pixel values ​​of each of the second to fifth sampling positions (kernel positions 1 to 4) in the same manner, which are written as partial pixel values ​​P, Q, R, Z in the second to fifth positions (buffer positions 1 to 4), respectively.

[0135] As will be appreciated by those skilled in the art, the kernel values ​​from the corresponding row of the kernel up to the last kernel row are used to iteratively repeat the process using the pixel values ​​of the first data stream for each subsequent image row of the main image. Fig. 9 94. Once the pixel value corresponding to the kernel value of the last kernel row has been processed for a sampling position, all pixel values ​​within the sampling window have been sampled, and the accumulated value output to the buffer 96 represents the total (weighted sum) pixel value. Therefore, when the basic kernel process 94 determines that it is using the last kernel row for subsampling, the output of the accumulated (weighted) pixel value is as follows with reference to Fig.10 Modify as described.

[0136] Fig.10 The output of all or complete accumulated pixel values ​​of the secondary image is schematically shown. This process can occur after each iteration using the kernel values ​​of the last kernel row and / or after the final iteration. In particular, once all pixel values ​​of the pixels in the sampling window at the kernel sampling position are subsampled, the total accumulated (weighted) pixel values ​​are determined. In particular, when the pixel values ​​of the image row in the first data stream are processed using the last row of kernel values ​​of the kernel in the second data stream 92, the basic kernel processing 94 outputs the total accumulated pixel values. As before, the accumulated (weighted) pixel values ​​are sequentially provided to the output buffer 96 to form the third data stream 98. However, instead of providing the accumulated pixel values ​​as feedback to the basic kernel processing 94, the third data stream 98 is fed to the optional division unit 126. The optional division unit 126 can determine the final subsampled pixel value for the secondary image by dividing each accumulated (weighted) pixel value of the third data stream 98 by the total kernel weight to determine the normalized pixel value as described above. The output stream 122 from the divide unit may be streamed to a data store for the secondary image, and / or may be sent in real time to the hologram engine for hologram calculations. Additionally, instead of feeding the third data stream 98 back to the basic kernel processing 94, a reverse data stream 124 including a sequence of null data values ​​(i.e., a sequence of zeros (0)) equal in number to the number of kernel positions used to subsample the image rows may be provided to the basic kernel processing 94. The reverse data stream 124 is provided simultaneously to the first and second data streams 90, 92 to synchronize the values ​​at corresponding positions in the respective data streams. Although not provided in Figure 7 , but a similar backward data stream 124 may be provided to the basic kernel processing 94 for processing pixel values ​​corresponding to the first image row.

[0137] Fig.12 is a flow chart illustrating a method 1200 for subsampling a main image based on kernel operation and buffering based on data streaming, according to some embodiments. As described above, the method 1200 receives synchronized first and second data streams of pixel values ​​and kernel values, respectively, and outputs a third data stream of partial (or full) pixel values ​​to an output buffer for feedback (or output). As described above, it can be combined with Fig.11 The method 1100 of data stream transmission is implemented by the method 1200. In particular, Fig.12 Certain steps of method 1200 may be obtained from Fig.11 The step of method 1100 receives an output signal. Fig.11In the method, the kernel comprises an array of M rows and N columns of kernel values, thus forming a sampling window of the M×N pixel array of the host image. In addition, a series of sampling positions of the kernel are defined with a stride of N pixels in the x direction and a stride of M pixels in the y direction, so as to subsample a continuous array of M×N pixels of the host image.

[0138] Fig.12 The method uses a kernel to subsample a continuous array of M×N pixels of a primary image to generate pixel values ​​of a secondary image with reduced resolution. In particular, for each array of M×N pixels of the primary image contained within a sampling window defined by the kernel at a sampling position, a single pixel value of the secondary image is determined.

[0139] In response to receiving a first value of the first and second data streams or a related trigger indicating the start of the data streaming process, the method starts at step 1205. In some embodiments, step 1205 may additionally receive information about the main image, in particular the number of rows and columns of image pixels and / or information about the kernel, in particular the number of rows and columns providing kernel values, in order to track pixel and kernel values ​​and kernel sampling positions, as described below.

[0140] Step 1210 sets the current storage position of the output buffer to the first kernel sampling position. It can be said that the first storage position in the output buffer matches the first sampling position of the kernel. As will be appreciated by those skilled in the art, as the kernel moves over the main image (e.g., in a raster scan path including strides in the x and y directions), the sampling positions of the kernel can be defined in a numerical order that increases by one (e.g., from kernel position 0 to kernel position X) according to an ordered sequence of sampling positions. As will be appreciated by those skilled in the art, in the illustrated embodiment, each storage position in the output buffer is dedicated to receiving an output (partial) pixel value associated with a corresponding sampling position.

[0141] Step 1220 receives a first pair of values ​​of the synchronized first and second data streams. For example, the first pair of values ​​may be received from Fig.11 The first and second data streams may be received at the output of method 1100 at "A". Thus, step 1220 receives a first pixel value of the main image from the first data stream and a corresponding first kernel value or weight of the kernel from the second data stream. As described above, the corresponding pairs of pixel and kernel values ​​may be synchronized by a clock counter.

[0142] Step 1222 determines a corresponding weighted pixel value by multiplying the pixel value and the kernel value received in step 1220. Then, step 1224 determines whether the weighted pixel value determined in step 1222 corresponds to the last pixel value of the image row at the current kernel sampling position. In particular, step 1224 can determine whether the processed value pair of the first and second data streams corresponds to the end of the kernel row (or the last kernel column). In the example shown, if step 1224 receives the pixel value from Fig.11 If "B" in method 1100 receives a "kernel row end" signal, step 1224 determines that the weighted pixel value determined in step 1222 corresponds to the last pair of values ​​for the current kernel sampling position. In other examples, a kernel column counter may be used (e.g., set to zero prior to step 1220), in which case step 1224 may directly compare the value of the kernel column counter to the number of kernel columns. As will be appreciated by those skilled in the art, in the example shown, the kernel column counter is N when the kernel operates on the last pixel value of the current sampling position. If step 1224 determines that the weighted pixel value determined in 1222 is not the last pixel value of the current sampling position, the method returns to step 1220, which receives the next pair of pixels and kernel values ​​for the first and second data streams. In examples using a column kernel counter, the kernel counter is incremented by 1 before the next pair of values ​​is read in step 1220. The method then continues in a loop including steps 1220 to 1224 until step 1224 determines that the previously determined weighted pixel value corresponds to the last pair of pixel kernel values ​​for the current sampling position, and the method proceeds to step 1230 .

[0143] Step 1230 determines the sum of the weighted pixel values ​​for the current kernel position. In some embodiments, for a given kernel sampling position, each of the N weighted pixel values ​​determined by the successive iterations of step 1224 is sent to a register (or other temporary data storage) by an adder. The adder adds the received weighted pixel value to the current value in the register and returns the result to the register, thereby updating the value stored therein. Therefore, in these embodiments, the N weighted pixel values ​​determined for a particular kernel sampling position are added together in real time, and step 1230 determines the sum of the weighted pixel values ​​as the final value stored in the register. In other embodiments, each weighted pixel value for a particular kernel sampling position can be stored in a register (or other temporary data storage) and added together in step 1230.

[0144] Step 1232 also adds the weighted pixel value sum of the current kernel position determined in step 1230 to the cumulative (partial pixel) value of the third data stream received as feedback from the output buffer. In the example shown, a third data stream of cumulative / partial pixel values ​​can be received from "F", as described below. As described above, when processing pixel values ​​in a row corresponding to the kernel value in the first row of the kernel, a stream of empty partial pixel values ​​can be provided as feedback. Therefore, the value determined in step 1232 corresponds to the updated cumulative (partial pixel) value of the current kernel position.

[0145] Step 1240 stores the updated accumulated (partial pixel) value determined in step 1222 in the current storage location of the output buffer. The method then continues to step 1250.

[0146] Step 1250 determines whether the pixel value of the first data stream received in the previous step 1220 corresponds to the pixel value of the last pixel in the image row of the main image. In the example shown, if step 1250 receives the pixel value of the last pixel in the image row of the main image, Fig.11 If "C" of method 1100 receives "end of image row", step 1250 determines that the pixel value received in the previous step 1220 is the last pixel in the image row of the main image and is therefore at the end of the image row. In other examples, method 1200 may receive information about the number of pixel columns of the main image and include an image column counter (not shown) to track the pixel values ​​of the first data stream received in step 1220 in order to identify the end of the image row. If step 1250 determines that the pixel value received in the previous step 1220 does not correspond to the last pixel in the image row, the method proceeds to step 1252, which increments the storage location of the output buffer by 1 and sets the new storage location to the next kernel sampling location, such as by incrementing the kernel sampling location by 1 (see step 1210). The method then returns to step 1220, which receives the next pair of pixels and kernel values ​​for the first and second data streams. Again, in the example of using a kernel column counter, the kernel column counter is reset to 0 before the next pair of values ​​is received in step 1220. The method then continues in an inner loop including steps 1220 to 1252 until step 1250 determines that the pixel value received in the previous step 1220 is at the end of an image line. When step 1250 determines that the last received pixel value is at the end of an image line, the method proceeds to step 1260.

[0147] Step 1260 determines whether the image row of pixel values ​​received in the previous step 1220 corresponds to the last image row of the main image and is therefore the pixel value of the last pixel of the main image in the first data stream. In the example shown, if step 1260 receives a pixel value from Fig.11If "E" of method 1100 receives an "end of image" signal, step 1260 determines that the pixel value received in the previous step 1220 is the pixel value of the last pixel of the first data stream.

[0148] If step 1260 determines that the pixel value received in the previous step 1220 is not the pixel value of the last pixel in the first data stream, then the subsampling of the primary image is not completed, and the method proceeds to step 1270. On the other hand, if step 1260 determines that the pixel value received in the previous step 1220 is the pixel value of the last pixel in the first data stream, then the subsampling of the primary image is completed, and the method proceeds to step 1280, which outputs the last row of accumulated values ​​of the third data stream from the output buffer as the full (or complete) pixel value of the (subsampled) secondary image. The method then ends at step 1285.

[0149] Returning to step 1270, method 1200 determines whether the kernel value or weight of the second data stream received in step 1220 is from the last row of the kernel (i.e., the last kernel value of the kernel), which means that the kernel sampling at the current sampling position is completed. In the example shown, if step 1270 is from Fig.11 If the "D" of method 1100 receives a "kernel end" signal, step 1270 determines that the kernel value received in the previous step 1220 is the last kernel value of the kernel. In other examples, a kernel row counter is used to track kernel rows, and step 1270 can directly compare the current value in the kernel row counter with the kernel row number M and determine that the kernel value received in the previous step 1220 is the last kernel value of the matching kernel. If step 1270 determines that the kernel value received in the previous step 1220 is not the last kernel value of the kernel, the method continues to step 1272, which provides the current row of accumulated values ​​stored in the output buffer as a third data stream of partial pixel values ​​as feedback at "F", which is again received in step 1232, as described above. The method then returns to step 1220, which receives the next pair of pixels and kernel values ​​of the first and second data streams. Again, in the example of using a kernel column counter, the kernel column counter is reset to 0 before the next pair of values ​​is received in step 1220. The method then continues in an outer loop including steps 1220 to 1272, which continues to subsample the kernel on the same set of sampling positions (i.e., rows of sampling positions) as the previous outer loop until step 1270 determines that the kernel value received in the previous step 1220 is the last kernel value of the kernel, and the method proceeds to step 1274.

[0150] When step 1270 determines that the kernel value received in the previous step 1220 is the last kernel value of the kernel, two results occur. First, the sampling window of the kernel will move the stride distance in the y direction to the next sampling position (i.e., the next row of sampling positions) and move in the x direction to the beginning of the next image row (because the pixel values ​​in the first data stream are read from the primary image in raster scan order). Second, the kernel operation has processed a sub-sample of the N pixel values ​​contained in the kernel at each sampling position in a row of sampling positions, so the accumulated value output to each corresponding storage position of the output buffer is the full (or complete) pixel value of the secondary image.

[0151] Therefore, step 1274 outputs the accumulated values ​​stored in the output buffer as a third data stream including all (or complete) pixel values ​​of the secondary image to, for example, Fig.10 The divider 120 shown is shown. This can be used to clear the storage location of the output buffer. The method 1200 then returns to step 1210, which sets the current storage location in the output buffer to the next kernel sample location. Therefore, the current first storage location in the output buffer matches the first sample location in the next row of sample locations of the kernel. The method 1200 then continues until step 1260 determines that the last image row has been processed, the last row of all (complete) pixel values ​​of the (sub-sampled) secondary image is output in step 1280, and the method ends in step 1285.

[0152] As will be appreciated by those skilled in the art, Fig.11 and 12 The flowchart shown is exemplary only. When implementing the present disclosure, many variations or modifications are possible and expected. For example, the steps may be performed in an order different from the order depicted in the flowchart.

[0153] Interlaced Scan

[0154] As described above, a plurality of secondary images may be generated by subsampling (undersampling) a primary image (source image or intermediate image) using a kernel. Each secondary image contains fewer pixels than the primary image. A hologram is determined for each of the plurality of secondary images, and each hologram is sequentially displayed on a display device to form a holographic reconstruction corresponding to each secondary image on a playback plane.

[0155] Thus, disclosed herein are techniques for interlacing multiple holographic reconstructions corresponding to a master image, optionally while compensating for deformation by subsampling deformed versions of the source images (ie, intermediate images).

[0156] In some embodiments, the speed of hologram calculation and interlacing is increased by means of a data streaming method according to the present disclosure. In particular, all or complete pixel values ​​of the secondary image may be streamed to the hologram engine in real time to start hologram calculation before all pixel values ​​of the secondary image are determined. For example, before all or complete pixel values ​​of the last row of the second image are output at step 1280, the pixel values ​​of the secondary image may be streamed to the hologram engine in real time to start hologram calculation. Fig.12 Step 1274 of method 1200 outputs all or complete pixel values ​​for each row of the secondary image, which are streamed in real time to the hologram engine to begin hologram calculations.

[0157] Therefore, a method for generating a secondary image by undersampling a primary image using a kernel having m rows and n columns of kernel values ​​is disclosed herein, wherein the kernel has a plurality of kernel sampling positions for each row of the primary image, each kernel sampling position of the row being separated by a stride distance of x pixels, the method comprising: forming a first data stream of pixel values, wherein the first data stream is formed by reading image pixel values ​​of the primary image row by row; forming a second data stream of kernel values, and synchronizing the pixel values ​​of the first data stream with the kernel values ​​of the second data stream so that each pixel value is paired with a corresponding kernel value of the kernel for a corresponding kernel sampling position.

[0158] In some embodiments, a display device including a holographic projector and an optical relay system, such as a head-up display, is provided. The optical relay system is arranged to form a virtual image reconstructed by each hologram. In some embodiments, the target image includes near-field image content in a first area of ​​the target image and far-field image content in a second area of ​​the target image. The virtual image of the holographically reconstructed near-field content is formed as a first virtual image distance from a viewing plane (e.g., an eye frame), and the virtual image of the holographically reconstructed far-field content is formed as a second virtual image distance from the viewing plane, wherein the second virtual image distance is greater than the first virtual image distance. In some embodiments, one hologram in a plurality of holograms corresponds to the image content of the target image to be displayed to the user in the near field (e.g., speed information), and another hologram in a plurality of holograms corresponds to the image content of the target image to be projected into the far field (e.g., a landmark indicator or a navigation indicator). The refresh rate of the image content in the far field may be higher than that of the image content in the near field, and vice versa.

[0159] System Diagram

[0160] Fig.139 is a schematic diagram illustrating a holographic system according to an embodiment. A spatial light modulator (SLM) 940 is arranged to display a hologram received from a controller 930. In operation, a light source 910 illuminates a hologram displayed on the SLM 940 and forms a holographic reconstruction in a playback field on a playback plane 925. The controller 930 receives one or more images from an image source 920. For example, the image source 920 can be an image capture device, such as a still camera arranged to capture a single still image or a video camera arranged to capture a video sequence of moving images.

[0161] The controller 930 includes an image processing engine 950, a hologram engine 960, a data frame generator 980, and a display engine 990. The image processing engine 950 receives a target image from an image source 920. The image processing engine 950 includes a data flow engine 952, which is arranged to receive the target image and the kernel and form a corresponding synchronized data stream of pixel values ​​and kernel values ​​or weights, as described herein. The image processing engine 950 includes a sub-image generator 955, which is arranged to generate multiple sub-images from a main image based on the target image using the synchronized data stream from the data flow engine 950, as described herein. The image processing engine 950 can receive a control signal or otherwise determine a kernel scheme for generating a sub-image used by the data flow engine 952. Therefore, each sub-image includes fewer pixels than the main image. The image processing engine 950 can use the source image as the main image to generate multiple sub-images. The source image can be an enlarged version of the target image, or the image processing engine can perform the enlargement as described herein. Alternatively, the image processing engine 950 can process the source image to determine an intermediate image, and use the intermediate image as the primary image. As described herein, the image processing engine 950 can generate multiple secondary images by undersampling the primary image. The image processing engine 950 can determine a first secondary image and a second secondary image. The image processing engine 950 passes the multiple secondary images to the hologram engine 960. In some embodiments, the image processing engine 950 can stream pixel values ​​of the secondary images to the hologram engine 960 in real time, as described herein.

[0162] As described herein, the hologram engine 960 is arranged to determine a hologram corresponding to each secondary image. The hologram engine 960 passes a plurality of holograms to a data frame generator 980. The data frame generator 980 is arranged to generate a data frame (e.g., an HDMI frame) including a plurality of holograms, as described herein. Specifically, the data frame generator 980 generates a data frame including hologram data for each of the plurality of holograms and a pointer indicating the beginning of each hologram. The data frame generator 980 passes the data frame to the display engine 990. The data frame generator 980 and the display engine 990 can in turn operate via data streaming. The display engine 990 is arranged to display each of the plurality of holograms on the SLM 940 in turn. The display engine 990 includes a hologram extractor 992, a tiling engine 970, and a software optical device 994. The display engine 990 extracts each hologram from the data frame using a hologram extractor 992, and tiles the hologram according to a tiling scheme generated by a tiling engine 970, as described herein. In particular, the tiling engine 970 can receive a control signal to determine the tiling scheme, or can additionally determine a tiling scheme for tiling based on the hologram. The display engine 990 can optionally use a software optics 994 to add a phase ramp function (a software grating function is also referred to as a software lens) to translate the position of the playback field on the playback plane, as described herein. Thus, for each hologram, the display engine 990 is arranged to output a drive signal to the SLM 940 to sequentially display each of the plurality of holograms, as described herein.

[0163] The controller 930 can dynamically control how the secondary image generator 955 generates the secondary image. The controller 930 can dynamically control the refresh rate of the hologram. The refresh rate can be considered as the frequency at which the hologram engine recalculates the hologram from the next target image in the sequence received by the image processing engine 950 from the image source 920. As described herein, dynamically controllable features and parameters can be determined based on external factors indicated by control signals. The controller 930 can receive control signals related to such external factors, or can include modules for determining such external factors and generating such control signals accordingly.

[0164] As will be appreciated by those skilled in the art, the above-described features of the controller 930 may be implemented in software, firmware or hardware, and any combination thereof.

[0165] Therefore, an image processing engine is provided, which is arranged to generate a secondary image by undersampling a primary image using a kernel having m rows and n columns of kernel values, wherein the kernel has a plurality of kernel sampling positions for each row of the primary image, each kernel sampling position of the row being separated by a stride distance of x pixels, wherein the image processing engine includes a data flow engine, the data flow engine being configured to: form a first data stream of pixel values, wherein the first data stream is formed by reading image pixel values ​​of the primary image row by row; form a second data stream of kernel values, and synchronize the pixel values ​​of the first data stream with the kernel values ​​of the second data stream so that each pixel value is paired with a corresponding kernel value of the kernel for a corresponding kernel sampling position.

[0166] Additional Features

[0167] The embodiments relate to electrically activated LCOS spatial light modulators by way of example only. The teachings of the present disclosure may equally be implemented on any spatial light modulator capable of displaying a computer-generated hologram according to the present disclosure, such as any electrically activated SLM, optically activated SLM, digital micromirror device, or micro-electromechanical device.

[0168] In some embodiments, the light source is a laser such as a laser diode. In some embodiments, the detector is a photodetector, such as a photodiode. In some embodiments, the light receiving surface is a diffuse surface or screen, such as a diffuser. The holographic projection system of the present disclosure can be used to provide an improved head-up display (HUD) or head-mounted display. In some embodiments, a vehicle is provided, which includes a holographic projection system installed in the vehicle to provide a HUD. The vehicle can be a motor vehicle, such as a car, a truck, a van, a delivery truck, a motorcycle, a train, an airplane, a ship, or a ship.

[0169] The examples describe illuminating the SLM with visible light, but one skilled in the art will appreciate that the light source and SLM may equally be used to direct infrared or ultraviolet light, for example, as disclosed herein. For example, a skilled person will be aware of techniques for converting infrared and ultraviolet light to visible light to provide information to a user. For example, the present disclosure extends to the use of phosphors and / or quantum dot technology for this purpose. The data streaming techniques are applicable to all such applications.

[0170] Some embodiments describe 2D holographic reconstructions by way of example only. In other embodiments, the holographic reconstruction is a 3D holographic reconstruction. That is, in some embodiments, each computer-generated hologram forms a 3D holographic reconstruction.

[0171] The methods and processes of data stream transmission described herein can be implemented in hardware to optimize processing speed. However, it will be appreciated by the technician that certain aspects of data stream transmission technology can also be implemented in software. Therefore, various aspects can be embodied in computer-readable media. The term "computer-readable medium" includes media arranged to temporarily or permanently store data, such as random access memory (RAM), read-only memory (ROM), buffer memory, flash memory and cache memory. The term "computer-readable medium" should also be considered to include any medium or a combination of multiple media that can store instructions for execution by a machine, so that when the instruction is executed by one or more processors, the machine is made to perform any one or more methods described herein in whole or in part.

[0172] The term "computer-readable medium" also encompasses cloud-based storage systems. The term "computer-readable medium" includes, but is not limited to, one or more tangible and non-transitory data storage repositories (e.g., data volumes) in the example form of solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof. In some example embodiments, instructions for execution may be conveyed by a carrier medium. Examples of such carrier media include transient media (e.g., propagated signals conveying instructions).

[0173] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope of the appended claims. The present disclosure covers all modifications and variations within the scope of the appended claims and their equivalents.

Claims

1. An image processing engine arranged to generate a secondary image by undersampling a primary image using a kernel having m rows and n columns of kernel values, wherein the kernel has a plurality of kernel sampling positions for each row of the primary image, each kernel sampling position of a row being separated by a stride distance of x pixels, in, The image processing engine includes a data flow engine, and the data flow engine is configured as follows: forming a first data stream of pixel values, wherein the first data stream is formed by reading image pixel values ​​of the primary image row by row, each pixel value corresponding to a row position and a column position within the kernel; forming a second data stream of kernel values, each kernel value corresponding to a row position and a column position within the kernel, and Pixel values ​​of the first data stream are synchronized with kernel values ​​of the second data stream such that each pixel value is paired with a corresponding kernel value corresponding to a same row position and column position within the kernel.

2. The image processing engine according to claim 1, wherein: There is a one-to-many correlation between pixel values ​​of the main image in the first data stream and kernel values ​​of the second data stream.

3. The image processing engine according to claim 1 or 2, wherein: The data flow engine is configured to form the second data flow using the following steps: (s1) repeatedly reads the kernel value of the first row of the kernel; (s2) repeatedly read the kernel value of the next row of the kernel; (s3) iteratively repeating step (s2) (m-2) times; (s4) returning to step (s1), and (s5) When there are no more pixel values ​​in the first data stream, stop steps (s1) to (s4).

4. The image processing engine according to claim 1 or 2, wherein: Each row of kernel values ​​of the kernel in the second data stream is paired with multiple rows of image pixels of the main image in the first data stream.

5. The image processing engine according to claim 1 or 2, further comprising a buffer, wherein: The image processing engine is further configured to: receiving, sequentially from the data stream engine, synchronized pairs of image pixel values ​​and kernel values ​​for the first and second data streams; processing each pixel value of the first data stream with a paired kernel value of the second data stream, and Processed pixel values ​​are accumulated for each kernel sampling location for storage in the buffer.

6. The image processing engine according to claim 5, configured as follows: The first row of pixel values ​​of the main image in the first data stream is processed using the following steps: (a) multiplying each pixel value by its paired kernel value of the second data stream to determine a sequence of corresponding weighted pixel values, (b) summing the n weighted pixel values ​​for each kernel sampling position in the first plurality of kernel sampling positions, (c) determining an accumulated weighted pixel value for each of the first plurality of kernel sampling locations, and (d) storing the accumulated weighted pixel values ​​for each of the first plurality of kernel sampling locations in consecutive storage locations in a buffer so as to form a sequence of partial pixel values ​​of the secondary image in the buffer, and Steps (a) to (d) are iteratively repeated to process the pixel values ​​of each subsequent row of the main image in the first data stream.

7. The image processing engine according to claim 6, configured as follows: iteratively repeating steps (a) to (d) to process pixel values ​​of (m-1) subsequent rows of the primary image in the first data stream to determine an accumulated weighted complete pixel value for each of the first plurality of kernel sampling locations, and For each kernel sampling position in the further plurality of kernel sampling positions, steps (a) to (d) are used to process a subsequent consecutive group of m rows of pixel values ​​of the main image in the first data stream.

8. The image processing engine of claim 6 or 7, further configured to process the pixel values ​​of each row of the main image in the first data stream using the following steps: (e) Feeding back a third data stream from the buffer, the third data stream comprising a sequence of partial pixel values ​​of the sub-image, for use in processing pixel values ​​of a next row of the main image in the first data stream.

9. The image processing engine of claim 8, configured to determine the accumulated weighted pixel value for each kernel sampling position in (c) by determining the sum of: (b) the n weighted pixel values ​​determined for the kernel sampling locations, and The feedback in (e) contains the corresponding partial sub-image pixel values ​​of the third data stream at the kernel sampling position.

10. The image processing engine according to claim 6 or 7, further configured to: A final sub-image pixel value corresponding to each of a plurality of kernel sampling locations is output from the buffer.

11. The image processing engine according to claim 1 or 2, wherein: The kernel moves in a raster scan path with a stride distance of n pixels in the x direction and a stride distance of m pixels in the y direction so that the kernel window subsamples a contiguous array of m×n pixels of the host image.

12. A method for generating a secondary image by undersampling a primary image using a kernel having m rows and n columns of kernel values, wherein the kernel has a plurality of kernel sampling positions for each row of the primary image, each kernel sampling position of a row being separated by a stride distance of x pixels, the method comprising: forming a first data stream of pixel values, wherein the first data stream is formed by reading image pixel values ​​of the primary image row by row, each pixel value corresponding to a row position and a column position within the kernel; forming a second data stream of kernel values, each kernel value corresponding to a row position and a column position within the kernel, and Pixel values ​​of the first data stream are synchronized with kernel values ​​of the second data stream such that each pixel value is paired with a corresponding kernel value corresponding to a same row position and column position within the kernel.

13. The method of claim 12, wherein: Forming the second data stream includes: (s1) repeatedly reads the kernel value of the first row of the kernel; (s2) repeatedly read the kernel value of the next row of the kernel; (s3) iteratively repeating step (s2) (m-2) times; (s4) returning to step (s1), and (s5) When there are no more pixel values ​​in the first data stream, stop steps (s1) to (s4).

14. The method according to claim 12 or 13, further comprising: Each row of kernel values ​​of the kernel in the second data stream is paired with multiple rows of image pixels of the main image in the first data stream.

15. The method of claim 12 or 13, further comprising: receiving synchronized pairs of image pixel values ​​and kernel values ​​of the first and second data streams sequentially from a data stream engine; Processing each pixel value of the first data stream with a paired kernel value of the second data stream; Accumulate the processed pixel values ​​at each kernel sampling location, and Each accumulated value is stored at a corresponding memory location in the buffer.

16. The method of claim 15, further comprising: The first row of pixel values ​​of the main image in the first data stream is processed using the following steps: (a) multiplying each pixel value by its paired kernel value of the second data stream to determine a sequence of corresponding weighted pixel values, (b) summing the n weighted pixel values ​​for each kernel sampling position in the first plurality of kernel sampling positions, (c) determining an accumulated weighted pixel value for each of the first plurality of kernel sampling locations, and (d) storing the accumulated weighted pixel values ​​for each of the first plurality of kernel sampling locations in consecutive storage locations in a buffer so as to form a sequence of partial pixel values ​​of the secondary image in the buffer, and Steps (a) to (d) are iteratively repeated to process the pixel values ​​of each subsequent row of the main image in the first data stream.

17. The method of claim 16, further comprising: iteratively repeating steps (a) to (d) to process pixel values ​​of (m-1) subsequent rows of the primary image in the first data stream to determine an accumulated weighted complete pixel value for each of the first plurality of kernel sampling locations, and For each kernel sampling position in the further plurality of kernel sampling positions, steps (a) to (d) are used to process a subsequent consecutive group of m rows of pixel values ​​of the main image in the first data stream.

18. The method according to claim 16 or 17, wherein: Processing pixel values ​​of each row of the main image in the first data stream comprises: (e) Feeding back a third data stream from the buffer, the third data stream comprising a sequence of partial pixel values ​​of the sub-image, for use in processing pixel values ​​of a next row of the main image in the first data stream.

19. The method of claim 18, wherein: Determining the accumulated weighted pixel value for each kernel sampling location in (c) includes determining the sum of: (b) the n weighted pixel values ​​determined for the kernel sampling locations, and The feedback in (e) contains the corresponding partial sub-image pixel values ​​of the third data stream at the kernel sampling position.

20. The method of claim 16 or 17, further comprising: (e) outputting from the buffer a final sub-image pixel value corresponding to each of the plurality of kernel sampling positions.

Images