Hologram calculation method

Abstract

To provide a hologram engine for calculating a hologram of a target picture comprising a plurality of image points.SOLUTION: The hologram engine is arranged to perform a first accumulation comprising, for a first subset of image points: determining, using a point cloud method, a value for every n-th pixel of a display device for displaying the hologram; and determining a value for at least some of the other pixels by performing a first interpolation; where n is greater than 1. Each image point of the first subset has an associated diffraction angle for use in the point cloud method, each diffraction angle being less than a maximum diffraction angle of the display device.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present invention relates to a hologram engine for calculating a hologram and a method thereof. More specifically, the present invention relates to a hologram engine for calculating a hologram using a point cloud method. Some embodiments relate to a holographic projector, an image generation unit, or a head-up display.

Background Art

[0002] The light scattered from an object contains information on both amplitude and phase. This amplitude and phase information can be captured on a photosensitive plate, for example, by well-known interference techniques, to form a holographic record including interference fringes, that is, a "hologram". The hologram is reconstructed by irradiating appropriate light to form a two-dimensional or three-dimensional holographic reconstruction representing the original object, that is, a reproduced image.

[0003] Computer-generated holography numerically simulates the interference process. Computer-generated holograms can be calculated by methods based on mathematical transforms such as Fresnel transform and Fourier transform. These types of holograms are sometimes called Fresnel / Fourier transform holograms, or simply Fresnel / Fourier holograms. Fourier holograms can be regarded as the Fourier domain / plane representation of an object, or the frequency domain / plane representation of an object. Computer-generated holograms can also be calculated, for example, by coherent ray tracing or the point cloud method.

[0004] Computer-generated holograms may be encoded on a spatial light modulator arranged to modulate the amplitude and / or phase of the incident light. Optical modulation can be realized, for example, using electrically addressable liquid crystals, optically addressable liquid crystals, or micromirrors.

[0005] A spatial light modulator typically consists of a plurality of individually addressable pixels, often also referred to as cells or elements. The light modulation method can be binary, multi-level, or continuous. Alternatively, the device may be continuous (i.e., not composed of pixels), in which case the light modulation is continuous across the entire device. The spatial light modulator may be reflective, in which case the modulated light is reflected and output. The spatial light modulator may similarly be transmissive, in which case the modulated light is transmitted and output.

[0006] Using the system described herein, a holographic projector can be provided. Such projectors have applications in head-up displays (HUDs).

SUMMARY OF THE INVENTION

[0007] Aspects of the present disclosure are defined in the appended independent claims.

[0008] Generally speaking, a hologram engine is provided that calculates the hologram of a target picture in a computationally efficient way, such that the hologram engine can achieve substantially real-time hologram calculation while having relatively low-cost and low-power hardware. In an embodiment, the hologram engine is configured to calculate the value of the hologram using a point cloud method. However, rather than calculating the values of all pixels of the hologram using the point cloud method as in the prior art, the hologram engine is configured to undersample the hologram during the point cloud stage of the calculation. That is, the hologram engine is initially configured to calculate only the hologram values of a subset of the pixels of the hologram. For example, the hologram engine can be initially configured to calculate only the hologram values every nth pixel, such as every second, third, fourth, or fifth pixel. Thus, each pixel (using the point cloud method) for which the hologram value is initially calculated may be separated by one or more intermediate pixels for which the hologram value is not initially calculated. Next, the hologram engine according to the present invention is configured to determine the values of at least some of the other pixels (i.e., at least some of the intermediate pixels) by performing interpolation. The inventors have discovered that interpolation is much lower in computational cost than the first hologram calculation (using the point cloud method). Thus, undersampling and interpolation significantly improve the computational speed and efficiency of hologram calculation (compared to calculating the hologram at full resolution using the point cloud method). The memory requirements of the hologram engine are also significantly reduced. As will be understood by those skilled in the art, it is generally desirable to reduce the computational cost of hologram calculation (or improve the computational efficiency). This is particularly important for achieving real-time hologram calculation with high-quality image resolution. In the context of holographic projectors such as vehicle head-up displays, it is particularly advantageous that this real-time hologram calculation can be achieved with as little computational power as possible. The hologram calculation method according to the present disclosure contributes to the realization of this real-time holography.

[0009] The inventor recognized that the undersampling and interpolation approach is only possible when the spatial frequency of the undersampled hologram is relatively low. The inventor recognized that undersampling and interpolation are impossible when the spatial frequency is high. This is because the phase / complex amplitude values in the display device can change very rapidly, so that the interpolation becomes (very) inaccurate and (severe) aliasing problems may occur. Therefore, the hologram engine of the present invention is configured such that the point cloud method is applied to the image points and the spatial frequency of the hologram calculated by the point cloud method becomes relatively low. In particular, the hologram engine of the present invention is configured such that the spatial frequency of the resulting hologram (calculated using the point cloud method) is lower than the maximum spatial frequency that can be displayed on the display device. This will be described in more detail herein.

[0010] Conventionally, the calculation of a point cloud hologram may involve virtually propagating waves from each image point to the center of the viewer's entrance pupil (within the display window of the optical system constituting the display device). In other words, conventionally, the waves may propagate along the ray path defined between the image point and the center of the entrance pupil. The calculation of the hologram is based on the characteristics (contributions) of the waves propagating along this ray path in the display device. The propagated waves intersect the display device at an angle corresponding to the position of the center of the entrance pupil and each image point. When the calculated hologram is appropriately illuminated, a holographic wavefront is formed, reconstructing the simulated / propagated waves so that each image point is holographically reconstructed and propagated to be displayed at the intended location (defined in the original target picture). In other words, when the propagated waves intersect the display device at a specific angle, as long as that angle is within the field of view of the display device, the holographically reconstructed points are displayed at positions depending on that angle within the field of view of the system including the display device. The field of view is typically determined by (at least) a) the wavelength of the light used to illuminate the display device, and b) the pitch of the pixels of the display device (i.e., the distance between the centers of adjacent pixels). The smaller the pitch, the wider the field of view. The display device may only be able to holographically reconstruct image points within the angular range of its field of view.

[0011] After thorough simulations and experiments, the inventors discovered that the spatial frequency of the hologram of the image points calculated using the conventional point group method generally depends on the angle formed between the optical path along which the wave propagates and the display device. The spatial frequency may be maximized when the ray / propagation path makes an angle equal to the maximum diffraction angle of the display device with respect to the display device. The spatial frequency may be minimized when the ray / propagation path makes an angle equal to zero with respect to the display device (i.e., when the propagation path is substantially perpendicular to the display device). The spatial frequency may refer to the frequency of the values (e.g., complex amplitude) in the display device. In the case of a hologram of only phase, the hologram of points with a relatively high spatial frequency may have a rapidly changing phase. For example, the phase of adjacent pixels of the display device for displaying the hologram may change by nearly π. The hologram of image points with a relatively low spatial frequency may have a relatively slowly changing phase. For example, the phase of adjacent pixels of the display device for displaying the hologram may change by only a relatively small fraction of π. Importantly, the inventors recognize that when the spatial frequency of the hologram is low, the initial hologram calculation can be performed at a low resolution. For example, the complex amplitude / phase of the display device may be sampled for a part of the pixels of the display device. The inventors recognize that when the spatial frequency is relatively low, the values of the (intermediate) pixels between the sampled pixels may be filled using interpolation. The inventors recognize that when the spatial frequency is relatively high, undersampling and interpolation are not possible because the phase / complex amplitude values of the display device may change very rapidly, making interpolation (very) inaccurate and causing a (significant) aliasing problem. Therefore, the hologram engine of the present invention is configured such that the point group method is applied to the image points so that the hologram calculated by the point group method has a relatively low spatial frequency.

[0012] The inventors recognized that if image points with appropriate associated diffraction angles are selected, the above-described undersampling and interpolation hologram calculations can be used. In particular, the inventors discovered that undersampling and interpolation can be applied to image points having an associated diffraction angle smaller than the maximum diffraction angle of the display device. Here, the associated diffraction refers to the angle used in the point group method. In other words, the associated diffraction angle refers to the angle created by the propagation path used in the point group method and along which the wave propagates.

[0013] In some embodiments, the hologram engine is configured to selectively apply an undersampling and interpolation approach to image points having an (initial) position within a target picture such that the diffraction angle of the image point is smaller than the maximum diffraction angle of the display device. In some embodiments, the hologram engine may be configured to process the image points so as to reduce the diffraction angle. This means that the image points are translated or moved such that the propagation path / ray path from the translated image points forms an angle smaller than the maximum diffraction angle of the display device with the display device. Once the hologram / complex value of the translated image points is determined (by undersampling and then interpolating as described above), the image points can be returned to their original intended positions using one or more lattice functions. The one or more lattice functions may actually be high spatial frequency functions added to the low spatial frequency hologram.

[0014] In some embodiments, the hologram engine may be configured such that the amount or degree of undersampling depends on the diffraction angle associated with the image point for which the hologram is being calculated. For example, in general terms, undersampling is defined as determining a value for every nth pixel of the display device that displays the hologram, where n is an integer greater than 1. In some embodiments, n may increase as the diffraction angle of the image point for which the hologram value is being calculated decreases. For example, as the diffraction angle approaches zero (or is equal to zero), n may be the highest.

[0015] The hologram engine according to the present invention is optional and has additional features that contribute to increasing the calculation efficiency and reducing the memory usage when executing the hologram calculation method.

[0016] For example, during the first data processing stage, also called the first accumulation, the hologram engine may be configured to determine the individual value of the n-th pixel for each pixel of the first subset of image points of the target picture. Thus, there may be as many values for each of the n-th pixels as there are image points in the first subset. The hologram engine may further be configured to combine (e.g., sum) these values at each of the n-th pixels. This may be performed before the first interpolation. The hologram engine may be configured to perform the first interpolation after the step of determining the combined values. In other words, the interpolation may be performed on the combined values at each pixel. The inventors recognized that by performing the first interpolation after the combination step, even if the point cloud method is applied to multiple image points, it may be necessary to perform the first interpolation only once during one iteration of the first accumulation. This has the advantage that, for example, compared to performing interpolation for each image point within the subset, the calculation time of the first accumulation is shortened. The inventors recognized that this effect is obtained because the sum of a finite number of slowly varying functions is generally another slowly varying function. Thus, a finite combination of values of a hologram with a low spatial frequency (i.e., having slowly varying values) results in another hologram with a low spatial frequency. Thus, the inventors recognized that interpolation can be performed without aliasing problems even after the combination step. The inventors have developed a more specific implementation of the hologram engine that includes a second accumulation and, optionally, a third accumulation.

[0017] In another example, the inventors recognized that the diffraction gratings may be important to apply after each interpolation. More specifically, the first interpolation can be performed in the first dimension / direction (filling in intermediate pixels that separate the nth pixel in the first dimension). The inventors recognized that when applying the first grating function to convert light in the first dimension, this should be done after the first interpolation. This is because the first grating function may be a relatively high-frequency function and thus should be applied after the first interpolation. Similarly, when applying the second grating function to convert light in the second dimension perpendicular to the first dimension, this should be done after the second interpolation in the second dimension.

[0018] In some examples, it may be desirable to apply different grating functions to different subsets of image points. The particular implementations of the first through third accumulators / accumulations described herein enable this in a computationally efficient manner.

[0019] In a first aspect of the present disclosure, a hologram engine is provided for calculating a hologram of a target picture including a plurality of image points. The hologram is for display on a display device including a plurality of pixels such as a pixel array. In other words, the hologram calculated by the hologram can include a plurality of pixel values including each hologram pixel value to be displayed on the pixels of the display device. In some embodiments, the pixel array of the display device is extended in at least a first dimension. In some embodiments, the pixel array of the display device is also extended in a second dimension. The pixel array may be a regular array.

[0020] The hologram engine is configured to perform a first processing stage, which can be referred to herein as the first accumulation. This means that the calculation of the hologram of the target picture includes the execution of the first accumulation by the hologram engine. The first accumulation includes determining values only for every nth pixel of the display device (e.g., the above-mentioned display device) using the point cloud method (n is an integer greater than 1). Therefore, it can be explained that the hologram is undersampled during the calculation of the point cloud method. This is because when n is greater than 1, values can be determined only for every other pixel of the display device. The values of the other / remaining pixels cannot be determined using the point cloud method.

[0021] The hologram engine is further configured to determine the values of at least some of the other pixels by performing a first interpolation. Here, the "other" pixels refer to the pixels whose values are not determined by the point cloud method. These pixels are also referred to as "intermediate" pixels throughout the present disclosure. The first interpolation can be based on the values determined using the point cloud method. In other words, the interpolation can include receiving the values of the nth pixels (determined using the point cloud method) and interpolating those values to determine the values of at least some of the intermediate or "other" pixels. As described above, this undersampling and interpolation can make the calculation of the hologram more efficient. For example, at least twice (or up to five times) the efficiency can be achieved compared to the case of determining the value of each pixel using the point cloud method.

[0022] The hologram engine is further arranged such that each image point of the first subset has a relevant diffraction angle smaller than the maximum diffraction angle of the display device for use in the point cloud method. As described above, this is to make the hologram of the image point have a spatial frequency lower than the maximum spatial frequency that can be displayed on the display device. This means that undersampling and interpolation can be performed without causing (significant) aliasing problems.

[0023] The inventors have discovered that the closer the diffraction angle is to zero, the larger the range within which the hologram can be undersampled when applying the point group method and then interpolating without significant aliasing problems. Therefore, the improvement in the efficiency of calculating the hologram according to the present disclosure may increase as the diffraction angle decreases. For example, as the diffraction angle decreases, n may increase. For example, as the diffraction angle associated with the image point approaches zero (e.g., the image point is substantially perpendicular to the display device), n may be as high as 3 or more, optionally 5 or more. This may mean that the hologram engine is configured to use the point group method to determine the values of every third or fifth pixel, respectively. n may decrease as the diffraction angle increases. For example, if the diffraction angle of the image point is equal to half of the maximum diffraction angle of the display, n may be equal to 2.

[0024] The hologram engine according to the first aspect has been described above in relation to a single image point. However, the first accumulation is typically applied to a plurality of image points (e.g., a first subset of image points). For example, the first accumulation may include, for a first subset of image points, using the point group method to determine a value for every nth pixel of a display device for displaying the hologram (where n is greater than 1), and determining the values of at least some of the other pixels by performing a first interpolation. Each image point in the first subset is associated with a diffraction angle for use in the point group method, and each diffraction angle is less than the maximum diffraction angle of the display device.

[0025] As used herein, the values determined by the hologram engine may be referred to as "complex values" or "hologram values". The values may include amplitude and / or phase information / values. The values may also be referred to as the gray levels assigned to each pixel. As described above, some of these values (for each n-th value) are determined using the point cloud method. One skilled in the art will be familiar with the point cloud method. Generally speaking, a hologram engine configured to execute the point cloud method virtually / computationally propagates waves (also referred to herein as "wavelets") associated with each image point along a propagation path, and then determines the contribution of the propagated waves at the display device. Based on the determined / simulated contribution of each image point at the display device, a hologram can be calculated. For example, an array of (complex) values of the wave at the intersection of the wave and the display device can be determined. Thereby, a hologram is calculated, and when appropriately illuminated, it spatially modulates the incident light to form a holographic wavefront that can reconstruct each image point. In the calculation of a phase-only hologram, an array of phase values of the pixels of the display device can be determined. Conventionally, in the point cloud method, values are calculated for all pixels of the hologram / display device. The hologram engine according to the present disclosure only calculates values for every n-th pixel of the display device (i.e., undersamples the hologram).

[0026] As used herein, accumulation refers to an iterative process as part of a hologram calculation in which multiple values are calculated and combined. For example, the first accumulation is iterated over the image points of the first subset. In particular, for each image point of the image points of the first subset, the first accumulation includes calculating a value for each of the nth pixels of the display device. The value calculated in a particular iteration is combined (e.g., added) with the value calculated in the previous iteration. Thus, at the end of the last iteration, the combined value for each of the nth pixels of the display device is output. Each combined value (for each pixel) represents the combination / sum of all the values determined at each pixel of the display device for each image point of the first subset. The hologram may be configured to perform further accumulations (e.g., second and third accumulations), as described in more detail below. The terms accumulation and data processing stage may be used interchangeably. In particular, the first accumulation may be referred to as the first data processing stage. The second accumulation may also be referred to as the second data processing stage, and the third accumulation may also be referred to as the third data processing stage.

[0027] As used herein, the diffraction angle associated with each image point refers to the angle formed by the propagation path used in the point group method for that image point with the display device. In particular, the angle may be the angle between the propagation path and the normal of the display device. As described above, the point group method includes determining the contribution of waves propagating along the propagation path. The determined values (for each of the nth pixels) each represent the contribution of the wave propagating at that pixel. The diffraction angle is less than the maximum diffraction angle of the display device. In some embodiments, the propagation path used in the point group method follows the ray path from the image point. In such embodiments, a diffraction angle less than the maximum diffraction angle may be achieved by selectively applying the method only to a first subset of pixels arranged at such an angle with the display device. In some embodiments, the hologram engine is configured to translate or move the starting point of the propagation path to reduce the angle (to be less than or significantly less than the maximum angle). This has the effect of moving the image point. As explained, when the angle is less than the maximum diffraction angle, the spatial frequency of the hologram / determined value is relatively low, enabling advantageous undersampling and interpolation with significant aliasing problems. In some embodiments, the diffraction angle is less than half of the maximum diffraction angle of the display device, optionally less than a quarter of the maximum diffraction angle of the display device, and optionally substantially equal to zero.

[0028] The maximum diffraction angle of the display device depends on the pitch of the pixels of the display device. In some embodiments, the maximum diffraction angle may be 2 degrees or more, optionally 3 degrees or more. As described above, the maximum diffraction angle may be defined with respect to the normal of the display device. This means that a display device with a maximum diffraction angle of 2 degrees has a 4-degree field of view. In some embodiments, the diffraction angle associated with the first subset of image points may be 50% or less of the maximum diffraction angle. For example, the diffraction angle associated with the first subset of image points may be 0.5 degrees or less.

[0029] As used herein, the maximum diffraction angle of a display device refers to the maximum angle at which light can be diffracted or deviate from its original path after interacting with the display device. Thus, the maximum diffraction angle of a display device can define the field of view of the system and the size of the reproduction field. The display device can reconstruct image points only within the range of the maximum diffraction angle. The maximum diffraction angle depends on the wavelength of the incident light (incident on the display device) and the pixel pitch of the display device. As the pixel pitch decreases, the maximum diffraction angle increases. In embodiments, since the pixel pitch of the display device is equal in both the first and second dimensions, the maximum diffraction angles in the first and second dimensions may be substantially equal. Throughout the present disclosure, the maximum diffraction angle is defined with respect to the normal of the display device. Thus, light can be emitted from the display device over a range of angles equal to ± the maximum diffraction angle.

[0030] In some embodiments, the display device is a spatial light modulator. In some embodiments, the display device is a liquid crystal on silicon spatial light modulator.

[0031] In some embodiments, the hologram engine is configured such that determining the value of each nth pixel includes determining the value of the nth pixel of the display device using a point cloud method for each image point in the first subset, and combining the values determined for each of the nth pixels. In such embodiments, the first interpolation is based on the combined values determined for each nth pixel. Thus, the first interpolation may be performed after the step of combining the values determined for each of the nth pixels. As described above, it may be advantageous to perform the first interpolation after the (plural) values determined for each (hologram) pixel (one value for each image pixel in the first subset) are combined. This may be because the interpolation only needs to be performed once for a plurality of image points. The inventors have recognized that this is possible because the hologram value of each image point is a low spatial frequency and thus a slowly varying function. A combination of a finite number of slowly varying functions may be another slowly varying function. Thus, the combined values may also represent a slowly varying function with a low spatial frequency, making them suitable for performing interpolation without the risk of aliasing problems.

[0032] In some embodiments, the value determined using the point cloud method for each image point represents the contribution of a wave propagating along a path that forms an angle equal to the diffraction angle associated with each image point with respect to the display device. In other words, the propagation path forms an angle smaller than the maximum diffraction angle of the display device with respect to the display device.

[0033] In some embodiments, the diffraction angle associated with each image point of the first subset is defined by a first path from the image point to the viewer's entrance pupil. This may be the case when a conventional point cloud method is used without preprocessing the image points. In such embodiments, the image points of the first subset may be selected such that the first path forms an angle smaller than the maximum diffraction angle of the display device. In other words, the image points of the first subset may not be substantially located at the edge of the reproduction field / viewing field. The image points of the first subset may be, for example, closer to the center of the target picture than to the edge of the target picture.

[0034] In some embodiments, the diffraction angle associated with each image point of the first subset is defined by a second path arranged as if each image point had been moved closer to the center of the target picture within the target picture. In such embodiments, some preprocessing may be performed on the target picture to move the image points closer to the center of the target picture. This can be done such that the second path forms a reduced angle with respect to the angle formed by the respective first path for that image point (and an angle smaller than the maximum diffraction angle of the display device) with respect to the display device.

[0035] In some embodiments, the maximum diffraction angle of the display device defines a reproduction field that includes a central portion surrounded by an outer portion. The central portion may include the center of the reproduction field. The center of the reproduction field may be concentric with the center of the central portion. In some embodiments, the central portion occupies less than 75% of the reproduction field, optionally less than 50% of the reproduction field, and optionally less than 25% of the reproduction field. The central portion represents the portion of the reproduction field where the hologram of any point in the central portion has a sufficiently low spatial frequency, and the hologram values can first be undersampled and then interpolation can be applied without causing significant aliasing problems.

[0036] In some embodiments, the first subset of image points is included only in the central portion of the reproduction field. In such embodiments, the diffraction angle associated with each image point of the first subset is defined by a first path from the image point to the viewer's entrance pupil. In other words, the image points in the central portion of the reproduction field may not require preprocessing to achieve undersampling and interpolation approaches without significant aliasing problems.

[0037] In some embodiments, the dot pattern method is configured as if the first subset of image points has been moved from the outer portion to the central portion of the reproduction surface. In such embodiments, the diffraction angle associated with each image point of the first subset is defined by a path according to the aforementioned second path. In other words, the hologram engine may be configured to preprocess the first subset of image points to translate or move them from the outer portion to the central portion (i.e., closer to the center of the reproduction field). Thereby, the angle formed by the second path and the display device decreases compared to the angle formed when the image point is in its original position. Thereby, the spatial frequency of the hologram calculated using the dot pattern method may decrease accordingly.

[0038] In some embodiments, the hologram engine is arranged such that at least some of the pixels whose values are determined using the dot pattern method are separated in a first dimension by one or more first intermediate pixels. In such embodiments, the display device can include an array of pixels extending at least in the first dimension. In some embodiments, a first interpolation is performed in the first dimension to determine the value of the first intermediate pixel.

[0039] In some embodiments, the first accumulation further includes applying a first lattice function arranged to convert incident light in a first direction parallel to the first dimension. The hologram engine may be configured to apply the lattice function after the first accumulation has performed a first interpolation. In some embodiments, the hologram engine is configured such that the first lattice function is applied to the nth pixel (of the hologram) and a first intermediate pixel.

[0040] In some embodiments, the hologram is for a display device including an array of pixels extending in a first dimension and a second dimension different from the first dimension. In some embodiments, the first dimension is perpendicular to the second dimension. In some embodiments, the hologram engine is arranged such that at least some of the pixels whose values are determined using the point group method are separated by one or more second intermediate pixels in the second dimension. In some embodiments, the hologram engine is arranged to determine the values of the second intermediate pixels by performing a second interpolation in the second dimension, and the hologram engine is arranged to perform the second interpolation after the first interpolation.

[0041] In some embodiments, the hologram engine is configured to perform a second accumulation. The second accumulation may include receiving the value determined by the first accumulation. If the first accumulation includes combining values at each pixel, the combined value may be received.

[0042] In some embodiments, a hologram engine configured to perform a second accumulation includes repeating a first accumulation for a subset of k image points of a target picture. The second accumulation can further include receiving respective values from the first accumulation for each of the subset of k image points. In some embodiments, each of the subset of k image points is within a discrete angular range defined by a first angle in a first dimension and a second angle in a second dimension. The second angle may be the same for each of the k subsets. In other words, each of the subset of k image points can together form a row or a column. Similar to the first subset of image points, each image point of the k subsets has an associated diffraction angle for use in a point cloud method, and each diffraction angle is less than the maximum diffraction angle of the display device.

[0043] In some embodiments, the hologram engine is further configured to combine the determined value for each of the nth pixels with the value determined for the first intermediate pixel during the second accumulation (if any).

[0044] In some embodiments, the second accumulation includes a second interpolation. The hologram engine may be configured such that the second interpolation is performed after the step of combining the values determined during the second accumulation. This advantage is similar to that described for the first accumulation. In particular, this configuration has the advantage that the second interpolation only needs to be performed once, even though there are k subsets of image points.

[0045] The second accumulation may further include applying a second grating function arranged to convert incident light in a second direction parallel to the second dimension. The hologram engine may be configured such that the second accumulation includes applying the second grating function after performing the second interpolation. The hologram engine can be configured such that the second grating function is applied to the nth pixel (of the hologram) and the first and second intermediate pixels.

[0046] In some embodiments, the hologram engine is configured to perform a third accumulation. The third accumulation includes receiving the value determined in the second accumulation. This can be performed after the steps of combining and interpolation.

[0047] In some embodiments, the third accumulation includes repeating the second accumulation for a subset of the image points of a target picture and receiving each respective value. The third accumulation further includes that for each of a subset of the image points corresponding to a discrete angular range defined by a first angle of a first dimension and a second angle of a second dimension, the second angle is different each time the second accumulation is repeated. Similar to the first subset of image points, each image point of a subset is associated with a diffraction angle for use in the point cloud method, and each diffraction angle is less than the maximum diffraction angle of the display device.

[0048] In some embodiments, the hologram engine is configured to divide the target picture into a plurality of subsets of image points by dividing the target picture into a plurality of discrete angular ranges of a first dimension and a second dimension.

[0049] In some embodiments, the hologram engine is configured to divide the target picture into an array of cells. Each cell may contain a subset of image points. For example, each cell may be defined by an angular range of a first dimension and a second dimension. The array of cells may extend in a first dimension and a second dimension. In some embodiments, the first accumulation is first performed on the image points of the first cell of the array of cells. The second accumulation may include repeating the first accumulation for each cell of a column or row of cells of the array of cells including the first cell. The third accumulation may include repeating the first accumulation and the second accumulation for the other of the rows or columns of each cell of the array of cells.

[0050] In a first accumulation, it may be necessary to store data in a memory that is relatively smaller than the memory required for a third accumulation (and a second accumulation). However, the memory used in the first accumulation may need to be accessed relatively frequently (for example, it may need to be accessed more frequently than the memory used in the third accumulation). Therefore, the inventors have discovered that it may be beneficial for the hologram engine to use different memories for different accumulations. For example, the first and / or second accumulations are implemented using registers, and the third accumulation is implemented using VRAM.

[0051] In a second aspect, a hologram engine is provided for calculating a hologram of a target picture including a plurality of image points. The hologram is displayed on a display device including an array of pixels extending in at least a first dimension. The display device has a maximum diffraction angle defined with respect to the normal of the display device. This may refer to the maximum diffraction angle in the first dimension. The hologram engine is configured to perform a first accumulation. For each image point in a first subset of the image points of the target picture, the value is determined only at a first subset of the pixels of the display device. In other words, the hologram is undersampled. The value represents the contribution of a wave propagating along a path intersecting the display device at an angle smaller than the maximum diffraction angle, optionally smaller than half of the maximum diffraction angle.

[0052] At least some of the pixels of the first subset are separated in a first dimension by one or more first intermediate pixels. The first intermediate pixels are complex values of a propagation wave that are initially undetermined. The first accumulation also includes, for each pixel of the first subset, combining the values determined at that pixel for each image point (if present). Combining means overlaying the values, summing them. If the values are complex values including an imaginary part and a real part, these parts can be combined separately. The first accumulation also includes determining the values of the first intermediate pixels by performing a first interpolation in the first dimension based on the combined values of the first accumulation (the first subset of pixels).

[0053] In a third aspect, a method for calculating a hologram of a target picture including a plurality of image points is provided. This method includes performing a first accumulation. The first accumulation includes, for a first subset of image points, using a point cloud method to determine a value for each nth pixel of a display device for displaying the hologram. Here, n is greater than 1. The first accumulation further includes determining values for at least some of the other pixels (for which values are not determined by the point cloud method) by performing a first interpolation (based on the hologram values determined using the point cloud method). Each image point of the first subset is associated with a diffraction angle for use in the point cloud method, and each diffraction angle is less than the maximum diffraction angle of the display device (defined with respect to the normal of the display device) (optionally, less than half of the maximum diffraction angle of the display device).

[0054] This method is such that the step of determining the value of each nth pixel includes, for each image point of the first subset, determining the value of the nth pixel of the display device using the point cloud method and combining the values determined at each of the nth pixels.

[0055] In some embodiments, the first interpolation is based on the combined values determined for each nth pixel.

[0056] In some embodiments, the value determined using the point cloud method for each image point is a value representing the contribution of a wave propagating along a path that forms an angle equal to the diffraction angle associated with each image point (i.e., an angle smaller than the maximum diffraction angle of the display device) with respect to the display device.

[0057] In some embodiments, the diffraction angle associated with each image point in the first subset is defined by a first path from the image point to the viewer's entrance pupil, or by a second path arranged as if each image point were moved within the target picture towards the center of the target picture, and the angle formed by the second path with the display device is smaller than the angle formed by each first path with respect to that image point (smaller than the maximum diffraction angle of the display device).

[0058] In some embodiments, the maximum diffraction angle of the display device defines a reproduction field that includes a central portion surrounded by an outer portion. The central portion can include the center of the reproduction field. The center of the reproduction field can be concentric with the center of the central portion. The central portion can occupy less than 75% of the reproduction field, optionally less than 50% of the reproduction field, and optionally less than 25% of the reproduction field.

[0059] In some embodiments, the first subset of image points may be included in the central portion of the reproduction field. In some embodiments, the point cloud method is configured as if the first subset of image points were moved from the outer portion to the central portion of the reproduction plane.

[0060] In some embodiments, the first accumulation is performed such that at least a portion of the pixels whose values are determined using the point cloud method are separated in a first dimension by one or more first intermediate pixels (where the display device comprises an array of pixels extending at least in the first dimension). The step of performing the first interpolation can be performed in the first dimension to determine the values of the first intermediate pixels.

[0061] In some embodiments, the first accumulation further includes applying a first lattice function arranged to convert incident light in a first direction parallel to the first dimension. In some embodiments, the method includes applying the lattice function after performing a first interpolation.

[0062] In some embodiments, the hologram engine is arranged such that the first lattice function is applied to the n-th pixel and the first intermediate pixel (of the hologram).

[0063] In some embodiments, the hologram is for a display device including an array of pixels extending in a first dimension and a second dimension (optionally perpendicular) different from the first dimension. In some embodiments, the method is such that at least some of the pixels whose values are determined using the point group method are separated by one or more second intermediate pixels in the second dimension. In some embodiments, the method further includes determining the values of the second intermediate pixels by performing a second interpolation in the second dimension, and the hologram engine is configured to perform the second interpolation after the first interpolation.

[0064] In some embodiments, the method further includes performing a second accumulation. This may be after performing the first accumulation (the first time). The second accumulation may include receiving the values determined in the first accumulation. This may be after the combining and / or interpolation steps. The second accumulation may further include repeating the first accumulation for m subsets of the image points of the target picture and receiving the respective (complex) values. Each of the m subsets of the image points may correspond to a discrete angular range defined by a first angle in the first dimension and a second angle in the second dimension. The second angle may be the same for each of the m subsets, and the m subsets may together form a column or row extending in the first dimension. The method may further include combining the values determined for each of the n-th pixel and the first intermediate pixel during the second accumulation. The second accumulation may include a second interpolation.

[0065] This method may perform a second interpolation after the step of combining the values determined during the second accumulation. The second accumulation may further include applying a second grating function configured to convert incident light in a second direction parallel to the second dimension.

[0066] In some embodiments, the method further includes performing a third accumulation. The third accumulation may include receiving the values determined by the second accumulation. This may be after the combining step of the second accumulation and / or after the second interpolation.

[0067] The third accumulation may include repeating the second accumulation for one subset of the image points of the target picture. The third accumulation may include receiving respective values for each of the one subset. Each of the one subset of image points may correspond to image points included in a discrete angular range defined by a first angle of the first dimension and a second angle of the second dimension. The second angle is different each time the second accumulation is repeated.

[0068] This method may include dividing the target picture into a plurality of subsets of image points by dividing the target picture into a plurality of discrete angular ranges of the first dimension and the second dimension.

[0069] In some embodiments, this method may include dividing the target picture into an array of cells. Each cell may include a subset of the target picture. The array of cells may include cells extending in the first dimension and the second dimension. This method may include first performing a first accumulation on the image points of the first cell of the array of cells. The second accumulation may include repeating the first accumulation for each cell of a column or row of cells of the array of cells including the first cell. The third accumulation may include repeating the first accumulation and the second accumulation for each other cell of a row or column of cells of the array of cells.

[0070] In the present disclosure, the term "replica" is only used to reflect that spatially modulated light is split and the composite light field is directed along a plurality of different optical paths. The word "replica" is used to refer to each occurrence or instance of the composite light field after a replication event such as partial reflective transmission by a pupil expander. Each replica moves along a different optical path. Some embodiments of the present disclosure relate to the propagation of light encoded by a hologram rather than an image, that is, to spatially modulated light by a hologram of an image rather than the image itself. Thus, it can be said that a plurality of replicas of the hologram are formed. Those skilled in the art of holography will understand that the composite light field associated with the propagation of light encoded by a hologram changes with the propagation distance. The use of the term "replica" here is independent of the propagation distance, and thus two optical branches or paths associated with a replication event are still called each other's "replicas" even if the lengths of the branches are different. As a result, the composite light field evolves in different forms along each path. That is, two composite light fields are still considered "replicas" according to the present disclosure even if they are associated with different propagation distances, provided that they are generated from the same replication event or a series of replication events.

[0071] The "diffraction light field" or "diffracted light field" according to the present disclosure is a light field formed by diffraction. The diffraction light field can be formed by irradiating the corresponding diffraction pattern. According to the present disclosure, an example of the diffraction pattern is a hologram, and an example of the diffraction light field is a light field that forms a holographic reconstruction of a holographic light field or an image. The holographic light field forms a (holographic) reconstruction of the image on the reproduction plane. It can be said that the holographic light field propagating from the hologram to the reproduction plane includes the light encoded by the hologram or the light within the hologram region. The diffraction light field is characterized by the diffraction angle determined by the minimum feature size of the diffraction structure and the wavelength of the light (of the diffraction light field). According to the present disclosure, it can also be said that the "diffraction light field" is a light field that forms a reconstruction on a plane spatially separated from the corresponding diffraction structure. In this specification, an optical system for propagating the diffraction light field from the diffraction structure to the observer is disclosed. The diffraction light field can form an image. The term "hologram" is used to refer to a record containing amplitude information or phase information about an object, or a combination thereof. The term "holographic reconstruction" is used to refer to the optical reconstruction of an object formed by irradiating a hologram. The system disclosed herein is described as a "holographic projector". This is because the holographic reconstruction is a real image and is spatially separated from the hologram. The term "replay field" is used to refer to a 2D region where the holographic reconstruction is formed and is fully focused. When a hologram is displayed on a spatial light modulator containing pixels, the replay field is repeated in the form of multiple diffraction orders, and each diffraction order is a replica of the zero-order replay field. Since the zero-order replay field is the brightest replay field, it generally corresponds to the preferred or primary replay field. Unless explicitly stated otherwise, the term "replay field" is interpreted to refer to the zero-order replay field. The term "replay plane" is used to refer to the plane of space containing all replay fields. The terms "image", "replay image", and "image region" refer to the region of the replay field illuminated by the light of the holographic reconstruction. In some embodiments, an "image" is composed of individual spots called "image spots" or, for convenience, "image pixels".

[0072] The terms "encode", "write", or "address" are used to describe a process of providing a plurality of control values that respectively determine the modulation levels of the plurality of pixels of the SLM. It can be said that the pixels of the SLM are configured to "display" a light modulation distribution in response to receiving the plurality of control values. Thus, the SLM is said to "display" a hologram, and the hologram can be regarded as an array of light modulation values or levels. It has been found that acceptable quality holographic reconstruction can be formed from a "hologram" that contains only the phase information related to the Fourier transform of the original object. Such a holographic record may be referred to as a phase-only hologram. Embodiments relate to phase-only holograms, but the present disclosure is equally applicable to amplitude-only holography.

[0073] The present disclosure is equally applicable to forming holographic reconstruction using the amplitude and phase information related to the Fourier transform of the original object. In some embodiments, this is achieved by complex modulation using a so-called full complex hologram that contains both the amplitude and phase information related to the original object. Such a hologram may be referred to as a full complex hologram because the value (gray level) assigned to each pixel of the hologram has both amplitude and phase components. The value (gray level) assigned to each pixel can be represented as a complex number having both amplitude and phase components. In some embodiments, a full complex computer-generated hologram is calculated.

[0074] The phase of a pixel of a computer-generated hologram or a spatial light modulator, referred to as a phase value, phase component, phase information, or simply phase, may be referred to as an abbreviation for "phase delay". That is, the phase value being described is actually a numerical value (for example, in the range from 0 to 2π) representing the amount of phase delay provided by that pixel. For example, a pixel of a spatial light modulator described as having a phase value of π / 2 delays the phase of the received light by π / 2 radians. In some embodiments, each pixel of the spatial light modulator is operable with one of a plurality of possible modulation values (for example, phase delay values). The term "gray level" may be used to refer to a plurality of available modulation levels. For example, the term "gray level" may be used to refer to a plurality of phase levels available in a phase-only modulator, even if different phase levels do not provide different shades of gray. The term "gray level" may also be used to refer to a plurality of complex modulation levels available in a complex modulator for convenience.

[0075] Accordingly, a hologram is composed of an array of gray levels, that is, an array of optical modulation values such as an array of phase delay values or complex modulation values. Since a hologram is a pattern that is displayed on a spatial light modulator and causes diffraction when irradiated with light having a wavelength comparable to (usually shorter than) the pixel pitch of the spatial light modulator, it can also be considered a diffraction pattern. Here, mention is made of combining a hologram with other diffraction patterns such as a diffraction pattern that functions as a lens or a grating. For example, a diffraction pattern that functions as a grating can be combined with a hologram to transform a reproduction field on a reproduction plane, or a diffraction pattern that functions as a lens can be combined with a hologram to focus holographic reconstruction on a reproduction plane in the near field.

[0076] In the following detailed description, different embodiments and groups of embodiments may be disclosed separately, but any feature of any embodiment or group of embodiments can be combined with any other feature or combination of features of any embodiment or group of embodiments. That is, all possible combinations and permutations of the features disclosed in this disclosure are envisioned.

Brief Description of the Drawings

[0077] Specific embodiments are described by way of example only with reference to the following figures.

[0078]

Figure 1

[0079]

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 14A

[0080]

Figure 14B

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 20A

Figure 20B

Figure 20C

[0081] The same reference numbers are used throughout the drawings to refer to the same or similar parts.

DETAILED DESCRIPTION OF THE INVENTION

[0082] The present invention is not limited to the embodiments described below, but extends to the full scope of the appended claims. That is, the present invention can be implemented in different forms and should not be construed as being limited to the embodiments described for the purpose of explanation.

[0083] The singular terms may include the plural unless otherwise specified.

[0084] A structure described as being formed on top of / below another structure, or on / under another structure, is to be interpreted as including cases where the structures are in contact, and further including cases where a third structure is disposed therebetween.

[0085] When describing a time relationship, for example, when the chronological order of events is described as "after", "subsequent", "next", "before", etc., unless otherwise specified, the present disclosure should be interpreted as including both continuous and discontinuous events. For example, unless words such as "just", "immediately", "directly" are used, the description should be interpreted as including cases where it is not continuous.

[0086] In this specification, terms such as "first" and "second" may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the appended claims, the first element can be called the second element, and similarly, the second element can be called the first element.

[0087] The features of different embodiments can be combined or combined with each other, either partially or wholly, and can be interoperated with each other in various forms. Depending on the embodiment, they can be executed independently of each other or together in an interdependent relationship.

[0088] In the present disclosure, when applied to the structural unit of the device, the term "substantially" can be construed as the technical features of the structural unit being generated within the technical tolerance range of the method used to manufacture it. Conventional optical configuration of holographic projection

[0089] FIG. 1 shows an embodiment in which a computer-generated hologram is encoded onto a single spatial light modulator. The computer-generated hologram is the Fourier transform of the object for reconstruction. Therefore, it can be said that the hologram is a Fourier domain, frequency domain, or spectral domain representation of the object. In this embodiment, the spatial light modulator is a reflective liquid crystal on silicon (LCOS) device. The hologram is encoded onto the spatial light modulator, and a holographic reconstruction is formed on a light receiving surface such as a screen or a diffuser, for example, a reproduction field.

[0090] A light source 110, such as a laser or a laser diode, is arranged to illuminate an SLM 140 via a collimating lens 111. The collimating lens causes a substantially planar wavefront of light to be incident on the SLM. In FIG. 1, the direction of the wavefront is not perpendicular (e.g., it is 2 or 3 degrees away from true orthogonality with respect to the plane of the transparent layer). However, in other embodiments, a substantially planar wavefront is provided with normal incidence, and a beam splitter arrangement is used to separate the input optical path and the output optical path. In the embodiment shown in FIG. 1, the light from the light source is reflected at the mirror backside of the SLM and is arranged to interact with the light modulation layer to form an output wavefront 112. The output wavefront 112 is applied to an optical system including a Fourier transform lens 120 focused on a screen 125. More specifically, the Fourier transform lens 120 receives the beam of modulated light from the SLM 140 and performs a frequency space transformation to generate a holographic reconstruction at the screen 125.

[0091] In particular, in this type of holography, each pixel of the hologram contributes to the overall reconstruction. There is no one-to-one correlation between a specific point (or image pixel) on the reconstruction field and a specific light modulation element (or hologram pixel). In other words, the modulated light emerging from the light modulation layer is dispersed over the entire reconstruction field.

[0092] In these embodiments, the position of the holographic reconstruction in space is determined by the refractive power (focusing power) of the Fourier transform lens. In the embodiment shown in FIG. 1, the Fourier transform lens is a physical lens. That is, the Fourier transform lens is an optical Fourier transform lens, and the Fourier transform is performed optically. Any lens can function as a Fourier transform lens, but the performance of the lens limits the accuracy of the Fourier transform performed. Those skilled in the art understand how to perform an optical Fourier transform using a lens. In some embodiments of the present disclosure, the lens of the observer's eye performs the conversion from the hologram to the image. Hologram calculation

[0093] In some embodiments, the computer-generated hologram is a Fourier transform hologram, or simply a Fourier hologram, or a Fourier-based hologram, and the image is reconstructed in the far-field by utilizing the Fourier transform characteristics of a positive lens. The Fourier hologram is calculated by a Fourier transform that returns the desired light field of the reproduction plane back to the lens plane. The computer-generated Fourier hologram can be calculated using a Fourier transform. Embodiments relate to Fourier holography and Gerchberg-Saxton type algorithms by way of example only. The present disclosure is similarly applicable to Fresnel holography and Fresnel holograms that can be calculated in a similar manner. In some embodiments, the hologram is a phase or phase-only hologram. However, the present disclosure is also applicable to holograms calculated by other techniques such as those based on the point cloud method.

[0094] In some embodiments, the hologram engine is configured to exclude the contribution of light blocked by the limiting aperture of the display system from the hologram calculation. UK Patent Application 2101666.2, filed on February 5, 2021 and incorporated herein by reference, discloses a first hologram calculation method that uses ray tracing and ray tracing to identify sub-areas of a display device for the calculation of point cloud holograms that eliminate ghost images. The sub-areas of the display device correspond to the apertures of the present disclosure and are used to exclude optical paths from the hologram calculation. UK Patent Application 2112213.0, filed on August 26, 2021 and incorporated herein by reference, discloses a second method based on a modified Gerchberg-Saxton type algorithm that includes the step of cropping the light field according to the pupil of the optical system during hologram calculation. The cropping of the light field corresponds to the determination of the limiting aperture of the present disclosure. UK Patent Application 2118911.3, filed on December 23, 2021 and incorporated herein by reference, discloses a third method of calculating a hologram that includes the step of determining the area of a so-called extended modulator formed by a hologram replicator. According to this disclosure, the area of the extended modulator is also an aperture.

[0095] In some embodiments, a real-time engine is provided that is configured to receive image data and calculate holograms in real time using an algorithm. In some embodiments, the image data is a video comprising a series of image frames. In other embodiments, the holograms are pre-calculated, stored in computer memory, and called as needed for display on the SLM. That is, in some embodiments, a repository of predetermined holograms is provided. Wide field of view using a small display device

[0096] Broadly speaking, the present disclosure relates to image projection. The present disclosure relates to a method of image projection and an image projector comprising a display device. The present disclosure also relates to a projection system comprising an image projector and a display system. In this projection system, the image projector projects or relays light to the display system from the display device. The present disclosure is equally applicable to monocular and binocular display systems. The display system can comprise the eyes of a viewer or multiple eyes. The display system comprises an optical element having an optical power (e.g., the lens of a human eye) and a display surface (e.g., the retina of a human eye). The projector may be referred to as an "optical engine". The display device and the images formed (or perceived) using the display device are spatially separated from each other. The images are formed on the display surface or are perceived by the viewer. In some embodiments, the images are virtual images and the display surface may be referred to as a virtual image plane. In other examples, the images are real images formed by holographic reconstruction and the images are projected or relayed onto the display surface. In these other examples, spatially modulated light of an intermediate holographic reconstruction formed on a screen or other light-receiving surface between the free space or the display device and the viewer propagates to the viewer. In either case, the images are formed by irradiating a diffraction pattern (such as a hologram or a kinoform) displayed on the display device.

[0097] The display device is composed of pixels. The pixels of the display can display a diffraction pattern or structure that diffracts light. The diffracted light forms an image on a plane spatially separated from the display device. According to well-known optics, the magnitude of the maximum diffraction angle is determined by other factors such as the size of the pixel and the wavelength of the light.

[0098] In an embodiment, the display device is a spatial light modulator such as a liquid crystal on silicon (LCOS) spatial light modulator (SLM). Light propagates from the LCOS towards a display entity / system such as a camera or an eye over a range of diffraction angles (e.g., from zero to the maximum diffraction angle). In some embodiments, an expansion technique can be used to expand the range of available diffraction angles beyond the conventional maximum diffraction angle of the LCOS.

[0099] In some embodiments, the light of the hologram itself reaches the eye. For example, the spatially modulated light of the hologram (holographic reconstruction, i.e., not yet fully converted into an image) can be said to be informally "encoded" by the hologram, but it reaches the viewer's eye directly. The viewer can perceive a real or virtual image. In these embodiments, no intermediate holographic reconstruction / image is formed between the display device and the viewer. In these embodiments, the eye lens is sometimes said to perform the conversion or transformation from the hologram to an image. The projection system or the light engine can be configured such that the viewer effectively sees the display device directly.

[0100] The specification refers to a "light field", which is a "complex light field". The term "light field" only refers to a pattern of light having a finite size in at least two orthogonal spatial directions, e.g., x and y. The term "complex" is used in this specification only to indicate that the light at each point within the light field is defined by an amplitude value and a phase value and can thus be represented by a complex number or a pair of values. For the purpose of hologram calculation, a complex light field is a two-dimensional array of complex numbers, where the complex numbers define the intensity and phase of the light at a plurality of discrete positions within the light field.

[0101] According to well-known optical principles, the range of angles of light propagated from a display device that can be observed by the eye or other observation target / system varies depending on the distance between the display device and the observation target. For example, at an observation distance of 1 meter, only a very small portion of the angles from an LCOS passes through the pupil of the eye and can form an image on the retina at a specific eye position. The range of angles of the light rays propagated from the display device is the range that can pass through the pupil of the eye and form an image on the retina, and determines the portion of the image that is "visible" to the observer. In other words, not all parts of the image are visible from any one point on the observation plane (for example, any one eye position within an observation window such as an eyebox).

[0102] In some embodiments, the image perceived by the viewer is a virtual image displayed upstream of the display device. That is, the viewer perceives the image as being farther away than the display device. Conceptually, the viewer can be considered to be looking at the virtual image through a very small "display device-sized window", such as 1 cm in diameter, at a relatively large distance, for example 1 m. Also, the user will be looking at the display device-sized window through a very small pupil of the eye. Therefore, the field of view is narrow, and the specific angular range that can be seen depends greatly on the eye position at any given time.

[0103] The pupil expander addresses the problem of how to expand the angular range of the light rays propagated from the display device and successfully pass through the pupil of the eye to form an image. The display device is generally (relatively) small, and the projection distance is (relatively) large. In some embodiments, the projection distance is at least one order of magnitude, for example at least two orders of magnitude, larger than the diameter or width of the entrance pupil and / or aperture of the display device (that is, the size of the pixel array).

[0104] When using a pupil expander, the viewing area (i.e., the user's eye box) expands horizontally, allowing the user to move their eyes while still being able to view the image. As will be understood by a skilled technician, in an image system, the viewing area (the user's eye box) is the area where the observer's eyes can recognize the image. The present disclosure is directed to a non-infinite virtual image distance, i.e., a near-field virtual image.

[0105] Conventionally, a two-dimensional pupil expander is composed of one or more one-dimensional optical waveguides each formed using a pair of opposing reflective surfaces, and the output light from the surface forms a display window or an eye box. The light received from the display device (e.g., spatially modulated light from an LCOS) is replicated by the waveguide or each waveguide such that the viewing field (or display area) is expanded in at least one dimension. In particular, the waveguide expands the display window by generating additional light rays or "replicas" through the splitting of the amplitude of the incident wavefront.

[0106] It may have an active area or display area of less than 10 cm, for example less than 5 cm or less than 2 cm. The propagation distance between the display device and the display system may be more than 1 m, for example more than 1.5 m or more than 2 m. The optical propagation distance within the waveguide may be at most 2 m, for example at most 1.5 m or at most 1 m. This method can receive an image and determine a corresponding hologram of sufficient quality in less than 20 ms, for example less than 15 ms or less than 10 ms.

[0107] In some embodiments, although described only as examples of diffractive light fields or holographic light fields according to the present disclosure, the hologram is configured to route light into a plurality of channels, each channel corresponding to a different portion (i.e., sub-area) of the image. The channels formed by the diffractive structure are herein simply referred to as "hologram channels" to reflect that they are channels of light encoded by the hologram with image information. The light of each channel can be said to be in the hologram region rather than in the image or spatial region. In some embodiments, the hologram is a Fourier or Fourier transform hologram, and thus the hologram region is the Fourier or frequency domain. The hologram may similarly be a Fresnel or Fresnel transform hologram. The hologram may be a point cloud hologram. In this specification, the hologram is described as routing light into a plurality of hologram channels to reflect that the image reconstructable from the hologram has a finite size and can be arbitrarily divided into a plurality of image sub-regions, each hologram channel corresponding to each image sub-region. Importantly, the hologram of this example is characterized by how it distributes the image content when illuminated. Specifically, the hologram divides the image content by angle. That is, each point on the image is associated with a unique ray angle in the spatially modulated light formed by the hologram when illuminated. At least, since the hologram is two-dimensional, it is a pair of unique angles. To avoid ambiguity, the operation of this hologram is different from the conventional one. The spatially modulated light formed by this special type of hologram, when illuminated, is divided into a plurality of hologram channels, each hologram channel defined by a range (two-dimensional) of ray angles. From the foregoing, it is understood that the hologram channels that may be considered in the spatially modulated light (i.e., sub-ranges of ray angles) are associated with each respective portion or sub-region of the image. That is, all the information necessary to reconstruct that portion or sub-region of the image is contained within the sub-range of angles of the spatially modulated light formed from the hologram of the image.When spatially modulated light is observed as a whole, evidence of multiple individual light channels does not necessarily exist.

[0108] Nevertheless, the hologram can be identified. For example, if only a continuous portion or sub-region of the spatially modulated light formed by the hologram is reconstructed, only a sub-region of the image should be visible. If different continuous portions or sub-regions of the spatially modulated light are reconstructed, different sub-regions of the image should be visible. A further distinguishing feature of this type of hologram is that the shape of the cross-sectional area of the hologram channel, at least in the correct plane in which the hologram was calculated, may vary in size but substantially corresponds to (i.e., is substantially the same as) the shape of the entrance pupil. Each light / hologram channel propagates from the hologram at a different angle or angular range. These are exemplary ways to characterize or identify this type of hologram, but other methods can also be used. In summary, the holograms disclosed herein are characterized by how image content is dispersed within the light encoded by the hologram and are distinguishable. Again, for the sake of clarity, the reference herein to holograms configured to direct light or angularly split an image into multiple hologram channels is for illustrative purposes only, and this disclosure is equally applicable to any type of holographic light field, and indeed to any type of diffractive light field or pupil expansion of a diffractive light field.

[0109] This system can be provided in a compact and rationalized physical form. This enables the realization of a system suitable for various practical applications, such as when space is limited and real estate values are high. For example, it can be implemented in a head-up display (HUD) such as those in vehicles or cars.

[0110] According to the present disclosure, pupil expansion is provided for diffracted light or diffracted light including a diverging light beam. The diffracted light field is defined by a "light cone". Thus, the size of the diffracted light field (defined on a two-dimensional plane) increases with the propagation distance from the corresponding diffractive structure (i.e., the display device). The pupil expander can be said to replicate a hologram or form a replication of at least one hologram, and convey that the light reaching the viewer is spatially modulated according to the hologram.

[0111] In some embodiments, two one-dimensional waveguide pupil expanders are provided, and each one-dimensional waveguide pupil expander is arranged to effectively expand the size of the system's exit pupil by forming a plurality of replicas or copies of the exit pupil (or the light of the exit pupil) of the spatial light modulator. It can be understood that the exit pupil is the physical area where light is output by the system. Also, it can be said that each waveguide pupil expander is arranged to expand the size of the system's exit pupil. Also, it can be said that each waveguide pupil expander is arranged to expand / increase the size of the eye box where the observer's eye can be located to view / receive the light output by the system. Light channeling

[0112] The hologram formed according to some embodiments can provide a plurality of hologram channels that angularly divide the image content and have a cross-sectional shape defined by the aperture of the optical system. The hologram is calculated to provide this channeling of the diffracted light field. In some embodiments, this is achieved during hologram calculation by considering the aperture (virtual or real) of the optical system as described above.

[0113] Figures 2 and 3 show examples of this type of hologram that can be used in combination with the pupil expansion device disclosed herein. However, this example does not limit the present invention.

[0114] FIG. 2 shows a projection image 252 including eight image regions / components V1 through V8. FIG. 2 shows eight image components as an example, and image 252 can be divided into any number of components. FIG. 2 also shows an encoded light pattern 254 (i.e., a hologram) that can reconstruct image 252 when transformed by the lens of a suitable display system. The encoded light pattern 254 is composed of first through eighth sub-holograms or components H1 through H8 corresponding to the first through eighth image components / regions V1 through V8. FIG. 2 further shows how the hologram decomposes the image content by angle. Thus, the hologram is characterized by the channelling of light that it performs. This is shown in FIG. 3. Specifically, the hologram in this example directs light into a plurality of individual regions. The individual regions are disks in the example shown, but other shapes are envisioned. The optimal disk size and shape may be related to the size and shape of the aperture of an optical system such as the entrance pupil of the observation system after propagation through the waveguide.

[0115] FIG. 4 shows a system 400 including a display device that displays a hologram calculated as shown in FIGS. 2 and 3.

[0116] System 400 includes a display device that in this arrangement includes an LCOS 402. The LCOS 402 displays a modulation pattern (or "diffraction pattern") including the hologram and projects the holographically encoded light toward an eye 405 including a pupil that functions as aperture 404, a lens 409, and a retina (not shown) that functions as a field plane. There is a light source (not shown) arranged to illuminate the LCOS 402. The lens 409 of the eye 405 performs the conversion from the hologram to an image. The light source may be of any suitable type. For example, it may be a laser light source. The visual system 400 further includes a waveguide 408 disposed between the LCOS 402 and the eye 405. Due to the presence of the waveguide 408, all angular content from the LCOS 402 is received by the eye even at the relatively long projection distances shown in the figure. This is because the waveguide 508 functions as a pupil expander. This method is well known and will only be briefly described here.

[0117] Briefly speaking, the waveguide 408 shown in FIG. 4 is substantially composed of an elongated structure. In this example, the waveguide 408 is composed of an optical slab of refractive material, although other types of waveguides are well known and may be used. The waveguide 408 is arranged, for example, at an oblique angle so as to intersect the light cone (i.e., the diffracted light field) projected from the LCOS 402. In this example, the size, location, and position of the waveguide 408 are configured such that light from each of the eight light beams within the light cone enters the waveguide 408. Light from the light cone enters the waveguide 408 through the first planar surface (closest to the LCOS 402) of the waveguide 408, is guided at least partially along the length of the waveguide 408, and then is emitted through the second planar surface (closest to the eye) substantially opposite the first surface. As is well understood, the second plane is partially reflective and partially transmissive. In other words, when each ray of light moves from the first plane into the waveguide 408 and strikes the second plane, a portion of the light is transmitted through the waveguide 408 and a portion is reflected by the second plane back to the first plane. The first plane is reflective, and all light that strikes the first plane from within the waveguide 408 is reflected to the second plane. Thus, some of the light is refracted and then transmitted between the two planes of the waveguide 408, while other light is reflected and undergoes one or more reflections (or "bounces") between the planes of the waveguide 408 before being transmitted.

[0118] Figure 4 shows a total of nine "reflection" points B0 to B8 along the length of the waveguide 408. As shown in Figure 2, the light associated with all points (V1 - V8) of the image is transmitted from the waveguide at each "reflection" from the second plane of the waveguide 408, but only the light from one angular portion of the image (e.g., the light from any of V1 to V8) has a trajectory that can reach the eye 405 from each "reflection" point B0 to B8. Further, the light from different angular portions V1 to V8 of the image reaches the eye 405 from each "reflection" point. Thus, in the example of Figure 4, each angular channel of the encoded light reaches the eye only once from the waveguide 408.

[0119] The waveguide 408 forms a plurality of replicas of the hologram at each "reflection" point B1 to B8 along its length corresponding to the direction of pupil dilation. As shown in Figure 5, the plurality of replicas are linearly extrapolated to the corresponding plurality of replicas or virtual display devices 402'. This process corresponds to the step of "unfolding" the optical path within the waveguide, and the light rays of the replicas are extrapolated to the "virtual surface" without internal reflection within the waveguide. Thus, the light of the enlarged exit pupil is considered to originate from a virtual surface (also referred to herein as an "expanded modulator") that includes the display device 402 and the replica display device 402'.

[0120] In this specification, we have generally described virtual images for which the received modulated light needs to be converted by the eye to form a perceived image, but the methods and configurations described herein are also applicable to real images. Two-dimensional pupil expansion

[0121] The arrangement shown in Figure 4 includes a single waveguide that provides pupil dilation in one dimension, but pupil dilation can be provided in multiple dimensions such as two dimensions. Further, in the example of Figure 4, holograms calculated to create channels of light corresponding to different portions of the image are used, but the present disclosure and the systems described below are not limited to such types of holograms.

[0122] FIG. 5A shows a perspective view of a system 500 including two replicators 504, 506 arranged to expand a light beam 502 two-dimensionally.

[0123] In the system 500 of FIG. 5A, the first replicator 504 is composed of a first pair of surfaces stacked parallel to each other and arranged to provide replication (or pupil expansion) similar to the waveguide 408 of FIG. 4. The first pair of surfaces have similar (in some cases identical) sizes and shapes to each other and are substantially elongated in one direction. The collimated light rays 502 are directed towards the input of the first replicator 504. As is well known to those skilled in the art, due to the internal reflection process between the two surfaces and the partial transmission of light from each of the multiple output points of one of the surfaces (the upper surface as shown in FIG. 5A), the light of the light rays 502 is replicated in a first direction along the length of the first replicator 504. Accordingly, a first plurality of replica light rays 508 are radiated from the first replicator 504 towards the second replicator 506.

[0124] The second replicator 506 includes a second pair of surfaces stacked parallel to each other, is arranged to receive each of the collimated light rays of the first plurality of light rays 508, and is further arranged to provide replication, i.e., pupil expansion, by expanding each of those light rays in a second direction substantially orthogonal to the first direction. The first pair of surfaces are of similar (and in some cases identical) size and shape and are substantially rectangular. The second replicator is implemented in a rectangular shape in order to have a length along the first direction to receive the first plurality of light rays 508, have a length along the second orthogonal direction, and provide replication in the second direction. By the process of internal reflection between the two surfaces and the partial transmission of light from each of the plurality of output points of one of the surfaces (the upper surface as shown in FIG. 5A), the light of each light ray within the first plurality of light rays 508 is replicated in the second direction. Accordingly, a second plurality of light beams 510 are emitted from the second replicator 506, and the second plurality of light beams 510 include replicas of the input light beam 502 along each of the first and second directions. Accordingly, the second plurality of light beams 510 can be regarded as including a two-dimensional grid or array of replica light beams.

[0125] Accordingly, it can be said that combining the first and second replicators 504, 505 of FIG. 5A provides a two-dimensional replicator (or “two-dimensional pupil expander”). Accordingly, the replica light beams 510 may be emitted along the optical path to an expanded eyebox of a display system such as a head-up display.

[0126] In the system of FIG. 5A, the first replicator 504 is a waveguide including a pair of elongated straight reflecting surfaces stacked parallel to each other, and similarly, the second replicator 504 is a waveguide including a pair of rectangular reflecting surfaces stacked parallel to each other. In other systems, the first replicator is a solid elongated straight waveguide, and the second replicator is a solid planar rectangular waveguide, and each waveguide includes an optically transparent solid material such as glass. In this case, the pair of parallel reflecting surfaces are formed by a pair of opposing major sidewalls, each optionally including a reflective and a reflection-transmissive surface coating, well known to those skilled in the art.

[0127] FIG. 5B shows a perspective view of a system 500 including two replication devices 520, 540 arranged to replicate an optical beam 522 in two dimensions, where the first replication device is a solid elongated waveguide 520 and the second replication device is a solid planar waveguide 540.

[0128] In the system of FIG. 5B, the first replicator / waveguide 520 is arranged such that its pair of elongated parallel reflecting surfaces 524a, 524b are perpendicular to the plane of the second replicator / waveguide 540. Accordingly, the system includes an optical coupler arranged to couple light from the output port of the first replicator 520 to the input port of the second replicator 540. In the illustrated arrangement, the optical coupler is a planar / bent mirror 530 arranged to bend and rotate the optical path of the light to achieve the necessary optical coupling from the first replicator to the second replicator. As shown in FIG. 5B, the mirror 530 is arranged to receive light including a one-dimensional array of replicas extending in a first dimension from the output port / reflection-transmissive surface 524a of the first replicator / waveguide 520. The mirror 530 is tilted at an angle that provides waveguiding and replica formation along the length of a second dimension to redirect the received light into the optical path to the input port of the (total) reflecting surface of the second replicator 540. The mirror 530 is an example of an optical element that can redirect light in the manner illustrated, and it is understood that one or more other elements can be used instead to perform this task.

[0129] In the illustrated arrangement, the (partially) reflective transmissive surface 524a of the first replicator 520 is adjacent to the input port of the first replicator / waveguide 520 and receives the input beam 522 at an angle to perform waveguide and replica formation along the length of the first dimension. Thus, the input port of the first replicator / waveguide 520 is disposed at the input end of the same surface as the reflective transmissive surface 524a. A skilled reader will understand that the input port of the first replicator / waveguide 520 may be disposed at other suitable locations.

[0130] Thus, with the arrangement of FIG. 5B, the first replicator 520 and the mirror 530 can be provided as part of a relatively thin first layer in the plane of the first and third dimensions (shown as the xz plane). In particular, the size or “height” of the first planar layer in which the first replicator 520 is disposed is reduced in the second dimension (shown as the y dimension). The mirror 530 is configured to deflect light away from the first layer / plane in which the first replicator 520 is disposed (i.e., the “first planar layer”) and direct the light upward of the first layer / plane and substantially parallel to the first layer / plane to the second layer / plane in which the second replicator 540 is disposed (i.e., the “second planar layer”). Thus, the overall size or “height” of the system including the first and second replicators 520, 540 and the mirror 530 disposed in the stacked first and second planar layers of the first and third dimensions (shown as the xz plane) is compact in the second dimension (shown as the y dimension). A skilled reader will understand that many variations of the arrangement of FIG. 5B for implementing the present disclosure are possible and have been considered.

[0131] The image projector can be arranged to project a diverging or diffracted light field. In some embodiments, the light field is encoded with a hologram. In some embodiments, the diffracted light field includes a diverging light beam. In some embodiments, the image formed by the diffracted light field is a virtual image.

[0132] In some embodiments, the pair of first parallel / complementary surfaces are relatively long along a first dimension and relatively short along a second dimension, e.g., relatively short along each of the other two dimensions, with each dimension being substantially orthogonal to each of the other dimensions, and are elongated or slender surfaces. The process of reflection / transmission of light between the pair of first parallel surfaces is configured such that light propagates within the first waveguide pupil expander, and the general direction of light propagation is the direction in which the first waveguide pupil expander is relatively long (i.e., its "elongated" direction).

[0133] Disclosed herein is a system that uses diffracted light to form an image and provides an eye box size and field of view suitable for actual applications (e.g., head-up displays in the automotive industry). Diffracted light is light that forms a holographic reconstruction of an image from a diffraction structure (e.g., a hologram such as a Fourier or Fresnel hologram). The use of diffraction and diffraction structures requires a high-density display device with very small pixels (e.g., 1 μm), which effectively means a small display device (e.g., 1 cm). The inventors have addressed the problem of a method for providing 2D pupil expansion using a diffracted light field (e.g., diffracted light including a diverging (non-parallel) light beam).

[0134] In some embodiments, the display system includes a pixelated display device, such as a spatial light modulator (SLM) or a liquid crystal on silicon (LCOS) SLM, configured to provide or form diffracted light or divergent light. In such an aspect, the aperture of the spatial light modulator (SLM) is the limiting aperture of the system. That is, the size of the aperture of the spatial light modulator, more specifically, the size of the region delimiting the array of light modulation pixels included within the SLM, determines the size (e.g., spatial extent) of the light beam that can emerge from the system. According to this disclosure, it is stated that the exit pupil of the system is enlarged to reflect that the exit pupil of the system (limited by a small display device having a pixel size for light diffraction) is made larger or larger or larger in spatial extent by the use of at least one pupil expander.

[0135] It can be said that the diffracted or divergent light field has a "light field size" defined in a direction substantially orthogonal to the propagation direction of the light field. Since light diffracts / diverges, the light field size increases with the propagation distance.

[0136] In some embodiments, the diffracted light field is spatially modulated according to a hologram. In other words, in such aspects, the diffracted light field includes a "holographic light field". The hologram may be displayed on a pixelated display device. The hologram may be a computer-generated hologram (CGH). This may be a Fourier hologram, a Fresnel hologram, a point cloud hologram, or any other suitable type of hologram. Optionally, the hologram may be calculated to form channels of holographic light. Each channel corresponds to a different respective part of the image intended for the viewer to see (or, in the case of a virtual image, to perceive). The pixelated display device may be configured to display a plurality of different holograms continuously or in sequence. Each of the aspects and embodiments disclosed herein may be applied to the display of a plurality of holograms.

[0137] The output port of the first waveguide pupil expander may be coupled to the input port of the second waveguide pupil expander. The second waveguide pupil expander may be arranged to direct a diffracted light field, which includes a part, preferably a majority, preferably all of the replica of the light field output by the first waveguide pupil expander, from its input port to its respective output ports by means of internal reflection between a pair of third parallel planes of the second waveguide pupil expander.

[0138] The first waveguide pupil expander can be arranged to provide pupil expansion or replication in a first direction, and the second waveguide pupil expander can be arranged to provide pupil expansion or replication in a second different direction. The second direction can be substantially orthogonal to the first direction. The second waveguide pupil expander can be arranged to maintain the pupil expansion provided by the first waveguide pupil expander in the first direction and expand (or replicate) a part, preferably most, preferably all of the replicas received from the first waveguide pupil expander in the second different direction. The second waveguide pupil expander can be arranged to receive the optical field directly or indirectly from the first waveguide pupil expander. One or more other elements can be provided along the propagation path of the optical field between the first and second waveguide pupil expanders.

[0139] The first waveguide tube pupil expander can be substantially elongated, and the second waveguide tube pupil expander can be substantially planar. The elongated shape of the first waveguide tube pupil expander may be defined by a length along a first dimension. The plane of the second waveguide tube pupil expander, i.e., the rectangular shape, may be defined by a length along a first dimension and a width, i.e., a width, along a second dimension substantially orthogonal to the first dimension. The size, i.e., the length, of the first waveguide tube pupil expander along the first dimension corresponds respectively to the length, i.e., the width, of the second waveguide tube pupil expander along the first dimension or the second dimension. Of the pair of parallel planes of the second waveguide tube pupil expander, the first surface including the input port may be shaped, sized, and / or arranged to correspond to the region defined by the output port on the first surface of the pair of parallel planes of the first waveguide tube pupil expander. Thereby, the second waveguide tube pupil expander is arranged to receive each replica output by the first waveguide tube pupil expander.

[0140] The first and second waveguide pupil expanders can jointly provide pupil expansion in a first direction and a second direction perpendicular to the first direction. Optionally, the plane including the first and second directions is substantially parallel to the plane of the second waveguide pupil expander. In other words, the first and second dimensions that respectively define the length and width of the second waveguide pupil expander can be parallel to the first and second directions (or the second and first directions) in which the waveguide pupil expander provides pupil expansion. The combination of the first waveguide pupil expander and the second waveguide pupil expander is sometimes generally referred to as a "pupil expander".

[0141] It can be said that the expansion / replication provided by the first and second waveguide expanders has the effect of expanding the exit pupil of the display system in each of two directions. The region defined by the expanded exit pupil defines an expanded eyebox region from which a viewer can receive light from the input diffraction or diverging light field. It can be said that the eyebox region is disposed on or defines the display surface.

[0142] The two directions in which the exit pupil is expanded may be in the same plane or parallel to the first and second directions in which the first and second waveguide pupil expanders provide replication / expansion. Alternatively, in an arrangement including other elements such as an optical combiner, for example, the windshield (or windscreen) of a vehicle, the exit pupil may be regarded as the exit pupil from other elements such as the windshield. In such an arrangement, the exit pupil may not be in the same plane and parallel to the first and second directions in which the first and second waveguide pupil expanders provide replication / expansion. For example, the exit pupil may be substantially perpendicular to the first and second directions in which the first and second waveguide pupil expanders provide replication / expansion.

[0143] The viewing surface and / or the eyebox region may be non-coplanar or non-parallel to the first and second directions in which the first and second waveguide pupil expanders provide replication / expansion. For example, the viewing surface may be substantially perpendicular to the first and second directions in which the first and second waveguide pupil expanders provide replication / expansion.

[0144] To provide appropriate emission conditions for realizing internal reflection within the first and second waveguide pupil expanders, the elongated dimension of the first waveguide pupil expander may be inclined with respect to the first and second dimensions of the second waveguide pupil expander. Combiner shape correction

[0145] The advantage of projecting a hologram onto an eyebox is that optical compensation can be encoded into the hologram (see, for example, European Patent No. 2936252 incorporated herein). The present disclosure is compatible with holograms that compensate for the complex curvature of an optical combiner used as part of a projection system. In some embodiments, the optical combiner is the front windshield of a vehicle. Details of this approach are described in European Patent No. 2936252, and the detailed features of those systems and methods are not essential to the novel teachings of the present disclosure and are merely illustrative of configurations that benefit from the teachings of the present disclosure, and thus will not be repeated here. Control device

[0146] The present disclosure is also compatible with an optical configuration including a control device (e.g., an optical shutter device) for controlling the delivery of light from a light-channeling hologram to a viewer. The holographic projector may further comprise a control device arranged to control the delivery of angular channels to the eye box position. UK Patent Application 2108456.1, filed on 14 June 2021 and incorporated herein by reference, discloses at least one waveguide pupil expander and a control device. The reader will understand from at least this prior disclosure that the optical configuration of the control device is essentially based on the user's eye box position and is compatible with any hologram calculation method for implementing the light channeling described herein. The control device can be said to be an optical shutter or an aperture device. The optical shutter device comprises a 1D array of apertures or windows, each of which can be independently switched between a light-transmissive state and a light-non-transmissive state in order to control the delivery of hologram light channels and their replicas to the eye box. Each aperture or window may comprise a plurality of liquid crystal cells or pixels. Point cloud hologram calculation

[0147] Figure 6 shows a schematic diagram representing the point cloud hologram method. Figure 6 consists of a point cloud 602 (i.e., a cloud of image points), a display device 604 (or hologram plane), and an entrance pupil 606. The display device 604 is disposed between the entrance pupil 606 and the point cloud 602 and, in this example, is a pixelated liquid crystal on silicon spatial light modulator. In this example, the position of the entrance pupil 606 is determined by the viewer's eye tracking. The method described here is executed virtually (i.e., it is a simulation). Thus, Figure 6 simply represents what is executed computationally during the calculation of the point cloud hologram. Further, each step of the method is executed virtually. For example, when a wave is described as propagating, the wave does not physically propagate through a physical system. Instead, a virtual wave is virtually propagated within a virtual system. The display device 604 in this example represents the plane on which the hologram is displayed after it is calculated. As will be described in detail below, the purpose of the point cloud hologram calculation is to determine a hologram that can be (physically) displayed on the display device 604. Thereby, when the hologram is appropriately (physically) illuminated with coherent light, a (physical) holographic wavefront is formed, which is relayed to the viewer's actual entrance pupil, giving the appearance of a holographic reconstruction of the three-dimensional point cloud.

[0148] In this example, the point cloud 602 is a three-dimensional point cloud, and the points represent objects or scenes. In particular, the point cloud 602 represents the target picture that is the basis for the calculation of the hologram. Four image points, a first image point 610, a second image point 612, a third image point 614, and a fourth image point 616, are shown in Figure 6. The first through fourth image points are distributed throughout the point cloud 604. This is merely representative. It should be understood that typically the point cloud of a point cloud hologram consists of thousands or millions of image points representing an image.

[0149] The point group hologram calculation method generally consists of simulating the propagation of light waves (or wavelets) along the propagation path from each image point of the point group 604 towards the center of the entrance pupil 606. The propagation paths from the first to the fourth image points 610 to 614 are represented by the dashed lines in FIG. 6. The first propagation path 620 extends from the first image point 610 to the entrance pupil 606, the second propagation path 622 extends from the second image point 612 to the entrance pupil 606, the third propagation path 624 extends from the third image point 614 to the entrance pupil 606, and the fourth propagation path 626 extends from the fourth image point 616 to the entrance pupil 606. Skilled readers will be well aware of the appropriate approaches and algorithms for performing this propagation of light waves.

[0150] It should be noted that the optical system in FIG. 6 is simplified. In most cases, one or more optical components are arranged between the point group 602 / display device 604 and the viewer. The propagation path may pass through these one or more optical components. For example, as mentioned above, one or more optical components may include one or more lenses, one or more mirrors, and / or waveguides. In the calculation of the point group hologram, it is necessary to consider each of these optical components when the wave from the image point propagates virtually. Furthermore, FIG. 6 is not drawn to scale. For example, in a vehicle head-up display, the distance between the display device 604 and the viewer may be one meter or more. The distance from the viewer to the point group 602 may be even longer.

[0151] After simulating the propagation of light waves, the next step in this method is to obtain complex values at each point on the (virtual) display device 604 or hologram plane. The display device 604 is pixelated. Conventionally, the steps of this method include determining the complex value of each pixel of the display device 604 (or at least each pixel of the display device where the wave is incident). In this example, a phase-only hologram is determined, and the phase value is extracted from the complex value of each pixel. Skilled readers will be proficient in determining these values. This example is iterative and iterates over each image point. Specifically, in each iteration, light waves are propagated from each image point, the complex value of each pixel of the display device 604 is determined, and then the phase value is determined. The phase values of each image point are combined by summing, superimposing, accumulating, or other methods, and a hologram of the target picture is output.

[0152] The inventors have discovered that the spatial frequency of the hologram of an image point depends on the position of that image point within the target picture. Generally, spatial frequency refers to the change in the wavefront / hologram phase and / or intensity across different positions within the hologram. In this example, since the hologram is a phase-only hologram, spatial frequency refers to the change in phase.

[0153] FIG. 7A shows the hologram of the first image point 610 of FIG. 6 alone. FIG. 7B shows the hologram of the third image point 614 of FIG. 6 alone. The upper pane 702 of FIG. 7A shows the complete hologram of the first image point 610. The lower pane 704 of FIG. 7A shows a close-up of a part of the complete hologram of the first image point 610. The upper pane 712 of FIG. 7B shows the complete hologram of the third image point 614. The lower pane 714 of FIG. 7B shows a close-up of a part of the complete hologram of the third image point 614. Both FIGS. 7A and 7B are in grayscale and both show how the phase delay applied to the hologram changes spatially. In this example, the phase changes from dark (black) representing a phase delay of 0 to bright (white) representing a phase delay of 2π.

[0154] As can be seen from FIGS. 7A and 7B, the spatial frequency of the hologram of the first image point 610 (see FIG. 7A) is much higher than the spatial frequency of the third image point 614 (see FIG. 7B). In other words, the phase of the hologram in FIG. 7A changes much more rapidly than the phase of the hologram in FIG. 7B. This difference in spatial frequency becomes particularly apparent when comparing the lower panels 704, 714 in FIGS. 7A and 7B. In the lower panel 704 of FIG. 7A, the phase changes by approximately π between adjacent pixel columns. For example, refer to the difference in darkness between the first pixel column 706 and the second pixel column 707 in FIG. 7A. However, in the lower panel 714 of FIG. 7B (on the same scale as the lower panel 704 of FIG. 7A), the phase change is much more gradual. The phase changes very smoothly from approximately 0 to 2π along the entire length of the lower panel 714.

[0155] The field of view of a pixelated display device (such as display device 604) is limited by the wavelength of light and, more importantly in the context of the present disclosure, by the pixel pitch (or pixel size) of the display device. As the pixel pitch decreases, the field of view increases. The display device 604 can holographically reconstruct only points within its field of view (corresponding to the maximum diffraction angle of the display device). The hologram of the first image point 610 has a very high spatial frequency. This is because the first image point 610 is towards the edge of the point group 602. In this example, this corresponds to being close to the edge of the field of view of the display device 604, and the propagation axis 620 passes through the display device 604 at an angle close to the diffraction limit of the display device 604. The high spatial frequency is necessary to diffract the light incident thereon to form the reconstructed first image point 610 close to the edge of the field of view (i.e., close to the diffraction limit of the display device 604). Conversely, the image point 610 is towards the center of the point group 602, and the associated wave propagates approximately perpendicular to the display device 604. Therefore, the spatial frequency is relatively very low (when compared to the hologram of the first image point 610).

[0156] The inventors recognized that when the spatial frequency of a hologram is low, the hologram can be calculated at a resolution lower than the native resolution of display device 604. In other words, a hologram with a low spatial frequency may be undersampled with only a portion of the calculated phase values. For example, there may be one or more empty pixel values between each phase value. The inventors recognized that since the phase of such a hologram changes slowly and smoothly, the hologram can be upscaled to the full resolution of display device 604 by accurately interpolating between the calculated phase values (without aliasing problems). The inventors utilized this concept in an improved method for calculating the point cloud holograms disclosed below. Improved point cloud hologram calculation method

[0157] It is desirable to be able to calculate point cloud holograms in real time so that a sequence of holograms of different target pictures can be calculated and continuously displayed at a relatively high frame rate (e.g., at least 30 frames per second) and at a resolution high enough that the reconstruction of the holograms looks of high quality. It is also desirable to achieve this with relatively limited computational resources. For example, a vehicle's head-up display may have relatively limited on-board computational resources. This has not been achievable heretofore using conventional point cloud hologram calculation methods. This is because calculating hologram values at high resolution using the point cloud method is very computationally costly. That is, such real-time calculation is not practical, and it is even more so when using a device with relatively limited on-board computational resources at high resolution. Therefore, the inventors devised the improved point cloud hologram calculation method described herein.

[0158] The method proposed by the inventors involves determining the hologram value only for every nth pixel, using the point cloud method, for a first subset of image points. Here, n is greater than 1 (for example, for every second, third, fourth, or fifth pixel). Since the point cloud method (including wavelet propagation and value determination) has a very high computational cost, reducing the number of sample points (i.e., the number of pixel values to be determined) significantly reduces the computational load of determining values for all pixels of the display device. The pixels between the nth pixels can be filled using interpolation based on the determined (nth) pixels.

[0159] The first subset of image points is arranged / selected such that each image point of the first subset has a related diffraction angle for use in the point cloud method, and each diffraction angle is smaller than the maximum diffraction angle of the display device. As described above, this allows the hologram to be undersampled and interpolated without aliasing problems. Here, two examples of image points forming such a first subset will be described. First example

[0160] In the first example, the first subset of image points, each having a related diffraction for use in the point cloud method that results in a relatively low spatial frequency, is realized by selecting the first subset of pixels as pixels relatively close to the center of the field of view / the center of the target picture and applying the improved method only to those pixels. For example, the first subset of image points is composed of image point 614, rather than image points 610 or 616. Since the first subset of image points is selected in this way, the calculated hologram has a low spatial frequency and can be undersampled as described above. A part of the undersampled hologram displayed on the display device is shown in FIG. 8.

[0161] The grid of FIG. 8 represents the pixels of a pixelated display device. Each square in the array represents a pixel. The black / filled squares 804 in the array represent the pixels for which the complex number / phase value has been determined in the initial hologram calculation (following the propagation of waves along the propagation path). The white / blank squares 806 represent the pixels for which the complex number / phase value has not been determined. In this example, each black square 804 is separated from the nearest adjacent black square by four white (blank) squares 806 in the x and y directions. In other words, the pixel values are determined for every fifth pixel. Thus, FIG. 8 shows a method of using undersampling to significantly reduce the number of pixels for which the complex number / phase value needs to be determined during the initial hologram calculation. This results in a significant savings in computation time and a significant improvement in the efficiency of hologram calculation. As described above, the hologram determined in FIG. 8 has a very low spatial frequency. Therefore, interpolation can be used to accurately determine the values of the empty squares / pixels 806 based on the known values of the black squares 804. For example, bicubic interpolation can be used for interpolation.

[0162] To further illustrate an improved method of calculating the hologram of a target picture, FIGS. 9 through 11 represent the steps of hologram calculation for a specific target picture (not the general single point 610 of FIG. 6). FIG. 9 shows the target picture 900 of this example. The point cloud of FIG. 6 is shown in the yz plane. This target picture 900 of FIG. 9 is instead in the xy plane. Thus, the view of the target picture 900 of FIG. 9 is effectively a front view of the target picture 900 from the perspective of the viewing system (not a side view). Thus, the target picture 900 of FIG. 9 is displayed two-dimensionally. In some examples, the target picture 900 has depth in the z direction and may be three-dimensional.

[0163] The target picture 900 in FIG. 9 is for a vehicle's head-up display and is composed of information useful to the vehicle driver. This includes speed information, the vehicle's charge state, direction information, etc. The target picture 900 is a cloud of points (or point cloud). However, since the target picture 900 is high-resolution, at the scale of the target picture 900 shown in FIG. 9, individual pixels cannot be distinguished.

[0164] In this example, the first step of the method for calculating the hologram is to divide the target picture into a plurality of parts or cells. FIG. 10 shows the target picture 900 divided into an array 1000 of cells. In this example, the array 1000 of cells is composed of 12x9 cells extending in the x-direction and y-direction respectively. In this example, dividing the target picture into a plurality of cells includes defining the angular range from the viewer to the target picture (using ray tracing). This step is performed virtually. Adjacent angular ranges are adjacent to each other (touching each other) and do not overlap. In this example, each angular range is equal in extent to all other angular ranges in both the x-direction and y-direction. In other words, the size of each cell is substantially equal.

[0165] This method further includes identifying which pixels of the target picture 900 correspond to which cells. In this example, some cells (such as cell 1002) do not contain pixels. Other cells (such as cell 1004) contain pixels. Cells that do not contain pixels can be substantially ignored in this method. The pixels contained in an individual cell may be referred to as a subset of pixels.

[0166] In this example, the method of calculating the hologram includes calculating the sub-holograms of each cell (or each cell containing at least image points). In other words, this method includes calculating the sub-holograms of each subset of the image points. In this example, different approaches are used to calculate the sub-holograms of different subsets of cells / image points. In particular, the sub-holograms of the cells in the central portion 1104 of the target picture are calculated using the improved method of the present disclosure, while the sub-holograms of the cells in the outer portion of the target picture are calculated using the conventional point cloud method.

[0167] FIG. 11 shows the outer portion 1102 and the central portion 1104. The boundary 1100 between the outer portion 1102 and the central portion 1104 is represented by a dashed line. The cells in the central portion 1104 are shown in gray for illustration purposes to distinguish them from the cells in the inner portion 1104 and the outer portion 1102.

[0168] The image points within the cells of the central portion 1104 are closer to the center of the target picture 900 than the image points within the cells of the outer portion 1102. Therefore, the angle formed by the propagation path of the image points within the cells of the central portion 1104 is closer to zero than the angle formed by the propagation path of the image points within the cells of the outer portion 1102 (the outer portion 1102 is closer to the maximum diffraction angle of the display device on which the hologram is displayed / displayable). This means that the spatial frequency of the sub-holograms of the cells in the central portion 1104 is lower than the spatial frequency of the sub-holograms of the cells in the outer portion 1102. Therefore, since the spatial frequency of the sub-holograms of the cells in the central portion 1104 is low enough that no significant aliasing problems occur due to interpolation, they can be calculated according to the method of the present disclosure (including undersampling and subsequent interpolation). Conversely, the spatial frequency of the sub-holograms of the cells in the outer portion 1102 may be too high to use interpolation without causing significant aliasing problems.

[0169] In summary, in the example of FIG. 11, different image points of the target picture are processed in different ways in hologram calculation according to how close their image points are to the center of the target picture. In particular, the inventors recognized that for points close to the center of the target picture, it is possible to first undersample the values of the subholograms. FIGS. 12 and 13A-13C illustrate an extension of this idea. In particular, the inventors recognized that there is generally a need to balance maximizing the initial undersampling (for computational efficiency) without causing aliasing problems. The inventors recognized that this balance may vary for each cell depending on how close the cell is to the center of the target picture. In the example shown in FIGS. 12 and 13A-13C, this effect is utilized and there are three different undersampling zones. The subholograms of cells in different zones or portions are undersampled to different degrees.

[0170] More specifically, the example of FIG. 12 consists of an outer portion 1202, a first central portion 1204, a second central portion 1206, and a third central portion 1208. Each portion is concentric with all other portions and the center of the target picture. There is a first boundary between the outer portion 1202 and the first central portion 1204. There is a second boundary between the first central portion 1204 and the second central portion 1206. There is a third boundary between the second central portion 1206 and the third central portion 1208. The first, second, and third central portions are represented by a gray shading that gets darker for portions closer to the center of the target picture. This shading is for illustrative purposes only.

[0171] As described above, the calculation of the hologram of the target picture involves calculating the sub-holograms of the image points of each cell. Similar to the previous example, the sub-holograms of the cells in the outer portion 1202 are calculated at full resolution using conventional point cloud technology (i.e., no undersampling and subsequent interpolation are performed to achieve full resolution). The sub-holograms of the respective cells in the first to third central portions 1204 to 1208 are calculated using the method according to the present disclosure (including first undersampling and subsequent interpolation). The range of interpolation increases as the central portion progresses.

[0172] In one example, to determine the sub-hologram of the image points of the cells in the first central portion 1204, it is first necessary to use the point cloud method to determine every other pixel (i.e., every second pixel) value. To determine the sub-hologram of the image points of the cells in the second central portion 1206, it is first necessary to use the point cloud method to determine every third pixel value. To determine the sub-hologram of the image points of the cells in the third central portion 1208, it is first necessary to use the point cloud method to determine every fourth pixel value. Such sub-holograms (before interpolation) are shown in FIGS. 13A to 13C. In particular, FIG. 13A shows a part of the sub-hologram 1302 of the cells in the first central portion 1204, and the sub-hologram value 1304 is determined for every other cell. There are undetermined or intermediate pixels 1306 (in both the x and y directions) between each value 1304. FIG. 13B shows a part of the sub-hologram 1312 of the cells in the second central portion 1206, and the sub-hologram value 1314 is determined for every third cell. There are two undetermined or intermediate pixels 1316 (in both the x and y directions) between each value 1314. FIG. 13C shows a part of the sub-hologram 1322 of the cells in the third central portion 1208, and the sub-hologram value 1324 is determined for every fourth pixel. There are three undetermined or intermediate pixels 1326 (in both the x and y directions) between each value 1324.

[0173] More generally, determining the sub-hologram of the image point of a cell can be described as including the step of determining a value for each n-th pixel of the hologram using the point group method. In the above example, n for the first central portion 1204 is 2, n for the second central portion 1206 is 3, and n for the third central portion 1208 is 4. It should be understood that these values of n are merely examples. Furthermore, the fact that there are three central portions is also an example. The number of central portions is arbitrary, and the value of n is arbitrary.

[0174] In summary, FIGS. 12 and 13A - 13C show an example where the value of n depends on the position of the cell. Specifically, n increases for cells closer to the center of the target picture. Second example

[0175] In the second example, the first subset of image points, each having associated diffraction and resulting in a relatively low spatial frequency when used in the point group method, is achieved by processing the first image point to reduce the angle of the propagation path. In particular, instead of propagating a wave along the propagation path from the image point towards the viewer's entrance aperture (where the propagation path forms a relatively large angle with the display device), another propagation path is used. The propagation path used in the point group method is arranged to form a small angle with the display device. This means that the starting point of the propagation path is not an image point within the target image. In other words, the first subset of image points is effectively moved within the target image. FIGS. 14A and 14B show this for a single image point.

[0176] Figures 14A and 14B both represent point group hologram calculations for a single image point. Both Figures 14A and 14B are composed of a point group 1402 and a display device 1404. The entrance pupil is not shown in Figures 14A and 14B. The point group in Figure 14A is composed of the first image point 610 in Figure 6, from which a ray path 1420 extends. The ray path 1420 corresponds to the propagation path 1420 in Figure 6 and, in this example, is the path determined using ray tracing through the display device 1404 from the first image point 710 to the entrance aperture (not shown in Figure 8A). The wave of the hologram does not propagate along the ray path 1420. Figure 14A shows an angle 1422 formed by the ray path 1420 and the normal of the display device 1404. The angle 1422 is exaggerated in Figure 14A. The point is that the angle is not zero and approaches the diffraction limit of the display device 1404.

[0177] Figure 14B shows the actual propagation path 830 (not the ray path 1420) used in the point group hologram calculation. The propagation path 1430 is arranged to be substantially perpendicular to the display device 1404. In other words, the angle 1422 is reduced (to almost zero). This means that the propagation path 1430 does not extend from the first image point 610. Instead, the propagation path 1430 has a starting point 1432 indicated by the dashed line in Figure 8B. When the wave is propagated along the propagation path 1430, complex values and phase values are determined on the display device 1404 or the hologram plane in a manner well known to those skilled in the art. The calculated complex number / phase values are actually the hologram values of the image point at the starting point 1432 of the propagation path 1430, not the values of the image point at the position of the first image point 610. Therefore, it can be said that the first image point 610 is substantially moved to the starting point 1432 as part of the hologram calculation method. This movement is represented by the arrow 1434 in Figure 14B. The hologram of the image point at the starting point 1432 has a relatively very low spatial frequency and actually resembles the hologram in Figure 7B rather than the hologram in Figure 7A.

[0178] If no further processing is performed, the hologram that is calculated as described in connection with FIG. 14B, displayed on the display device, and appropriately illuminated forms a reconstruction of a first image point 610 substantially at the center of the system's field of view. This does not coincide with the position of the first image point 610 within the target image that is located towards the end of the point cloud 1402 (as shown in FIGS. 6 and 14A). The inventors recognize that the original or intended position within the point cloud 1402 can be recovered by further processing the hologram to move the reconstructed points. For example, one or more grating functions or phase ramp functions can be added to the hologram. In some examples, a first grating function can be added to the hologram to move the reconstructed points in a first dimension (e.g., the x-direction), and a second grating function can be added to the hologram to move the reconstructed points in a second dimension (e.g., the y-direction). Transforming the image points using a phase ramp function will be well known to the skilled reader. Since the reduction in angle is already known from the hologram calculation process, it is straightforward to select a gradient function or phase ramp function that corrects for the reduction in the angle of the propagation path 1430. Thus, the phase ramp function that corrects the reduction inversely can be easily determined / selected.

[0179] The second example can be used to calculate the sub - holograms of the cells of the target picture in the same way as described above in connection with FIGS. 9 and 10. In particular, this method includes a first step of dividing the target picture into a plurality of parts or cells. In this example, dividing the target picture into a plurality of cells includes defining the angular range from the viewer to the target picture (using ray - tracing). This step is performed virtually. Adjacent angular ranges are adjacent to each other (touch each other) and do not overlap. In this example, each angular range is equal in extent in both the x - direction and the y - direction to all other angular ranges. In other words, each cell is substantially of equal size. In this example, this method further includes identifying which image points of the target picture 900 correspond to which cells. In this example, some cells (such as cell 1002) do not contain image points. Other cells (such as cell 1004) contain image points. In this method, cells that do not contain image points are substantially ignored. The image points contained in an individual cell may be referred to as a subset of the image points.

[0180] In some examples, the method of calculating the hologram includes calculating the sub - hologram of each cell (or at least each cell containing image points). In other words, this method includes calculating the sub - hologram of each subset of the image points. In some examples, the approach of the second example is applied to a plurality of cells. In some examples, the approach of the second example is applied to substantially all cells (perhaps excluding the cell exactly at the center of the target picture which is already as close to the center as possible).

[0181] The advantage of the second example compared to the first example is that the second example can be used to calculate holograms or sub-holograms for all image points of the target picture at originally low spatial frequencies. In particular, since the second example can be applied to the image points displayed in the outermost part of the target picture, the spatial frequencies of the holograms of these points become as low as those of the holograms of the image points closer to the center. Another advantage of the second example compared to the first example is that the undersampling range can always be maximized. For example, as explained above in relation to the first example, it is necessary to balance between maximizing interpolation without causing aliasing problems. "n" (i.e., the parameter that defines the ratio of pixels calculated by the dot pattern method) increases as the image point approaches the center (therefore, various zones or parts are shown in FIG. 12). However, in the second example, regardless of where the image point actually is within the target picture, the dot pattern method can be executed as if the image point were near the center of the target picture. That is, before applying the dot pattern method, any image point can be arbitrarily brought closer to the center of the target picture. Therefore, in the second example, n can be maximized for all image points.

[0182] In some examples, the hologram calculation may include a combination of the above two examples. For example, the first example is applied to a subset of the image points close to the center of the target picture, and the second example is applied to a subset of the image points close to the outer periphery of the target picture. For example, referring to FIG. 11, the first example is applied to a subset of the image points within the cells of the central portion 1104, and the second example is applied to a subset of the image points within the cells of the outer portion 1102. Implementation of the improved point cloud method

[0183] So far, examples of the improved point group method have been described in relatively general terms. In particular, several examples have been described for calculating a hologram (or sub-hologram) of a subset of the first image points of the target picture. In these examples, since the diffraction angle associated with the subset of the first image points is smaller than the maximum diffraction angle of the display device, the spatial frequency of the hologram is sufficiently low, and the hologram can be first undersampled and then interpolated to full resolution. Here, specific details of examples implementing the improved point group method will be described. These examples are particularly advantageous. The inventors have developed these methods to be highly computationally efficient.

[0184] Specific implementation examples of the improved point cloud hologram include three accumulators or integrators. Each accumulator is for executing an iterative process included in hologram calculation. As described above, the first step of the method for calculating a hologram includes dividing a target picture into an array of cells, where each cell includes a plurality of image points. Generally speaking, the first accumulator is configured to iterate over all the image points of a subset of the image points (for example, iterate over all the image points within the first cell of the array of cells). Thus, the first accumulator determines the hologram value for each subset of the image points or for the cells. In this example, the hologram determined by the first accumulator (using point cloud technology) is undersampled in both the first direction and the second direction (for example, the x direction and the y direction). In this example, the first accumulator is configured to interpolate in the first direction but not in the second direction. The second accumulator is configured to repeat the first accumulation for each cell of an array of cells extending in the first direction (that is, for each cell of a row or column of cells). The second accumulator is configured to receive and combine the hologram values determined in each iteration of the first accumulator. Next, the second accumulator is configured to interpolate the values in the second direction. The third accumulator is configured to repeat the second accumulation for a set of cells of the array of cells extending in the first direction. Thus, if the second accumulator is configured to repeat the first accumulation for the first cell row, the third accumulator repeats the second accumulation for each remaining cell row, and vice versa. After the third accumulation, a full-resolution hologram of the complete target picture is output.

[0185] The above implementation is excellent in calculation efficiency. It minimizes the number of interpolation steps that need to be executed to realize a full-resolution hologram and efficiently implements the above improved hologram calculation method (including undersampling and interpolation).

[0186] Furthermore, with this implementation, it is possible to apply a grating function (or a phase ramp function) to the hologram while minimizing the number of interpolation steps to be executed. In particular, with this implementation, different grating functions can be applied to sub-holograms of different cells or portions of the target picture. As is well known to those skilled in the art, it may be desirable to apply a grating function to the hologram in order to achieve a beam steering effect. This is to move the content within the playback field of the holographic projector as needed. For example, this movement may be based on an instruction from the user. The movement may be, for example, to correct or calibrate a physical change of the holographic projector resulting from, for example, a temperature change of the holographic projector or a distortion of the optical combiner. In this case, it may be necessary to apply different grating functions to the (sub)holograms of the image points included in different cells (for example, because the distortion may be non-uniform). Another example where different grating functions need to be applied to different cells is when using a grating function to correct the movement of the image points of the target picture by the point group method (see FIGS. 14A and 14B). In such a case, the image points of different cells may be moved by different amounts in order to move to approximately the center of the target picture. In any case, the inventors recognize that it is not possible to simply add a grating function at the end.

[0187] In any case, the inventors recognized that the required movement can be divided into the x and y directions. A first grating function can be used to achieve the required movement in the x direction. A second grating function can be used to achieve the required movement in the y direction. The inventors have developed a way to perform this in a computationally efficient manner.

[0188] The implementation (including three accumulators) will be described in more detail with reference to FIG. 15 and later. First accumulator

[0189] Figure 15 is a flow diagram representing a first accumulation 1500 applied to individual cells of a target picture. In the following description, the first accumulation 1500 is applied to a cell 1602 of a target picture 900 as shown in Figure 16A. In Figure 16, the cell 1602 is shown in gray for illustrative purposes. The cell 1602 is composed of image points that form part number 9.

[0190] Step 1502 of the first accumulation 1500 includes setting a counter j to zero. This counter is stored in the memory of a hologram engine (not shown in the figure) arranged to execute a hologram calculation method.

[0191] Step 1504 of the first accumulation 1500 includes comparing the current value of the counter j with the number x of image points included in the cell 1602. Optionally, the first accumulation includes determining the number x of image points. Alternatively, the number x of image points in the cell 1602 may have been previously determined or specified and may be included in the memory of the hologram engine. For example, the number of image points in each cell may have been previously specified when the target picture was divided into an array of cells. The hologram engine may be configured to count the number of image points in each cell and store that count in the memory.

[0192] If j is less than x, the first accumulation moves to step 1506. If j is greater than or equal to x, the first accumulation moves to step 1512.

[0193] The first accumulation step 1506 of 1500 involves (virtually) propagating a wave along a propagation path associated with the j-th image point of the first cell 1602 using a point cloud method. In some examples, the propagation path is defined from the j-th. The image points of the first cell 1602 are projected onto the aperture of a possible viewing system determined using a line-of-sight tracking device. This is consistent with what was described in FIG. 6. In other examples, some processing may be applied such that the angle between the propagation path and the display device decreases. In practice, the j-th image point may be moved so as to approach or substantially be at the center of the target picture 900. This is consistent with the examples described in relation to FIGS. 14A and 14B. The processing of the j-th image point and / or the associated propagation path may be performed before the first accumulation or during the first accumulation step 1506. In other examples, each propagation path associated with the j-th image point used in the point cloud method is made to form an angle smaller than the maximum diffraction angle of the display devices 604, 1404, or optionally significantly smaller. The maximum diffraction angle of the display devices 604, 1404 is determined by the pixel pitch of the display device and is defined with respect to the normal of the display device. Optionally, the angle may be made substantially zero by each propagation path and the display device. In any case, the angle of the propagation path associated with the j-th image point is made to form an angle smaller than the maximum diffraction angle of the display devices 604, 1404, or optionally significantly smaller. The maximum diffraction angle of the display devices 604, 1404 is determined by the pixel pitch of the display device and is defined with respect to the normal of the display device. Optionally, the angle may be made substantially zero by each propagation path and the display device. The image points are those where the spatial frequency of the hologram values is relatively low. Next, in this example, the hologram values are determined for every fifth pixel in both the x - direction and the y - direction of the display device. In other words, the hologram values determined in step 1506 are determined at a lower resolution in both the x - direction and the y - direction compared to the native resolution of the display devices 604, 1604. In other words, the hologram values are undersampled.

[0194] The output of step 1506 is shown in FIG. 16B. FIG. 16B shows a part of the hologram where the hologram values are determined for every fifth pixel. The pixel 1614 for which the hologram value was determined in step 1506 is represented by a black square in FIG. 16B. The pixel 1616 for which the hologram value was not determined in step 1506 is represented by a white square in FIG. 16B.

[0195] Step 1508 of the first accumulation 1500 includes storing the hologram values output in step 1506 in the memory of the hologram engine. If the hologram value of each pixel is already stored in the memory (because the iteration of the first accumulation has already been executed for the previous image point 1614), step 1508 includes adding (i.e., accumulating or summing or overlaying) the hologram values of each pixel of the j - th image point 1614. The image point is converted to the currently stored hologram value of that pixel. Thus, each value of each pixel becomes the running total that combines the values calculated for that pixel for each image point.

[0196] Step 1510 of the first accumulation includes incrementing the counter j by 1. Thereafter, the first accumulation returns to step 1504.

[0197] The wave propagates along the j-th propagation axis. The values of the image point and the hologram are determined for the j-th. The image point is drawn with reduced resolution in both x and y. In each iteration, new hologram values are determined in step 1506. These are added to the hologram values already stored in memory in step 1508. Thus, the hologram values of pixel 1614 are accumulated in memory over time (thus, the term "accumulator" is used). The first accumulator continues to repeat steps 1504 to 1510 for each image point until j becomes equal to x. When j becomes equal to x, the first accumulator moves to step 1512 instead of step 1506.

[0198] In this example, step 1512 of the first accumulator 1500 includes performing interpolation only in the x direction. In the example, a bicubic interpolation method can be used for the interpolation. The interpolation is used to determine the hologram values of pixel 1616 in the x direction between pixels 1614. This result is represented in FIG. 16C where the pixels 1614 for which the hologram values are determined spread over consecutive rows. In other examples, step 1512 of the first accumulator 1500 includes performing interpolation only in the y direction.

[0199] In this example, step 1514 of the first accumulator 1500 includes applying the first lattice function to the pixels 1614 in FIG. 16C (but not to the pixels 1616 in FIG. 16C between the rows of pixels 1614). It is important to perform the interpolation in step 1512 before applying the first lattice function in step 1514. This is because the first lattice function is a substantially high-frequency function. Thus, if the first lattice function is applied before the interpolation, aliasing problems may occur due to the high frequency of the first lattice function during the interpolation.

[0200] In step 1516 of the first accumulator 1500, the hologram value from step 1514 is output to the second accumulator, and the memory used by the first accumulator is cleared. The hologram value output in step 1516 has substantially full resolution in the x direction (since interpolation is performed in the x direction), but the resolution decreases in the y direction. The hologram value also includes the first lattice function.

[0201] When the output of step 1516 is displayed on a display device (such as liquid crystal on a silicon spatial light modulator) and appropriately illuminated, a holographic reconstruction of the image points of cell 1602 is formed. This is shown in FIG. 16D.

[0202] In step 1518 of the first accumulation 1500, the memory used in the first accumulation is cleared. Second accumulator

[0203] FIG. 17 is a flow diagram representing a second accumulation 1700 applied to a row of cells of a target picture. In the following description, the second accumulation 1700 is applied to row 1902 of cells of target picture 900, as shown in FIG. 19A. In FIG. 19, row 1902 of cells is shown in gray for illustration purposes. Row 1902 of cells is composed of image points that form the upper part of the digit 9 and the upper part of two boundary graphics.

[0204] Step 1702 of the second accumulation 1700 includes setting counter k to zero. This counter is stored in the memory of a hologram engine (not shown in the figure) configured to execute a hologram calculation method.

[0205] Step 1704 of the second accumulation 1700 consists of comparing the current value of counter k with the number y of cells in cell row 1902. In this example, there are 8 individual cells in cell row 1902. Thus, in this example, y = 8.

[0206] If k is less than y, the second accumulation 1700 moves to step 1706. If k is greater than or equal to y, the second accumulation 1700 moves to step 1712.

[0207] Step 1706 of the second accumulation includes performing the first accumulation 1500 on the k-th cell of cell row 1902. Thus, the second accumulation 1700 substantially includes performing the first accumulation 1500 (optionally, for multiple iterations).

[0208] According to step 1516 of the first accumulation 1500, the hologram value determined in the first accumulation 1500 is output to the second accumulator. Thus, step 1708 of the second accumulation 1700 includes receiving the hologram value determined in the first accumulation. Step 1708 of the second accumulation 1700 includes storing the hologram value in the memory of the hologram engine. If the hologram value of each pixel is already stored in the memory (because the first accumulation has already been performed on the cells before the cell row 1902), step 1708 includes adding (i.e., accumulating or summing or overlaying) the hologram value of each pixel 1614 to the currently stored hologram value of that pixel. Thus, each value of each pixel may become the running total that combines the values calculated for that pixel for each cell of the cell row.

[0209] In step 1710 of the second accumulation, the counter k is incremented by 1. Thereafter, the second accumulation returns to step 1704.

[0210] Steps 1704 through 1710 of the second accumulation 1700 are repeated for each cell k of cell row 1902. In each iteration, the first accumulation is performed on the k-th cell, a new hologram value is determined in the first accumulation, and added to the hologram value already stored in the memory of the hologram engine for the second accumulation.

[0211] The hologram values at the second accumulation step 1708 are shown in FIG. 19B. Since the hologram values determined at step 1708 correspond to the output of the first accumulation, FIG. 19B corresponds to FIG. 16B. In other words, FIG. 19B shows a part of the hologram in which the hologram values are determined for every fifth pixel. The pixel 1914 for which the hologram value was determined at step 1708 is represented by a black square in FIG. 19B. The pixel 1916 for which the hologram value has not been determined is represented by a white square in FIG. 19B.

[0212] The second accumulator continues to repeat steps 1704 to 1710 for each cell until k becomes equal to y. When k becomes equal to y, the second accumulator moves to step 1712 instead of step 1706.

[0213] In this example, step 1712 of the second accumulation 1700 includes performing interpolation in the direction opposite to that performed by the first accumulator. In this example, since the interpolation of the first accumulation was only in the x direction, the interpolation performed by the second accumulation is only in the y direction in this example. In the example, a bicubic interpolation method can be used for interpolation. The interpolation is used to determine the hologram values of the pixels 1616 in the y direction between the rows of the pixels 1614. The result is represented in FIG. 19C where the pixels 1614 for which the hologram values have been determined spread into a continuous two-dimensional block. In other examples, step 1712 of the second accumulation 1700 includes performing interpolation only in the y direction (when the interpolation performed in the first accumulation is only in the x direction).

[0214] In this example, step 1714 of the second accumulation 1700 includes applying a second lattice function to the pixels 1614 in FIG. 19C. It is important to perform the interpolation in step 1712 before applying the second lattice function in step 1714. This is because the second lattice function can potentially be a high-frequency function. Thus, if the second lattice function is applied before interpolation, aliasing problems may occur in the interpolation due to the high frequency of the second lattice function.

[0215] In step 1716 of the second accumulation 1700, the hologram value of step 1714 is output to the third accumulation, and the memory used by the second accumulator is cleared. The hologram value output in step 1516 has substantially full resolution in the x and y directions (since interpolation is performed in the x and y directions in the first and second accumulations respectively). The hologram value also includes one or more lattice functions.

[0216] When the output of step 1716 is displayed on a display device (such as liquid crystal on a silicon spatial light modulator) and appropriately illuminated, a holographic reconstruction of row 1702 of the cells is formed. This is shown in FIG. 19D. In step 1718 of the second accumulation 1700, the memory for the second accumulation is cleared. Third accumulator

[0217] FIG. 18 is a flow diagram representing a third accumulation 1800 applied to a 2D array of cells of a target picture. In the following description, the third accumulation 1800 is applied to a 2D array 2002 of cells of a target picture 900, as shown in FIG. 20A. In FIG. 20A, for the sake of explanation, the 2D array 2002 of cells is shown in gray. The 2D array 2002 of cells corresponds to the cells of the central portion 1104 of FIG. 11.

[0218] Step 1802 of the third accumulation 1800 includes setting counter l to zero. This counter is stored in the memory of a hologram engine (not shown in the figure) configured to execute a hologram calculation method. of the memory.

[0219] Step 1804 of the third accumulation 1800 consists of comparing the current value 1 of the counter with the number of rows z of the cells of the 2D array 2002. In this example, there are four individual cell rows in the array 2002. Therefore, in this example, z = 4.

[0220] If l is less than z, the third accumulation 1800 moves to step 1806. If l is greater than or equal to z, the third accumulation 1800 moves to step 1812. Step 1806 of the third accumulation 1800 includes performing the second accumulation 1700 on the cells in the l-th row of 2002. Thus, the third accumulation 1800 substantially includes performing the second accumulation 1700 (optionally, for multiple iterations).

[0221] According to step 1716 of the second accumulation 1700, the hologram value determined in the second accumulation 1700 is output to the third accumulator. Thus, step 1808 of the third accumulation 1800 includes receiving the hologram value determined in the second accumulation. Step 1808 of the third accumulation 1800 includes storing the hologram value in the memory of the hologram engine. If the hologram value for each pixel is already stored in the memory (because the second accumulation has already been performed on the previous cell row of array 2002), step 1808 includes adding (i.e., accumulating, summing, or overlaying) the hologram value of each pixel 1614 to the currently stored hologram value of that pixel. Thus, each value of each pixel may become the running total that combines the values calculated for that pixel for each cell in the cell row.

[0222] In step 1810 of the second accumulation, the counter l is incremented by 1. Thereafter, the third accumulation 1800 returns to step 1804.

[0223] Steps 1804 through 1810 of the third accumulation 1800 are repeated for each cell l in cell row 2002. In each iteration, the second accumulation is performed on the l-th cell row, a new hologram value is determined in the second accumulation, and is added to the hologram value already stored in the memory of the hologram engine for the third accumulation.

[0224] The hologram values at the third accumulation step 1808 are shown in FIG. 20B. Since the hologram values determined at step 1808 correspond to the output of the second accumulation, FIG. 20B corresponds to FIG. 19C.

[0225] Accumulation continues to repeat steps 1804 to 1810 for each cell until l is equal to z. When l is equal to z, the third accumulation 1800 moves to step 1812 instead of step 1806.

[0226] Step 1812 of the third accumulation 1800 is to output a hologram. The hologram is at full resolution in the x and y directions. The hologram values also include one or more lattice functions.

[0227] When the output of step 1812 is displayed on a display device (such as liquid crystal on a silicon spatial light modulator) and appropriately illuminated, a holographic reconstruction of the image points of cell row 1902 is formed. This is shown in FIG. 20C.

[0228] Step 1814 of the third accumulation 1800 includes clearing the memory used for the third accumulation.

[0229] With the three accumulators described above, the hologram of the cell array 1802 shown in FIG. 20A corresponding to the central portion shown in FIG. 11 can be determined. In this portion, since the hologram values associated with the image points have a sufficiently low spatial frequency, the hologram values may be undersampled first. In the example, the hologram values of the remaining cells corresponding to the outer portion shown in FIG. 11 can be calculated using a dot - matrix hologram by a conventional method (such as without undersampling or interpolation). These values can be added to the hologram output at step 1812 of the third accumulation.

[0230] In some examples, the array of cells can include all the cells of the target picture. Thus, the first accumulator is repeatedly applied (via the second and third accumulators) to each cell of the array of cells. In such embodiments, prior to applying the accumulation, the image points of each cell can be moved or otherwise processed to bring them closer to the center of the target picture. Next, the first and second lattice functions can be used to correct that processing. The skilled reader will understand that different first and second lattice functions may need to be applied to each cell (in the first and second interpolations) to return the cells and / or rows of cells to their respective original positions.

[0231] In some examples, undersampling may vary from cell to cell, according to what was described in connection with FIG. 12. Additional functions

[0232] The methods and processes described herein can be incorporated into a computer-readable medium. The term "computer-readable medium" includes media configured to temporarily or permanently store data, such as random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, cache memory, and the like. Also, the term "computer-readable medium" is to be construed to include any medium, or combination of media, capable of storing instructions for execution by a machine, which instructions, when executed by one or more processors, cause the machine to perform all or part of one or more of the methods described herein.

[0233] The term "computer-readable medium" includes cloud-based storage systems. The term "computer-readable medium" includes, but is not limited to, one or more tangible and non-transitory data repositories (such as data volumes) such as solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof. In some embodiments, the executable instructions may be transmitted by a carrier medium. Examples of such carrier media include transient media (such as a propagated signal that transmits instructions).

[0234] It will be apparent to those skilled in the art that various modifications and changes can be made without departing from the scope of the appended claims. The present disclosure covers all modifications and changes within the scope of the appended claims and their equivalents.

Claims

1. 1. A hologram engine for computing a hologram of a target picture comprising a plurality of image points, the hologram engine calculating for a first subset of the image points: determining a value for every nth pixel of a display device for displaying the hologram using a point cloud method, n being greater than 1; performing a first interpolation to determine values ​​of at least some of the other pixels; configured to perform a first data processing stage comprising: a hologram engine, wherein each image point of the first subset has an associated diffraction angle for use in the point cloud method, each diffraction angle being less than a maximum diffraction angle of the display device.

2. Determining the value of every n-th pixel determining, for each image point of the first subset, a value of the nth pixel of the display device using a point cloud method; combining the values ​​determined for each image point at each of the nth pixels; The hologram engine of claim 1 , wherein the hologram engine is configured to include:

3. The hologram engine of claim 2 , wherein the first interpolation is based on a combined value determined for every nth pixel.

4. A hologram engine as described in any one of claims 1 to 3, wherein the value determined for each image point using the point cloud method represents the contribution of a wave propagated along a path that makes an angle with the display device equal to the associated diffraction angle of the respective image point.

5. The diffraction angle associated with each image point of the first subset is: by a first path from the image point to a viewer's entrance pupil; or by a second path, in which each of the image points is positioned as if translated closer to a center of the target picture such that the second path makes an angle with the display device that is reduced relative to the angle made by each of the first paths for that image point; A hologram engine as claimed in any one of claims 1 to 4 as defined above.

6. 6. The hologram engine of claim 1, wherein the maximum diffraction angle of the display device defines a replay field including a central portion surrounded by an outer portion, the central portion including a center of the replay field, and the central portion occupies less than 50% of the replay field.

7. 7. The hologram engine of claim 6, wherein the first subset of image points is included in the central portion of the replay field, or the point cloud method is arranged as if the first subset of image points were translated from the outer portion of the replay field to the central portion.

8. 8. The hologram engine of claim 1, wherein the hologram engine is configured such that at least a portion of the pixels whose values ​​are determined using a point cloud method are separated in a first dimension by one or more first intermediate pixels, and the first interpolation is performed in the first dimension to determine values ​​for the first intermediate pixels.

9. 9. The hologram engine of claim 8, wherein the first data processing stage further comprises applying a first diffraction grating function configured to translate light incident on the hologram engine in a first direction parallel to the first dimension, the diffraction grating function being applied after the first interpolation.

10. the hologram comprising an array of pixels extending in the first dimension and a second dimension perpendicular to the first dimension; 10. The hologram engine of claim 1, wherein the hologram engine is configured such that at least some of the pixels whose values ​​are determined using point cloud methods are separated in the second dimension by one or more second intermediate pixels.

11. 11. The hologram engine of claim 10, wherein the hologram engine is configured to determine a value for the second intermediate pixel by performing a second interpolation in the second dimension, the hologram engine being configured to perform the second interpolation after the first interpolation.

12. The hologram engine is configured to perform a second data processing stage, the second data processing stage comprising: receiving the values ​​determined in the first data processing stage; repeating the first data processing step for k subsets of image points of the target picture and receiving the respective values, each of the k subsets of image points corresponding to a discrete angular range defined by a first corner in the first dimension and a second corner in the second dimension, the second corner being the same for each of the k subsets; The hologram engine of claim 1 , comprising:

13. 13. The hologram engine of claim 12, wherein the hologram engine is further configured to combine values ​​determined for each of the nth pixel and the first intermediate pixel during the second data processing stage.

14. The hologram engine of claim 12 when dependent on claim 11, wherein the second data processing stage includes the second interpolation, and the hologram engine is configured such that the second interpolation is performed after the step of combining the values ​​determined during the second data processing stage.

15. 15. The hologram engine of claim 12, wherein the second data processing stage further comprises applying a second diffraction grating function configured to translate light incident on the hologram engine in a second direction parallel to the second dimension.

16. The hologram engine of claim 15 when dependent on claim 14, wherein the hologram engine is configured such that the second data processing stage includes applying the second diffraction grating function after performing the second interpolation.

17. The hologram engine is configured to perform a third data processing stage, the third data processing stage comprising: receiving said values ​​determined in said second data processing step; repeating the second data processing step for l subsets of image points of the target picture to receive the respective values, each of the l subsets of image points corresponding to a discrete angular range defined by a first corner in the first dimension and a second corner in the second dimension, the second corner being different each time the second data processing step is repeated; 17. The hologram engine of claim 12, comprising:

18. 18. A hologram engine according to claim 1, wherein the hologram engine is configured to divide the target picture into an array of cells, each cell containing a subset of image points.

19. 20. The hologram engine of claim 18, wherein the first data processing stage is first performed on the image point of a first cell of the array of cells.

20. 20. A hologram engine as claimed in claim 18 or 19 when dependent on claim 12, wherein the second data processing stage comprises repeating the first data processing stage for each cell in a column or row of cells of the array of cells that includes the first cell.

21. 21. The hologram engine of claim 20 when dependent on claim 17, wherein the third data processing stage includes repeating the first and second data processing stages for cells in each column or row of the array of cells.

22. 1. A method for computing a hologram of a target picture comprising a plurality of image points, the method comprising the steps of: for a first subset of the image points: determining a value for every nth pixel of a display device for displaying the hologram using a point cloud method, n being greater than 1; performing a first interpolation to determine values ​​of at least some of the other pixels; performing a first data processing step including: A method wherein each image point of the first subset has an associated diffraction angle for use in the point cloud method, each diffraction angle being less than a maximum diffraction angle of the display device.

Images