Method, apparatus, computer program and computer-readable medium for determining a computer-generated hologram

Abstract

A computer-implemented method of determining a computer-generated hologram is described. The method comprises: receiving (1) first data representing a three-dimensional scene from a particular viewpoint; supplementing (5) the first data with additional data representing at least part of the three-dimensional scene that is occluded as a result of the particular viewpoint, thereby to generate second data representing the three-dimensional scene; and determining (6) hologram data using the second data and a diffraction-based occlusion algorithm. A holographic display, computer program and computer-readable medium implementing the method are also described.

Description

[0001] METHOD, APPARATUS, COMPUTER PROGRAM AND COMPUTER-READABLE

[0002] MEDIUM FOR DETERMINING A COMPUTER-GENERATED HOLOGRAM

[0003] Technical Field

[0004] The present invention relates to computer-generated holograms and more specifically to methods, systems and computer programs for improving image fidelity of computer-generated holograms created using occluded source data.

[0005] Background

[0006] Computer-generated holography presents images that preserve depth cues when viewed. In this way, images with depth can be viewed more comfortably, with a viewer focussing naturally at the perceived depth and with the image exhibiting defocus and parallax which is not possible with conventional displays.

[0007] One method proposed in VividQ Limited’s patent application WO2020 / 148521A1 uses image data separated into layers to generate a hologram for display. This has efficiency benefits compared to ray-tracing techniques from a full 3D model and also works well with captured real-world data. The source data typically includes the visible data from a particular viewpoint and does not include data of features which are hidden (occluded) behind other objects in the view. Source data may, for example be a point cloud or so-called “2.5D” or “RGBZ” data where each pixel is associated with a particular depth (the “Z”). For example, depth may be encoded at the capture or rendering stage, or added in postprocessing for 2D images that do not include depth data. Processing the hologram data by separating the source data into layers of different depths is a computationally efficient way of generating holograms, enabling substantially real-time generation of holograms and / or reduced computational resources and consequent reduced power requirements.

[0008] As well as providing correct depth cues to a viewer, holograms have a further advantage that a single hologram inherently encodes parallax data, so that a slight shift of viewing position results in the position of closer objects changing relative to those further away. However, when a computer-generated hologram (CGH) is created from data from a single viewpoint, viewing it from even a slightly different position or angle may reduce the image fidelity. Effects that can reduce image fidelity in CGH using data from a single viewpoint include: • Bright areas, where the shift in view causes a foreground object to overlap the background so the viewer perceives light from both the foreground and background at a single point resulting in increased brightness.

[0009] • Dark areas, where the shift in view causes a foreground object to move so that the previously occluded background is exposed and appears dark or black because nothing is encoded in the hologram for that point.

[0010] • Incorrect appearance of foreground defocus when focussing on the background. Foreground objects may appear sharper than they should do when focussing on the background due to the in-focus outlines of the foreground object’s occlusion being present in the background layer.

[0011] • Incorrect appearance of background defocus when focussing on the foreground. The background will reveal areas of missing information in occluded areas when it defocuses.

[0012] It would be desirable to improve the image fidelity of computer-generated holograms so that one or more of these effects are reduced.

[0013] Summary

[0014] According to a first aspect of the present disclosure, there is provided a computer- implemented method of determining a computer-generated hologram. The method comprises: receiving first data representing a three-dimensional scene from a particular viewpoint; supplementing the first data with additional data representing at least part of the three-dimensional scene that is occluded as a result of the particular viewpoint, thereby to generate second data representing the three-dimensional scene; and determining hologram data using the second data and a diffraction-based occlusion algorithm.

[0015] Many data sources for CGH represent a three-dimensional scene rendered from a particular viewpoint, such as a single perspective, or a single viewpoint. As mentioned above, a CGH inherently encodes parallax, so that the perceived image changes when viewed from a different viewpoint or position. However, because the data used to generate the CGH is from a single viewpoint it does not include data of hidden or occluded objects. The resulting hologram may exhibit one or more of bright areas, dark areas and incorrect defocus as discussed above. The method proposed herein improves image fidelity by encoding data of occluded parts of the scene in the hologram itself. It combines supplementing the first data and diffraction-based occlusion techniques in the generation of the hologram. More specifically, by supplementing the first data with additional data of occluded portions or regions, and then using that second, supplemented data with diffraction-based occlusion to determine the hologram data for display, wavefront effects from the occluded elements can be more accurately reflected in the resultant hologram data. The resulting hologram data exhibits improved image fidelity when viewed both off-axis and on-axis. Bright regions and dark regions are reduced when viewed off-axis and defocus is improved when viewed both on- axis and off-axis.

[0016] Geometric occlusion is typically chosen for rendering source material for CGH because it is computationally efficient. Compared to geometric occlusion, diffraction-based occlusion might seem a poor choice; it is computationally complex, requiring additional Fourier transforms and lensing functions not needed in geometric occlusion. However, the combination of supplementing data with additional data of occluded portions or regions and using diffraction-based occlusion provides significant image quality improvements.

[0017] Any suitable diffraction-based occlusion algorithm can be used. Some examples use a wave-based occlusion algorithm which can be used when determining the hologram data, for example the “ping-pong” algorithm proposed in Ichioka et. al., “ Scanning Halftone Plotter and Computer-Generated Continuous-Tone Hologram". Applied Optics 10, 403-11, 1971. An overview of other suitable wave-based occlusion techniques is in section 3.4.1 of Chen, R. H. (2010). Computer-generated holograms for 3-D holographic display (Doctoral thesis). https: / / doi.or / 10.17863 / CAM.14125. Another example of diffraction-based occlusion is ray-based occlusion, such as discussed in Shi et al. "Towards real-time photorealistic 3D holography with deep neural networks". Nature, Vol 591, 11 March 2021 pp 234-239, available from https: / / doi.org / 10.1038 / s41586-020-03152-0. Shi et al. introduces an “occlusion-aware point-based method” which uses the cone of propagation and a boundary check condition to perform occlusion on any wavefronts. Another ray -based occlusion technique is discussed in “Full parallax three-dimensional computer generated hologram with occlusion effect using ray casting technique" Hao Zhang et al 2013 J. Phys.: Conf. Ser. 415 012048, available from https: / / iopscience.iop.Org / article / T 0.1088 / 1742- 6596 / 415 / 1 / 012048. While the hologram data may be determined in any suitable way, in some examples it comprises a layer-based algorithm, such as the method described in WO2020 / 148521, incorporated herein by reference for all purposes. A layer-based algorithm is an efficient way of determining hologram data. Layer-based algorithms provide additional benefits in combination with supplementing the first data and using a diffraction-based occlusion; the layer-based algorithm may reduce the computational burden allowing more processing to be available for the diffraction-based occlusion and / or reducing overall processing resources required. In some examples, the second data may already be in layer-based form, such as associated with a number of layers, each layer having a predetermined depth from the viewpoint. In other examples, the second data may be processed into layer-based data, such as by grouping pixels, elements or content of similar depth from the viewpoint into a layer. The layers may not be equally spaced, for example being spaced more closely for near depths than far depths from the viewpoint. The layers may also be tilted.

[0018] The diffraction-based occlusion algorithm may use at least one mask determined from the first data or the second data. This can allow the diffraction-based occlusion to be applied more efficiently only to regions where diffraction-based occlusion will change the hologram data, such as around the edges of an occluded region or portion. The mask may be an alpha mask, such as a mask of parts of the first or second data at a given depth. While the mask may be for parts at a closer depth, this is not always the case when reflective and refractions are considered. In some examples a binary alpha mask can give good results, while other examples use alpha masks with more gradations, such as to improve anti-aliasing around edges or encode other semi-transparencies. In some examples an alpha mask may be built up by moving from closest to furthest depth in the data. For example, data at a first, closest depth data is generally not occluded, but data at a second, next furthest depth may be occluded by at least part of the data at the first, closest depth data. Data at a third, next further depth from the second depth may be occluded by at least part of the data at the first depth and / or second depth. In this way, there may be one or more masks associated with particular depths. Some examples may also use three-dimensional or tilted masks.

[0019] In some examples, determining hologram data comprises determining hologram data with a layer-based algorithm and then applying a diffraction-based occlusion algorithm using at least one mask. The first data may comprise or be associated with depth data. For example, the first data may have been pre-rendered or recorded as a “point cloud”, “2.5D” or “RGBZ” data, wherein elements such as pixels have a depth as well as spectrum / colour values. Some examples may use an associated depth map. The first data may also be rendered in real-time, such as for gaming, virtual reality, augmented reality, or head-up displays. The first data may inherently be geometrically occluded, such as when captured by a camera, or be generated using geometric occlusion techniques, such as rendering a 3D model from a viewpoint.

[0020] In other examples, depth data may be determined based on the first data, whether or not the first data already comprises or has associated depth data. This may be useful to add depth data where there is none initially, or to improve the resolution of depth data. For example, content analysis techniques can be used to identify a distance from the viewpoint to parts of the data and associate a view. Temporal data may also be used in some examples, such as how the data changes over time as relative positions change from the viewpoint.

[0021] A mask determined from the first data and the depth data may be used in the supplementing the first data. This may have any of the features discussed above for the possible mask in the occlusion algorithm, and the same mask may be used for both supplementing the first data and the diffraction-based occlusion algorithm. When a mask is used in the supplementing the data, it can define an area, region or portion for supplementing with additional data. The additional data may be generated by image or data synthesis. At its simplest, it could simply fill with a single colour value, such as the average of other occupied pixels in a layer. Other examples include image blurring, colour stretching, image pyramid algorithms, Content-Aware Fill, or neural network generated imagery. Another example is the algorithm described in “PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing", Barnes et al, 2009 available from https: / / pixl.cs.princeton.edu / pubs / Bames 2009 PAR / patchmatch.pdf. Where the image data comprises a plurality of frames, temporal methods may also be used to supplement the first data.

[0022] Supplementing the first data may use a first algorithm at a first depth and a second algorithm at a second depth, wherein the first algorithm is different from the second algorithm and the first depth is different from the second depth. The processing resources required to supplement the first data varies greatly between different methods and it can be useful to improve computational efficiency by using higher quality, more computationally intensive algorithms for generating additional data where the resulting increase in quality is more likely to be perceived. For example, closer data may be supplemented using higher quality algorithms than those of data at a further depth. In other examples, a depth may determine an algorithm for supplementing the first data.

[0023] In some examples, supplementing the first data may use a machine learning algorithm having an input of the first data and an output of the second data. The machine learning algorithm may be trained with a library of data that contains “flawed” input data (the first data), either 2.5D, volumetric / points etc, which has missing data for occlusions and corresponding “perfect” scene data (the second data) including the occluded sections. Such training data can be generated from computer models encoding a three-dimensional scene, for example.

[0024] Another aspect may use a machine learning algorithm which receives first data representing a three-dimensional scene from a particular viewpoint and outputs hologram data for display. Such a machine learning algorithm may be trained with a library of data that contains “flawed” input data (the first data), either 2.5D, volumetric / points etc, which has missing data for occlusions and corresponding “perfect” hologram data (the hologram data), such as a hologram generated using ray-tracing or diffraction-based occlusion techniques including the missing data from the first data. Such training data can be generated from computer models encoding a three-dimensional scene, for example.

[0025] The computer generated hologram generated by the methods discussed herein may be a hologram forming part of a multiview hologram. For example, the computer-generated hologram may be a sub-hologram or hogel of a multiview hologram. A multiview hologram includes a plurality of holograms from different viewpoints displayed so that a viewer perceives the hologram closest to their actual viewpoint. Multiview holograms improve image fidelity because the viewer cannot be as far “off-axis” as when viewing a single hologram.

[0026] When used with multiview holograms, the methods discussed herein combine improved image fidelity with reduced processing. The multiview hologram can comprise fewer sub -holograms, also referred to as hogels, when those are generated according to the methods discussed herein because image fidelity is improved across a wider range of viewing angles. As well as reducing processing resources, this can also improve resolution, such as when the hogels are spatially tiled because each can occupy a larger area. This is a surprising result, because multiview systems generally add more views to improve image fidelity.

[0027] The method may further comprise displaying the hologram data. Alternatively, or additionally, the hologram data may be stored for later display. Any suitable computer readable storage medium can be used to store the hologram data, if required.

[0028] The methods discussed above can be applied to a holographic display system comprising: an illumination source emitting at least partially coherent light; a spatial light modulator illuminated by the illumination source; an optical system for focussing the output of the spatial light modulator; and a controller. The controller is configured to implement the method described above, with or without optional features also described, and cause the spatial light modulator to display the hologram data.

[0029] The holographic display system may form part of a headset for augmented reality or virtual reality. Some examples may include two holographic display systems, one for a right eye and one for a left eye. The holographic display system may also form part of a large screen display. A “large screen display” is used to refer to a screen which may be viewed by both a viewer’s eyes at once, with a typical human interpupillary distance of 60mm to 70mm. Such a “large screen display” may have a largest dimension, such as diagonal measurement of a rectangular display, of at least 10cm, 12.5cm, or 15cm (4 inches, 5 inches or 6 inches). The holographic system may also form part of a heads-up display, such as for a vehicle driver or pilot.

[0030] In another aspect, a computer program is provided which comprises computer program code that, when executed by a processor, causes the processor to implement the method described above, with or without optional features also described. Some examples include a computer-readable medium, such as a non-transitory computer-readable medium, comprising instructions embodied thereon that, when executed by a processor, instruct the processor to perform the method described above, with or without optional features also described. This may allow retro-fitting to existing holographic display systems, for example.

[0031] Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only, which is made with reference to the accompanying drawings. Brief Description of the Drawings

[0032] Figure 1 shows a diagrammatic representation of a method according to an embodiment;

[0033] Figure 2 shows a diagrammatic representation of input data and the input data with masked areas;

[0034] Figure 3 shows a diagrammatic representation of the output of Figure 1 when viewed from different perspectives;

[0035] Figure 4 shows a diagrammatic representation of another method according to an embodiment;

[0036] Figure 5 shows a diagrammatic representation of the output of Figure 1 when viewed from different positions using multi-view;

[0037] Figures 6A to 6F are example simulations of a two layer hologram with and without the processing according to the embodiment of Figure 1 or 4;

[0038] Figures 7A and 7B are example simulations showing the further benefits of the embodiment of Figure 1 or 4 on defocussing of foreground objects;

[0039] Figures 8A and 8B are photographs of an example hologram viewed off axis both with and without the processing according to the embodiment of Figure 1; and

[0040] Figure 9 is an example holographic display system for use with the methods described herein.

[0041] Detailed Description

[0042] Improved Image fidelity for computer-generated holograms based on occluded source data

[0043] Figure 1 is a diagrammatic representation of a method according to a first embodiment, by which the image fidelity of a computer-generated hologram based on occluded source data may be improved. The method of Figure 1 determines a layer-based hologram. Layer-based holograms are known and are a computationally efficient method of determining a computer-generated hologram (CGH). In layer-based holograms, the source data is divided, grouped or separated into layers to determine the CGH.

[0044] At block 1, first data representing a three-dimensional scene is received or otherwise presented to a holographic display system. The first data is depicted as having a plurality of layers in Figure 1, but this need not necessarily be the case. For example, the data may have associated depth data but not be grouped into layers, or each point or element of data can be considered to occupy its own layer. (An example holographic display system for use with the method of Figure 1 is described later, with reference to Figure 9). In this example a single view of a scene is rendered, or captured, from a single perspective, also referred to as a viewpoint. A “single perspective” is used to mean that the first data is of a scene viewed from a particular position and direction. This single perspective means that some of the data of the three-dimensional scene is occluded in the first data.

[0045] A hologram pre-processing system first analyses the received image data to in-fill, also referred to as synthesise, data of occluded regions in at least one layer, using the process of blocks 2 to 5. At block 2, the display system is instructed, or determines, a number of layers from the first data. The techniques described herein can be applied to any number of layers, providing that there are at least two layers. The number of layers may be the same as those in the received data, or there may be more layers or fewer layers than in the received data. Fewer layers may be generated by consolidating layers of similar depth, or data associated with similar depth, into a smaller number of layers. This can simply involve gathering the data in those layers into a set, for example. More layers can be generated by image analysis, for example to identify and extract foreground and background content. In this example, the image data comprises three layers , with foreground on the left side of block 2 and background on the right side of block 2. Unoccupied regions in a particular layer are coloured black.

[0046] At block 3, at least one layer is analysed to determine what it obscures in layers at other distances or depths from the viewing point. In this example a mask, and more specifically an alpha mask is generated. In this example, no mask is generated for the background layer, but other examples may also generate a mask for the background. Masks are depicted for both of the layers in front of the background, although other examples may not process all layers, perhaps to balance image quality against processing resources. In this example an alpha mask is used to describe the region of a layer in the shadow of forward layers. Other examples may also use the mask for reflections and refractions, and the skilled person will appreciate that the disclosure is not limited only to shadows of forward layers. An alpha mask uses an additional image channel in addition to the colour and depth data to indicate an opacity of an element. Here, the alpha mask is binary, with a level of zero or false (“0”) indicating that no object is present and a level of true or “1” indicating that an object is present. As is known to the skilled person, the alpha channel may also encode values between these two extremes which can improve the anti-aliasing around edges, for example.

[0047] The alpha mask may be transmitted explicitly with the image data as an alpha channel or layer. The alpha mask may also be inferred from the received image data, with any unoccupied pixel in a layer being assumed to have a value of zero.

[0048] At block 4, the mask is then used to determine or identify the areas, regions or parts of each layer that require supplementing with additional synthesised data for where no data is present because it is occluded by other layers. It can be seen that, for the middle layer, the identified occluded region corresponds directly to the mask of the foreground layer. Meanwhile, for the background layer, the identified occluded region corresponds to the union or sum of the alpha masks of the foreground and middle layers.

[0049] At block 5, the occluded areas from block 4 go through a method of image synthesis (potentially using methods such as, but not limited to, image blurring, colour stretching, Image Pyramid algorithms, Content-Aware Fill, or neural network generated imagery) to supplement them with additional data. The supplemented data joins with the first data to generate second data. In this example, the foreground layer is unaltered as nothing was occluded in that layer, but both the middle and background layers have additional data. It can also be seen how the synthesis for the background layer has provided additional data over substantially the whole area of the occluded region from block 4. In contrast, the occluded region middle layer is not completely full of additional data across its whole area, this is because the image fill determined that a large area of the occluded region contained no image content. Put another way, the synthesis algorithm may “fill” at least some of the occluded area with content.

[0050] The hologram pre-processor has now completed and the second data, including the first data and the additional data, is passed to the hologram algorithm at block 6. Here, holograms are generated for each layer at the correct associated depth. For example, the methods discussed in WO2020 / 148521, incorporated herein by reference for all purposes, can be followed. In this example the same layers are used as were used in blocks 2 to 5, but other examples may use a different number of layers in the hologram algorithm, grouping, segmenting or dividing the second data as necessary. At block 7, each layer then goes through an occlusion algorithm (such as listed below), this requires a mask area to determine the regions that can be holographically occluded. In this example, the masks from block 3 can be reused. Other examples may generate updated masks again from the second data of block 5. Any suitable occlusion algorithm can be used including the “Ping-Pong” algorithm and the algorithms discussed in:

[0051] • R. Dorsch, A. W. Lohmann, and S. Sinzinger. Fresnel ping-pong algorithm for two- plane computer-generated hologram display. Applied Optics Vol. 33, No. 5, 1994.

[0052] • A. Jendral. Computer-Generated Image Holograms for 3D-Display, 1995.

[0053] • A. Gilles. Fast hologram synthesis methods for realistic 3D visualization. INSA de Rennes, 2016

[0054] The layers may or may not be combined during this occlusion process but do need to be combined in some way for display. In this example, a single hologram is determined by merging the holograms having applied the occlusion algorithm. Alternatively, each layer may be displayed individually across-time (time-multiplexing), making use of persistence of vision to combine them at a viewer’s eye.

[0055] At block 8, the resulting hologram data is displayed.

[0056] Referring now to Figure 2, the layers 202, 204, 206 formed from the first data (as depicted in block 2 of Figure 1) are shown adjacent the layers 202’, 204’ and 206’ showing the combination of the first data and the masks from block 4 of Figure 1. The foreground layer 202’ has not changed because it has no synthesis requirements. In other words, layer 202’ contains only the first data that was already present in layer 202. The middle layer 204’ has additional data requirements indicated by a mask section in solid white. In other words, in addition to the first data already present in 204, there is mask data corresponding to the region occluded by the foreground layer 202 (visible as white ellipse in 204’). The background layer 206’ also has additional data requirements indicated by a mask section in solid white. In other words, in addition to the first data already present in 206, there is mask data corresponding to the region occluded by the foreground layer 202 and / or the middle layer 204 (visible as a white rectangle and ellipse). While Figure 2 has considered occlusion in the direction of foreground to background, other examples may additionally or alternatively occlusions in the direction of background to foreground (this may be relevant for some lens effects in an image, for example). Figure 3 is a diagrammatic representation of one of the image fidelity improvements from the processing of the method of Figure 1, showing how dark regions are reduced when the hologram is viewed “off-axis” or from a different perspective than it was rendered. Representations of what the viewer sees when viewing a hologram generated according to blocks 6 and 7 of Figure 1, but without the pre-processing of blocks 2 to 5 of Figure 1 is depicted at 302, 304 and 306. This uses an occlusion algorithm without the pre-processing. The “on-axis” view 304 shows that when viewed from the rendered perspective image fidelity is good. Similarly, when viewed from the left (representation 302) or right (representation 306) of the rendered perspective, the occlusion processing means that there are no bright areas. However, gaps or artefacts 308 are visible in the image due to parallax. In representations 302 and 306 the foreground objects have moved relative to the background and reveal areas occluded at the rendering perspective. The occluded areas have no information in the hologram and appear as dark regions. As can be seen in representations 302’, 304’ and 306’, when the pre-processing of Figure 1 is applied these dark regions or artefacts are not visible when viewed off axis despite the same change in relative position of foreground and background.

[0057] Figure 4 is a diagrammatic representation of a method according to a second embodiment, by which the image fidelity of a computer-generated hologram based on occluded source data may be improved. The method of Figure 4 is the same as the method of Figure 1, but a different pre-processor is used to supplement the first data. Rather than the layer-based image fill techniques of the Figure 1, Figure 4 uses a machine learning system trained to take inputs of the first data and output the second data directly. This may be useful as processing systems - including CPUs, GPUS and system-on-chip and system- in-package - increasingly include acceleration hardware for machine learning algorithms and neural networks. However, hardware acceleration is not essential and the method of Figure 4 may also be used without it.

[0058] At block 401, first data representing a three-dimensional scene is received or otherwise presented to the holographic display system. As discussed above for block 1 of Figure 1, although the data is depicted as comprising layers, this is not essential and the first data may be 2.5D, as in this example, or a full 3D point-cloud with a viewing position specified, or similar volumetric representation, such as VDB. In some examples, the first data may not even contain depth data, for it to be synthesised, generated or determined as part of the machine learning algorithm of block 403.

[0059] Block 402 shows diagrammatically how, as shown by the preview regions at the right hand side of block 402, the first data has missing data behind occlusions. Occlusions include objects occluding other objects or where an object spans multiple regions.

[0060] At block 403, the first data is input to an Artificial Intelligence (Al) / Machine Learning (ML) algorithm which has been trained to accept input data representing a 3D scene and output second data of an updated scene including additional data in occluded parts of the first data. Any suitable algorithm may be used and can be trained via training data that contains occluded input data, either as 2.5D, volumetric / points etc, with corresponding “perfect” or full 3D scene data.. Such training data can be generated from rendering models which can output a 2.5D render at a viewpoint and also store the full three-dimensional data of the scene, including occlusions omitted from the 2.5D render

[0061] Where the first data includes no depth data, the AI / ML algorithm may be trained on 2D data output from a rendering model at a particular viewpoint, with a corresponding full 3D scene data (including depth information and occluded elements missing from the 2D data).

[0062] The second data output from the AI / ML algorithm is depicted in block 405 and may be made up of spectrum / colour, alpha, depth and other data. As depicted at block 405, this is sliced into layers and passed to the hologram algorithm in block 406. In other examples it may be directly passed to the hologram algorithm without explicitly being allocated to layers.

[0063] At block 406, the hologram algorithm generates holograms, and performs the diffraction-based occlusion based on the second data at block 407 in the same way as discussed above for blocks 6 and 7 of Figure 1.

[0064] At block 408, the resulting hologram is displayed on a holographic display system and demonstrates the same image fidelity improvements discussed above with reference to Figure 3.

[0065] As can be seen, in the method of Figure 4, a single AI / ML algorithm may encompass steps 3 through to 5 of the method of Figure 1. In another example, a single AI / ML algorithm may encompass steps 3 to 7. Multi-view holograms

[0066] As discussed so far, the method of Figure 1 and the method of Figure 4 have been applied to one hologram rendered from a single perspective. The method can equally be applied to multiview holograms, where a plurality of holograms (also referred to as subholograms or hogels) are displayed substantially simultaneously and show the same scene from different viewpoints. The benefits to image fidelity of the methods of Figures 1 and 4 mean that fewer hogels are required than prior methods, as a single hogel can be viewed with improved fidelity over a wider range of angles.

[0067] Figure 5 shows a diagrammatic representation of the benefits of applying the method of Figure 1 or 4 to each hologram in a multi -view display. The position of a viewer’s eye 502 is depicted relative to the replay field of the holographic display system. As a reference, in view 504 the entire replay field is used for a single hologram, and the viewer’s eye is offset from the “on axis” viewing position indicated by a cross 505 in the centre of the replay field. With a layer-based hologram without the process of Figure 1 or 4, the off axis viewing creates parallax. Dark regions or artefacts are visible in view 506 as with Figure 3. With the processing of Figure 1 or 4, these artefacts are not present, as shown in the representation 508.

[0068] Representations 504a, 506a and 508a show a multi-view hologram when viewed on axis. In the representation 504a, 16 sub-holograms or hogels 510 of the same scene are tiled over the field of view, each with their own rendering position depicted by a cross. (For clarity only two of the hogels 510 are indicated by reference signs on Figure 5.) As can be seen, the viewer’s eye 502 is aligned on axis with one of the multi -view holograms so both the standard layer-based hologram, represented by 506a, and one using the processing of Figure 1 or 4, represented by 508a, show substantially no parallax artefacts.

[0069] While the viewer’s eye cannot move as far from the ideal perspective in a multi -view hologram, viewing off-axis is still possible. The set of representations 504b, 506b, 508b show the benefits of the processing of Figure 1 or 4 when a viewer’s eye straddles two Multiview holograms, as shown in 504b. With a standard layer-based hologram dual dark regions or artefacts are visible, as represented by 506b. This is because two holograms combine and elements which are viewed off axis may be partially filled with information from one of holograms. Applying the processing of Figure 1 or 4 removes this effect as shown in representation 508b. Finally, the processing of Figure 1 or 4 also provides the same benefits when a viewer’s eye is positioned off axis within one hologram of the multi -view hologram. Representation 504c gives an example viewing position. Representation 506c shows the dark regions or artefacts from standard layer-based processing and representation 508c shows how these artefacts are not present when the processing of Figure 1 is applied.

[0070] Examples showing the benefits of processing as described herein

[0071] Some further examples are now included to show the benefits of the image processing when applied to example simulations and photographs of displayed holograms.

[0072] Figures 6A to 6F depict various stages and simulated results for a two-layer hologram, so that the benefits of the processing of Figure 1 or 4 can be demonstrated. In the example of Figures 6A to 6F an object (a spiral) is placed in front of a background (a room scene). From the rendered image perspective, there is a gap in information about the background, as it is occluded by the foreground object. The background is effectively in the shadow of the foreground object from the camera or rendering perspective. When a front hologram occludes the background, image fidelity is reduced when viewed from another perspective than the rendering perspective.

[0073] Figure 6A shows an input first data including two layers (foreground and background) from the rendering perspective. Figure 6B shows the background layer independently. In a holographic display, both the background and foreground layers will be turned into holograms and displayed at the correct depth to preserve the perceptually correct depth.

[0074] In real life, a viewer expects to be able to focus on a distant object and have closer objects defocus in a physically correct way. In a hologram generated from the above input, the defocus is physically correct, but the way that the foreground occludes background within its defocus is incorrect, even when viewed on axis. This is due to the missing information from the background image. The resulting view, while focussing on the background, is simulated and shown in Figure 6C. The foreground object (the spiral) appears to remain incorrectly in focus due to image data from the background becoming visible when it defocuses but this information is not present in the background layer. As a result, the foreground object appears sharper than it should, which confuses the viewer. The processing of Figure 1 or 4 solves the defocus problem depicted in Figure 6C by incorporating synthesised image information within the background layer in order to prevent the artefact becoming visible. Figure 6D depicts the result of supplementing the background layer with additional data. A map of the foreground layer is used to synthesise additional data in occluded parts of the background layer. In this example, an “image pyramid” method is used. Other methods can also be used to balance quality against available processing resources for the synthesis. For example, “Image Pyramid” may be used as mid-ground between quality and processing resources, neural network synthesis is higher quality but more resource intensive. A colour blur technique gives lower quality but uses fewer processing resources.

[0075] As discussed with reference to the method of Figure 1 or 4, the background layer including additional information (Figure 6D), can form second data used to generate a hologram rather than the source background layer of the first data (Figure 6B). A hologram can then be generated using the second data and diffraction-based occlusion with the additional data meaning that data is present to reduce or eliminate dark patches due to parallax when viewed off-axis.

[0076] In this example, a layer-based hologram is generated using a Fast Fourier Transform, with Zemike polynomials to set the depth information for the layer. The resulting hologram then appears to be perceptually correct to the viewer and is correctly presented with occlusion (using the “ping-pong” method, in this example) that utilises the additional data in the second data.

[0077] The simulated images of Figures 6E and 6F demonstrate the results when viewed. In Figure 6E, the focus is on the background and this results in a correctly defocussed foreground, unlike that of Figure 6C. In Figure 6F, the focus is on the foreground, showing the defocus in the background.

[0078] Figures 6A to 6F therefore show how image fidelity is improved when holograms generated by the method of Figure 1 are viewed on-axis.

[0079] Figures 7A and 7B are simulation results to show the benefits of using the method of Figure 1 or 4 over applying diffraction-based occlusion techniques directly on the first data. In this simulated example, a cylinder has been added in a layer closer than the spiral of Figures 6A to 6F. A cuboid has been placed in a further layer closer than cylinder. Figure 7A shows a layer-based hologram which uses a wave-based occlusion technique on the first data without supplementing the first data with additional data. In Figure 7A, the focus is on the background and the occlusion achieves some correct defocus, but there are still the same “hard” or “sharp” edges on parts the foreground objects. Figure 7B shows the results when the method of Figure 1 or 4 is used, combining diffraction-based occlusion with supplementing the first data with additional data. The image fidelity improves to show correct defocus of foreground objects.

[0080] Figure 8A is a photograph captured of a CGH determined using a standard layerbased technique without the method of Figure 1 or 4. It is taken from a real holographic display system. The hologram consists of 2 layers and shows a scene containing a cube with a dog image on each face on the front layer, and a portion of a sphere. The sphere is split across two layers, with the closest part being on the same layer as the cube. As the camera moves around it simulates the viewer moving as they view the hologram. Figure 8A shows an off-axis view exhibiting both dark regions indicated by 801 and bright regions indicated by 802.

[0081] Figure 8B is a photograph from the same off-axis position when the CGH has been determined using the method of Figure 1. The dark and bright regions are not visible and image fidelity has improved.

[0082] An example holographic system to which the method of Figure 1 or 4 can be applied is depicted in Figure 9. Figure 9 shows, in general terms, a holographic optical system 900. The system 900 comprises an illumination system or light source 902 configured to generate at least partially coherent light. The system 900 further comprises a spatial light modulator (SLM) 904 arranged to be illuminated by the at least partially coherent light. A lens or optical system 906 focusses the light from the SLM for viewing.

[0083] The light source 902 may, for example, comprise a laser module or an LED. The light source 902 is configured to generate at least partially coherent light at a plurality of wavelengths in sequence (corresponding to red, green and blue, for example).

[0084] The SLM 904 may be configured to modulate at least one of the phase, amplitude, binary phase and binary amplitude of the light. The SLM 904 may be, for example, a DMD, an LCD, an amplitude LCoS or a phase LCoS.

[0085] As shown, the SLM 904 and lens 906 are coaxial. Other configurations may also be used, such as a folded light path that may allow a more compact display.

[0086] A controller 908 processes received image data to generate a CGH for display. For clarity, Figure 9 depicts a transmissive SLM, it will be understood that the principles discussed here are not limited to this and can equally be applied to reflective SLMs. Likewise, the same principles apply to other types of modulators than a SLM.

[0087] The above embodiments are to be understood as illustrative examples of the invention. Further embodiments of the invention are envisaged. Although the discussion above has not considered colour, the method of Figure 1 or 4 can equally be applied to colour holograms, whether those created by simultaneous or time-sequential display of individual colours, such as red, green and blue components. The method of Figure 1 or 4 could also be applied to any combination of wavelengths, whether visible to the human eye or not. When applied to colour holograms, or more generally multi-channel holograms, the mask and image synthesis steps may be applied to the colours or channels as a whole, or to the individual colours or channels independently.

[0088] The methods described herein therefore improve the image fidelity of holograms when viewed both on and off-axis.

[0089] It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

CLAIMS1. A computer-implemented method of determining a computer-generated hologram, the method comprising: receiving first data representing a three-dimensional scene from a particular viewpoint; supplementing the first data with additional data representing at least part of the three-dimensional scene that is occluded as a result of the particular viewpoint, thereby to generate second data representing the three-dimensional scene; and determining hologram data using the second data and a diffraction-based occlusion algorithm.

2. A method according to claim 1, wherein determining the hologram data comprises a layer-based algorithm.

3. A method according to claim 1 or 2, wherein the diffraction-based occlusion algorithm uses at least one mask determined from the first data or the second data.

4. A method according to any preceding claim, wherein the first data comprises or is associated with depth data.

5. A method according to any preceding claim, further comprising determining depth data based on the first data.

6. A method according to claim 4 or 5, comprising using a mask determined from the first data and the depth data in the supplementing the first data.

7. A method according to claim 6, wherein the supplementing the first data uses a first algorithm at a first depth and a second algorithm at a second depth, wherein the first algorithm is different from the second algorithm and the first depth is different from the second depth.

8. A method according to any of claims 1 to 5, wherein the supplementing the first data uses a machine learning algorithm having an input of the first data and an output of the second data.

9. A method according to any preceding claim, wherein the computer-generated hologram is part of a multiview hologram.

10. A holographic display system comprising: an illumination source emitting at least partially coherent light; a spatial light modulator illuminated by the illumination source; an optical system for focussing the output of the spatial light modulator; and a controller configured to implement the method of any of claims 1 to 9 and cause the spatial light modulator to display the hologram data.

11. A headset for augmented reality or virtual reality, a large screen display, or a headsup display comprising the holographic display system according to claim 10.

12. A computer program comprising computer program code that, when executed by a processor, causes the processor to implement the method of any of claims 1 to 9.

13. A non-transitory computer-readable medium comprising instructions embodied thereon that, when executed by a processor, instruct the processor to perform the method of any of claims 1 to 9.

Images