Adjustment-invariant near-eye display

Abstract

According to an exemplary embodiment, there is provided a stereoscopic near-eye display (NED) assembly for a NED device, the NED device comprising a pair of NED assemblies, the NED assemblies comprising: a two-dimensional (2D) display for rendering an image for viewing by eyes of a viewer; a lens assembly arranged at a predefined distance from the 2D display to allow viewing of an image rendered via the 2D display therethrough, where the lens assembly comprises a diffractive optical element (DOE) arranged to provide different phase delays through a plurality of positions of an aperture of the diffractive optical element; and a display controller 10 comprising a pre-processing section arranged to derive a pre-processed image based on the received image via application of a pre-processing program, the pre-processing program is arranged to apply image region position dependent pre-processing in derivation of different sub-regions of an image region of the pre-processed image to 15 account for different transmission characteristics of different sub-apertures through a lens aperture of the lens assembly and to supply the pre-processed image for rendering via the 2D display, and wherein the pre-processing program is arranged to apply image region position dependent pre-processing in derivation of different sub-regions of the image region of the pre-processed image, the pre-processing program and the lens assembly are arranged to display, when viewed through the lens assembly, a pre-processed image as an image perceived as clear for a plurality of adjustment depths residing within an extended depth of field (DoF) of the NED assembly in order to facilitate providing an adjustment-invariant 3D presentation based on the pre-processed image via operation of the NED device.

Description

Technical Field

[0001] The present invention relates to near eye displays (NEDs). Background Art

[0002] Stereoscopic near-eye displays (NEDs) are of particular interest for use in various virtual reality (VR) and augmented reality (AR) applications. Since some users experience visual discomfort, it is a challenge to ensure that the viewing experience via wearable NED devices is comfortable yet natural or nearly natural.

[0003] One of the key challenges in ensuring a comfortable viewing experience involves resolving the vergence-accommodation conflict (VAC), which may occur when a NED device is used to display an object such that its vergence distance does not match its accommodation distance. Such a situation often occurs when viewing VR or AR content using a conventional NED device, because the display distance of an object having a three-dimensional (3D) image with its corresponding position in the image space (i.e., the corresponding vergence distance) is different from the distance to the position of the (virtual) image plane of the NED device (i.e., the accommodation distance).

[0004] While NED devices provided with techniques for addressing VAC (such as varifocal, multifocal, light field, holographic, and Maxwell) are known in the art, they are all subject to their own tradeoffs, such as limited eye movement range, limited image quality (e.g., low spatial resolution, speckle noise), and / or complex device requirements (e.g., bulky and / or fast optics, precise eye tracker). Therefore, there is a continuing need for NED devices that can mitigate VAC in a manner that provides high user comfort without compromising image quality and device complexity. Summary of the invention

[0005] It is an object of the present invention to provide a technique for reducing or eliminating user discomfort caused by Vergence Accommodation Conflict (VAC) in NED devices without significantly degrading the resulting perceived image quality.

[0006] According to an exemplary embodiment, a stereoscopic near-eye display (NED) assembly for a NED device is provided, the NED device comprising a pair of NED assemblies, the NED assembly comprising: a two-dimensional (2D) display for rendering an image for viewing by an eye of a viewer; a lens assembly arranged at a predefined distance from the 2D display to allow viewing of the image rendered via the 2D display through the lens assembly, wherein the lens assembly comprises a diffractive optical element (DOE) arranged to provide different phase delays through a plurality of positions of an aperture of the diffractive optical element; and a display controller comprising a preprocessing portion arranged to process the image via a preprocessing process. A preprocessor is provided for deriving a preprocessed image based on the received image by applying a preprocessor, the preprocessor being arranged to apply image area position dependent preprocessing in the derivation of different sub-areas of the image area of ​​the preprocessed image to take into account different transmission characteristics of different sub-apertures of the lens aperture through the lens assembly and to provide the preprocessed image for rendering via a 2D display, and wherein the preprocessor and the lens assembly are arranged to display the preprocessed image as an image perceived as sharp for multiple predefined accommodation depths within an extended depth of field (DoF) of the NED assembly when viewed through the lens assembly, so as to facilitate providing accommodation-invariant 3D presentation based on the preprocessed image via operation of the NED device.

[0007] According to another exemplary embodiment, a device for deriving a preprocessor and a phase delay distribution map for a stereoscopic NED component according to the aforementioned exemplary embodiment is provided, the device being arranged to apply a corresponding learning model to jointly derive at least one artificial neural network (ANN) acting as a preprocessor and a phase delay distribution map defining corresponding phase delays for multiple positions of a DOE via an iterative learning procedure based on multiple training images, wherein the device is arranged to perform the following operations at multiple iteration rounds: process each of the multiple training images through the learning model, the learning model being arranged to perform the following operations: for the corresponding iteration rounds, select one or more sub-apertures, and determine corresponding one or more sub-regions of the image area of ​​the corresponding training image, the one or more sub-regions spatially corresponding to the selected one or more sub-apertures; taking into account the corresponding training images The one or more sub-regions of the image are determined at corresponding positions within the image area, and the one or more sub-regions are processed by at least one ANN using current weights of at least one ANN to determine one or more sub-regions of the pre-processed image that are spatially corresponding; according to the current phase delay distribution map and taking into account the adjustment depth selected for the corresponding iteration round, the one or more sub-regions of the pre-processed image are processed by the display and the eye model into one or more sub-regions of the simulated retinal image that are spatially corresponding; and the difference between the one or more sub-regions of the simulated retinal image and the one or more sub-regions of the corresponding reference retinal image is determined by a predefined loss function; and based on the corresponding differences determined for multiple training images, the weights of at least one ANN and a portion of the phase delay distribution map that is spatially corresponding to one or more selected sub-apertures are updated.

[0008] According to another exemplary embodiment, a method for deriving a preprocessor and a phase delay distribution map for a stereoscopic NED component according to the aforementioned exemplary embodiment is provided, wherein the method includes performing an iterative learning procedure based on multiple training images via application of a corresponding learning model to jointly derive at least one ANN serving as a preprocessor and a phase delay distribution map defining corresponding phase delays for multiple positions of a DOE, the method including for multiple iterations: processing each of the multiple training images through a learning model, the learning model including the following steps: for the corresponding iterations, selecting one or more sub-apertures, and determining corresponding one or more sub-areas of an image area of ​​the corresponding training image, the one or more sub-areas spatially corresponding to the selected one or more sub-apertures; taking into account the determined one or more sub-apertures of the corresponding training image; The invention relates to a method for determining the spatially corresponding one or more sub-regions of a pre-processed image by processing the one or more sub-regions at corresponding positions of the sub-regions within the image area, using the current weights of the at least one ANN, the one or more sub-regions being processed by the at least one ANN to determine the spatially corresponding one or more sub-regions of the pre-processed image; processing the one or more sub-regions of the pre-processed image into the spatially corresponding one or more sub-regions of the simulated retinal image by the display and the eye model according to the current phase delay distribution map and taking into account the adjustment depth selected for the corresponding iteration round; and determining the difference between the one or more sub-regions of the simulated retinal image and the spatially corresponding one or more sub-regions of the corresponding reference retinal image through a predefined loss function; and updating the weights of the at least one ANN and a portion of the phase delay distribution map that spatially corresponds to the one or more selected sub-apertures based on the corresponding differences determined for the plurality of training images.

[0009] According to another exemplary embodiment, a computer program is provided, the computer program comprising a computer readable program code configured to cause the method according to the aforementioned exemplary embodiment to be performed when the program code is executed on one or more computing devices.

[0010] The computer program according to the above exemplary embodiments may be embodied on a volatile or non-volatile computer-readable recording medium, for example as a computer program product comprising at least one computer-readable non-transitory medium having a program code stored thereon, which, when executed by one or more computing devices, causes the computing devices to at least perform the method according to the foregoing exemplary embodiments.

[0011] The exemplary embodiments of the invention presented in this patent application should not be interpreted as limiting the applicability of the appended claims. The verb "comprise" and its derivatives are used in this patent application as open-ended limitations that do not exclude the presence of unrecorded features. Unless explicitly stated otherwise, the features described below can be freely combined with each other.

[0012] Certain features of the invention are set forth in the appended claims. However, the invention in its various aspects, both as to its construction and its method of operation, together with additional objects and advantages thereof, will be best understood from the following description of certain exemplary embodiments when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which:

[0014] Figure 1A schematically illustrates some aspects of a stereoscopic near-eye display assembly according to an example;

[0015] Figure 1B schematically illustrates some aspects of a stereoscopic near-eye display assembly according to an example;

[0016] Figure 2A schematically illustrates some aspects of a stereoscopic near-eye display assembly according to an example;

[0017] Figure 2B A block diagram showing some components of a near-eye display assembly;

[0018] Figure 3A schematically illustrates light from a pixel of a display being received at an eye through a sub-aperture of a lens plane;

[0019] Figure 3B schematically illustrates light from a pixel of a display being received at an eye through a sub-aperture of a lens plane;

[0020] Figure 4 shows a height diagram of a dispersive optical element according to an example;

[0021] Figure 5 A block diagram showing some components of a learning model according to an example; and

[0022] Figure 6 A block diagram showing some components of a device according to an example is shown. DETAILED DESCRIPTION

[0023] Figure 1A and Figure 1BA conventional stereoscopic near-eye display (NED) assembly 101 and some characteristics of a viewer's eye 110 are schematically shown. The NED assembly 101 comprises a two-dimensional (2D) display 102 and a magnifying lens 103, which are arranged in front of the viewer's eye 110 when the NED assembly 101 is operated to view an image rendered on the 2D display 102 through the magnifying lens 103. The 2D display 102 is arranged in a fixed position with respect to the magnifying lens 103, wherein the 2D display 102 is located closer to the magnifying lens 103 than its focal length. Therefore, the 2D display 102 is mapped to a (virtual) image plane 104, which is located at a fixed distance behind the 2D display 102. The NED device may include a pair of NED components 101 (i.e., a respective NED component 101 for each eye 110 of a viewer) and a display controller arranged to provide respective images of a stereoscopic image pair for rendering on respective 2D displays 102 of the pair of NED components 101 to provide a three-dimensional (3D) presentation of a scene captured in the stereoscopic image pair.

[0024] In theory, assuming aberration-free optics, visual content displayed at the 2D display 102 can be shown with a resolution up to the diffraction limit at the image plane 104. At depths offset relative to the image plane 104 (i.e., distance from the 2D display 102), a frequency response is observed to decrease rapidly with increasing depth due to defocus blur (or defocus aberration). Thus, if the accommodation depth of the eye 110 coincides with the image plane 104, then the viewer observes a high-resolution, sharp image, whereas as the offset between the accommodation depth of the eye 110 and the image plane 104 increases, the image observed by the viewer becomes increasingly blurred. In this regard, Figure 1A The example of shows a scene in which the accommodation depth 105a of the eye 110 coincides with the image plane 104 and enables the viewer to observe a clear image, while Figure 1B The example of shows a scene in which the adjustment depth 105b is offset relative to the image plane 104 and thus results in a blurred image. The adjustment depth may also be referred to as the adjustment distance.

[0025] Defocus blur is the primary driver of the eye's 110 accommodation depth in viewing situations, such that the eye 110 generally tends to accommodate at the distance at which the image appears sharpest. Figure 1A and Figure 1B In the arrangement shown in the corresponding example of FIG. 1 , the adjustment depth tends to coincide with the image plane 104 or fall in the immediate vicinity thereof, regardless of the convergence distance of the object displayed in the 3D rendering of the displayed image data, so that Figure 1A The clear image of the object is perceived in the scene, resulting in Figure 1BA blurred image of an object is perceived in the scene. Another factor that has an impact on accommodation depth is binocular parallax: although binocular parallax primarily drives the convergence distance, it also affects the accommodation depth. The present disclosure describes a method in which the coupling between accommodation depth and convergence distance is employed to eliminate or at least significantly reduce retinal blur in images displayed by the NED component 101, so as to create a viewing situation in which the accommodation depth of the eye 110 is primarily driven by binocular parallax, thereby aligning the accommodation depth with the convergence distance and thus solving VAC. Alternatively, the method can be considered as an extension of the depth of field (DoF) of the NED component 101.

[0026] The improved stereoscopic NED component according to the present disclosure utilizes wavefront coding to extend the DoF in a manner that produces a substantially uniform frequency response across multiple (virtual) image planes within the depth range of the NED component, thereby substantially providing an accommodation-invariant (AI) NED component. As an example in this regard, Figure 2A An accommodation-invariant NED assembly 201 according to the present disclosure and some characteristics of the eye 110 of a viewer are schematically shown. In the following, the accommodation-invariant NED assembly 201 is mainly referred to as the NED assembly 201 for short. The NED assembly 201 includes a 2D display 202 and a lens assembly 203 arranged at a predefined distance from the 2D display 202. A pair of NED assemblies 201 may be provided, for example as respective elements of a NED device (such as an AR headset or a VR headset), which allows each of the two NED assemblies 201 to be positioned in front of a respective eye 110 of a viewer for viewing a respective image of a stereoscopic image pair displayed via a respective 2D display 202 through a respective lens assembly 203, so as to perceive a 3D presentation of a scene captured in the stereoscopic image pair. In the following, for the sake of brevity and clarity, the description mainly relates to the singular NED assembly 201, while the description implicitly applies to the two NED assemblies 201 of the NED device.

[0027] Hereinafter, when referring to the position of the 2D display 202 relative to other elements of the NED assembly 201 and various concepts associated with the characteristics of the NED assembly 201, the term "display plane" shall be used to refer to the position of the 2D display 202. Along similar lines, the term "lens plane" shall be used to refer to the position of the lens assembly 203 relative to other elements of the NED assembly 201 and various concepts associated with the characteristics of the NED assembly 201.

[0028] The lens assembly 203 includes at least a diffractive optical element (DOE), and the lens assembly 203 is arranged to provide DoF extension via implementing wavefront coding, which wavefront coding focuses the pre-processed image displayed on the 2D display 202 so as to provide a substantially uniform frequency response within the depth range of the NED assembly 201. According to an example, the lens assembly 203 may include separate optical subassemblies for providing imaging / magnification and DoF extension, respectively, while according to another example, the lens assembly 203 may include a single optical component arranged to provide both imaging / magnification and DoF extension. In an example where separate optical subassemblies are applied for imaging / magnification and DoF extension, the imaging / magnification may be provided by, for example, a refractive lens, a Fresnel lens, or a pancake lens, and the DoF extension may be provided by the DOE. In an example where a single optical component is arranged to provide both imaging / magnification and DoF extension, a DOE also arranged to provide imaging / magnification may be applied.

[0029] At this point, Figure 2A The illustration of provides a non-limiting example, wherein the lens assembly 203 includes a refractive lens 203a and a DOE 203b arranged on the same optical axis, wherein the refractive lens 203a provides imaging / magnification and the DOE 203b provides DoF expansion via waveform encoding. The refractive lens 203a may include, for example, a magnifying lens, while the DOE 203b may be provided as, for example, an optical element that may be integrated into the refractive lens 203a or as an element physically separated from the refractive lens 203 and arranged at a predefined distance from the refractive lens 203a (and in close proximity to the refractive lens). The optical properties of the lens assembly 203 formed by the refractive lens 203a and the DOE 203b are such as to provide a substantially uniform frequency response within the depth range of the NED assembly 201, which is Figure 2A 204-1, 204-2, 204-3, 204-4, and 204-5, which correspond to respective adjustment depths within the depth range of the NED component 201. In this regard, the image planes 204-1, 204-2, 204-3, 204-4, 204-5 represent a plurality of image planes (corresponding to respective plurality of adjustment depths) and they may be collectively referred to via reference numeral 204, while any individual image plane in the plurality of image planes 204 may be referred to via reference numeral 204-k.

[0030] In the following, reference is made to Figure 2A Various aspects related to the lens assembly 203 and the DOE therein are described with reference to the non-limiting example of the lens assembly 203 and the DOE 203b therein, and these aspects can be easily summarized as having similarities to those for Figure 2AThe lens assembly 203 and / or DOE having different characteristics than those described by way of non-limiting examples. Furthermore, it is noted that although the description herein refers to a plurality of discrete (virtual) image planes 204-k, this is a selection made for illustrative purposes only, and in a realistic implementation of the NED assembly 201, the plurality of image planes 204 may include a predefined set of discrete (virtual) image planes 204-k at corresponding distances from the display plane, or the plurality of image planes 204 may include a (virtual) image plane 204-k that may be located at any distance from the display plane within the depth range of the NED assembly 201.

[0031] Figure 2B A block diagram of some components of a display controller 210 is shown, the display controller being adapted to pre-process an input image for rendering on a 2D display 202 in a manner that facilitates providing a viewer with an adjustment-invariant 3D presentation of an image provided as input to the display controller 210 when the viewer views through a lens assembly 203. The display controller 210 may be provided as an element of a NED assembly 201 and / or as an element of a NED device utilizing a pair of NED assemblies 201. The display controller 210 comprises a control portion 211 and an image pre-processing portion 212, wherein the image pre-processing portion 212 may be arranged to operate at least in part under the control of the control portion 211, and the control portion 211 may be arranged to control at least some aspects of the operation of the 2D display 202 relating to the manner in which images are displayed thereon. The operation of the control portion 211 may be controlled at least in part via control inputs provided thereto (e.g., via a user interface of the NED device). According to an example, the display controller 210 may be implemented via operation of a computing device including one or more processors and one or more memories for storing one or more computer programs, wherein the one or more computer programs are arranged to cause the computing device to operate as a display controller 210 according to the present disclosure when executed by the one or more processors.

[0032] The display controller 210 (e.g., the image pre-processing portion 212 therein) may receive a stereoscopic image pair time sequence and derive, based on the stereoscopic image pair time sequence, a corresponding pre-processed image pair time sequence suitable for providing an adjustment-invariant 3D presentation of a scene captured in the stereoscopic image pair time sequence via rendering the stereoscopic image pair time sequence on respective 2D displays 202 of two NED components 201 of the NED device. Thus, for each of the two NED components 201 of the NED device, the image pre-processing portion 212 may be arranged to receive a respective input image time sequence I(t) and derive, based on the respective received input image time sequence I(t), a respective corresponding pre-processed image time sequence I(t). d(t) for display via the 2D display 202 of the corresponding NED component 201 so as to provide a rendering of the 3D scene.

[0033] In this regard, any single image in the input image time series I(t) may be referred to as the input image I, and the preprocessed image time series I d Any single image in (t) can be referred to as a preprocessed image I d In this paper, the input image I refers to the pixel values ​​of the underlying input image, and the preprocessed image I d Refers to the pixel value of the underlying pre-processed image. In a non-limiting example, the input image I and the corresponding pre-processed image I d This may include separately in the red, green and blue channels for the respective images I, I d The image preprocessing section 212 may derive the corresponding preprocessed image time sequence I(t) based on the input image time sequence I(t). d (t) in the depth map time series D(t). In the following, various aspects related to the operation of the image preprocessing portion 212 of the display controller 210 are mainly described by referring to one of the two NED components 210 of the NED device and also by referring to receiving a single input image I and deriving a corresponding preprocessed image I based on the received input image I. d , and the processing of a single input image I can be easily summarized as performing corresponding processing on a plurality of images in the corresponding input image time series I(t) of each of the two NED components 201 of the NED device. In this regard, the pre-processed image I is derived based on the received input image I. d The aspect can also be considered as processing the received input image I into the corresponding pre-processed image I d .

[0034] The image preprocessing section 212 may be arranged to process the input image I into a corresponding preprocessed image I via application of a predefined preprocessing procedure. d, the predefined preprocessing procedure is matched to the optical characteristics of the lens assembly 203 and in particular to the DOE therein, so as to facilitate accommodation-invariant 3D rendering at the image plane 204-k within the depth range of the NED assembly 201, which exhibits an extended DoF compared to a conventional stereo NED. The preprocessing procedure can be considered to provide a method for encoding an input image I into a corresponding preprocessed image I d , and the lens assembly 203 may be considered to provide a pre-processed image I within the depth range of the NED assembly 201 d In this regard, at least some characteristics of the pre-processing routine and the diffraction characteristics of the DOE may be co-designed to ensure that a sufficient degree of adjustment invariance is provided for operation of the NED assembly 201 .

[0035] The preprocessing procedure may be arranged to process the input image I into corresponding preprocessed images I in different ways for different positions of the display plane. d . In other words, the preprocessing procedure may be performed in a display plane position-aware manner, which may result in providing different preprocessing characteristics for different sub-regions of the display plane. Such non-uniform processing of sub-regions of the image area at different locations across the display plane is used to compensate for light transmission characteristics that differ from one display plane location to another due to differences in transmission characteristics through the lens assembly 203 at different locations of the lens plane. For example, by dividing the image area into a plurality of sub-regions and deriving the preprocessed image I in a manner that depends on their position P within the image area, d Display plane position aware pre-processing may be provided for each of the multiple sub-areas of the image area.

[0036] In this regard, the display plane position awareness pre-processing stems from the fact that the pupil 110a of the viewer's eye 110 is smaller than the lens aperture of the lens assembly 203, which has at least the following consequences:

[0037] - only a sub-portion of the light from each pixel displayed on the display plane and transmitted through the lens assembly 203 enters through the pupil 110a and meets the retina of the eye 110; and

[0038] - Only a sub-portion of the lens and DOE aperture participates in transmitting light from a certain pixel of the display plane through the lens assembly 203 and through the pupil 110a to the retina of the eye 110, where this sub-portion can be referred to as the (virtual) sub-aperture associated with the corresponding pixel.

[0039] With respect to the latter point above, the position of the sub-aperture through which light from a certain pixel of the display plane passes to enter the retina depends on the display plane position of the corresponding pixel, the distance between the display plane and the lens assembly 203, and the (hypothetical) distance between the lens assembly 203 and the viewer's eye 110. Specifically, the position of the sub-aperture (e.g., its center point) in the lens plane and the angle of incidence (θ, φ) of light passing therethrough received at the pupil 110a vary with the pixel position (ξ, η) on the display plane. Thus, light from different positions of the display plane reaches the retina of the eye 110 via different paths and via different (virtual) sub-apertures of the lens aperture, which results in sub-aperture-dependent light transmission characteristics through the lens assembly 203. These light transmission characteristics can be modeled via a display point spread function (PSF) at the reference plane 206, which depends on the position of the corresponding subaperture on the lens plane (e.g., the distance of the subaperture to the central axis of the lens assembly 203), and a display plane position-aware pre-processing routine can be arranged to account for these differences in light transmission characteristics, thereby helping to maintain a wide field of view (FoV) without compromising DoF.

[0040] Figure 3A and Figure 3B The corresponding examples of FIG. 1 show the corresponding subapertures at two different pixel positions of the display plane, where the subapertures are depicted as corresponding ellipses overlaid on the refractive lens 203a. In both examples, the pre-processed image I d The image is imaged at a reference plane 206 at a distance z from the lens plane. Figure 3A The example shows receiving a signal from a first pixel position (ξ 1 ,η 1 ) and therefore, the light from the first pixel position (ξ 1 ,η 1 ) at the reference plane 206 (x 1 ,y 1 ) and reaches the pupil 110a at an incident angle of zero and thus passes through the position (s 1 ,t 1 ) at a first sub-aperture centered on the central axis of the lens assembly 203. Figure 3B The example shows receiving a second pixel position (ξ 2 ,η 2 ) and therefore, the light from the second pixel position (ξ 2 ,η 2 ) at the reference plane 206 (x 2 ,y 2 ) and at a non-zero incident angle (θ 2,φ 2 ) reaches the pupil 110a and thus passes through the lens at a position (s) offset relative to the central axis of the lens assembly 203 and at the lens plane. 2 ,t 2 ) as the center.

[0041] Therefore, the preprocessing procedure can be designed by considering multiple sub-apertures, which can have shapes and sizes close to a typical (e.g., average) pupil 110a and together cover the entire lens aperture without any gaps between them. Following the ideas described above, the sub-apertures of the lens plane are at least conceptually mapped to corresponding sub-regions of the display plane (e.g., to corresponding sub-groups of pixels of the display plane), and the sub-regions of the display plane are also mapped to the preprocessed image I d Considering the mapping to the preprocessed image I d The preprocessing program is applied to the input image I to derive a preprocessed image I d The processing of a certain sub-region of the derived image region may be different from one sub-region to another, so that the pre-processing procedure provides different pre-processing characteristics for different sub-regions of the derived image region.

[0042] For example, the pre-processing procedure may be provided via the application of at least one artificial neural network (ANN) arranged (e.g. trained) to process the input image I into a corresponding pre-processed image I in a manner that takes into account the sub-apertures of the lens aperture of the lens assembly 203. d , as described in more detail below, thereby generating a display plane position aware preprocessor. As a non-limiting example in this regard, the at least one ANN employed by the image preprocessing portion 212 to implement the preprocessor may be provided as at least one convolutional neural network (CNN) or as at least one derivative of a CNN. However, a CNN (or its derivative) serves as a non-limiting example of an applicable ANN, and in other examples, a different kind of ANN may be employed instead without departing from the scope of the NED component 201 according to the present disclosure.

[0043] According to an example of applying at least one ANN, a corresponding pre-processed image I may be provided via a single ANN. d Different processing of different sub-regions of the image region, the single ANN is trained to dEach sub-region of the image region of interest is processed separately based on its position within the image region. The training of such an ANN is described below via the examples provided. In the example, such an ANN may take as input an input image I and an indication of an image sub-region of interest and provide a pre-processed image I d The corresponding sub-region of the image is taken as output. In a variation of this method, the input to the single ANN can also include depth information related to the sub-region under consideration. d can be derived by applying a single ANN to each of the sub-regions of the image region separately and combining the sub-regions, so that the pre-processed image I is obtained d .

[0044] According to another example of applying at least one ANN, the corresponding pre-processed image I can be provided via the application of multiple ANNs. d Different processing of different sub-regions of the image region, each ANN is trained to process the preprocessed image I d The training of such a plurality of ANNs is described below via the examples provided. In the example, each of such ANNs can take as input the input image I and the corresponding sub-region and provide a pre-processed image I d The corresponding sub-region of the image I is taken as output. In a variation of the method, the input to each of the ANNs may also include depth information related to the sub-region processed by the corresponding ANN. d The preprocessed image I can be derived by applying multiple ANNs to corresponding sub-regions of the image region and combining the sub-regions, thus obtaining d .

[0045] Following the ideas described above, according to a non-limiting example, the DOE may include an optical element arranged to implement wavefront coding, the wavefront coding being intended to be based on a pre-processed image I displayed on the 2D display 202. d to provide a substantially uniform frequency response over the depth range of the NED assembly 201. In this regard, the DOE may be arranged such that the lens assembly 203 introduces a lens plane position-dependent phase delay to light transmitted therethrough. In other words, the DOE may be arranged to provide phase delays that may be different at different locations (e.g., sub-portions) of the lens plane to facilitate providing an adjustment-invariant 3D rendering of the input image I across the depth range of the NED assembly 201. The phase delays at different locations of the lens plane may be represented as Φ(s, t), where the tuple (s, t) is used to indicate a 2D coordinate at the lens plane. Variations in the phase delays at different sub-portions of the DOE may be provided via non-uniform thickness of the optical element, such as in Figure 2A203b. As an example, such a non-uniform thickness can be achieved by a profile defined via a height map d(s, t) that defines the thickness of the DOE 203b across the lens plane. In this regard, the phase delay Φ(s, t) can be converted to the height map d(s, t), and the height map d(s, t) can be converted to the phase delay Φ(s, t), taking into account the optical properties of the material used to implement the lens element used as the DOE 203b.

[0046] In other examples, the DOE included in the lens assembly 203 may be different from Figure 2A As an example in this regard, the DOE can be provided via the application of a phase modulating metamaterial arranged as, for example, a metasurface, where a desired phase delay across a position of a lens aperture of the lens assembly 203 can be provided via correspondingly controlling at least one aspect (e.g., orientation and / or size) of the nanostructures of the metasurface.

[0047] According to a non-limiting example, the phase delay Φ(s,t) may be rotationally symmetric about the central axis of the lens assembly 203. Figure 2A In the example scenario of , this can be achieved via the application of a height map d(s,t) that is rotationally symmetric with respect to the central axis of the lens assembly 203 (and therefore with respect to the central axis of the DOE 203b). This is an option that simplifies the design of the DOE, taking into account the fact that the defocus aberrations of the lens assembly 203 and the display PSF at positions across the lens plane are generally rotationally symmetric about the central axis of the lens assembly 203. According to other examples, the phase delay Φ(s,t) may not exhibit symmetry about the central axis of the lens assembly 203 (or about any other axis), which generally leads to increased design complexity of the DOE (and pre-processing procedure), but enhances the possibility to control the phase delay Φ(s,t) at positions across the lens plane. Taking into account Figure 2A The example framework of DOE 203b, Figure 4 A non-limiting example of a height map d(s,t) is shown that is symmetric about the central axis of the lens assembly 203 and thus can be applied to define the DOE 203b as an element that is symmetric about its central axis.

[0048] Along the lines described above, the phase delay characteristics of the DOE's position across the lens plane of the lens assembly 203 can be matched with the display plane position perception pre-processing characteristics provided via the operation of the pre-processing program to facilitate driving the depth of accommodation of the viewer's eye 110 in a desired manner. In this regard, certain characteristics of the pre-processing program and the DOE's phase delay Φ(s,t) can be designed via a common program to ensure such matching characteristics.

[0049] As described above, in various examples, the pre-processing procedure may be provided via application of an ANN, and the weights of the ANN implementing the pre-processing procedure may be determined via a training procedure performed prior to application of the pre-processing procedure in the image pre-processing portion 212 of the display controller 210. The match between the pre-processing characteristics provided by the pre-processing procedure and the phase delay characteristics of the DOE may be provided via modeling and optimizing the phase delay Φ(s, t) as part of the training procedure applied to determine the weights of the ANN used to provide the pre-processing procedure.

[0050] Figure 5 A learning model 300 according to an example is shown, wherein the learning model 300 can be applied to implement an iterative learning procedure relying on supervised learning in order to determine the weights of an ANN for implementing a preprocessing procedure of the image preprocessing portion 212, and at the same time determine a phase delay profile, which defines the corresponding phase delays of a plurality of positions (s, t) of the lens plane of the lens assembly 203 and can therefore be applied as the phase delay Φ(s, t) of the DOE. Following the ideas described above, if implemented as an element with varying heights across the DOE aperture, for example according to Figure 2A For example, the phase delay Φ(s, t) can be applied to determine the corresponding height map d(s, t) of DOE 203b. The learning procedure can apply a training data set, which includes a plurality of training images I t (typically thousands of images) and wherein the training dataset comprises color images (e.g., RGB images) depicting natural scenes, which may be selected based on the intended use of the NED component 201 and / or a NED device utilizing the NED component 201. According to an example, before starting the iterative learning procedure, the weights and / or phase delay profile of the ANN may be initialized using (pseudo) random values, while in another example, the initial values ​​of the weights and / or phase delay profile of the ANN may include corresponding values ​​obtained via a previous learning procedure.

[0051] The training dataset may include images, where each training image I t Corresponding pixel values ​​may be defined and provided as input to the learning model 300 (e.g. Figure 5 In some examples, the training data set may also include a corresponding depth map D t (t), which can also be used as input in the learning process and the corresponding training image I t Provided together to the learning model 300 ( Figure 5 ).

[0052] According to an example, the learning procedure may be performed via operation of a computing device comprising one or more processors and one or more memories for storing one or more computer programs, wherein the one or more computer programs are arranged to cause the computing device to perform the learning procedure described in the present disclosure when executed by the one or more processors. According to another example, the learning procedure may be performed via use of a plurality (i.e., two or more) of the computing devices of the above kind (mutatis mutandis).

[0053] The learning model 300 applies the eye model 302 to transform each training image I t Processed into the corresponding reference retinal image I r , which corresponds to the training image I t The direct (or natural) viewing of the image is modeled without considering the effects of the NED component 201 caused by the combined effects of the pre-processing procedure and the lens component 203. r is derived for a given accommodation state of the eye, where the eye is assumed to be accommodated at a reference plane that is within the depth range of the NED assembly 201 and represents a conjugate plane of the retina of the viewer's eye 110. Reference retinal image I r The corresponding training image I in the framework of the learning model 300 t The eye model 302 may be arranged to model the chromatic aberrations of the (typical) eye optics and the diffraction-limited resolution of the eye 110 due to the finite size of the pupil 110a, thereby deriving a reference retinal image I r As an image that explains at least some of the optical limitations of the eye 110. This can be achieved, for example, via convolution of the corresponding training image with a diffraction-limited PSF representing the chromatic aberrations of the eye 110 with respect to the viewer. t Exported reference retinal image I r remains constant throughout the iterations, and thus the learning procedure may involve calculating for each training image I of the training data set before entering the iterative learning procedure or (only) during the first iteration of the iterative learning procedure t To determine the corresponding reference retinal image I r .

[0054] At each iteration of the learning procedure, each training image I t All of them are learned by the model 300, which makes the corresponding training image I t Do the following:

[0055] - For the corresponding iteration round, one or more sub-apertures of the lens plane are selected, and the corresponding training image I is determined tcorresponding one or more sub-regions of the image area of ​​, the one or more sub-regions spatially corresponding to the selected one or more sub-apertures,

[0056] - Considering the corresponding training image I t The corresponding position P of the one or more sub-regions determined in the image region n , using the current weights of at least one ANN 304, the at least one ANN 304 processes the one or more sub-regions into a pre-processed image I d One or more sub-regions corresponding in space,

[0057] - The pre-processed image I is converted by the display and eye model 306 according to the current phase delay profile of the display model and taking into account the selected modulation depth z for the corresponding iteration round. d The one or more sub-regions are processed into a simulated retinal image one or more spatially corresponding sub-regions of

[0058] - Determine the simulated retinal image through the loss function 308 The one or more sub-regions of the reference retinal image I r The difference between one or more spatially corresponding sub-regions.

[0059] At the end of each iteration, based on the simulated retinal image The corresponding one or more sub-regions are compared with the reference retinal image I r The weights of at least one ANN 304 and at least a portion of the phase delay profile corresponding to the one or more sub-apertures selected for the corresponding iteration round are updated based on the determined difference between the one or more sub-regions corresponding to the spatially selected one or more sub-apertures. As an example in this regard, the updating can be performed via the use of a gradient descent method. The learning procedure can be performed until the simulated retinal image is satisfied. and the corresponding reference retinal image I r , and / or until a predefined number of iterations have been performed. Once the learning procedure has converged and / or the predefined number of iterations have been completed, the weights of at least one ANN 304 at the end of the last iteration may be adopted as the weights of at least one ANN applied to implement the preprocessing procedure in the image preprocessing portion 212 of the display controller 210, and the phase delay profile at the end of the last iteration may be adopted as the phase delay Φ(s,t) of the DOE of the NED component 201.

[0060] According to an example, the learning model 300 may directly consider the parameters defining the phase delay profile, rather than the phase delay profile itself. In an example using a DOE, the phase delay profile is characterized by its material properties and thickness (e.g., according to Figure 2A In an example DOE 203b), parameters considered during the learning procedure via application of the learning model 300 may include a height map d(s,t) of positions across a lens plane of the lens assembly 203. In another example, a phase delay profile of an optical element used as a DOE may be considered via corresponding coefficients of a set of predefined phase functions in a suitable signal space (e.g., Zernike polynomials). In an example where a DOE is provided via the use of metamaterials / metasurfaces, phase delay definition parameters considered during the learning procedure may include one or more geometrical properties (such as radius and / or orientation) of nanostructures of the metasurface that implement positions (s,t) across the lens plane. Regardless of the manner in which the phase delay profile is defined during the learning procedure, the phase delay profile may be defined via defining corresponding phase delays for positions (s,t) of the lens plane, where the positions (s,t) may be from a substantially uniform grid above the lens plane or positions (s,t).

[0061] As described above, at each iteration round, the learning model 300 considers one or more sub-apertures of the lens plane and the training image I t One or more sub-regions of the image area of ​​, which one or more sub-regions spatially correspond to one or more sub-apertures considered at the corresponding iteration round. In this regard, as described above, for example with reference to Figure 3A and Figure 3B , each subaperture of the lens plane maps (to a corresponding subarea of ​​the display plane and hence) to a corresponding subarea of ​​the image area, wherein the mapping depends on the spatial relationship between the respective positions of the 2D display 202 and the lens assembly 203 and their distances from the pupil 110a. In this regard, Figure 5 The illustration of suggests that the corresponding one or more incident angles (θ n ,φ n ) to define one or more subapertures, the one or more incident angles being mapped to the image region at position P n The corresponding one or more sub-regions at.

[0062] According to the example, a single subaperture and a training image I are considered at each iteration round t At this point, in a certain iteration of the learning procedure, for all training images I tThe single sub-aperture of the applied lens plane may be selected (substantially) randomly from a plurality of predefined sub-apertures, while in another example, the sub-aperture applied for a certain iteration round may vary from one iteration round to another via selecting one of the plurality of predefined sub-apertures according to a predefined rule. The plurality of sub-apertures available for consideration in the learning procedure may have corresponding predefined positions within the lens plane, and the plurality of sub-apertures may have predefined shapes and sizes close to a typical (e.g., average) pupil 110a. As an example, the plurality of sub-apertures may together cover the entire lens aperture without any gaps between them. As non-limiting examples in this regard, the sub-apertures may have a rectangular or hexagonal shape, and the sub-apertures may be non-overlapping, or they may be partially overlapping.

[0063] According to another example, at each iteration round, a set of two or more sub-apertures and a training image I may be applied. t The method of claim 1 , wherein a set of two or more sub-apertures applied at a certain iteration round may be (substantially) randomly selected from a plurality of predefined sub-apertures, or a set of two or more sub-apertures applied for a certain iteration round may be selected from a plurality of predefined sub-apertures according to a predefined rule. As a specific example of the latter, the set of two or more sub-apertures selected for a certain (and each) iteration round may include a plurality of sub-apertures that together cover the entire lens aperture.

[0064] For each training image I t , the input to at least one ANN 304 at a given iteration round may include the corresponding training image I t The pixel values ​​within the image sub-region considered in the corresponding iteration round and the image region position P identifying the image sub-region considered n The output of at least one ANN 304 may include the preprocessed image I d The corresponding pixel value of the corresponding image sub-region. The position P of the image sub-region within the image region n This can be achieved, for example, via the corresponding incident angles (θ, φ) at the pupil 110a (e.g. Figure 5 As shown in FIG. 1 ), via corresponding pixel positions (ξ, η) on the display plane, or via corresponding positions (s, t) on the lens plane. As previously mentioned, in some examples, the learning procedure may also include defining the corresponding pixel positions (ξ, η) on the display plane, or via corresponding positions (s, t) on the lens plane. t (t) Associated depth map D t (t), and thus, the input to at least one ANN 304 may also include depth information of the image sub-region considered in the corresponding iteration round, which may be obtained from the depth information of the sub-region of the image considered.t The associated depth map D t Following the idea described above, at least one ANN 304 may include, for example, at least one CNN or at least one modified CNN.

[0065] The display and eye model 306 may include a physics-based differentiable simulation model that simulates the pre-processed image I d The transmission from the display plane through the lens assembly 203 is combined with the eye model to derive the corresponding simulated retinal image at the reference plane At this point, from the corresponding preprocessed image I d Exported simulated retina image The effects of the lens assembly 203 as well as the aberrations of the (typical) eye optics and the diffraction-limited resolution of the eye are modeled, thereby modeling the corresponding training image I via the NED assembly 201. t of viewing. Following the ideas described above, the modeling of the lens assembly 203 involves modeling the optical properties of one or more sub-apertures of the lens plane selected for the corresponding iteration round via the application of the current phase delay distribution map and taking into account the adjustment depth z selected for the corresponding iteration round, and the eye modeling can be performed in a similar manner to that described above for the eye model 302 with necessary modifications. Specifically, the modeling of optical properties may include modeling the display PSF according to the corresponding lens plane positions of the one or more sub-apertures selected for the corresponding iteration round and according to the adjustment depth z selected for the corresponding iteration round. The applied one or more sub-apertures of the lens aperture can be defined, for example, via: the corresponding incident angle (θ, φ) at the pupil 110a (such as Figure 5 as shown); via the corresponding pixel position (ξ,η) on the display plane; or via the corresponding position (s,t) in the lens plane (which defines the center point of the subaperture considered at the corresponding iteration round).

[0066] According to an example, the adjustment depth z applied for all training images in a certain iteration round of the learning procedure can be (substantially) randomly selected from a set of predefined adjustment depths or from a predefined range of adjustment depths (which covers the (desired) depth range of the NED component 201), while in another example, the adjustment depth applied for a certain iteration round can vary from one iteration round to another according to a predefined rule by selecting one of the predefined adjustment depths or by selecting the adjustment depth from a predefined range.

[0067] The loss function 308 can determine the simulated retinal image and the corresponding reference retinal image I rThe difference between them is used as an objective error metric, such as the L1 loss or based on the two images considered I r The mean square error (MSE) is derived from the pixel-by-pixel error (or difference) between the two images. Additionally or alternatively, the difference may be determined as a perceptual error metric, such as I r According to an example, the loss function 308 may also involve determining the corresponding image I r The so-called neural contrast sensitivity function (NCSF) is applied before comparing the difference between the two images in order to incorporate neural factors into the learning model. The use of NCSF can help by achieving an improved trade-off between spatial resolution and DoF provided by the NED component 201.

[0068] Although the above description of various examples relating to the application of the learning model 300 in an iterative learning procedure relates to general image processing, the images processed by the preprocessing procedure may include corresponding RGB images involving three separate color channels (as described above), and therefore, the training model 300 may be applied to consider the three color channels separately from each other during the iterative learning procedure.

[0069] The examples relating to the learning procedure provided above (implicitly) assume that the operation of the at least one ANN 304 involves the training of a single ANN arranged to learn from the pre-processed image I d The position P of each sub-region of the image region within the image region n Thus, as described above, during the learning procedure, the input to a single ANN may include the corresponding training image I t The pixel values ​​in the image sub-region considered in the corresponding iteration round and the information P identifying the image sub-region considered n , and the output of a single ANN can include the preprocessed image I d The corresponding pixel values ​​of the corresponding image sub-region (where the input to the single ANN may optionally also include corresponding depth information).

[0070] According to other examples, the operation of the at least one ANN 304 may involve training a plurality of ANNs, each ANN being arranged to process the pre-processed image I d Thus, at each iteration, one or more ANNs corresponding to the corresponding one or more image subregions considered at the corresponding iteration are trained and / or updated. Following the ideas described above, during the learning procedure, the input to the corresponding ANN may include the corresponding training image It The pixel values ​​in the corresponding image sub-region considered in the corresponding iteration round, and the output of the corresponding ANN may include the preprocessed image I d The corresponding pixel values ​​of the corresponding image sub-region (wherein the input to the corresponding ANN may optionally also include corresponding depth information).

[0071] Figure 6 A block diagram of some components of a device 400 is shown, which may be employed to implement at least some aspects of the display controller 210 or to perform the iterative learning procedures described above. Although described herein with reference to a single device 400, at least some aspects of the display controller 210 or the iterative learning procedures described above may be implemented via the joint operation of two or more devices 400 arranged to provide a cloud-based computing service.

[0072] The device 400 includes a processor 410 and a memory 420. The memory 420 can store data and computer program code 425. The device 400 can also include a communication device 430 for wired or wireless communication with other devices and / or user I / O (input / output) components 440, which can be arranged to provide a user interface together with the processor 410 and a part of the computer program code 425, for receiving input from a user and / or providing output to the user. Specifically, the user I / O components can include user input devices, such as one or more keys or buttons, keyboards, touch screens or touch pads, etc. The user I / O components can include output devices, such as displays or touch screens. The components of the device 400 are coupled to each other via a bus 450, which realizes the transmission of data and control information between the components.

[0073] The memory 420 and a portion of the computer program code 425 stored therein may also be arranged to, together with the processor 410, cause the device 400 to operate as the display controller 210 or to perform (where applicable) the iterative learning procedure described above. The processor 410 is configured to read from and write to the memory 420. Although the processor 410 is depicted as a corresponding single component, it may be implemented as one or more corresponding separate processing components. Similarly, although the memory 420 is depicted as a corresponding single component, it may be implemented as one or more corresponding separate components, some or all of which may be integrated / removable, and / or may provide permanent / semi-permanent / dynamic / cached storage.

[0074] The computer program code 425 may include computer executable instructions that, when loaded into the processor 410, implement at least some aspects of the display controller 210 or perform (where applicable) the iterative learning procedure described above. As an example, the computer program code 425 may include a computer program consisting of one or more sequences of one or more instructions. The processor 410 is capable of loading and executing the computer program by reading one or more sequences of one or more instructions included in the computer program from the memory 420. The one or more sequences of one or more instructions may be configured to cause the device 400 to operate as the display controller 210 or perform (where applicable) the iterative learning procedure described above when executed by the processor 410. Therefore, the device 400 may include at least one processor 410 and at least one memory 420 including computer program code 425 for one or more programs, the at least one memory 420 and the computer program code 425 being configured to cause the device 400 to perform at least some aspects of the display controller 210 or (where applicable) the iterative learning procedure described above together with the at least one processor 410.

[0075] Computer program code 425 may be provided, for example, as a computer program product comprising at least one computer readable non-transitory medium having computer program code 425 stored thereon, which, when executed by processor 410, causes device 400 to perform at least some aspects of display controller 210 or (where applicable) the iterative learning procedure described above. The computer readable non-transitory medium may include a memory device or a recording medium tangibly embodying the computer program. As another example, the computer program may be provided as a signal configured to reliably transmit the computer program.

[0076] References to processor(s) herein should be understood to cover not only programmable processors, but also special purpose circuits such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processors, etc. Features described in the foregoing may be used in combinations other than those explicitly described.

Claims

1. A stereoscopic near-eye display (NED) component (201) for a NED device, the NED device comprising a pair of NED components (201), the NED component (201) comprising: A two-dimensional 2D display (202) for rendering an image for viewing by a viewer's eyes (110); a lens assembly (203) arranged at a predefined distance from the 2D display (202) to allow viewing of the image rendered via the 2D display (202) through the lens assembly, wherein the lens assembly (203) comprises a diffractive optical element (DOE) (203b) arranged to provide different phase delays through a plurality of positions of an aperture of the diffractive optical element; and A display controller (210) comprising a preprocessing section (212) arranged to derive a preprocessed image (I) based on a received image (I) via application of a preprocessing program d ), the application of the preprocessing procedure being arranged to apply an image region position dependent preprocessing to the preprocessed image (I d ) in order to take into account the different transmission characteristics of the different sub-apertures of the lens aperture through the lens assembly (203) and to supply the pre-processed image (I d ) for rendering via the 2D display (202), and wherein the preprocessing program and the lens assembly (203) are arranged to convert the preprocessed image (I d ) is to display an image perceived as sharp for a plurality of predefined adjusted depths residing within an extended depth of field (DoF) of the NED component (201) so as to facilitate operation of the NED device based on the preprocessed image (I d ) to provide adjustment-invariant 3D rendering.

2. The NED assembly (201) according to claim 1, wherein: The preprocessing part (212) comprises at least one artificial neural network ANN, arranged to implement the preprocessing procedure.

3. The NED assembly (201) according to claim 2, wherein: The at least one ANN includes at least one convolutional neural network (CNN).

4. The NED assembly (201) according to any one of claims 1 to 3, wherein: The at least one ANN includes a single ANN, wherein the single ANN is trained to generate a plurality of images according to the pre-processed image (I d ) is processed by positioning each of the multiple sub-regions of the image region within the image region.

5. The NED assembly (201) according to any one of claims 1 to 3, wherein: The at least one ANN includes a plurality of ANNs, each ANN being trained to process the preprocessed image (I d ) of the image area.

6. The NED assembly (201) according to any one of claims 1 to 5, wherein: The DOE (203b) comprises an optical element having thicknesses individually defined for the plurality of locations to provide the phase delays that are different for the plurality of locations through an aperture of the optical element.

7. A device for deriving a pre-processing procedure and a phase delay distribution map for a stereoscopic near-eye display (NED) component (201) according to claim 1, the device being arranged to apply a corresponding learning model to obtain a phase delay distribution map based on a plurality of training images (I t ) jointly derive at least one artificial neural network ANN (304) acting as the preprocessor and the phase delay distribution map defining corresponding phase delays for a plurality of positions of the DOE (203b) via an iterative learning procedure, wherein, The apparatus is arranged to perform the following operations at a plurality of iteration rounds: Each of the plurality of training images is processed by a learning model (300), the learning model being arranged to perform the following operations: For the corresponding iteration round, one or more sub-apertures are selected, and the corresponding training images (I t ), the one or more sub-regions spatially corresponding to the selected one or more sub-apertures, Considering the corresponding training image (I t ) of the one or more sub-regions determined by the image region (P n ), using the current weights of at least one ANN (304), processing the one or more sub-regions by the at least one ANN (304) to determine a pre-processed image (I d ), one or more sub-regions corresponding in space to The pre-processed image (I ) is converted by the display and eye model (306) according to the current phase delay profile and taking into account the selected accommodation depth (z) for the corresponding iteration round. d ) into a simulated retinal image one or more spatially corresponding sub-regions of The simulated retinal image is determined by a predefined loss function (308) The one or more sub-regions of the corresponding reference retinal image (I r ) between one or more spatially corresponding sub-regions; and Based on the plurality of training images (I t ) to update the weights of the at least one ANN (304) and a portion of the phase delay distribution map that spatially corresponds to one or more selected sub-apertures based on the corresponding differences determined.

8. Apparatus according to claim 7, arranged to update the weights of the at least one ANN and the phase delay profile via use of a gradient descent method.

9. The device according to claim 7 or 8, arranged to derive the corresponding reference retinal image (I) via application of an eye model (302). r ), the eye model represents the eye relative to the corresponding training image (I t ) of one or more optical limitations.

10. The apparatus according to any one of claims 7 to 9, arranged to perform one of the following operations: selecting one of a plurality of predefined adjustment depths as the adjustment depth (z) for the corresponding iteration round, The adjustment depth (z) for the respective iteration round is selected from a predefined range of adjustment depths.

11. The apparatus according to any one of claims 7 to 10, arranged to select one of a plurality of predefined subapertures for the respective iteration round, in, Each predefined sub-aperture has a corresponding position within the lens aperture and has a corresponding predefined shape and size, and The multiple sub-apertures together cover the entire DOE aperture.

12. The apparatus according to any one of claims 7 to 10, arranged to select two or more predefined sub-apertures of a plurality of predefined sub-apertures for the respective iteration rounds, in, Each predefined sub-aperture has a corresponding position within the lens aperture and has a corresponding predefined shape and size, and The multiple sub-apertures together cover the entire DOE aperture.

13. The apparatus according to any one of claims 7 to 12, wherein: The display and eye model (306) is arranged to take into account the accommodation depth (z) selected for the respective iteration round, via the application of phase delay, based on the pre-processed image (I d ) to derive the pre-processed retinal image One or more spatially corresponding sub-regions of.

14. The device according to claim 13, wherein: The applying of the optical property comprises applying a point spread function PSF, which is selected according to the respective positions of the one or more subapertures selected for the respective iteration round, taking into account the modulation depth (z) selected for the respective iteration round.

15. The apparatus according to any one of claims 7 to 14, wherein: The at least one ANN comprises at least one convolutional neural network (CNN).

16. Apparatus according to any one of claims 7 to 15, wherein: The at least one ANN (304) includes a single ANN, the single ANN being trained to generate a plurality of images based on the pre-processed image (I d ) is processed by positioning each of the multiple sub-regions of the image region within the image region.

17. Apparatus according to any one of claims 7 to 15, wherein: The at least one ANN (304) includes a plurality of ANNs, each ANN being trained to process the pre-processed image (I d ) of the image area.

18. A method for deriving a pre-processing program and a phase delay profile for a stereoscopic near-eye display (NED) component (201) according to claim 1, wherein: The method comprises: based on a plurality of training images (I t ) to perform an iterative learning procedure to jointly derive at least one artificial neural network ANN (304) acting as the preprocessor and the phase delay distribution map defining corresponding phase delays for a plurality of positions of the DOE (203b), the method comprising for a plurality of iteration rounds: Each of the plurality of training images is processed by a learning model (300), the learning model comprising the steps of: For the corresponding iteration round, one or more sub-apertures are selected, and the corresponding training images (I t ), the one or more sub-regions spatially corresponding to the selected one or more sub-apertures, Considering the corresponding training image (I t ) of the one or more sub-regions determined by the image region (P n ), using the current weights of the at least one ANN (304), processing the one or more sub-regions by the at least one ANN (304) to determine a pre-processed image (I d ), one or more sub-regions corresponding in space to The pre-processed image (I ) is converted by the display and eye model (306) according to the current phase delay profile and taking into account the selected accommodation depth (z) for the corresponding iteration round. d ) into a simulated retinal image one or more spatially corresponding sub-regions of The simulated retinal image is determined by a predefined loss function (308) The one or more sub-regions of the corresponding reference retinal image (I r ) between one or more spatially corresponding sub-regions; and Based on the plurality of training images (I t ) to update the weights of the at least one ANN (304) and a portion of the phase delay distribution map that spatially corresponds to one or more selected sub-apertures based on the corresponding differences determined.

19. A computer program (425) comprising computer readable program code configured to cause the method according to claim 18 to be performed when the program code is executed on one or more computing devices (400).

Images