Enhanced computational hyperspectral imaging

Abstract

According to an embodiment, a method for determining via an iterative training procedure, a phase modulation function, a spectral transmission function and a post-processing procedure that are useable in a hyperspectral imaging, HSI, apparatus (100) is provided, wherein the HSI apparatus (100) comprises a lens assembly (110) comprising a diffractive optical element, DOE, (112) arranged to implement said phase modulation function that results in a phase delay that varies with the spatial position of an aperture of the DOE (112), an image sensor assembly (120) arranged to receive light through the lens assembly (110), the image sensor assembly (120) comprising a two-dimensional,2D, color filter array, CFA, (124) arranged on an image sensor (122) such that the light received through the lens assembly (110) enters the image sensor (122) through the CFA (124), wherein the CFA (124) is arranged to implement said spectral transmission function that results in a spectral transmission that varies with a wavelength and with a spatial position of the CFA (124); and a controller (130) comprising an image reconstruction portion arranged to derive the HS image based on light received on the image sensor (124) via application of said post-processing procedure that comprises an artificial neural network, ANN, the method comprising processing a plurality of training items through a training network (300) that represents an arrangement of the lens assembly (110), the CFA (124) and the post-processing procedure in the HSI apparatus (100) and that comprises respective training models (302, 304, 306) for determining the phase modulation function, the spectral transmission function and weights of the ANN, and wherein the training model (304) for determining the spectral transmission function accounts for one or more predefined implementation constraints of the CFA (124).

Description

[0001]Enhanced computational hyperspectral imaging TECHNICAL FIELD The present invention relates to hyperspectral imaging. BACKGROUND Hyperspectral imaging (HSI) provides an approach for combining spatial and spectral information to provide data that is applicable in applications such as remote sensing. A hyperspectral (HS) image combines two-dimensional (2D) spatial information with one-dimensional (1D) spectral information to form a three-dimensional (3D) HS data cube. A traditional approach for acquiring HS images includes scanning along one of the axes of the 3D HS data cube and accumulating measurements in other dimensions along the respective axis. While this is a proven method for acquiring HS images at a high spectral resolution, this is accomplished with the cost of either a compromised spatial resolution or a relatively slow image acquisition process. In most applications also a high spatial resolution is important, thereby resulting in HS image acquisition methods that suffer from relatively long image acquisition time. While this is a shortcoming that can be tolerated to some extent in applications that involve capturing HS images of static scenes, on the other hand it makes such scanning-based HS image acquisition methods inapplicable e.g. for applications that require capturing HS images of dynamic scenes. Computational HSI, also known as snapshot HSI, addresses this shortcoming of the scanning-based HSI via capturing the 3D HS data cube that constitutes the HS image within a single detector integration period via optical encoding of the captured image data through additional optical elements, which enables recovering the 3D HS data cube from the optically encoded image data using a post-processing algorithm designed for the purpose. The snapshot HSI technique enables significant decrease in the image acquisition time while ensuring high spatial and spectral resolution, thereby constituting a HSI technique that is also applicable for capturing HS images of dynamic scenes. Nevertheless, there is a further demand for improvement in terms of improving spatial and / or spectral resolution of the acquired HS images, in terms of reducing the image acquisition time, in terms of improved optical design of the equipment applied in capturing the images and / or in terms of characteristics of the post-processing algorithm applied in the image acquisition process. SUMMARY It is an object of the present invention to provide a technique for enhanced hyperspectral imaging. According to an embodiment, a system is provided, the system comprising one or more apparatuses arranged for carrying out an iterative training procedure to determine a phase modulation function, a spectral transmission function and a post-processing procedure that are useable in a hyperspectral imaging (HSI) apparatus provided for capturing a hyperspectral (HS) image of a scene, wherein the HSI apparatus comprises: a lens assembly comprising a diffractive optical element (DOE) arranged to implement said phase modulation function that results in a phase delay that varies with the spatial position of an aperture of the DOE; an image sensor assembly arranged to receive light through the lens assembly, the image sensor assembly comprising a two-dimensional (2D) color filter array (CFA) arranged on an image sensor such that the light received through the lens assembly enters the image sensor through the CFA, wherein the CFA is arranged to implement said spectral transmission function that results in a spectral transmission that varies with a wavelength and with a spatial position of the CFA; and a controller comprising an image reconstruction portion arranged to derive the HS image based on light received on the image sensor via application of said post-processing procedure that comprises an artificial neural network (ANN), wherein the one or more apparatuses are arranged to carry out the iterative training procedure via processing a plurality of training items through a training network that represents an arrangement of the lens assembly, the CFA and the post- processing procedure in the HSI apparatus and that comprises respective training models for determining the phase modulation function, the spectral transmission function and weights of the ANN, and wherein the training model for determining the spectral transmission function is arranged to account for one or more predefined implementation constraints of the CFA. According to another embodiment, a method for determining via an iterative training procedure, a phase modulation function, a spectral transmission function and a post-processing procedure that are useable in a hyperspectral imaging (HIS) apparatus is provided, wherein the HSI apparatus comprises a lens assembly comprising a diffractive optical element (DOE) arranged to implement said phase modulation function that results in a phase delay that varies with the spatial position of an aperture of the DOE, an image sensor assembly arranged to receive light through the lens assembly, the image sensor assembly comprising a two-dimensional (2D) color filter array (CFA) arranged on an image sensor such that the light received through the lens assembly enters the image sensor through the CFA, wherein the CFA is arranged to implement said spectral transmission function that results in a spectral transmission that varies with a wavelength and with a spatial position of the CFA; and a controller comprising an image reconstruction portion arranged to derive the HS image based on light received on the image sensor via application of said post-processing procedure that comprises an artificial neural network (ANN) the method comprising processing a plurality of training items through a training network that represents an arrangement of the lens assembly, the CFA and the post-processing procedure in the HSI apparatus and that comprises respective training models for determining the phase modulation function, the spectral transmission function and weights of the ANN, and wherein the training model for determining the spectral transmission function accounts for one or more predefined implementation constraints of the CFA. According to another embodiment, a computer program is provided, the computer program comprising computer readable program code configured to cause performing the method according to the embodiment described in the foregoing when said program code is executed on one or more computer apparatuses. The computer program according to the above-described example embodiment may be embodied on a volatile or a non-volatile computer- readable record medium, for example as a computer program product comprising at least one computer readable non-transitory medium having the program code stored thereon, which, when executed by one or more computing apparatuses, causes the computing apparatuses at least to perform the method according to the example embodiment described in the foregoing. The exemplifying embodiments of the invention presented in this patent application are not to be interpreted to pose limitations to the applicability of the appended claims. The verb "to comprise" and its derivatives are used in this patent application as an open limitation that does not exclude the existence of also unrecited features. The features described hereinafter are mutually freely combinable unless explicitly stated otherwise. Some features of the invention are set forth in the appended claims. Aspects of the invention, however, both as to its construction and its method of operation, together with additional objects and advantages thereof, will be best understood from the following description of some example embodiments when read in connection with the accompanying drawings. BRIEF DESCRIPTION OF FIGURES The embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, where Figure 1 schematically illustrates the concept of hyperspectral images via an example; Figure 2 schematically illustrates some aspects of a hyperspectral imaging apparatus according to an example; Figure 3 illustrates a block diagram of some components of a hyperspectral imaging apparatus according to an example; Figure 4 illustrates a block diagram that represents a training model according to an example; and Figure 5 illustrates a block diagram of some components of an apparatus according to an example. DESCRIPTION OF SOME EMBODIMENTS Along the lines described in the foregoing, a HS image may be represented as a 3D HS data cube, which combines 2D spatial information with 1D spectral information. In other words, a HS image provides a respective spectrum for each pixel of a 2D image. The HS image may be considered as an extension of a traditional RGB image, where respective magnitudes of red (R), green (G) and blue (B) components are provided for each pixel of the image. This concept is schematically illustrated in Figure 1 with the illustration (A) on the left representing a traditional RGB image and the illustration (B) on the right representing a HS image: the illustration (A) shows the R, G and B images, i.e. three 2D images that each represent pixel magnitudes at respective points of the frequency axis (i.e. at different wavelengths), whereas the illustration (B) shows a high number of 2D images the that each represent pixel magnitudes at respective points of the frequency axis. Typically, the 2D images that constitute a HS image cover a predefined range of wavelengths (e.g. from 400 to 1100 nanometers) at relatively small spacing in the wavelength axis (e.g.10 nanometers or less). Figure 2 schematically illustrates a HSI apparatus 100 according to an example. The HSI apparatus 100 is provided for capturing a HS image of a scene and it comprises a lens assembly 110, an image sensor assembly 120 and a device controller 130. The lens assembly 110 comprises a diffractive optical element (DOE) 112, which defines an aperture through which the light enters the HIS apparatus 100. The DOE 112 is arranged to implement a predefined phase modulation function that results in a phase delay that varies with a spatial position of the aperture of the DOE 112. The image sensor assembly 120 is positioned with respect to the lens assembly 110 such that it receives the through the lens assembly 110 and the image sensor assembly 110 comprises an image sensor 122 and a 2D color filter array (CFA) 124. In the following, the 2D CFA 124 is referred to, in brief, as the CFA 124 The CFA 124 is arranged on the image sensor 122 such that the light entering the HIS apparatus 100 through the lens assembly 110 enters the image sensor 122 through the CFA 124. The CFA 124 is arranged to implement a predefined spectral transmission function that results in a spectral transmission that varies with wavelength and with a spatial position on the CFA 124. The device controller 130 comprises an image reconstruction portion 132 is arranged to derive the HS image based on light received at the image sensor 122 via applying a predefined post-processing procedure, where the post-processing procedure is implemented as an artificial neural network (ANN). The phase modulation function implemented by the DOE 112 of the lens assembly 110 and the spectral transmission function implemented by the CFA 124 of the image sensor assembly 120 provide an optical encoding for the light entering the HSI apparatus 100 through the lens assembly 110, whereas the image reconstruction portion 132 of the device controller 130 serves to reconstruct the optically encoded 2D image captured at the image sensor 122 into the HSI image that represents the scene. The HSI apparatus 100 hence provides computational HSI (a.k.a. snapshot HSI) that allows for capturing a HS image of a scene of interest within a single detector integration period, thereby rendering the HSI apparatus 100 suitable for capturing HS images of both static and dynamic scenes. Along the lines described in the foregoing, the lens assembly 110 comprises at least the DOE 112 that is arranged to implement the predefined phase modulation function for the light transmitted therethrough. According to an example, the DOE 112 may be provided as a lens element arranged to implement the predefined phase modulation function via having a varying thickness at different positions of its aperture to provide a respective desired phase delay through the respective different positions of the aperture, whereas in another example the DOE 112 may be provided via application of phase- modulating metamaterial arranged e.g. as a meta-surface, where a nanostructure forming the meta-surface is arranged to provide a respective desired phase delay through a plurality of positions of the meta-surface. In this regard, the desired phase modulation function may be provided via controlling at least one aspect (e.g. orientation and / or size) of a nanostructure of the meta- surface at different positions of its ‘aperture’ accordingly. In the context of the present disclosure, also the meta-surface implementation may be considered as an optical element due to its function in shaping optical characteristics of the light transmitted therethrough. In addition to the phase modulation function implemented by the DOE 112, the lens assembly 110 may be further arranged to provide one or more further optical functions such as magnification and / or focusing. Assuming implementation of the DOE 112 as a lens element, in some examples, a single optical component may be applied to implement both the DOE 112 and the one or more further optical functions, whereas in other examples separate optical components may be applied for implementing the DOE 112 and the one or more further optical functions. If, instead, assuming provision of the DOE 112 as a meta-surface of phase-modulating metamaterial, a dedicated optical element may be applied to implement the one or more further optical functions. In a particular example in this regard, the lens assembly 110 may comprise a refractive lens 114, as in the example of Figure 2, and an optical element that implements the DOE 112 in the same optical axis, where the refractive lens 114 may be one that provides a desired magnification and / or focusing. In such a design, in one example the lens element that implements the DOE 112 may be integrated to the refractive lens 114, whereas in another example the optical element that implements the DOE 112 may be physically separated from the refractive lens 114. Referring back to the phase modulation function provided by the DOE 112, the phase delay through the DOE 112 may vary from one position of its aperture to another. In this regard, the phase delay through a position (s, t) of the aperture of the DOE 112 may be denoted as ^ ^(s, t). If using a lens element to implement the DOE 112, as described in the foregoing, the phase modulation function may be provided via a non-uniform thickness of the lens element such that lens thickness at a position (s, t) is selected such that it provides a desired phase delay ^ ^(s, t) therethrough. The thickness of the lens element serving to provide the DOE 112 across the positions (s, t) of its aperture may be defined as a height map d(s, t). While in theory the phase delay ^ ^(s, t) may be a continuous function of the position (s, t) of the lens aperture, in a practical implementation the phase delay ^ ^(s, t) may be discretized such that the lens aperture positions (s, t) are sampled along the ’s axis’ by Δsand along the ‘t axis’ by Δt. The phase delay ^ ^(s, t) providing the desired phase modulation function may be found via a training procedure described in the following, whereas the phase delay ^ ^(s, t) may be translated into the height map d(s, t) in consideration of optical characteristics of the material applied for implementing the lens element serving as the DOE 112. Along the lines described in the foregoing, the image sensor assembly 120 comprises the CFA 124 arranged on the surface of the (2D) image sensor 122 such that the light entering the HSI apparatus 100 through the lens assembly 110 enters the image sensor 122 through the CFA 124. The predefined spectral transmission function implemented by the CFA 124 results in a spectral transmission through the CFA 124 that varies with a wavelength and with a spatial position on the CFA 124. In this regard, the spectral transmission function at a spatial position (x, y) of the CFA 124 at a wavelength ^ may be denoted as ^ ^(x, y, ^). The CFA 124 may be aligned with the image sensor 122 such that the spatial position (x, y) of the CFA 124 spatially coincides with a spatial position (x, y) on the image sensor 122. In this regard, the positions (x, y) on the image sensor 122 may correspond to pixels of the (2D) image sensor 122, the CFA 124 thereby having a spatial resolution that matches that of the image sensor 122. While in theory the spectral transmission function ^ ^(x, y, ^) may be a continuous function of the spatial position (x, y) of the CFA 124, in a practical implementation the spectral transmission function ^ ^(x, y, ^) may be discretized such that the CFA 124 positions (x, y) are sampled along the ’x axis’ by Δx and along the ‘y axis’ by Δywith the spectral sampling Δ^. In some examples, the spectral transmission function ^ ^(x, y, ^) may be spatially periodic with N pixels along the x and y directions, where a suitable value for N may be found before the training procedure (by setting it as a hyperparameter) or the value for N may be determined as part of the training procedure. The suitable value for N depends on many factors, such as the size of the pixels on the image sensor 122, the numerical aperture of the HSI apparatus 100, the desired number of spectral channels to be resolved, etc. and value of N is typically larger than 3 pixels. In an example, the CFA 124 may be implemented as a transmissive meta- optical element that has different spectral transmission characteristics through a plurality of its spatial positions (x, y) such the predefined spectral transmission function ^ ^ ^(x, y, ^) is provided over a predefined range of wavelengths ^. In a non-limiting example in this regard, the transmissive meta- optical element comprises a transmissive metal-insulator-metal (MIM) structure having a non-uniform thickness that is defined separately for the plurality of spatial positions of (x, y) of the CFA 124 such that the MIM structure is arranged to provide the predefined spectral transmission function ^ ^(x, y, ^) via suitable selection of respective thicknesses for the insulator and metal layers of the MIM structure at the plurality of spatial positions (x, y). Alternatively or additionally, the spectral transmission function ^ ^(x, y, ^) may provide different spectral transmission characteristics through different spatial positions (x, y) the MIM structure via variations in respective materials applied to provide the insulator layer and / or the metal layers of the MIM structure. According to an example, the spectral transmission function ^ ^(x, y, ^) through a pixel at spatial position (x, y) at the wavelength ^ of a transmissive meta- optical element serving as the CFA 124 may be modeled as a Gaussian function defined as where ^ denotes a mean value of a Gaussian spectral transmission profile and ^ denotes a standard deviation of the Gaussian spectral transmission profile. The parameters ^ and ^ enable modeling of the spectral transmission characteristics of the pixels of the meta-optical element, and they may be defined via the training procedure described in the following. In another example in this regard, the spectral transmission function ^ ^(x, y, ^) through a pixel at spatial position (x, y) at the wavelength ^ of the transmissive meta-optical element may be modeled as a Gaussian function defined as where ^ ^( ^) denotes a predefined function that defines the standard deviation of the Gaussian spectral transmission profile as a function of the mean value ^ of a Gaussian spectral transmission profile. In this another example, the training procedure is applicable for defining the mean parameter ^ while the standard deviation parameter ^ is definable via the predefined function ^ ^( ^) in dependence of the mean parameter ^. This approach serves to model the constraint arising from the structure of the meta-optical element that may enable implementing a Gaussian function with a given mean value ^ together with a specific standard deviation ^ ^( ^) associated with the respective mean value ^. While the examples described above refer to using a Gaussian function with its parameters defined separately for the spatial positions (x, y) of the transmissive meta-optical element to model the spectral transmission function ^ ^(x, y, ^), in other examples the spectral transmission function ^ ^(x, y, ^) may be modeled via usage of a non-Gaussian function with its one or more parameters defined separately for the spatial positions (x, y) of the transmissive meta-optical element, where the training procedure is applicable for defining the one or more parameters of such a function for each of the spatial positions (x, y). Further in this regard, one or more physical parameters or characteristics of the transmissive meta-optical element may be chosen such that they implement respective desired Gaussian or non-Gaussian spectral transmission functions through respective spatial positions (x, y) of the transmissive meta-optical element. As an example in this regard, respective thicknesses of the metal layers and the insulation layer at the spatial positions (x, y) of the MIM structure may be chosen such that the respective desired Gaussian or non-Gaussian spectral transmission function therethrough is provided. Non-limiting examples of applicable non-Gaussian spectral transmission functions include ones based on an asymmetric Gaussian function, on a skewed generalized t distribution, and on a spline. As an example of the latter, the spectral transmission function may be modelled as a uniform spline of degree n, represented by a linear combination of B-splines of the same degree: where ^^^^( ^^) denotes a B-spline of degree n, ^^[ ^^] denotes expansion coefficients of the linear combination, and^^ ^^denotes the wavelength at which the spline is centered and where a B-spline of degree n centered at the origin is defined as where ^^+^^denotes the truncated power function max(0, ^^)^^. According to another example, the spectral transmission through meta-optical element serving as the CFA 124 relies on selecting one of a plurality of predefined spectral transmission profiles for each of the plurality of spatial positions (x, y) of the CFA 124, where the training procedure may be applied to make the selection from the plurality of predefined spectral transmission profiles for each spatial position (x, y) of the CFA 124. Figure 3 illustrates a block diagram of some components of the device controller 130 according to an example, including a control portion 134 and the image reconstruction portion 132. Along the lines described in the foregoing, the image reconstruction portion 132 is arranged to derive the HS image based on the 2D image captured on the image sensor 122 via application of the post- processing function implemented as an ANN. In this regard, the ANN takes the optically encoded image captured on the image sensor 122 as input and provides the corresponding reconstructing HS image as output. Herein, any ANN known in the art may be applied after having been trained for the purpose using the training procedure described in the following. In a non-limiting example in this regard, the ANN comprises a deep neural network (DNN), such as a residual dense network or a deep unrolled network. Usage of a deep unrolled network as the ANN is advantageous due to providing accurate reconstruction of the optically encoded 2D image and due to its high computation speed. This is enabled via each layer of a deep unrolled network being based on a mathematical model, while spatial and spectral features are directly extracted from the data. The control portion 134 may be arranged to control various aspects related to operation of the HSI apparatus 100. As a particular example in this regard, the image reconstruction portion 132 may operate at least partially under control of the control portion 134. In this regard, the control portion 134 may be arranged to instruct the image reconstruction portion 132 to a (2D) image from the image detector 122, apply the post-processing procedure to the image acquired from the image detector 122 to derive the corresponding HS image, and supply the derived HS image for further processing by another entity and / or for storage in a memory provided in or otherwise accessible by the device controller 130. In this regard, operation of the HSI apparatus 100 may involve capturing an optically encoded image I’(x, y, ^) that represents a scene that can be defined by an ‘ideal’ HS image I(x’, y’, ^) at the image sensor 122 and applying the post-processing procedure of the image reconstruction portion 132 to reconstruct a HS image Î(x, y, ^) based on the optically encoded image Is(x, y). Operation of the control portion 134 may be at least partially controlled via control input provided thereto, e.g. via a user interface associated with the HSI apparatus 100. According to an example, the device controller 130 may be implemented via operation of a computer apparatus that comprises one or more processors and one or more memories for storing one or more computer programs, where the one or more computer programs are arranged to cause the computer apparatus to operate as the device controller 130 according to the present disclosure when executed by the one or more processors. Along the lines described in the foregoing, respective parameters that define the predefined phase modulation function implemented by the DOE 112 of the lens assembly 110, the spectral transmission function implemented by the CFA 124 of the image sensor assembly 120 and the post-processing procedure implemented by the image reconstruction portion 134 of the device controller 130 may be determined via the training procedure. In particular, an iterative training procedure may be applied to determine the phase modulation function, the spectral transmission function and the post-processing procedure, where respective training models are applied to determine the phase modulation function, the spectral transmission function and the post- processing procedure in a joint procedure. According to an example, each of the phase modulation function and the spectral transmission function comprise a respective differentiable function that enables application of a backpropagation model that takes advantage of differentiability of these functions. According to an embodiment, the training procedure may be carried out by a training system assigned for the purpose, where the training system may comprise a single computer apparatus arranged to carry out the training procedure or the training system may comprise an arrangement of two or more computer apparatuses arranged jointly carry out the training procedure. In the latter approach, the two or more computer apparatuses may constitute a cloud computing service. Each apparatus involved in providing the training system may comprise one or more processors and one or more memories for storing one or more computer programs, where the one or more computer programs are arranged to cause the computer apparatus to operate as the respective apparatus of the training system when executed by the one or more processors. The training procedure relies on supervised learning carried out based on a plurality of training items, which may be also referred to as training images and the plurality of training items (or training images) forms a training dataset, whereas the training procedure involves processing the training items through a training network that represents the arrangement of the lens assembly 110, the CFA 124 and the post-processing procedure. A particular aspect of the training network is that it accounts for the implementation constraints of the CFA 124, thereby directly resulting in a spectral transmission function that can be implemented substantially without deviation further approximations or deviations from its intended form with the phase modulation function and the post-processing procedure optimized accordingly. While such an approach may not result in a theoretically optimal spectral transmission function for the CFA 124, it enables optimizing the phase modulation function of the DOE 112 and the post-processing procedure of the image reconstruction portion 134 to account for any compromises in the spectral transmission function arising from its implementation constraints considered in the training model applied for derivation of the spectral transmission function of the CFA 124. The training dataset may comprise N training items, where a training item of the training dataset may be denoted as ^^^^(x’, y’, ^) and it may comprise a respective ground-truth (GT) HS image, which may be provided e.g. as a respective 3D HS data cube. Figure 4 illustrated a block diagram that represents a training network 300) that represents the arrangement of the lens assembly 110, the CFA 124 and the post-processing procedure in the HSI apparatus (100) and that comprises respective training models for determining the phase modulation function of the DOE 112, the spectral transmission function of the CFA 124 and the weights of the ANN of the post-processing procedure. In particular, the training network 300 may be applied to process each training item ^^^^(x’, y’, ^) through a lens assembly model 302, a CFA and sensor model 304 and an ANN 306 to derive a corresponding reconstructed training item ^^^^(x, y, ^) in each iteration round of the iterative training procedure. The reconstructed training item ^^^^(x, y, ^) may be also referred to as a respective reconstructed training image ^^^^(x, y, ^). Moreover, the training network 300 involves a loss function 308 for determining a difference ^^^^( ^^^^, ^^^^) between the reconstructed training item ^^^^(x, y, ^) and the corresponding training item^^ ^^(x’, y’, ^), whereas the differences^^ ^^(^^ ^^,^^ ^^) are backpropagated through the ANN 306 and through the CFA and sensor model 304 to the lens assembly model 302 and respective parameters in each of the lens assembly model 302, the CFA and sensor model 304 and the ANN 306 are updated based on the differences ^^^^( ^^^^, ^^^^) after each iteration round. In a particular example, each of the phase delay ^ ^(s, t), the spectral transmission function ^ ^(x, y, ^) and weights wi,j of the ANN are initialized into respective suitable values at the beginning of the training procedure and at each iteration round of the training procedure, each training item ^^^^(x’, y’, ^) of the training dataset is passed through the training network 300, which results in carrying out the following operations based on the respective training item ^^^^(x’, y’, ^): - process the respective training item ^^^^(x’, y’, ^) by the lens assembly model 302 into a corresponding phase-encoded training image^^′ ^^(x, y, ^) in accordance with current phase delay t); - process the phase-encoded training image ^^′^^(x, y, ^) by the CFA and sensor model 304 into a corresponding optically encoded training image y) in accordance with current spectral transmission function ^ ^(x, y, ^); - process the optically encoded training image y) by the ANN 306 into a corresponding reconstructed training item ^^^^(x, y, ^) using current weights wi,j of the ANN 306; and - determine a difference ^^^^( ^^^^, ^^^^) between the reconstructed training item ^^^^(x, y, ^) and the corresponding training item ^^^^(x’, y’, ^) by the loss function 308. The respective initial values of the phase delay t), the spectral transmission function ^ ^(x, y, ^) and the weights wi,j of the ANN may comprise, for example, respective random or pseudo-random values. In another example, the respective initial values for one or more of the phase delay ^ ^(s, t), the spectral transmission function ^ ^(x, y, ^) and the weights wi,j may be initialized into (or based on) respective values obtained from an earlier application of the training procedure. Along the lines described in the foregoing, each of the training items In(x’, y’, ^) may comprise a respective HS image, which may be provided as a respective 3D HS data cube. Along similar lines, each of the phase-encoded training image ^^′^^(x, y, ^) and the reconstructed training item may comprise a respective HS image (likewise provided as a respective 3D HS data cube). The optically encoded training image y) may comprise a respective 2D image, provided e.g. as a 2D array of pixel values. At the end of each iteration round, the phase delay ^ ^(s, t), the spectral transmission function ^ ^(x, y, ^) and the weights wi,j of the ANN are updated based on the determined differences ^^^^( ^^^^, ^^^^) obtained at the respective iteration round. As an example in this regard, the updating may be carried out via usage of a gradient descent method known in the art. The iteration rounds according to the training procedure may be carried out until one or more predefined convergence criteria that pertain to the differences ^^^^( ^^^^, ^^^^) are met and / or until a predefined number of iteration rounds have been carried out. Once the training procedure has converged and / or the predefined number of iteration rounds have been completed, the phase delay ^ ^(s, t) at the end of the final iteration round may be adopted for implementation by the DOE 112, the spectral transmission function ^ ^(x, y, ^) at the end of the final iteration round may be adopted for implementation by the CFA 124, and the weights of the ANN 306 at the end of the final iteration round may be adopted as the weights of the ANN of the post-processing function of the image reconstruction portion 132. According to an example, the training network 300 may directly consider the phase delay ^ ^(s, t) as part of the lens assembly model 302, whereas in another example the training network 300 may consider one or more parameter that indirectly define the phase delay ^ ^(s, t). In scenarios where a lens element is applied to implement the DOE 112 and hence (known) properties of the material applied to implement the lens element and thickness of the lens element at different positions (s, t) of its aperture may be applied to characterize the phase delay ^ ^(s, t), according to an example, the parameter(s) considered in the training procedure may comprise the height map d(s, t) across positions of the lens aperture of the lens element serving to implement the DOE 112. In another example, the parameters that indirectly define the phase delay ^ ^(s, t) for a lens element implementing the DOE 112 may comprise respective coefficients of a set of predefined phase functions in a suitable signal space, e.g. Zernike polynomials. In scenarios where the DOE 112 is provided via usage of a metamaterial / meta-surface, the parameter(s) that define the phase delay ^ ^(s, t) may comprise one or more geometric characteristics (such as radius and / or orientation) of the nanostructure implementing the meta-surface across positions (s, t) of the aperture of the DOE 112. According to an example, the training network 300 may directly consider the spectral transmission function ^ ^(x, y, ^) as part of the CFA and sensor model 304, whereas in another example the training network 300 may consider one or more parameters that indirectly define the spectral transmission function ^ ^(x, y, ^). As a particular example of the latter approach, the training network 300 may consider one or more parameters that define the meta-optical element, such as the MIM structure, that is applied to implement the CFA 124. This is advantageous in that it enables convenient manner of accounting for the implementation constraints of the meta-optical element in order to find parameter(s) that can be realized in the practical implementation of the CFA 124, which in turn enables fine-tuning the phase delay ^ ^(s, t) of the phase modulation function implemented by the DOE 112 and the weights wi,j of the ANN serving as the post-processing procedure of the image reconstruction portion 132 to match the actual implementation of the CFA 124. In case the spectral transmission function ^ ^(x, y, ^) through the meta-optical element serving as the CFA 124 is considered via the Gaussian function described in the foregoing, the parameters learned via application of the CFA and sensor model 304 comprise the mean parameter ^ and the standard deviation parameter ^. In this regard, the CFA and sensor model 304 may be constrained to consider mean parameter ^ and standard deviation parameter ^^ values within respective predefined ranges, i.e. ^min < ^ < ^max and ^^min < ^^ < ^^max, respectively, to ensure ending up with respective values for the mean parameter ^ and standard deviation parameter^^that can be implemented as part of the meta-optical element. In case the spectral transmission function ^ ^(x, y, ^) through the meta-optical element serving as the CFA 124 is considered via the Gaussian function described in the foregoing, the parameter(s) learned via application of the CFA and sensor model 304 comprise the mean parameter ^, whereas the standard deviation parameter associated with a given value of the mean parameter ^ is derived by using the predefined function ^ ^( ^). As described in the foregoing, this approach serves to model the constraint arising from the structure of the meta-optical element that may enable implementing a Gaussian function with a given mean value ^ together with a specific standard deviation ^ ^( ^) associated with the respective mean value ^. Moreover, the CFA and sensor model 304 may be further constrained to consider mean parameter ^ values within the predefined range, i.e.^min<^<^max(as described in the foregoing with the previous example). In case the spectral transmission function ^ ^(x, y, ^) through the meta-optical element serving as the CFA 124 is considered via the plurality of predefined spectral transmission profiles (as also described in the foregoing), the parameters learned via application of the CFA and sensor model 304 include selection of one of the plurality of predefined spectral transmission profiles for each spatial position of the CFA 124. In case the meta-optical element comprises the MIM structure, in one example, the spectral transmission profiles selected for the plurality of spatial positions of the CFA 124 map to the respective insulator thicknesses of the respective spatial positions of the MIM structure, whereas in another example the spectral transmission profiles selected for the plurality of spatial positions of the CFA 124 map to the respective combinations of metal layer thicknesses and insulator thickness in the respective spatial positions of the MIM structure. In examples that make use of the MIM structure as the meta optical element serving as the CFA 124, the parameter(s) learned via application of the CFA and sensor model 304 include respective one or more layer thicknesses for the spatial positions (x, y) of the MIM structure. Examples in this regard include the following: - Learning, for each spatial position (x, y) of the MIM structure, a respective insulator layer thickness while assuming uniform thickness for the metal layers throughout the spatial positions (x, y) of the MIM structure. In other words, the assumption of uniform metal layer thickness throughout the MIM structure may be applied as an implementation constraint of the MIM structure accounted for in the CFA and sensor model 304. - Learning, for each spatial position (x, y) of the MIM structure, respective metal layer thicknesses while assuming uniform thickness for the insulation layer throughout the spatial positions (x, y) of the MIM structure. In other words, the assumption of uniform insulation layer thickness throughout the MIM structure may be applied as an implementation constraint of the MIM structure accounted for in the CFA and sensor model 304. - Learning, for each spatial position (x, y) of the MIM structure, a respective insulator layer thickness and respective metal layer thicknesses. Along the lines described in the foregoing, the layer thickness(es) selected for a spatial position (x, y) of the MIM structure may implement a Gaussian transmission function or a non-Gaussian transmission function. In this regard, the CFA and sensor model 304 may be constrained, for example, via considering a respective predefined set of insulator thicknesses and metal layer thickness (as applicable) or via considering insulator thicknesses and metal layer thicknesses (as applicable) within a respective predefined thickness range, where the predefined set(s) of thicknesses or the predefined range(s) of thicknesses (as applicable) are chosen such that learned values may be also implemented in practice. As the examples described above relate to learning that results in determining the thicknesses for the insulator layer and / or for the metal layers of the MIM structure with an implicit assumption of using a predefined metal (having predefined spectral transmission characteristics) for implementing the metal layers throughout the MIM structure, such learning may be considered to be (further) constrained via preselection of the metal material (and its known spectral transmission characteristics) to be applied for implementing the MIM structure. In further examples, alternatively or additionally, the learning may consider a choice of metal to be applied for implementing the metal layers at different spatial positions (x, y) of the MIM structure, where the choice of metal is made from a predefined set of two or more metals (with respective predefined spectral transmission characteristics). Examples in this regard include the following: - Learning, for each spatial position (x, y) of the MIM structure, a respective metal to be applied for implementing one of the metal layers of the MIM structure while assuming the predefined metal for implementing the other one of the metal layers of the MIM structure. Hence, the learning is constrained via the set of two or more available metals for the one metal layer are via the preselection of the metal for the other metal layer. - Learning, for each spatial position (x, y) of the MIM structure, a respective first metal to be applied for implementing one of the metal layers of the MIM structure and a respective second metal to be applied for implementing the other one of the metal layers of the MIM structure. Hence, the learning is constrained via the set of two or more available metals for the two metal layers. Along the lines described in the foregoing, the input to the ANN 306 of the training network 300 takes the optically encoded training image y) as input and provides the corresponding reconstructed training image ^^^^(x, y, ^) as output. Hence, the ANN 306 of the training network 300 may directly model the ANN applied as the post-processing procedure of the image reconstruction portion 132. The loss function 308 may determine the difference^^ ^^(^^ ^^,^^ ^^) between the reconstructed training image ^^^^(x, y, ^) and the corresponding training image ^^^^(x’, y’, ^) as an objective error measure such as the L1-loss or the mean squared error (MSE) derived based on pixel-wise error (or difference) between the respective pluralities of 2D images that constitute the respective HS images under consideration. Additionally or alternatively, the difference may be determined as a perceptual error measure such as the structural similarity measure (SSIM) that aims at predicting perceived error (or difference) between the two images^^ ^^(x, y, ^) and^^ ^^(x’, y’, ^) under consideration. In an exemplifying scenario, the training items^^ ^^(x’, y’, ^) of the training dataset applied for the training procedure are respective HS images that represent respective planar objects located at a fixed (predefined) depth from the HSI apparatus 100. In an example within this scenario, the lens assembly 110 may comprise the refractive lens 114 focused at said fixed (predefined) depth. Consequently, the training procedure results in optimizing the phase modulation function of the DOE 112, the spectral transmission function of the CFA 124 and the post-processing procedure of the image reconstruction portion 132 for a depth of field (DoF) that corresponds to the fixed (predefined) depth applied for the training items^^ ^^(x’, y’, ^) of the training dataset. In another exemplifying scenario, the training items ^^^^(x’, y’, ^) of the training dataset applied for the training procedure include respective HS images that represent 3D scenes that reside with a predefined depth range from the HSI apparatus 100, which is extended around a fixed (predefined) depth focused by the refractive lens 114 of the lens assembly 110. Consequently, the training procedure results in optimizing the phase modulation function of the DOE 112, the spectral transmission function of the CFA 124 and the post-processing procedure of the image reconstruction portion 132 for an extended depth of field (EDoF) in the predefined depth modeled by the training items ^^^^(x’, y’, ^) of the training dataset. While various aspects of the iterative training procedure are described in the foregoing with references to the training system that comprises one or more computer apparatuses arranged to carry out the training procedure, according to another embodiment operations that constitute the training procedure may be provided as a method for determining the phase modulation function, the spectral transmission function and the post-processing procedure useable in the HIS apparatus 100, where the method comprises processing the plurality of training items through the training network 300 described in the foregoing via a number of examples to apply the lens assembly model 302, the CFA and sensor model 304 and the ANN 306 to determine, respectively, the phase modulation function, the spectral transmission function and weights of the ANN such that CFA and sensor model 304 is applied to account for one or more predefined implementation constraints of the CFA 124. Figure 5 illustrates a block diagram of some components of an apparatus 400 that may be employed to implement at least some aspects of the device controller 130 or to serve as a computer apparatus of the training system described in the foregoing. Although described herein with references to a single apparatus 400, the at least some aspects of the device controller 130 (e.g. those pertaining to the image reconstruction portion 132) or the training system may be implemented via joint operation of two or more apparatuses 400 arranged to provide a cloud-based computing service. The apparatus 400 comprises a processor 410 and a memory 420. The memory 420 may store data and computer program code 425. The apparatus 400 may further comprise communication means 430 for wired or wireless communication with other apparatuses and / or user I / O (input / output) components 440 that may be arranged, together with the processor 410 and a portion of the computer program code 425, to provide the user interface for receiving input from a user and / or providing output to the user. In particular, the user I / O components may include user input means, such as one or more keys or buttons, a keyboard, a touchscreen or a touchpad, etc. The user I / O components may include output means, such as a display or a touchscreen. The components of the apparatus 400 are communicatively coupled to each other via a bus 450 that enables transfer of data and control information between the components. The memory 420 and a portion of the computer program code 425 stored therein may be further arranged, with the processor 410, to cause the apparatus 400 to operate as the device controller 130 or as a computer apparatus of the training system (as applicable). The processor 410 is configured to read from and write to the memory 420. Although the processor 410 is depicted as a respective single component, it may be implemented as respective one or more separate processing components. Similarly, although the memory 420 is depicted as a respective single component, it may be implemented as respective one or more separate components, some or all of which may be integrated / removable and / or may provide permanent / semi- permanent / dynamic / cached storage. The computer program code 425 may comprise computer-executable instructions that implement at least some aspects of the device controller 130 or a computer apparatus of the training system described in the foregoing (as applicable) when loaded into the processor 410. As an example, the computer program code 425 may include a computer program consisting of one or more sequences of one or more instructions. The processor 410 is able to load and execute the computer program by reading the one or more sequences of one or more instructions included therein from the memory 420. The one or more sequences of one or more instructions may be configured to, when executed by the processor 410, cause the apparatus 400 to operate as the device controller 130 or as a computer apparatus of the training system (as applicable). Hence, the apparatus 400 may comprise at least one processor 410 and at least one memory 420 including the computer program code 425 for one or more programs, the at least one memory 420 and the computer program code 425 configured to, with the at least one processor 410, cause the apparatus 400 to perform at least some aspects of the device controller 130 or a computer apparatus of the training system (as applicable). The computer program code 425 may be provided e.g. as a computer program product comprising at least one computer-readable non-transitory medium having the computer program code 425 stored thereon, which computer program code 425, when executed by the processor 410 causes the apparatus 400 to perform at least some aspects of the device controller 130 or a computer apparatus of the training system (as applicable). The computer-readable non- transitory medium may comprise a memory device or a record medium that tangibly embodies the computer program. As another example, the computer program may be provided as a signal configured to reliably transfer the computer program. Reference(s) to a processor herein should not be understood to encompass only programmable processors, but also dedicated circuits such as field- programmable gate arrays (FPGA), application specific circuits (ASIC), signal processors, etc. Features described in the preceding description may be used in combinations other than the combinations explicitly described.

Claims

Claims 1. A system comprising one or more apparatuses for carrying out an iterative training procedure to determine a phase modulation function, a spectral transmission function and a post-processing procedure that are useable in a hyperspectral imaging, HSI, apparatus (100) provided for capturing a hyperspectral, HS, image of a scene, wherein the HSI apparatus (100) comprises: a lens assembly (110) comprising a diffractive optical element, DOE, (112) arranged to implement said phase modulation function that results in a phase delay that varies with the spatial position of an aperture of the DOE (112); an image sensor assembly (120) arranged to receive light through the lens assembly (110), the image sensor assembly (120) comprising a two-dimensional, 2D, color filter array, CFA, (124) arranged on an image sensor (122) such that the light received through the lens assembly (110) enters the image sensor (122) through the CFA (124), wherein the CFA (124) is arranged to implement said spectral transmission function that results in a spectral transmission that varies with a wavelength and with a spatial position of the CFA (124); and a controller (130) comprising an image reconstruction portion arranged to derive the HS image based on light received on the image sensor (124) via application of said post-processing procedure that comprises an artificial neural network, ANN, wherein the one or more apparatuses are arranged to carry out the iterative training procedure via processing a plurality of training items through a training network (300) that represents an arrangement of the lens assembly (110), the CFA (124) and the post-processing procedure in the HSI apparatus (100) and that comprises respective training models (302, 304, 306) for determining the phase modulation function, the spectral transmission function and weights of the ANN,wherein the CFA (124) comprises a transmissive metal-insulator-metal, MIM, structure and wherein the training model (304) for determining the spectral transmission function is arranged determine respective at least one layer thickness separately for a plurality of predefined spatial positions of the MIM structure to implement the predefined spectral transmission function, and wherein the training model (304) for determining the spectral transmission function is arranged to account for one or more predefined implementation constraints of the MIM structure serving as the CFA (124).

2. A system according to claim 1, wherein the phase modulation function defines a respective phase delay for a plurality of predefined spatial positions within the aperture of the DOE (112) and wherein the training model (302) for determining the phase modulation function is arranged to determine the respective phase delay for said plurality of predefined spatial positions within the aperture of the DOE (112).

3. A system according to claim 1 or 2, wherein the spectral transmission function defines a respective spectral transmission characteristics for a plurality of predefined spatial positions of the CFA (124) at a plurality of wavelengths and wherein the training model (304) for determining the spectral transmission function is arranged to determine the respective spectral transmission characteristics for said plurality of predefined spatial positions of the CFA (124) at said plurality of wavelengths.

4. A system according to any of claims 1 to 3, wherein the ANN comprises a deep neural network, DNN and wherein the training model (306) for determining the weights of the ANN is arranged to determine weights of the DNN.

5. A system according to any of claims 1 to 4, wherein the DOE (112) comprises a lens element having a non-uniform thickness that is defined separately for a plurality of predefined spatial positions within the aperture of the DOE to provide a respective desired phase delay through the respective spatial positions of the aperture of the DOE.

6. A system according to any of claims 1 to 4, wherein the DOE (112) comprises phase-modulating metamaterial arranged into a meta-surface, where a nanostructure forming the meta-surface is arranged to provide a respective desired phase delay through a plurality of spatial positions of the meta-surface.

7. A system according to any of claims 1 to 6, wherein the training model (304) for determining the spectral transmission function is constrained to determine a respective insulator thickness separately for said plurality of spatial positions of the MIM structure.

8. A system according to claim 7, wherein said implementation constraints comprise a predefined set of insulator thicknesses.

9. A system according to claim 7, wherein said implementation constraints comprise a predefined range of insulator thicknesses.

10. A system according to any of claim 7 to 9, wherein the training model (304) for determining the spectral transmission function is arranged assume uniform metal layer thicknesses throughout said plurality of spatial positions of the MIM structure.

11. A system according to any of claims 7 to 9, wherein the training model (304) for determining the spectral transmission function is arranged determine respective metal layer thicknesses separately for said plurality of spatial positions of the MIM structure.

12. A system according to claim 11, wherein said implementation constraints comprise a predefined set of metal layer thicknesses.

13. A system according to claim 11, wherein said implementation constraints comprise a predefined range of metal layer thicknesses.

14. A system according to any of claims 1 to 13, wherein the lens assembly further comprises a refractive lens (114) disposed with respect to the DOE (112) such that the light entering the HSI apparatus (100) enters the DOE (112) through the refractive lens (114) and having its focus distance at a predefined depth.

15. A system according to any of claims 1 to 14, wherein each of the plurality of training items comprises a respective ground-truth HS image.

16. A system according to claim 15, wherein the plurality of ground-truth HS images represent objects that reside at a predefined depth from the HSI apparatus, the iterative training procedure thereby optimizing operation of the HSI apparatus (100) for the predefined depth.

17. A system according to claim 16, wherein the plurality of ground-truth HS images represent objects that reside within a predefined depth range from the HSI apparatus, the iterative training procedure therebyoptimizing operation of the HSI apparatus (100) for the predefined depth range.

18. A system according to any of claims 15 to 17, wherein the iterative training procedure comprises the following for a plurality of iteration rounds: processing each of the ground-truth HS images through the training network (300) to derive a corresponding reconstructed HS image, and updating each of the phase modulation function, the spectral transmission function and the weights of the ANN based on differences between the plurality of ground-truth HS images and the corresponding reconstructed HS images derived therefrom. 19 A method for determining, via an iterative training procedure, a phase modulation function, a spectral transmission function and a post- processing procedure that are useable in a hyperspectral imaging, HSI, apparatus (100) provided for capturing a hyperspectral, HS, image of a scene, wherein the HSI apparatus (100) comprises: a lens assembly (110) comprising a diffractive optical element, DOE, (112) arranged to implement said phase modulation function that results in a phase delay that varies with the spatial position of an aperture of the DOE (112); an image sensor assembly (120) arranged to receive light through the lens assembly (110), the image sensor assembly (120) comprising a two-dimensional, 2D, color filter array, CFA, (124) arranged on an image sensor (122) such that the light received through the lens assembly (110) enters the image sensor (122) through the CFA (124), wherein the CFA (124) is arranged to implement said spectral transmission function that results in a spectral transmission that varies with a wavelength and with a spatial position of the CFA (124); anda controller (130) comprising an image reconstruction portion arranged to derive the HS image based on light received on the image sensor (124) via application of said post-processing procedure that comprises an artificial neural network, ANN, wherein the method comprises processing a plurality of training items through a training network (300) that represents an arrangement of the lens assembly (110), the CFA (124) and the post-processing procedure in the HSI apparatus (100) and that comprises respective training models (302, 304, 306) for determining the phase modulation function, the spectral transmission function and weights of the ANN, wherein the CFA (124) comprises a transmissive metal-insulator-metal, MIM, structure and wherein the training model (304) for determining the spectral transmission function is arranged determine respective at least one layer thickness separately for a plurality of predefined spatial positions of the MIM structure to implement the predefined spectral transmission function, and wherein the training model (304) for determining the spectral transmission function accounts for one or more predefined implementation constraints of the MIM structure serving as the CFA (124).

20. A computer program comprising computer readable program code configured to cause performing the method according to claim 19 when said program code is executed on one or more computer apparatuses.

Images