Image processing device and method involving a rotating point spread function and a neural network for estimating pixel-width depths and deblurring

Abstract

An information processing device, including: an imaging unit including: an image sensor configured to generate image data, a phase mask configured to introduce a phase delay on light that is incident on the image sensor according to a superposition of Gauss-Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects such that the image sensor generates image data representing a blurred image of the scene in which the point objects are represented blurred according to the RPSF; and an image processing unit configured to input the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

Description

[0001] IMAGE PROCESSING DEVICE AND METHOD INVOLVING A ROTATING POINT SPREAD FUNCTION AND A NEURAL NETWORK FOR ESTIMATING PIXEL-WIDTH DEPTHS AND DEBLURRING

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to an information processing device, an information processing method, a method, computer programs and computer-readable media.

[0004] TECHNICAL BACKGROUND

[0005] Generally, phase masks are known (also known as phase-shift masks). Typically, phase masks are used for improving an image resolution or increasing an imaging contrast by introducing specific phase shifts in light waves passing through the phase mask such that a light intensity pattern (e.g. on an image sensor or on a substrate) is manipulated by interference of the light waves.

[0006] Basically, two different types of aperture masks may be distinguished: phase masks and amplitude masks. The phase masks are typically transparent and exploit regions with different optical thickness to achieve specific phase shifts. The amplitude masks may let only a small amount of light pass through certain regions of the aperture mask for manipulating the light intensity pattern.

[0007] Recently, engineered phase masks are used in cameras for introducing specific phase shifts such that a specifically engineered point spread function is applied to light originating from a scene.

[0008] Monocular Depth Estimation (“MDE”) is the task that concerns estimating the depth of a scene from a single RGB (“Red-Green-Blue”) image. Occlusion may, however, cause problems since objects in the foreground can occlude objects in the background.

[0009] The ambiguity and scale ambiguity arise due to the fact that multiple 3D scenes can be projected onto the same 2D image and consequently can only provide relative depth information rather than absolute depth information.

[0010] Overall, the state of the art in MDE is focused on improving the accuracy and robustness of depth estimation algorithms while also exploring new approaches that can handle the inherent challenges in this task.

[0011] Apart from this, a method is known that makes use of rotating PSF (“Point Spread Function”) phase mask for depth estimation. However, conventional signal processing techniques are applied for depth and image recovery, which may have, in some cases, some limitations regarding accuracy of the recovery.

[0012] Although there exist techniques for monocular depth estimation, it is generally desirable to improve the existing techniques.

[0013] SUMMARY

[0014] According to a first aspect, the disclosure provides an information processing device, comprising: an imaging unit including: an image sensor configured to generate image data, a phase mask configured to introduce a phase delay on light that is incident on the image sensor according to a superposition of Gauss-Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects such that the image sensor generates image data representing a blurred image of the scene in which the point objects are represented blurred according to the RPSF; and an image processing unit configured to input the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0015] According to a second aspect, the disclosure provides An information processing method, comprising: generating, by an image sensor, image data representing a blurred image of a scene in which point objects are represented blurred according to a rotating point spread function, RPSF, wherein, by a phase mask, a phase delay according to a superposition of Gauss-Laguerre modes is introduced on light incident on the image sensor such that the RPSF is applied on light originating from point objects in the scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects; inputting the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0016] According to a third aspect, the disclosure provides a method for generating simulated image data representing a blurred image of a scene, comprising: obtaining ground-truth image data representing a ground-truth image of the scene, wherein the ground-truth image data include pixel-wise depth values of the scene; and applying a rotating point spread function, RPSF, on the ground-truth image data for generating the blurred image in which point objects are represented blurred according to the RPSF, wherein the RPSF is based on a superposition of Gauss-Laguerre modes and has one or more peaks which rotate with respect to a reference axis as a function of the depth value.

[0017] According to a fourth aspect, the disclosure provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of the second aspect.

[0018] According to a fifth aspect, the disclosure provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the information processing method of the second aspect.

[0019] According to a sixth aspect, the disclosure provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the third aspect.

[0020] According to a seventh aspect, the disclosure provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the third aspect.

[0021] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0024] Fig. 1 schematically illustrates an embodiment of a Gauss-Laguerre modal plane;

[0025] Fig. 2 schematically illustrates in a block diagram an embodiment of an information processing device;

[0026] Fig. 3 schematically illustrates in Fig. 3 A an embodiment of a point spread function without a phase mask and in Fig. 3B and Fig. 3C an embodiment of a rotating point spread function;

[0027] Fig. 4 schematically illustrates in a block diagram an embodiment of a training stage of a neural network;

[0028] Fig. 5 schematically illustrates in a block diagram an embodiment of an image processing pipeline for generating simulated image data; Fig. 6 schematically illustrates in a flow diagram an embodiment of a method for generating simulated image data representing a blurred image of a scene; and

[0029] Fig. 7 schematically illustrates in a flow diagram an embodiment of an information processing method.

[0030] DETAILED DESCRIPTION OF EMBODIMENTS

[0031] Before a detailed description of the embodiments under reference of Fig. 2 is given, general explanations are made.

[0032] As mentioned in the outset, aperture masks are known, wherein basically two different types of aperture masks are distinguished as generally known: phase masks and amplitude masks. Typically, phase masks introduce specific phase shifts in light waves passing through the phase mask such that a light intensity pattern (e.g., on an image sensor or a substrate) is manipulated by interference, for example, for increasing an imaging resolution.

[0033] As further mentioned in the outset, engineered phase masks are used in cameras for introducing specific phase shifts such that a specifically engineered point spread function is applied to light originating from a scene.

[0034] It has been recognized that, when amplitude masks are used, the amount of light that may reach an image sensor may be limited, whereby a light efficiency may be low in some cases.

[0035] It has further been recognized that phase masks may be engineered which apply an engineered point spread function to light originating from point objects in a scene such that the engineered point spread function encodes a depth or distance of the point object to the camera in a blurred image of the scene.

[0036] Moreover, it has been recognized that a sharp image should be obtained in addition to depth information of the scene. Thereby, one image acquisition allows to obtain a sharp image of a scene and depth information of the scene.

[0037] As further mentioned in the outset, conventional signal processing techniques are typically applied in the state of the art for depth and image recovery, which may have, in some cases, some limitations regarding accuracy of the recovery.

[0038] Hence, some embodiments pertain to an information processing device, including: an imaging unit including: an image sensor configured to generate image data, a phase mask configured to introduce a phase delay on light that is incident on the image sensor according to a superposition of Gauss-Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects such that the image sensor generates image data representing a blurred image of the scene in which the point objects are represented blurred according to the RPSF; and an image processing unit configured to input the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0039] The information processing device may be camera or a mobile device (e.g., a smartphone, a laptop, etc.) for example. The information processing device may include two separated parts, for instance, a camera or camera module including the imaging unit and a computer or server including the image processing unit such that the camera transmits generated image data via a wireless or wired connection to the computer or server for analysis.

[0040] The imaging unit typically includes an aperture, the phase mask, a lens system and the image sensor for capturing an image of a scene, wherein the image is a blurred image and depth information of the scene is encoded in the blurred image. The imaging unit may include a data bus (interface) (e.g., a Camera Serial Interface (CSI) in accordance with MIPI (Mobile Industry Processor Interface) specifications (e.g., MIPII CSL2 or the like) or the like). The imaging unit may include a data bus interface for transmitting (and receiving) data over a data bus.

[0041] The image sensor may be a CCD (“Charge-Coupled Device”) image sensor or a CMOS (“Complementary Metal Oxide Semiconductor”) image sensor or the like. The image sensor includes a plurality of image pixels which are configured to generate an electric signal in response to light that is incident onto them, as generally known.

[0042] The image sensor includes image sensor circuitry for generating image data. The image sensor circuitry may include read-out circuitry for reading the electric signals of the image pixels, signal processing circuitry for signal amplification, correlated double sampling, or the like, and analog- digital-conversion circuitry, for example, for converting the electric signals to digital values (pixel values), thereby generating image data which include the pixel values of the plurality of image pixels which represent an image of a scene.

[0043] The image sensor may be configured to generate image data representing a grey image or a monochromatic image or a multispectral image or a color image (e.g., red-green-blue image). The image sensor may include a Bayer color filter array arranged on the plurality of image pixels and the plurality of image pixels may thus be arranged in Bayer arrays, as generally known, such that red-green-blue pixel values are generated.

[0044] In some embodiments, the image data include red-green-blue pixel values, and the image processing unit is further configured to perform demosaicing on the image data prior to its input into the neural network.

[0045] As mentioned above, the image sensor may include a Bayer color filter array arranged on the image pixels and the image pixels may be arranged in a Bayer array such that a pixel value from each image pixel represents only one value for a specific color (e.g., red, green, blue). Therefore, demosaicing is performed, as generally known, which includes, for example, an interpolation of the pixel values of the other image pixels of the Bayer array to obtain all color values for each image pixel.

[0046] The phase mask may be made of glass or another transparent material, wherein its surface can be (mi cro- / nano-) structured (e.g., by etching, lithography, etc.), for example, a flat thin substrate which is structured to have a specific height profile. The phase mask may be arranged in an aperture plane of the imaging unit.

[0047] In some embodiments, the phase mask is integrated in an optical lens.

[0048] The height profile of the phase mask is (computed) such that the phase mask introduces a phase delay on light that is incident on the image sensor according to a superposition of Gauss- Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects.

[0049] The image processing unit includes circuitry configured to achieve the functions described herein.

[0050] The circuitry may be based on or may include or may be implemented as integrated circuity logic or may be implemented by a CPU (central processing unit), an application processor, a graphical processing unit (GPU), a microcontroller, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit) or the like. The functionality may be implemented by software executed by a processor such as an application processor or the like.

[0051] The circuitry may be based on or may include or may be implemented by typical electronic components configured to achieve the functionality as described herein. The circuitry may be based on or may include or may be implemented in parts by typical electronic components and integrated circuitry logic and in parts by software.

[0052] The circuitry may include data storage capabilities to store data such as memory which may be based on semiconductor storage technology (e.g. RAM, EPROM, etc.) or magnetic storage technology (e.g. a hard disk drive) or the like.

[0053] The circuitry may include a data bus (interface) (e.g. a Camera Serial Interface (CSI) in accordance with MIPI (Mobile Industry Processor Interface) specifications (e.g. MIPII CSI-2 or the like) or the like). The circuitry may include a data bus interface for transmitting (and receiving) data over a data bus.

[0054] It is generally known that Gauss-Laguerre modes are a set of solutions for the paraxial Helmholtz equation in cylindrical coordinates. The Gauss-Laguerre modes may be denoted by GL(m, n), wherein m is the radial index and n is the azimuthal index.

[0055] The combinations of m and n define the Gauss-Laguerre modal plane, wherein each point in the Gauss-Laguerre modal plane corresponds to a Gauss-Laguerre mode.

[0056] In some embodiments, the superposition, i.e. a linear combination, is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

[0057] The slanted line thus covers at least two different values of the azimuthal index n.

[0058] For enhancing the general understanding of the present disclosure, this is discussed under reference of Fig. 1, which schematically illustrates an embodiment of a Gauss-Laguerre modal plane 70.

[0059] The azimuthal index n is depicted on the vertical axis and the radial index is depicted on the horizontal axis. Each point in the Gauss-Laguerre modal plane 70 corresponds to a Gauss- Laguerre mode.

[0060] For obtaining phase delays which result in a RPSF, a linear combination of Gauss-Laguerre modes which lie on a slanted line in the Gauss-Laguerre modal plane 70 are used, thus Gauss- Laguerre modes with at least two different values of the azimuthal index n are used.

[0061] For example, the slanted lines 71, 72 and 73 provide a linear combination of Gauss-Laguerre modes which results in a RPSF.

[0062] The slanted line 71 includes GL(0, 0), GL(2, 4) and GL(4, 8).

[0063] The slanted line 72 includes GL(1, 1), GL(2, 4) and GL(3, 7).

[0064] The slanted line 73 includes GL(1, 1), GL(3, 5) and GL(5, 9). Returning to the general explanations, as generally known, the point spread function describes the response of an imaging system to a point source or point object (also known as the impulse response of an optical system). Typically, the point spread function is spatially narrow when a point object is in focus of the optical system and spatially more extended (“blurred”) when the point object is in defocus of the optical system.

[0065] In contrast, in some embodiments, the RPSF has one or more peaks which appear at a reference position with respect to the reference axis when the point object is in focus, however, once the point object is in defocus, the peaks rotate as a function of the amount of defocus around the reference axis, thereby depth is encoded in a blurred image.

[0066] The reference axis may be an optical axis. The optical axis may be, for example, a central axis of the imaging unit such that aperture, phase mask, lens system and image sensor are symmetrically arranged on the optical axis.

[0067] The neural network may be based on or may include a U-Net, which is or includes a deep neural auto-encoder, and the neural network may be based on or may include one or more layers implementing a Wiener deconvolution deep neural network, which may me trained to perform non-blind and non-uniform image deblurring. The neural network may further include one or more fully connected layers, convolutional layers, etc.

[0068] Hence, in some embodiments, both a sharp image of a scene and depth information of the scene are obtained from a single image.

[0069] In some embodiments, parameters of a height profile model representing a height profile of the phase mask are obtained based on a joint optimization of the parameters and weights of the neural network in a training of the neural network.

[0070] In such embodiments, the phase mask is simulated as an additional optical layer, wherein its output is input to the neural network and then both are optimized via backpropagation during the training. The optical layer includes parameters which describe the phase shifts introduced by the phase mask which results in the RPSF applied by the phase mask and simulated by the optical layer. The parameters are connected with a specific height profile of the phase mask.

[0071] In some embodiments, ground-truth image data are used as training data in such a training which represent ground-truth images of scenes, wherein the ground-truth image data include pixel-wise depth values of a scene for each ground-truth image.

[0072] The ground-truth image data may be computer-generated scenes without noise or may be image data generated by high-resolution cameras which were image processed to remove noise. In some embodiments, in a training of the neural network, simulated image data are (additionally) used as training data which represent blurred images of scenes on which an image processing pipeline has been applied.

[0073] Additionally, it has been recognized that simulated image data should be more realistic, for example, to generate training data for training a neural network.

[0074] Hence, some embodiments pertain to a method for generating simulated image data representing a blurred image of a scene, including: obtaining ground-truth image data representing a ground-truth image of the scene, wherein the ground-truth image data include pixel-wise depth values of the scene; and applying a rotating point spread function, RPSF, on the ground-truth image data for generating the blurred image in which point objects are represented blurred according to the RPSF, wherein the RPSF is based on a superposition of Gauss-Laguerre modes and has one or more peaks which rotate with respect to a reference axis as a function of the depth value.

[0075] In some embodiments, the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

[0076] In some embodiments, the method further includes applying an image processing pipeline on the blurred image.

[0077] In some embodiments, the image processing pipeline includes simulating Bayer color filter array mosaicing and demosaicing.

[0078] In some embodiments, the image processing pipeline includes simulating an addition of image sensor noise.

[0079] In some embodiments, the image processing pipeline includes simulating analog-to-digital conversion.

[0080] Hence, the simulated image data represent blurred images which may be more realistic (a realistic model for digital image transformation) due to a complete simulation of a real image processing pipeline.

[0081] Thus, the image deblurring neural network is trained based on more realistic simulated image data.

[0082] Moreover, the image deblurring (by the image deblurring neural network) may be more robust against noise in the image capturing process.

[0083] Some embodiments pertain to a corresponding information processing method, including: generating, by an image sensor, image data representing a blurred image of a scene in which point objects are represented blurred according to a rotating point spread function, RPSF, wherein, by a phase mask, a phase delay according to a superposition of Gauss-Laguerre modes is introduced on light incident on the image sensor such that the RPSF is applied on light originating from point objects in the scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects; inputting the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0084] The information processing method may be performed by the information processing device as described herein.

[0085] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0086] Returning to Fig. 2, there is schematically illustrated in a block diagram an embodiment of an information processing device 1, which is discussed in the following.

[0087] The information processing device l is a mobile device and includes an imaging unit 2 and an image processing unit 3.

[0088] The imaging unit 2 includes an aperture 4, a phase mask 5, a lens system 6, and a Bayer color filter array 7 arranged on an image sensor 8 which includes an analog-to-digital converter 9.

[0089] The image processing unit 3 includes a demosai cing unit 10 and a neural network unit 11.

[0090] The aperture 4 gathers light 13 originating from point objects in a scene 14.

[0091] The phase mask 5 is arranged in the aperture plane and made of glass, wherein the glass is etched to have a specific height profile which introduces a phase delay on the light according to a superposition of Gauss-Laguerre modes (the height profile may be pre-computed as discussed, e.g., under reference of Figs. 3 and 4). The gathered light 13 passes through the phase mask 5, thereby experiencing phase shifts. In other words, the phase masks 5 applies a rotating point spread function, RPSF, on the light 13 incident on the image sensor 8, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects. The phase mask 5 introduces a phase delay on the light according to a superposition of Gauss- Laguerre modes such that the RPSF is applied.

[0092] The lens system 6 images the light 13 through the Bayer color filter array 7 on the image sensor 8.

[0093] The image sensor 8 has a plurality of image pixels which generate electric signals in response to the light 13 incident on the plurality of image pixels. Further, the analog-to-digital converter 9 of the image sensor 8 generates image data based on the electric signals of the plurality of image pixels.

[0094] Here, the image data include red-green-blue pixel values from Bayer array image pixels. The image data represent a blurred image of the scene 14 in which the point objects are represented blurred in accordance with the RPSF due to the light 13 passing through the phase mask 5.

[0095] The image data is obtained by the demosai cing unit 10.

[0096] The demosai cing unit 10 performs demosai cing on the image data prior to its input into the neural network unit 11.

[0097] The neural network unit 11 is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0098] Thereby, one image acquisition allows to obtain a sharp all-in-focus image (deblurred image) of the scene 14 and depth data representing depth information of the scene 14.

[0099] Fig. 3 schematically illustrates in Fig. 3 A an embodiment of a point spread function (20-PSF and 21-PSF) without the phase mask 5 and in Fig. 3B and Fig. 3C an embodiment of the RPSF, which is discussed in the following.

[0100] The embodiment is based on the information processing device 1 of Fig. 2.

[0101] Referring to Fig. 3 A, light originating from a first point object 20 and a second point object 21 on an optical axis 22 is imaged on the image sensor 8 by the lens system 6. The optical axis 22 is along the z-direction and the image sensor 8 is in the x-y-plane.

[0102] The first point object 20 is in defocus of the lens system 6 such that, when light originating from the first point object 20 is imaged on the image sensor 8 by the lens system 6, a first point spread function 20-PSF is observed on the image sensor 8 in the imaging plane (x-y-plane).

[0103] The second point object 21 is in focus of the lens system 6 such that, when light originating from the second point object 21 is imaged on the image sensor 8 by the lens system 6, a second point spread function 21-PSF is observed on the image sensor 8 in the imaging plane (x-y-plane). In other words, the lens system 6 applies the first point spread function 20-PSF and the second point spread function 21-PSF on light incident on the image sensor 8, the light originating from the first point object 20 and the second point object 21, respectively, such that the image sensor 8 generates image data representing an image of the scene 14 in which first point objects 20 and second point objects 21 are represented in accordance with the first point spread function 20-PSF and the second point spread function 21-PSF, respectively.

[0104] As can be seen in Fig. 3 A, the first point spread function 20-PSF is spatially much broader in the imaging plane (x-y -plane) than the second point spread function 21-PSF due to the defocus.

[0105] Hence, it may be difficult to recover a sharp image when the focus of the lens system 6 is not adapted to the scene and, moreover, depth information may not be reliably encoded in the image. In some instances, defocus blur within a conventional imaging system conveys some depth related information and depth maps can be obtained from the defocus blur (Depth From Defocus DFD) but, in some embodiments, such depth information is not as strong or as reliable as the one encoded by the RPSF. Moreover, DFD requires two or more images, whereas only one image is required with RPSF depth encoding.

[0106] Thus, the phase mask 5 is included in the information processing device 1.

[0107] Referring to Fig. 3B, the first point object 20 is in defocus of the lens system 6 such that, when light originating from the first point object 20 passes through the phase mask 5 and is then imaged on the image sensor 8 by the lens system 6, a first shape of the RPSF (20-RPSF) is observed on the image sensor 8 in the imaging pane (x-y-plane).

[0108] The second point object 21 is in focus of the lens system 6 such that, when light originating from the first point object 20 passes through the phase mask 5 and is then imaged on the image sensor 8 by the lens system 6, a second shape of the RPSF (21 -RPSF) is observed on the image sensor 8 in the imaging plane (x-y-plane).

[0109] In other words, the phase mask 5 and the lens system 6 apply the RPSF on light incident on the image sensor 8, the light originating from point objects 20 and 21, respectively, such that the image sensor 8 generates image data representing a blurred image of the scene 14 in which point objects 20 and 21 are represented in accordance with the RPSF.

[0110] As illustrated in Fig. 3B, the first shape of the RPSF (20-RPSF) is substantially equal to the second shape of the RPSF (21 -RPSF) when rotationally corrected and both shapes are spatially broad in the imaging plane such that a blurred image is generated. As illustrated in Fig. 3C, the first shape and the second shape of the RPSF (20-RPSF and 21- RPSF) both have two peaks ((20-RPSF-l, 20-RPSF-2) and (21-RPSF-l, 21-RPSF-2)) in the imaging plane which rotate with respect to the optical axis 22 (z-axis, reference axis here) as a function of defocus of the point objects 20 and 21. Thereby, depth is encoded in the blurred image.

[0111] The phase mask 5 has a height profile in the x-y -plane which introduces specific phase delays such that the phase mask 5 applies the RPSF on light incident on the image sensor 8.

[0112] A height profile model representing the height profile of the phase mask 5 will be discussed in the following.

[0113] The height profile is given by: mod 2K), wherein is the wavelength of the light, n is the refractive index of the phase mask 5, arg is the complex argument function and maskis the phase delay introduced by the phase mask 5, which is here selected according to a superposition of Gauss-Laguerre modes such as schematically depicted in Fig. 1. The average wavelength of the light may be used to compute the height profile of the phase mask 5.

[0114] The parameters of the height profile include the coefficients of the linear combination of the Gauss-Laguerre modes.

[0115] Fig. 4 schematically illustrates in a block diagram an embodiment of a training stage of a neural network 32-t, which is discussed in the following.

[0116] In the training stage, ground-truth image data 30 are used as training data which represent ground-truth images of scenes, wherein the ground-truth image data 30 include pixel values 30a for each ground-truth image and pixel-wise depth values 30b of a scene for each ground-truth image.

[0117] Moreover, simulated image data 44 are used as training data which are generated according to the method as discussed under reference of Fig. 5 further below. The simulated image data 44 are blurred image data obtained based on each ground-truth image, thereby representing a simulated blurred image of the corresponding ground-truth image.

[0118] The pixel-wise depth values 30b may be defocus values (defocus parameters) or depth values with respect to a predetermined focus depth. An optical layer in optimization stage 31-t obtains the pixel values 30a and the pixel-wise depth values 30b.

[0119] The optical layer in optimization stage 31-t includes parameters of the height profile model discussed under reference of Fig. 3. The parameters further describe the RPSF, as discussed.

[0120] Hence, for each pixel value in the pixel values 30a of a ground-truth image, the optical layer in optimization stage 31-t calculates the shape of the RPSF based on the associated depth values 30b.

[0121] The optical layer in optimization stage 31-t then applies the shape of the RPSF by convolution. The convolutions are summed up to generate blurred image data 31a representing a blurred image of a ground-truth image of a scene.

[0122] The neural network in the training stage 32-t is configured to generate deblurred image data 32a and to estimate pixel -wise depths 32b from the blurred image data 31a and 44.

[0123] A loss function 33 obtains the deblurred image data 32a, the estimated pixel-wise depths 32b, the pixel values 30a, the pixel-wise depth values 30b.

[0124] Based on a difference between the deblurred image data 32a and the pixel values 30a and based on a difference between the estimated pixel-wise depths 32b and the pixel-wise depth values 30b, the loss function 33 generates weight updates 33a for the neural network in the training stage 32-t and parameter updates 33b for the optical layer in optimization stage 31-t.

[0125] Once the training stage is completed, a neural network 32 and the optical layer 31 are obtained. The optical layer 31 including optimized parameters such that the phase mask 5 can be generated having a height profile according to the optimized parameters.

[0126] Hence, the parameters of a height profile model representing the height profile of the phase mask 5 are obtained based on a joint optimization of the parameters and weights of a neural network in a training of the neural network for pixel-wise depth estimation from a blurred image and deblurring of the same.

[0127] Fig. 5 schematically illustrates in a block diagram an embodiment of an image processing pipeline for generating simulated image data 44, which is discussed in the following.

[0128] For generating more realistic training data, an image processing pipeline is applied on the ground-truth image data 30. The image processing pipeline includes the optical layer 31, a Bayer color filter array mosaicing layer 40, an image sensor noise layer 41, an analog-to-digital conversion layer 42 and a Bayer color filter array demosai cing layer 43.

[0129] The optical layer 31 applies the RPSF on the ground-truth image data 30 for generating a blurred image of a scene, wherein the RPSF has peaks which rotate with respect to a reference axis as a function of a depth value of the scene.

[0130] The Bayer color filter array mosaicing layer 40 simulates the Bayer color filter array mosaicing.

[0131] The image sensor noise layer 41 simulates an addition of sensor noise, e.g., based on a Gaussian noise model or the like.

[0132] The analog-to-digital conversion layer 42 simulates the analog-to-digital conversion, e.g., by accounting for a quantization error.

[0133] The Bayer color filter array demosaicing layer 43 simulates the Bayer color filter array demosaicing and generates the simulated image data 44 representing a blurred image of the scene.

[0134] The simulated image data 44 can be used, for example, as realistic training data to fine-tune the neural network 32 of Fig. 4.

[0135] Fig. 6 schematically illustrates in a flow diagram an embodiment of a method 100 for generating simulated image data representing a blurred image of a scene, which is discussed in the following.

[0136] The method 100 may be performed by a computer or server or the like.

[0137] At 101, ground-truth image data representing a ground-truth image of the scene are obtained, wherein the ground-truth image data include pixel-wise depth values of the scene, as discussed herein.

[0138] At 102, a rotating point spread function, RPSF, is applied on the ground-truth image data for generating the blurred image in which point objects are represented blurred according to the RPSF, wherein the RPSF is based on a superposition of Gauss-Laguerre modes and has one or more peaks which rotate with respect to a reference axis as a function of the depth value, as discussed herein.

[0139] At 103, an image processing pipeline is applied on the blurred image, as discussed herein.

[0140] At 104, Bayer color filter array mosaicing and demosaicing is simulated, as discussed herein.

[0141] At 105, an addition of image sensor noise is simulated, as discussed herein. At 106, analog-to-digital conversion is simulated, as discussed herein.

[0142] Fig. 7 schematically illustrates in a flow diagram an embodiment of an information processing method 200, which is discussed in the following.

[0143] The information processing method 200 may be performed by the information processing device as described herein.

[0144] At 201, by an image sensor, image data representing a blurred image of a scene are generated in which point objects are represented blurred according to a rotating point spread function, RPSF, wherein, by a phase mask, a phase delay according to a superposition of Gauss-Laguerre modes is introduced on light incident on the image sensor such that the RPSF is applied on light originating from point objects in the scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects, as discussed herein.

[0145] At 202, the image data are input into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image, as discussed herein.

[0146] At 203, demosaicing is performed on the image data prior to its input into the neural network, as discussed herein.

[0147] Returning to the general explanations, the approach exploits the Gauss-Laguerre modes to build phase masks, and which codes the point spread function in a way that it has a depth dependency.

[0148] During the decode phase, in some embodiments, the RPSF is known because the characteristics of the phase mask are known. This may allow exploiting the point spread function to recover all in-focus images and the depth estimation of the scene using machine learning based depth and image estimation providing good quality depth and image restoration performance.

[0149] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0150] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software. In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0151] Note that the present technology can also be configured as described below.

[0152] (1) An information processing device, including: an imaging unit including: an image sensor configured to generate image data, a phase mask configured to introduce a phase delay on light that is incident on the image sensor according to a superposition of Gauss-Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects such that the image sensor generates image data representing a blurred image of the scene in which the point objects are represented blurred according to the RPSF; and an image processing unit configured to input the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0153] (2) The information processing device of (1), wherein the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

[0154] (3) The information processing device of (1) or (2), wherein the image data include red- green-blue pixel values, and wherein the image processing unit is further configured to perform demosaicing on the image data prior to its input into the neural network.

[0155] (4) The information processing device of anyone of (1) to (3), wherein the image data represent a grey image or a monochromatic image or a multispectral image or a color image.

[0156] (5) The information processing device of anyone of (1) to (4), wherein the neural network is trained to perform non-blind and non-uniform image deblurring to deblur the blurred image.

[0157] (6) The information processing device of anyone of (1) to (5), wherein the phase mask is integrated in an optical lens.

[0158] (7) An information processing method, including: generating, by an image sensor, image data representing a blurred image of a scene in which point objects are represented blurred according to a rotating point spread function, RPSF, wherein, by a phase mask, a phase delay according to a superposition of Gauss-Laguerre modes is introduced on light incident on the image sensor such that the RPSF is applied on light originating from point objects in the scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects; inputting the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

[0159] (8) The information processing method of (7), wherein the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

[0160] (9) The information processing method of (7) or (8), wherein the image data include red- green-blue pixel values, and further including performing demosaicing on the image data prior to its input into the neural network.

[0161] (10) The information processing method of anyone of (7) to (9), wherein the image data represent a grey image or a monochromatic image or a multispectral image or a color image.

[0162] (11) The information processing method of anyone of (7) to (10), wherein the neural network is trained to perform non-blind and non-uniform image deblurring to deblur the blurred image.

[0163] (12) A method for generating simulated image data representing a blurred image of a scene, including: obtaining ground-truth image data representing a ground-truth image of the scene, wherein the ground-truth image data include pixel-wise depth values of the scene; and applying a rotating point spread function, RPSF, on the ground-truth image data for generating the blurred image in which point objects are represented blurred according to the RPSF, wherein the RPSF is based on a superposition of Gauss-Laguerre modes and has one or more peaks which rotate with respect to a reference axis as a function of the depth value.

[0164] (13) The method of (12), wherein the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

[0165] (14) The method of (12) or (13), including applying an image processing pipeline on the blurred image.

[0166] (15) The method of (14), wherein the image processing pipeline includes simulating Bayer color filter array mosaicing.

[0167] (16) The method of (14) or (15), wherein the image processing pipeline includes simulating Bayer color filter array demosaicing. (17) The method of anyone of (14) to (16), wherein the image processing pipeline includes simulating an addition of image sensor noise.

[0168] (18) The method of anyone of (14) to (17), wherein the image processing pipeline includes simulating analog-to-digital conversion. (19) A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of anyone of (7) to (H).

[0169] (20) A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the information processing method of anyone of (7) to (H).

[0170] (21) A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of anyone of (12) to (18).

[0171] (22) A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of anyone of (12) to (18).

Claims

CLAIMS1. An information processing device, comprising: an imaging unit including: an image sensor configured to generate image data, a phase mask configured to introduce a phase delay on light that is incident on the image sensor according to a superposition of Gauss-Laguerre modes such that a rotating point spread function, RPSF, is applied on light originating from point objects in a scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects such that the image sensor generates image data representing a blurred image of the scene in which the point objects are represented blurred according to the RPSF; and an image processing unit configured to input the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

2. The information processing device of claim 1, wherein the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

3. The information processing device of claim 1, wherein the image data include red-green- blue pixel values, and wherein the image processing unit is further configured to perform demosaicing on the image data prior to its input into the neural network.

4. The information processing device of claim 1, wherein the image data represent a grey image or a monochromatic image or a multispectral image or a color image.

5. The information processing device of claim 1, wherein the neural network is trained to perform non-blind and non-uniform image deblurring to deblur the blurred image.

6. The information processing device of claim 1, wherein the phase mask is integrated in an optical lens.

7. An information processing method, comprising: generating, by an image sensor, image data representing a blurred image of a scene in which point objects are represented blurred according to a rotating point spread function, RPSF, wherein, by a phase mask, a phase delay according to a superposition of Gauss-Laguerre modes is introduced on light incident on the image sensor such that the RPSF is applied on light originating from point objects in the scene, wherein the RPSF has one or more peaks which rotate with respect to a reference axis as a function of defocus of the point objects;inputting the image data into a neural network, wherein the neural network is trained to estimate pixel-wise depths from the blurred image and to deblur the blurred image.

8. The information processing method of claim 7, wherein the superposition is based on at least two Gauss-Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

9. The information processing method of claim 7, wherein the image data include red-green- blue pixel values, and further comprising performing demosaicing on the image data prior to its input into the neural network.

10. The information processing method of claim 7, wherein the image data represent a grey image or a monochromatic image or a multispectral image or a color image.

11. The information processing method of claim 7, wherein the neural network is trained to perform non-blind and non-uniform image deblurring to deblur the blurred image.

12. A method for generating simulated image data representing a blurred image of a scene, comprising: obtaining ground-truth image data representing a ground-truth image of the scene, wherein the ground-truth image data include pixel-wise depth values of the scene; and applying a rotating point spread function, RPSF, on the ground-truth image data for generating the blurred image in which point objects are represented blurred according to the RPSF, wherein the RPSF is based on a superposition of Gauss-Laguerre modes and has one or more peaks which rotate with respect to a reference axis as a function of the depth value.

13. The method of claim 12, wherein the superposition is based on at least two Gauss- Laguerre modes lying on a slanted line in the Gauss-Laguerre modal plane.

14. The method of claim 12, comprising applying an image processing pipeline on the blurred image.

15. The method of claim 14, wherein the image processing pipeline includes simulating Bayer color filter array mosaicing.

16. The method of claim 14, wherein the image processing pipeline includes simulating Bayer color filter array demosaicing.

17. The method of claim 14, wherein the image processing pipeline includes simulating an addition of image sensor noise.

18. The method of claim 14, wherein the image processing pipeline includes simulating analog-to-digital conversion.

19. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the information processing method of claim 7.

20. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the information processing method of claim 7.

21. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 12.

22. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 12.

Images