Method and apparatus for reconstructing an image from a raw image and arrangement for training an apparatus
The method for lensless cameras uses pre-initialized weights for deconvolution and neural network processing to achieve efficient single-shot image reconstruction with multiple focal planes, addressing the limitations of existing technologies in lensless camera image reconstruction.
Patent Information
- Application Number
- PCT/EP2024/059117
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-03
- Filing Date
- 2024-04-04
- Publication Date
- 2024-11-07
AI Technical Summary
Existing lensless cameras with known image reconstruction algorithms are limited to fixed-object distance reconstruction and require significant computational resources, storage, and are not compatible with flexible and transparent devices, especially when attempting to achieve all-in-focus or 3D image reconstruction.
A method involving a lensless camera with a coded aperture that captures a raw image in a single shot, using pre-initialized two-dimensional weights for deconvolution to generate intermediate images, which are then processed by a trained neural network to produce output images with multiple focal planes, reducing computational power and storage needs.
Enables efficient reconstruction of images with multiple object distances in a single shot, enhancing image quality and reducing computational requirements, while being compatible with flexible and transparent devices.
Smart Images

Figure EP2024059117_07112024_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] METHOD AND APPARATUS FOR RECONSTRUCTING AN IMAGE FROM A RAW IMAGE AND ARRANGEMENT FOR TRAINING AN APPARATUS
[0003] Technical field
[0004] The present disclosure relates to the field of imaging, especially imaging based on lensless coded aperture cameras. Specifically, the disclosure is directed to a method for reconstructing an image from a raw image, an apparatus for reconstructing an image from a raw image and an arrangement for training an apparatus .
[0005] Background
[0006] In the field of imaging, so called lensless cameras have gained increasing importance for realizing small, low-weight and low-cost cameras. Such cameras may be used e.g. in smart phones, smart displays, smart watches, smart glasses, smart surfaces, laptops, tablets, computer displays, medical displays, transparent displays, augmented reality and virtual reality devices for realizing lensless imaging, touchless interaction, touchless user interfaces, object tracking, people tracking, gesture recognition, object recognition or 3D imaging, for example. Instead of a lens such cameras employ a so-called coded aperture, which is implemented by a physical mask placed in front of the sensor, i.e. between the sensor and the scene. Such cameras produce inherently coded images. Suitable algorithms are used to reconstruct an image from the raw image captured by the sensor.
[0007] Existing cameras with known image reconstruction algorithms only target fixed-object distance reconstruction. All-in- focus or 3D image reconstruction is reported for other types of lensless systems - using phase masks and microlens arrays but using either acquisition of multiple encoded images , or strategies with signi ficant cost in compute time , storage and system development . Moreover, such phase-based implementations are not compatible with flexible and transparent devices .
[0008] An obj ective could therefore be seen in providing a solution which enhances imaging based on lensless cameras . A further obj ective could be seen in providing a solution which enables image reconstruction achieving images with more than one focal plane .
[0009] The obj ective is achieved by the subj ect-matter of the independent claims . Embodiments and developments are defined in the dependent claims .
[0010] The definitions provided above also apply to the following description unless stated otherwise .
[0011] Summary
[0012] In one embodiment a method for reconstructing an image from a raw image comprises the following steps : capturing, by a lensless camera with coded aperture , the raw image in a single-shot , deconvolving the raw image by using a first set of weights comprising a number of N two-dimensional weights initiali zed in a preceding calibration phase and therefrom providing a number of N intermediate images , wherein N is a natural number larger than or equal to two , processing each of the N intermediate images by a trained neural network according to a second set of weights acquired in a preceding training phase and therefrom providing at least one output image . Therein each of the number of N preinitiali zed two-dimensional weights corresponds to a discrete obj ect distance .
[0013] The raw image is captured by the lensless camera with coded aperture in a single shot , i . e . by reading out the data from a sensor comprised by the camera only once for the at least one output image . In the deconvolution the raw image is trans formed into the N intermediate images . The deconvolution inverts the coding which is inherent to the raw image due to the use of the mask in the coded aperture . By using the first set of weights , which comprises N two-dimensional weights which have been pre-initiali zed beforehand, N intermediate images are generated in the deconvolution . As each of the number of N pre-initiali zed two-dimensional weights of the first set corresponds to a di f ferent discrete obj ect distance , each of the N intermediate images also corresponds to a reconstruction of the raw image matching the discrete obj ect distance of the weight used in the deconvolution . The processing of the N intermediate images by the trained neural network using the second set of weights results in the at least one output image . This at least one output image represents a reconstruction of the raw image , which comprises more than one focal plane due to the usage of the N two- dimensional weights of the first set of weights , although the raw image has been captured in j ust a single shot . Thereby, the quality of the at least one output image is enhanced when compared to the state of the art .
[0014] In a development , each of the number of N pre-initiali zed two-dimensional weights of the first set depends on a point spread function resulting from a picture of a point light source through the mask used in the coded aperture of the camera. Each of the number of N pre-initialized two- dimensional weights of the first set is initialized in the calibration phase, in which the point light source is adjusted to a distinct, and each time different, distance with respect to the camera and the picture is captured. Then, each of said pre-initialized two-dimensional weights is accorded to one plane of the output image. Said plane matches the distance between the camera and the point light source and corresponds to the object distance.
[0015] The point spread function, PSF, results from the use of the physical mask in the lensless camera, as known by the skilled person. In the deconvolution step of the claimed method, the N PSFs are used as the N deconvolution kernels to invert the raw image captured by the lensless camera, to provide the N intermediate images. The N pre-initialized weights of the first set of weights are each initialized or calibrated in the calibration phase by arranging the point light source to N different distances with respect to the camera and each time capturing the picture with the lensless camera. Each of the N pre-initialized weights of the first set of weights is derived from the resultant picture containing a raw image, and is subsequently used in the method for reconstructing the output image in the deconvolution or inversion step.
[0016] In other words, the PSF of a lensless camera is the pattern projected by the mask onto the sensor when illuminated by the point light source. The PSF shifts when the point light source moves laterally, i.e. away and towards, the sensor.
[0017] The sensor measurement, i.e. the raw image, is the weighted sum of various shifted PSFs, leading to a convolutional model . In some implementations the trained neural network is reali zed as a U-Net .
[0018] Therein, the U-Net , which is known as a standard neural network for image processing, is implemented as described in the document "U-Net : Convolutional Networks for Biomedical Image Segmentation" by Olaf Ronneberger, Philipp Fischer and Thomas Brox, 18 May 2015 , cited in Medical Image Computing and Computer-Assisted Intervention (MICCAI ) , Springer, LNCS , Vol . 9351 : 234-241 , 2015 . In the contracting path, the U-Net captures context from each of the N intermediate images . This context is used in the subsequent stepwise decompression reali zed in the U-Net ' s expanding path to provide the at least one output image .
[0019] In a development , each of the number of N intermediate images represents an instance of the output image according to the plane indicated by the corresponding weight of the number of N pre-initiali zed two-dimensional weights of the first set of weights .
[0020] The proposed method enables reconstruction of multiple obj ect distances reflected in the N intermediate images which form the basis for the at least one output image . The lensless camera used in the proposed method employs a low rank separable mask pattern, which breaks down the complexity of the modulation performed by the mask . This enables simpli fication of the deconvolution of the raw image to the N intermediate images and reduces the inversion or deconvolution to simple matrix multiplications in the optical encoder for each level of the 3D space . A conversion to the Fourier space is avoided . Consequently, the N two-dimensional weights of the first set of weights do not need to be stored in the Fourier domain, which greatly reduces storage space . Also , due to the innovative concept proposed herein, reconstruction of the output image is achieved faster and with considerably less computing power .
[0021] In a development , the processing of each of the N intermediate images by the trained neural network is performed in parallel using stacked matrices .
[0022] In a development , the calibration phase is performed before the training phase . Optionally, the first set of weights obtained in the calibration phase is fine-tuned in the subsequent training phase .
[0023] After the training phase , the weights of the first set of weights , the weights of the second set of weights , as well as the structure of the U-Net , remain unchanged .
[0024] In a development , the processing of each of the N intermediate images by the trained neural network results in exactly one output image . In the corresponding training phase of this development , the neural network acquires the second set of weights based on a number of M pairs . Each pair consists of a training input image and a target image . Therein, M is a natural number larger than or equal to 1 . Each training input image is a raw image captured by the lensless camera .
[0025] Each of the M pairs used as training data for training the neural network, for instance the U-Net , consists of one training input image and an accorded target image , which may also be referred to as a reference image . In a development , each target image is a natural image with a single plane in focus , which is captured by a lensed camera . Each pair of training input image and target image is captured simultaneously using an arrangement comprising the lensed camera, the lensless camera and a beam-splitter, by placing a target obj ect at a distinct distance in relation to the beam-splitter and adj usting a focal distance of the lensed camera to the distinct distance of the target obj ect .
[0026] The beam-splitter may be reali zed as a prism or as a combination of mirrors as known to a skilled person . It achieves that the same scene comprising the target obj ect can be captured simultaneously by the lensed camera and the lensless camera . For each pair of training input image and target image , the target obj ect is placed at a random position, called the distinct distance . This position changes for each pair . Each pair of training input and target image is captured simultaneously, while the focal distance of the lensed camera is set such that the target obj ect is in focus .
[0027] Based on the training phase using the above described training data, the exactly one output image provided by the proposed method in this option represents a single focus image with continuous depth resolution . The plane which is in focus in the output image is thereby not necessarily the plane of j ust one intermediate image , but may rather be the result of an interpolation between di f ferent planes , i . e . di f ferent intermediate images . This interpolation is reali zed by the neural network based on the second set of weights resulting from the training . In an alternative solution, each target image is a natural image with at least two planes in focus , which is captured by a lensed camera, which is prepared to have a wide range of focus . Optionally, in the target image , a whole scene is in focus . Each pair of training input image and target image is captured simultaneously using the arrangement comprising the lensed camera, the lensless camera and a beam-splitter, by placing a target obj ect at a di f ferent random distance in relation to the beam-splitter .
[0028] In this alternative , a lensed camera is employed with an allin- focus capability for taking the target image .
[0029] The resulting output image which is provided by the proposed method after said training phase represents an all-in- focus image , which inherently includes information about the distance , i . e . the focus , of each pixel .
[0030] In another alternative , each target image is a depth map image of a scene , which is captured by a depth camera . Each pair of training input image and target image is captured simultaneously using the arrangement comprising the depth camera, the lensless camera and the beam-splitter . Alternatively, for capturing each pair of training input image and target image the lensless camera and the depth camera are arranged close together, for example side by side , or above or below each other, in a rigid frame .
[0031] The beam-splitter is setup such that it does not interfere with the depth camera, for example a direct Time-of- f light , dToF, indirect ToF, iToF, structured-light , active-stereo , light detection and ranging, LiDAR, camera . In an alternative solution the two cameras , i . e . the lensless and the depth camera are arranged close together, for example side by side , or above or below each other, in a rigid frame , such that the cameras cannot move . The relative poses , i . e . 3D position, rotations , lens distortions , of both cameras are calibrated, such that a 3D scene captured by the depth camera can be remapped in the lensless camera referential . The resulting slight of fset , which likely contributes to some errors or bias in the training pairs , especially in very close or very distant distances , is tolerable . This approach could also be extended to multiple depth cameras , with di f ferent viewpoints , arranged in a way such that the scene of interest is perfectly covered by all the viewpoints . A depth map can then be extracted from the resulting merged point cloud, to serve as the target image . Calibration for relative positioning of all the cameras as known to those skilled in the art is required for this case as well .
[0032] In another alternative , the processing of each of the N intermediate images by the trained neural network provides a number of N output images , wherein each of these N output images represents a plane corresponding to a weight of the first set of weights . The method further comprises selecting an image from the number of N output images according to a given distance and providing the selected image as a final output image .
[0033] In the described alternative , the method provides more than one , in other words N output images . Each of these N output images reflects the plane of the corresponding weight of the first set of weights which was used in generating the corresponding intermediate image . The method in this case consequently provides a stack of output images , from which the final output image is subsequently selected based on the distance indication .
[0034] The resulting final output image represents a single focus image with discrete depth resolution .
[0035] In a development , the given distance is determined by measuring a distance of an obj ect in the scene , which is captured by the lensless camera .
[0036] The distance indication which is used for selecting the image from the output image stack is measured, for instance , by a time-of- f light sensor, a stereovision sensor, a laser or a LIDAR sensor . Based on this measurement , the closest available plane of the N output images in the output stack is then selected and provided as the final output image . In this case , no interpolation is used .
[0037] In a development in the training phase , the neural network acquires the second set of weights based on a number of combinations , each combination consisting of one training input image and a number of N target images . Each training input image is a raw image captured by the lensless camera, each target image is a natural image with a single plane in focus , which is captured by a lensed camera . Each combination of training input image and N target images is captured using the arrangement comprising the lensed camera, the lensless camera and the beam-splitter, by placing a target obj ect at a distinct distance in relation to the beam-splitter, capturing the training input image and capturing the N target images in direct succession, while continuously increasing or decreasing a focal distance of the lensed camera for every capture . Each target image of the N target images of the combination represents the lensed camera ground tooth and matches one of the N distance planes . For each input training image , N target images are taken as described, by adj usting the focal distance , which prevents the need to change the lensed camera' s position for each target image .
[0038] In an embodiment , an apparatus for reconstructing an image from raw image data comprises a lensless camera with coded aperture , and processing means which are coupled to the camera . The apparatus is configured to perform the abovedescribed method .
[0039] The processing means may be implemented by a microprocessor, for example by a state of the art graphics processing unit , GPU .
[0040] From the above it can be followed that the proposed apparatus enables to reconstruct 3D output images or 2D with selective focus output images originating from a single shot acquisition with the lensless camera with coded aperture .
[0041] This is even reali zed with limited additional cost , computing power and storage space .
[0042] In one embodiment an arrangement for training an apparatus comprises the apparatus as described above , the beam-splitter and the lensed camera . The lensless camera of the apparatus , the beam-splitter and the lensed camera are arranged in a triangle configuration such that a target obj ect or scene is captured in the same way by the lensed camera and the lensless camera . Brief description of the drawings
[0043] The text below explains the proposed solution in detail using exemplary embodiments with reference to the drawings . Components and elements that are functionally identical or have an identical ef fect bear identical reference numbers . Insofar as parts or components correspond to one another in their function, a description of them will not be repeated in each of the following figures . Therein,
[0044] Figure 1 shows an exemplary embodiment of a method for reconstructing an image from a raw image as proposed,
[0045] Figure 2 shows a first exemplary embodiment of an apparatus for reconstructing an image from a raw image as proposed,
[0046] Figure 3 shows a second exemplary embodiment of an apparatus for reconstructing an image from a raw image as proposed,
[0047] Figure 4 shows a third exemplary embodiment of an apparatus for reconstructing an image from a raw image as proposed,
[0048] Figure 5 shows a fourth exemplary embodiment of an apparatus for reconstructing an image from a raw image as proposed, and
[0049] Figure 6 shows an exemplary embodiment of an arrangement for training any of the apparatuses described . Detailed description
[0050] Figure 1 shows an exemplary embodiment of a method for reconstructing an image from a raw image as proposed . The method comprises the following steps :
[0051] S I : capturing, by a lensless camera with coded aperture , the raw image in a single-shot ,
[0052] S2 : deconvolving the raw image by using a first set of weights comprising a number of N two-dimensional weights initiali zed in a preceding calibration phase and therefrom providing a number of N intermediate images , wherein N is a natural number larger than or equal to two ,
[0053] S3 : processing each of the N intermediate images by a trained neural network according to a second set of weights acquired in a preceding training phase ,
[0054] S4 : therefrom providing at least one output image , wherein each of the number of N pre-initiali zed two- dimensional weights corresponds to a discrete obj ect distance .
[0055] Optionally the method further comprises the steps of :
[0056] S5 : selecting an image from the number of N output images according to a given distance , and
[0057] S 6 : providing the selected image as a final output image .
[0058] The method may be implemented by one of the exemplary embodiments depicted in Figures 2 to 6 , which are described in the following .
[0059] Figure 2 shows a first exemplary embodiment of an apparatus for reconstructing an image from a raw image , as proposed . The apparatus comprises the lensless camera 10 and processing means 100 , which are coupled to the lensless camera 10 . The apparatus of Figure 2 is configured to perform the method as proposed and as described above, e.g. in Figure 1. The lensless camera 10 is realized as a lensless camera with coded aperture as known to the skilled person.
[0060] In an implementation of the apparatus, the camera 10 uses a low rank separable mask pattern and does not need a programmable mask.
[0061] The raw image lin is captured by the camera 10 in a single shot. Said raw image lin is deconvolved in the step S2 using the first set of weights comprising a number of N two- dimensional weights Wl, W2, ... WN, which results in the number N intermediate images lintl, Iint2, ..., lintN. The N intermediate images lintl, Iint2, ..., lintN are input to a neural network UN and processed in said neural network UN according to a second set of weights UW. At least one output image lout is accordingly provided. In the depicted exemplary embodiment for one raw image lin captured by the camera 10 exactly one output image lout is provided. The weights Wl, ..., WN of the first set of two-dimensional weights have been initialized in a preceding calibration phase, such that each of these weights corresponds to a different discrete object distance of a scene that is captured in the single shot by the lensless camera 10.
[0062] The lensless camera 10 comprises a mask and a sensor, the mask being mounted to the sensor. The scene which is photographed by the lensless camera 10 is captured after being modulated by the mask, which realizes a specific PSF. This modulation is inverted in the deconvolution step S2 using the first set of weights Wl, ..., WN, reflecting the different distance planes of the photographed scene. The neural network UN, which may be reali zed as a U-Net , uses the second set of weights UW acquired in the training phase to generate the single output image Tout .
[0063] The lower part of Figure 2 shows a training data configuration which is used for the embodiment of Figure 2 in the training phase to acquire the second set of weights UW . For the training, a number of M pairs are used . Each pair consists of a training input image and a target image . For example , the first pair consists of training input image Ti l and target image Tol . The Mthpair consists of training input image TiM and target image ToM . M is a natural number larger than or equal to 1 . Each training input image Ti l , ..., TiM is a raw image captured by the lensless camera 10 . Each target image Tol , ..., ToM is a natural image with a single plane in focus , which is captured by a lensed camera . For this capturing, an arrangement which is depicted in Figure 6 is used in the training phase . The first pair of training input image Ti l and target image Tol is captured by placing a target obj ect Obj at a first distance z l in relation to the beam-splitter 30 of the arrangement of Figure 6 and adj usting a focal distance of the lensed camera to said first distance z l of the target obj ect Obj . Next , the second pair having the second training input image Ti2 and target image To2 is captured simultaneously after placing the target obj ect Obj at a second distance z2 in relation to the beam-splitter 30 and adj usting the focal distance of the lensed camera to said second distance z2 of the target obj ect Obj . Said procedure is repeated until the Mthpair . For the Mthpair, the target obj ect Obj is placed at an Mthdistance zM in relation to the beam-splitter 30 , the focal distance of the lensed camera 20 is adj usted to said Mthdistance zM of the target obj ect Obj and the Mthtraining input image TiM and the Mthtarget image ToM are captured at the same time by the lensless camera 10 and the lensed camera 20 , respectively . In practice , about a thousand pairs of training data, which are prepared as described above , may be used in the training phase .
[0064] The embodiment shown in Figure 2 reali zes the reconstruction of a single focus output image lout with continuous depth resolution from a raw input image l in in j ust a single capture .
[0065] Figure 3 shows a second exemplary embodiment of an apparatus as proposed . The apparatus depicted in Figure 3 coincides with the apparatus of Figure 2 , except for the training data used in this case . The training data for the embodiment of Figure 3 is depicted in the lower part of Figure 3 . Again the arrangement shown in Figure 6 is used for generating the training data . For the embodiment of Figure 3 , a lensed camera 20 , which is prepared to have a wide range of focus , is used for generating the target images Tol , > ToM, which are natural images in each case . In an ideal case , each of the target images Tol , ..., ToM is a natural all-in- focus . The M pairs of training input images Ti l , ..., TiM and target images Tol , ..., ToM are used in the training phase for the apparatus depicted in the upper part of Figure 3 , to enable said apparatus to provide the exactly one output image lout as an all-in- focus image , based on j ust one shot of a scene with the lensless camera 10 and processing of the same according to the method as described in Figure 1 .
[0066] Figure 4 shows a third exemplary embodiment of an apparatus as proposed . The depicted apparatus coincides with those described in Figures 2 and 3 , except for the training data . The training data depicted in the lower part of Figure 4 also comprises a set of M pairs of training input images Ti l to TiM and target images Tol to ToM . In contrast to Figures 2 and 3 , each of the target images Tol to ToM is a depth map image of a scene which is photographed by a depth camera .
[0067] By training the apparatus depicted in the upper part of Figure 4 with the training data comprising the depth map target images Tol to ToM, the apparatus is enabled to provide a depth map image for a scene captured in a single shot by the lensless camera 10 .
[0068] Figure 5 shows a fourth exemplary embodiment of the apparatus as proposed . Said apparatus coincides with the apparatuses depicted in Figures 2 to 4 , except that the apparatus provides more than one output image . As shown in Figure 5 , the apparatus provides a number of N output images loutl , Iout2 , loutN after processing the N intermediate images l intl to IntN by the U-Net UN using the second set of weights UW . Each of the N output images represents a plane of the photographed scene . This plane corresponds to one of the weights of the first set of weights Wl , ... WN used in deconvolving the raw image l in .
[0069] The processing means 100 in this embodiment are furthermore prepared to perform steps S5 and S 6 shown in Figure 1 , i . e . to select an image from the number of N output images loutl to loutN according to a given distance and therefrom provide the selected image as final output image loutf . As already described, the given distance may be determined as known by a skilled person employing, for example , a time-of- f light sensor, which indicates the di f ference between an obj ect of the photographed scene and the camera 10 . Said distance may be used as a kind of index in the selection of the corresponding output image from the number of N output images . In detail , the output image of the number of N output images loutl , loutN is selected, the plane of which is closest to the distance indicated by the distance measurement .
[0070] In order to accomplish said task, the apparatus is trained using the training data depicted in the lower part of Figure 5 . The training data in this case comprises a number of L combinations Cl to CL . L is a natural number greater than one . The first combination Cl consists of a first training input image Ti l and a first number of N target images Tol to ToN . The Lthcombination CL consists of the training input image TiL and an Lth number of N target output images Tol to ToN . Therein N identi fies the number of distance planes , and L identi fies the number of the training combinations . Each training input image Tix of each combination Cl to CL is a raw image captured by the lensless camera 10 . Each target image Tox of every combination Cl to CL is a natural image with a single plane in focus , which is captured by the lensed camera 20 using the arrangement depicted in Figure 6 . In preparation of the training data, each combination Cl to CL is captured using the arrangement of Figure 6 by placing a target obj ect Obj at a distinct distance in relation to the beam-splitter 30 , capturing the training input image Tix of the combination Cx and capturing the N target images Tol , ..., ToN of said combination Cx in direct succession, while continuously increasing or decreasing the focal distance of the lensed camera 20 for every capture .
[0071] After training with the described training data, the apparatus of Figure 5 is enabled to provide a single focus image with discrete depth resolution after taking j ust one shot of a scene Sc using the proposed method .
[0072] Figure 6 shows an exemplary embodiment of an arrangement for training any of the apparatuses as described above . Said arrangement comprises the lensless camera 10 , the lensed camera 20 , and the beam-splitter 30 . The lensless camera 10 of the apparatus as depicted above in Figures 2 to 4 , the beam-splitter 30 and the lensed camera 20 are arranged in a triangle configuration as shown in Figure 6 , such that a target obj ect Obj in a scene Sc is captured in the same way by the lensed camera 20 and the lensless camera 10 .
[0073] The described method and apparatus reali ze a single shot 3D reconstruction, all-in- focus and / or selective focus 2D output images with multiple obj ect distances . Compared to the known solutions , the apparatus is not trained to a fixed distance which cannot be altered without retraining . These reconstructions are provided with considerably lower computing power compared to the known state of the art .
[0074] It will be appreciated that the invention is not limited to the disclosed embodiments and to what has been particularly shown and described hereinabove . Rather, features recited in separate dependent claims or in the description may advantageously be combined . Furthermore , the scope of the invention includes those variations and modi fications which will be apparent to those skilled in the art and fall within the scope of the appended claims . The term " comprising" used in the claims or in the description does not exclude other elements or steps of a corresponding feature or procedure . In the case that the terms " a" or " an" are used in conj unction with features , they do not exclude a plurality of such features. Moreover, any reference signs in the claims should not be construed as limiting the scope.
[0075] This patent application claims the priority of German patent application 102023111405.0, the disclosure content of which is hereby incorporated by reference.
[0076] References
[0077] SI, S2, S3, S4, S5, S6 method step
[0078] 10, 20 camera 30 beam-splitter
[0079] 100 processing means lin raw image lintl, ..., IlntN intermediate image lout, loutl, ..., loutN output image loutf final output image
[0080] Wl, W2, ..., WN weight
[0081] UW second set of weights
[0082] UN neural network
[0083] Ob j target object Sc scene
[0084] Til, TiM training input image
[0085] Tol, ToM, ToN target image
[0086] Cl, ...., CL combination
Claims
Claims1. A method for reconstructing an image from a raw image comprising the following steps: capturing (SI) , by a lensless camera (10) with coded aperture, the raw image (Tin) in a single-shot, deconvolving (S2) the raw image (Tin) by using a first set of weights comprising a number of N two-dimensional weights (Wl, ..., WN) initialized in a preceding calibration phase and stored in spatial domain, and therefrom providing a number of N intermediate images (lintl, ..., lintN) , wherein N is a natural number larger than or equal to two, processing (S3) each of the N intermediate images (lintl, ..., lintN) by a trained neural network (UN) according to a second set of weights (UW) acquired in a preceding training phase and therefrom providing at least one output image (lout) , wherein each of the number of N pre-initialized two- dimensional weights (Wl,..., WN) corresponds to a discrete object distance (zl, ..., zM) .
2. The method according to claim 1, wherein each of the number of N pre-initialized two- dimensional weights of the first set depends on a point spread function resulting from a picture of a point light source through a mask used in the coded aperture of the camera, wherein each of the number of N pre-initialized two- dimensional weights of the first set is initialized in the calibration phase, in which the point light source is adjusted to a distinct distance with respect to the camera and the picture is captured, wherein each of the number of N pre-initialized two- dimensional weights is accorded to one plane of the outputimage , wherein said plane matches the distance between the camera and the point light source .3 . The method according to claim 1 or 2 , wherein the trained neural network is reali zed as a U-Net .4 . The method according to claim 2 or 3 , wherein each of the number of N intermediate images represents an instance of the output image according to the plane indicated by the corresponding weight of the number of N pre-initiali zed two-dimensional weights of the first set .5 . The method according to any of claims 1 to 4 , wherein the processing of each of the N intermediate images by a trained neural network is performed in parallel using stacked matrices .6 . The method according to any of claims 1 to 5 , wherein the calibration phase is performed before the training phase , and wherein optionally the first set of weights obtained in the calibration phase is fine-tuned in the subsequent training phase .7 . The method according to any of claims 1 to 6 , wherein from the processing of each of the N intermediate images by the trained neural network exactly one output image is provided, and wherein in the training phase the neural network acquires the second set of weights based on a number of M pairs , each pair consisting of a training input image ( Ti l , TiM) and a target image ( Tol , ..., ToM) , wherein M is a natural numberlarger than or equal to one, and wherein each training input image is a raw image captured by the lensless camera.
8. The method according to claim 7, wherein each target image is a natural image with a single plane in focus, which is captured by a lensed camera (20) , wherein each pair of training input image and target image is captured simultaneously using an arrangement comprising the lensed camera (20) , the lensless camera (10) and a beamsplitter (30) , by placing a target object (obj) at a distinct distance (zl, ..., zM) in relation to the beam-splitter (30) and adjusting a focal distance of the lensed camera (20) to the distinct distance (zl, ..., zM) of the target object (Obj) .
9. The method according to claim 7, wherein each target image is a natural image with at least two planes in focus, which is captured by a lensed camera (20) which is prepared to have a wide range of focus, optionally in the target image a whole scene (Sc) is in focus , wherein each pair of training input image and target image is captured simultaneously using an arrangement comprising the lensed camera (20) , the lensless camera (10) and a beamsplitter (30) , by placing a target object (Obj) at a different random distance (zl, ..., zM) in relation to the beam-splitter (30) .
10. The method according to claim 7, wherein each target image is a depth map image of a scene, which is captured by a depth camera, wherein each pair of training input image and target image is captured simultaneously using the arrangement comprising the depth camera, the lensless camera and the beam-splitter, orwherein each pair of training input image and target image is captured by the lensless camera and the depth camera being arranged close to each other, for example side by side, or above or below each other, in a rigid frame.
11. The method according to any of claims 1 to 6, wherein the processing of each of the N intermediate images by the trained neural network provides a number of N output images (loutl, loutN) , wherein each of the number of N output images represents a plane corresponding to a weight of the first set of weights, wherein the method further comprises selecting (S5) an image from the number of N output images according to a given distance, and providing (S6) the selected image as a final output image ( loutf ) .
12. The method according to claim 11, wherein the given distance is determined by measuring a distance of an object (Obj) in the scene (Sc) , which is captured by the lensless camera (10) .
13. The method according to claim 11 or 12, wherein in the training phase the neural network acquires the second set of weights (UW) based on a number of combinations (Cl, ..., CL) , each combination consisting of one training input image and a number of N target images, and wherein each training input image is a raw image captured by the lensless camera (10) , wherein each target image is a natural image with a single plane in focus, which is captured by a lensed camera (20) , wherein each combination of training input image and N target images is captured using an arrangement comprising the lensedcamera (20) , the lensless camera (10) and a beam-splitter(30) , by placing a target object (Obj) at a distinct distance in relation to the beam-splitter (30) , capturing the training input image and capturing the N target images in direct succession, while continuously increasing or decreasing a focal distance of the lensed camera (20) for every capture.
14. An apparatus for reconstructing an image from raw image data comprising a lensless camera (10) with coded aperture, and processing means (100) which are coupled to the camera, wherein the apparatus is configured to perform the method according to any of claims 1 to 13.
15. An arrangement for training an apparatus comprising the apparatus according to the preceding claim, a beam-splitter (30) and a lensed camera (20) , wherein the lensless camera (10) of the apparatus, the beamsplitter (30) and the lensed camera (20) are arranged in a triangle configuration such that a target object (Obj) of a scene (Sc) or the scene (Sc) is captured in the same way by the lensed camera (20) and the lensless camera (10) .
Citation Information
Patent Citations
Method and Apparatus for Reconstructing Hyperspectral Image Using Artificial Intelligence
US20190096049A1
DE102023111405A
