Plenoptic imaging device with virtual intermediate image
The plenoptic imaging device addresses the low numerical aperture issue by placing the object between the focal plane and the main lens, enhancing resolution and depth of field for improved 3D X-ray image reconstruction.
Patent Information
- Application Number
- FR2021010579
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-10-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-10-06
AI Technical Summary
Conventional plenoptic imaging devices, particularly in the field of X-ray imaging, suffer from low numerical aperture, leading to significant loss of longitudinal resolution and complicating 3D reconstruction due to large voxel dimensions associated with depth, limiting their effectiveness in capturing and reconstructing 3D images of partially transparent samples.
A plenoptic imaging device configuration where the object is placed between the object focal plane and the main lens, forming a virtual image on the side of the object, with a main optical assembly and a light field sampling assembly to improve lateral, longitudinal, and angular resolution, utilizing a microlens matrix and photodetector to capture spatio-directional information.
Enhances lateral, longitudinal, and angular resolution, and depth of field, enabling more effective 3D reconstruction of X-ray images by increasing the object numerical aperture and reducing device size, while maintaining compactness and improving image clarity.
Smart Images

Figure 00000014_0000 
Figure 00000014_0001 
Figure 00000015_0000
Abstract
Description
Title of the invention: Plenoptic imaging device with virtual intermediate image Technical field
[0001] The present invention relates to the field of plenoptic imaging devices and the field of X-ray imaging. Prior art
[0002] Conventional digital cameras provide a two-dimensional image of a three-dimensional scene. This two-dimensional image represents the total amount of light illuminating each point on the photodetector within the camera. However, this two-dimensional image does not contain any information about the direction of the light illuminating the photodetector.
[0003] Conversely, plenoptic imaging devices sample the 4-D light field from a scene. The 4D light field or radiance (or "light-field" in English) is a representation of all the light rays in free space. It is a function r(#, p) or P are vectors that represent the position and direction of a ray intersecting a plane transverse to the optical axis. In a plenoptic imaging device, the sampling of the direction of the light field is carried out by measuring the inclination of the rays, for example via a microlens array.
[0004] Figures 1A and 1D illustrate two plenoptic imaging device configurations known to those skilled in the art, referred to respectively as the "traditional" configuration and the "focused" configuration.
[0005] In the traditional configuration, illustrated in Figure 1A, the device comprises a main objective lens 100, a microlens array 101 and a matrix photodetector 102. In this configuration, the microlens array is arranged in a plane optically conjugated by the main lens with the object plane, i.e. JL + A — ~L, with the distance between the object 103 and the main lens 100, 411 the distance between the main lens 100 and the microlens array 101 and / j the focal length of the main lens. In this way the image of the object, intermediate image, is formed directly in the plane of the microlens array 101. In the traditional configuration, the photodetector 102 is placed at a distance b from the microlens array 101 equal to the focal length f2 of each microlens. In this way, the microlens array is focused to infinity.As the distance between the main lens 100 and the microlens array 101. is much greater than the focal length f, this amounts to saying that the microlens array is focused on the main lens 100. Thus, the image recorded by the photodetector 102 is formed from a plurality of micro-images of the main lens.
[0006] As illustrated in [Fig.lB], the microlens array 101 makes it possible to separate the spatial components from the angular components. The spatial components are sampled directly by the microlenses because each microlens is associated with a different portion of the image formed by the main lens. The detection of the angular components is carried out by the pixels of each micro-image 104. Indeed, each microlens separates a beam coming from the main lens into rays coming from different portions of the aperture of the main lens. Each of these rays is detected by a pixel of the photodetector and the pixels forming each micro-image 104 are associated with different angles of the rays.
[0007] [Fig.lC] shows an example of a "raw" plenoptic image or light field image obtained by a plenoptic device and formed by all the micro-images, taken from Ren Ng. Digital Light Field Photography. PhD thesis, Stanford University, CA, USA, 2006. AAI3219345 or in Lumsdaine, A., & Georgiev, T. (2009, April). From the angular information contained in this light field image, it is possible to reconstruct an image focused at a certain depth with refocusing algorithms. To do this, the light field is extracted from the raw plenoptic image and back-projected into the object space. This step can be carried out in different ways (stereo-estimation, shift-and-sum algorithm, rendering with blending, "superresolution" method, etc.). The algorithm synthetically mimics the action of optical focusing on a sensor, as if the image were acquired by a conventional camera.The resulting image contains the in-focus elements that correspond to the refocusing depth, but also the out-of-focus elements from adjacent depth planes. A reconstruction with the refocusing algorithm has the same properties as with conventional photography: the in-focus elements are reconstructed sharply, while the out-of-focus elements are blurred. A detailed description of some refocusing algorithms can be found in Ren Ng. Digital Light Field Photography. PhD thesis, Stanford University, CA, USA, 2006. AAI3219345, Todor Georgiev, Georgi Chunev, and Andrew Lumsdaine. Superresolution with the focused plenoptic camera. SPIE Electronic Imaging,7873:1-13, 2011, or in Lumsdaine, A., & Georgiev, T. (2009, April). The focused plenoptic camera. In 2009 IEEE International Conference on Computational Photography (ICCP) (pp. 1-8). IEEE. .
[0008] Unlike the traditional configuration, in the focused configuration, the intermediate image is not formed in the plane of the ML microlens array. This array can be respectively placed before or after (as illustrated in [Fig.lB]) the intermediate image formed by the main lens. The main difference between the two methods is that, in the focused configuration, the properties of the micro images vary depending on the depth of the object in the object space. Thus, the magnification of each micro image varies depending on the depth of the object in the object space. In addition, there is a lateral shift of the position imaged by adjacent micro images depending on the depth. The focused configuration allows for greater spatial resolution but lower angular resolution than the traditional configuration (see Lumsdaine, A., & Georgiev, T. (2009, April). The focused plenoptic camera. In 2009 IEEE International Conference on Computational Photography (ICCP) (pp. 1-8). IEEE.).
[0009] Plenoptic imaging devices theoretically offer the possibility of reconstructing a 3D image of a sample partially transparent to X-rays by refocusing a 2D image of the sample in different planes. However, for plenoptic imaging devices which have resolutions limited by the diffraction of the main lens, the lateral and longitudinal resolutions vary in 1 / NA and 1 / NA2 with NA the object numerical aperture of the main lens. However, in the hard X-ray domain (therefore for photon energies greater than a few keV), the optics currently have very low numerical apertures (typically less than 0.001). This low numerical aperture then produces voxels having a dimension associated with the depth which is several orders of magnitude larger than the other two dimensions associated with the lateral resolution.3D reconstruction is then very complicated due to a very high loss of longitudinal resolution.
[0010] The invention aims to overcome certain problems of the prior art. To this end, an object of the invention is a plenoptic imaging device in which the object is placed between the object focal plane and the main lens in order to create a virtual image placed on the side of the object for the main lens. This new configuration produces an improvement in the lateral, longitudinal and angular resolution and in the depth of field compared to a device of the prior art with a focused configuration, which is a major advantage for the production of plenoptic cameras in X-rays which are currently very handicapped by the low numerical aperture of X-ray optics. Compared to a device of the prior art with a traditional configuration, the device of the invention allows an improvement in the lateral and longitudinal resolution. Summary of the invention
[0011] For this purpose, an object of the invention is a plenoptic imaging device for forming an image of the light field of an object comprising: - a sample holder suitable for holding said object, - a main optical assembly, arranged to focus rays from the light field of said object, at a distance called object from the sample holder less than an object focal distance of said main optical assembly so as to form an intermediate virtual image of said object in an intermediate image plane at a distance from said optical assembly, - a light field sampling assembly adapted to acquire spatio-directional information of the rays forming the virtual image and to form said light field image.
[0012] In a preferred embodiment of the invention, said light field sampling assembly comprises - a matrix photodetector - a microlens array arranged at a distance a from the intermediate image plane, the photodetector being arranged at a distance b from the microlens array, each microlens having a diameter d2 and a focal length f2 and being adapted to form a micro-image of a respective part of the virtual image of the object on a respective portion of the photodetector, each said portion comprising a plurality of pixels, the microimages forming said image of the light field.
[0013] Preferably, in this preferred embodiment, an image numerical aperture NAim^P of the main optical assembly is less than or equal to an image numerical aperture NAimML of each microlens.
[0014] Preferably, in this preferred embodiment, the photodetector is in a plane conjugated to the intermediate image plane by the microlens matrix.
[0015] Preferably, in this preferred embodiment, we have di _ £2, with *' i+t'+b & the diameter of the aperture of the main optical assembly.
[0016] Preferably, in this preferred embodiment, a distance c between the microlens matrix and the main optical assembly is greater than said distance
[0017] According to one embodiment of the device of the invention, the device comprises a processor adapted to implement a refocusing algorithm making it possible to reconstruct a depth image or a 3D image of the object from said image of the light field.
[0018] According to one embodiment of the device of the invention, the main optical assembly and the microlens matrix are adapted for X-rays.
[0019] According to one embodiment of the device of the invention, the main optical assembly is a “photon sieve”.
[0020] According to one embodiment of the device of the invention, the device of the invention comprises an X-ray source illuminating said object so as to form said light field of the object.
[0021] Another object of the invention is a method of using a plenoptic imaging device comprising a main optical assembly and a light field sampling assembly, said method comprising the following steps: - placing said device so that a so-called object distance between an object to be imaged and said main optical assembly is less than an object focal distance f of said main optical assembly so as to form an intermediate virtual image, - acquire, with the light field sampling set, spatio-directional information of the rays forming the virtual image in order to form an image of the light field of the object. Brief description of the drawings
[0022] Other characteristics, details and advantages of the invention will emerge from reading the description given with reference to the appended drawings given by way of example and which represent, respectively:
[0023] [Fig.lA] a schematic view of a plenoptic imaging device known to those skilled in the art,
[0024] [Fig.lB], a schematic view of rays forming a micro-image in a plenoptic imaging device known to those skilled in the art,
[0025] [Fig.lC], an image of the light field acquired by a plenoptic imaging device known to those skilled in the art,
[0026] [Fig.lD] a schematic view of a plenoptic imaging device known to those skilled in the art,
[0027] [Fig.2] a schematic view of a plenoptic imaging device according to the invention,
[0028] [Fig.3] a representation of the numerical apertures of the main optical assembly and an image numerical aperture of each microlens in the plenoptic imaging device according to the invention.
[0029] In the figures, unless otherwise indicated, the elements are not to scale. Description of the embodiments
[0030] [Fig. 2] illustrates a plenoptic imaging device 1 according to the invention, for forming an image of the light field of an object O particularly adapted to operate in the X-ray domain. As a reminder, by "image of the light field" is meant an image comprising both information on the intensity of the light field and information on the direction of the rays forming the light field. To produce this image, like the devices of the prior art, the plenoptic imaging device according to the invention comprises a main optical assembly LP and a light field sampling assembly LFS. In addition, the device 1 comprises a sample holder SH adapted to hold the object O in place.
[0031] The main optical assembly LP, with optical axis OA, is arranged to focus rays from the light field of the object. The LP assembly is placed at an object distance from the sample holder SH which is less than an object focal distance / j of the main optical assembly LP. Thus, the LP assembly forms an intermediate virtual image Iv of the object at a distance from the optical assembly, in an intermediate image plane. The magnification of the main optical assembly LP is defined as M — zj. The main optical assembly LP is an objective formed from a set of one or more refractive or diffractive optical components depending on the wavelength of the rays forming the light field of the object.
[0032] Unlike prior art plenoptic devices, the distance zo is less than the object focal distance f Y. This choice makes it possible to increase the object numerical aperture of the main optical assembly. As will be explained later, this produces two advantages compared to prior art plenoptic imaging devices operating with a focused configuration forming an image on the image side of the main lens: an improvement in angular resolution and better lateral and longitudinal resolution. Compared to a prior art plenoptic imaging device operating with a conventional configuration, the lateral and longitudinal resolution are improved. For clarification, "longitudinal resolution" corresponds to the minimum distance separating two planes of the object reconstructable by the device, the planes being perpendicular to the optical axis OA.
[0033] Finally, the device comprises a light field sampling assembly LFS adapted to acquire spatio-directional information of the rays forming the virtual image and to form the image of the light field. This LFS assembly is known to those skilled in the art. According to a preferred embodiment of the invention, illustrated in FIG. 2, this assembly comprises a matrix photodetector Det and a microlens matrix ML. The matrix ML comprises identical microlenses, of diameter d2 and focal length / 2. As a non-limiting example, the photodetector Det can be a CCD or CMOS sensor, adapted according to a wavelength of the rays forming the light field of the object.
[0034] Figure 2 illustrates different parameters of the device. The microlens array ML is arranged at a distance a from the intermediate image plane and the photodetector is arranged at a distance b from the microlens array. As in the configuration focused illustrated in Figure 1D, in the device 1 of the invention, the photodetector Det is in a plane conjugated to the intermediate image plane by the microlens matrix, so that -+- = ~. The magnification of each microlens is defined as m — L. Each microlens of the matrix ML is adapted to form a microimage / / / of a respective part of the virtual image Iv of the object on a respective portion of the photodetector MP. This micro-image is not visible in [Fig.2] but is represented in [Fig.3]. This portion MP of the photodetector, also called macropixel, comprises a plurality of pixels. All of the micro-images form the image of the light field of the object.
[0035] The device of the invention is an evolution of the device of the prior art with a focused configuration illustrated in [Fig.1D]. In the following section, we detail how this evolution allows an improvement of certain parameters of the device of the invention.
[0036] In the device of the invention, as in a device of the prior art with a focused configuration, the lateral resolution is given by the maximum value between 3 dimensions representing the diffraction by the main optical assembly, the diffraction by each microlens and the dimension of a pixel of the photodetector. In the intermediate image plane, the resolution given by the diffraction by the main assembly according to the Sparrow criterion is x M' with 'a wavelength of the rays forming the light field. Similarly, in the plane of the photodetector, the resolution given by diffraction by a microlens according to the Sparrow criterion is xm, with AM^^a numerical aperture object of a microlens. By comparing these dimensions in the photodetector space, we obtain an effective pixel size Ap^ which is:
[0037] Ape / / = max(mJ?oè; ; robj; Ap) (Eql).
[0038] Assuming that the pixel size is not the element limiting the resolution and placing ourselves in the object space of the main lens, we then obtain that the lateral resolution riat of the device is worth:
[0039] _ max(mRr)!,j-,robj1 (Eq2) Mum
[0040] Similarly, it can be proven that the longitudinal resolution is:
[0041] r, — -ÏA—।--4— (Eq3).
[0042] In order to compare the lateral and longitudinal resolutions between the device of the invention and the device of the prior art in focused configuration, it is therefore necessary to compare how the different parameters of equations Eq2 and Eq3 evolve.
[0043] Prior art plenoptic imaging devices operating with a confi focused guration have a magnification of the main optical assembly M = ^ / ¾ lower than that of the invention (because is lower in the invention and is greater in the invention). However, they have a magnification m = b / a larger than the invention. In addition, the device of the invention has a smaller NAOML aperture and a larger NA„LP aperture than the prior art device with a focused configuration.
[0044] In the device of the invention, preferably the microlens matrix is adapted so that the lateral resolution riaf is fixed by Robj, that is to say so that > r^;. Thus, the lateral resolution is worth r — .....9.4.7½..... So that JJ lai mxNA^ ' the device of the invention allows an improvement in the lateral resolution 1 i«t compared to the device of the prior art with a focused configuration, it is necessary that the product mx NA^p remains greater than 1 when moving from a device of the prior art with a focused configuration to the device of the invention. This amounts to choosing the distance b in the invention, such that »? x NA^p remains greater than 1 when moving from a device of the prior art with a focused configuration to the device of the invention, or else b > ax
[0045] Thanks to equation Eq3 it is clear that the device of the invention makes it possible to greatly improve the longitudinal resolution compared to the device of the prior art with a focused configuration, when the lateral resolution also decreases.
[0046] We now turn our attention to the angular resolution. As a reminder, the angular resolution corresponds to the quantity of different angular components - that is to say directions - collected for each spatial position. For a given spatial position in the intermediate image plane, the associated directions are distributed over several microlenses (see Figure 2). Also, estimating the angular resolution amounts to estimating the number of microlenses that capture these directions. In the device of the invention and in a device of the prior art with a focused configuration, the angular resolution Nu is given by the following relationship:
[0047]
[0048] We define a — c - Zj < = > - 1, with c the distance between the micro matrix lens and the main optical assembly (see Figure 2). It is deduced from Eq4 that the angular resolution increases thanks to the configuration of the invention compared to that of a prior art device with a focused configuration, if C > 2z^.
[0049] The plenoptic depth of field corresponds to the depth range over which it is possible to reconstruct a sharp image from a single acquisition. In the device of the invention and in a prior art device with a focused configuration, the plenoptic depth of field DOFp is given by the following relationship:
[0050] dof„ = n,-^- + <eq5) i UNA„LP NA2^
[0051] Equation 5 implies that with a suitable geometry, when the device of the invention allows an increase in angular resolution ^greater than a decrease in lateral resolution ri«t, the plenoptic depth of field increases compared to that of a device with a focused configuration, taking into account a larger NAOrLP aperture.
[0052] Thus, thanks to an appropriate optical assembly, the optical device of the invention allows an improvement in the lateral, longitudinal and angular resolution and the depth of field compared to a device of the prior art with a focused configuration. In the field of X-ray plenoptic imaging, this makes it possible to partially circumvent the limitation imposed by the low numerical aperture of the optics usually used.
[0053] Furthermore, the device of the invention is more compact than a prior art device with a focused configuration, because the distance b is reduced.
[0054] In the device of the invention, it is desirable to avoid an overlap of the micro-images in the plane of the photodetector which would induce a loss of spatio-directional information and would harm the focused reconstruction of the image. For this, it is necessary that an image numerical aperture NAimLP of the main optical assembly is less than or equal to an image numerical aperture NAim^L of each microlens. Figure 3 illustrates the preferred embodiment of the invention, in which the image numerical aperture NAimLP of the main optical assembly is equal to the image numerical aperture NAim^L of each microlens. For this, in this embodiment, we have _ ^2 Thus, the micro-images are adjacent in the Zj+aH-i» b ' plane of the photodetector and we optimize the use of the number of pixels of the photodetector. We note that the previous equality is different from that specified by the prior art (see for example Todor Georgiev and Andrew Lumsdaine. Reducing plenoptic camera artifacts. Computer Graphics Forum, 29(6):1955-1968, 2010.) which carried out the approximation a, b, which is incorrect in the device of the invention.
[0055] According to a preferred embodiment of the invention, the sample holder SH is adapted to be moved relative to the main optical assembly. Furthermore, preferably, the microlens array and the photodetector are mounted on respective translation stages. This embodiment makes it possible to easily control the distances 4o, c and b and therefore makes it possible to modify the resolutions and the plenoptic depth of field of the device.
[0056] According to one embodiment of the invention, the main optical assembly and the Microlens arrays are suitable for X-rays (from about 0.01 / tm to 10nm). For example, the microlens array is formed from a Fresnel lens array (Fresnel Zone Plate in English) and the main optical assembly includes one or more Fresnel lenses or a photon sieve element. By "photon sieves" is meant here a membrane opaque to radiation and comprising a plurality of apertures arranged appropriately in the Fresnel zone of a beam, so as to focus it to a size smaller than the dimension of the apertures by playing on the constructive and destructive interferences of the light diffracted by the apertures (see for example Kipp, L., Skibowski, M., Johnson, RL, Berndt, R., Adelung, R., Harm, S., & Seemann, R. (2001). Sharper images by focusing soft X-rays with photon sieves. Nature, 414(6^60), 184-188.).
[0057] Alternatively, according to another embodiment, the main optical assembly comprises one or more refractive lenses and the microlens array comprises a plurality of refractive lenses.
[0058] According to one embodiment of the invention, the device 1 comprises a processor connected to the photodetector and adapted to implement a refocusing algorithm making it possible to reconstruct a depth image or a 3D image of the object from the image of the light field. Indeed, when the object is partially transparent to the radiation illuminating it, it is possible to reconstruct the internal structure of the object by refocusing a 2D image of the sample in different planes perpendicular to the optical axis and each separated by a distance corresponding to the longitudinal resolution. The refocusing algorithms are not the subject of the invention and are known to those skilled in the art (see Ren Ng. Digital Light Field Photography. PhD thesis, Stanford University, CA, USA, 2006. AAI3219345 or Lumsdaine, A., & Georgiev, T. (2009, April). The focused plenoptic camera. In 2009 IEEE International Conference on Computational Photography (ICCP) (pp. 1-8). IEEE).
[0059] According to another embodiment of the invention, different from that illustrated in Figure 2, the sampling assembly of the LFS light field comprises a matrix of circular apertures instead of a microlens matrix. In this case, the microimages are formed by the circular apertures on the photodetector provided that the diameter d2 of the apertures allows it. In a manner known per se, it is preferable that the diameter d2 of the apertures is — 1.562^2 at i 10%.
[0060] Another object of the invention is a method of using a plenoptic imaging device comprising a main optical assembly LP and a light field sampling assembly LFS. This method comprises the following steps: - place the device so that the distance between the object to be imaged and said main optical assembly is less than the object focal length f} of the main optical assembly, so as to form the intermediate virtual image Iv placed on the object side of the main optical assembly, - acquire, with the light field sampling assembly, the spatio-directional information of the rays forming the virtual image in order to form the image of the light field of the object.
[0061] As explained above, this method makes it possible to increase the object numerical aperture NAoLP of the main lens, which makes it possible, depending on the configuration of the device, to increase the different resolutions of the device and the plenoptic depth of field.
Claims
Claims
1. A plenoptic imaging device (1) for forming an image of the light field of an object (0) comprising: - a sample holder (SH) adapted to hold said object, - a main optical assembly (LP), arranged to focus rays of the light field of said object, at a distance 7,1 called object from the sample holder less than an object focal distance f of said main optical assembly so as to form an intermediate virtual image (Zv) of said object in an intermediate image plane at a distance from said optical assembly, - a light field sampling assembly (LFS) adapted to acquire spatio-directional information of the rays forming the virtual image and to form said image of the light field, said light field sampling assembly comprising - a matrix photodetector (Det) - a microlens matrix (ML) arranged at a distance a from the intermediate image plane,the photodetector being arranged at a distance b from the microlens array, each microlens being of diameter d2 and having a focal length / 2 and being adapted to form a micro-image (id) of a respective part of the virtual image of the object on a respective portion of the photodetector, each said portion comprising a plurality of pixels, the micro-images forming said image of the light field.,
2. Device according to the preceding claim, in which an image numerical aperture NAimLP of the main optical assembly is less than or equal to an image numerical aperture NAim^L of each microlens.
3. Device according to claim 1 or 2, in which the photodetector is in a plane conjugated to the intermediate image plane by the microlens array.
4. Device according to any one of the preceding claims, in which _ d2, with d} the diameter of the opening of the main optical assembly.
5. A device according to any preceding claim, wherein a distance c between the microlens array and the main optical assembly is greater than said distance zi.
6. Device according to any one of the preceding claims, comprising a processor adapted to implement a refocusing algorithm making it possible to reconstruct a depth image or a 3D image of the object from said image of the light field.
7. A device according to any preceding claim, wherein the main optical assembly and the microlens array are adapted for X-rays.
8. Device according to the preceding claim, in which the main optical assembly is a “photon sieve”.
9. Device according to claim 7 or 8, comprising an X-ray source illuminating said object so as to form said light field of the object.
10. A method of using a plenoptic imaging device according to any one of claims 1 to 9, said method comprising the following steps: A. placing said device so that a so-called object distance between an object to be imaged and said main optical assembly is less than an object focal distance / 1 of said main optical assembly so as to form an intermediate virtual image (Iv), B. acquiring, with the light field sampling assembly, spatio-directional information of the rays forming the virtual image in order to form an image of the light field of the object.
