Imaging device and optical element

Abstract

This imaging device (10) has: an optical element (12) having a transparent substrate, a plurality of structures positioned on or within the transparent substrate in the planar direction of the transparent substrate, and a transparent layer for covering a plurality of pixels, each of which includes a photoelectric conversion element; an imaging element (11) in which are positioned the plurality of pixels that include photoelectric conversion elements; and a signal processing unit that generates an image signal on the basis of an electrical signal obtained from the imaging element (11). The optical element (12) outputs light in a state having differing point spread functions for each wavelength, and forms, on the imaging element (11), images in which the point spread function for each wavelength is convoluted. The plurality of structures have the same height when viewed from the side. The signal processing unit (13) reconstructs the images in which the point spread function for each wavelength is convoluted.

Description

Technical Field

[0001] The present invention relates to an imaging device and an optical element.

Background Art

[0002] Generally, an imaging device acquires optical information that can be obtained as a two-dimensional image of three colors, R (red), G (green), and B (blue). On the other hand, in recent years, hyperspectral cameras have been put into practical use as cameras that acquire more detailed color information (wavelength spectrum), and efforts have been made to extract new valuable information from more diverse optical information. Generally, as a hyperspectral camera that has been put into practical use, there is a line scan type. In addition, research on a compressive sensing type hyperspectral camera is underway.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] A line scan type hyperspectral camera allows only one-dimensional spatial light to enter through a slit, separates wavelengths by a spectroscopic element (prism or grating), and acquires one-dimensional space × one-dimensional wavelength using a two-dimensional image sensor. The line scan type hyperspectral camera generates an image of two-dimensional space × one-dimensional wavelength by taking multiple shots while scanning the position of the slit (or the imaging target itself).

[0005] In the case of the line scan type, since wavelength information is directly obtained by wavelength separation using a spectroscopic element, it has the advantage of good accuracy of the wavelength spectrum. On the other hand, the line scan type hyperspectral camera has the disadvantages that the device becomes large due to the long optical path length required for wavelength separation and the scanning mechanism, and the time resolution is low (it cannot image moving objects) due to multiple shootings.

[0006] A compressive sensing type hyperspectral camera performs optical encoding (operation / arithmetic of optical information) on the imaging target for imaging, and restores spatial information and wavelength information using a reconstruction process that utilizes the sparsity of natural images.

[0007] Since the compressive sensing type hyperspectral camera accurately restores the original information from a small amount of information, compared with the other methods described above, it has the advantages of less lack of information amount (without sacrificing spatial and time dimensions) and generally being a single-shot operation. On the other hand, the compressive sensing type hyperspectral camera has the disadvantage that a large-scale and complex optical system is generally required for optical encoding, so the manufacturing man-hours are many and the device also becomes large. Although it is also possible to perform encoding with a simplified optical system using a multi-stage diffractive optical element that requires multi-stage lithography, the manufacturing man-hours are many, and there are restrictions on the light transmittance and the numerical aperture (or F value) of the optical system due to principle constraints (such as the shadow effect).

[0008] The present invention has been made in view of the above, and an object thereof is to provide a hyperspectral imaging device having a simple device configuration and an optical element for realizing a hyperspectral imaging device having a simple device configuration.

Means for Solving the Problems

[0009] In order to solve the above-described problems and achieve the object, an imaging device according to the present invention includes an optical element having a transparent substrate and a plurality of structures arranged in a plane direction of the transparent substrate on or in the transparent substrate, an imaging element in which a plurality of pixels including a photoelectric conversion element are arranged, and a signal processing unit that generates an image signal based on an electrical signal obtained from the imaging element. The optical element outputs light in a state having a different point spread function for each wavelength, forms an image in which the point spread functions of each wavelength are convolved on the imaging element, the plurality of structures have the same height when viewed from the side, and the signal processing unit is characterized by reconstructing an image in which the point spread functions of each wavelength are convolved.

[0010] Further, an optical element according to the present invention is an optical element having a transparent substrate and a plurality of structures arranged in a plane direction of the transparent substrate on or in the transparent substrate, the optical element outputs light in a state having a different point spread function for each wavelength, forms an image in which the point spread functions of each wavelength are convolved on the imaging element, and the plurality of structures are characterized by having the same height when viewed from the side.

Effects of the Invention

[0011] According to the present invention, a hyperspectral imaging device having a simple device configuration can be realized.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0013] Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings. In the following description, each drawing only schematically shows the shape, size, and positional relationship to the extent that the content of the present invention can be understood. Therefore, the present invention is not limited to only the shape, size, and positional relationship illustrated in each drawing. Also, in the description of the drawings, the same parts are denoted by the same reference numerals. Hereinafter, when it is described as “^A” with respect to A, it is considered to be equivalent to “a symbol in which ‘^’ is written immediately above ‘A’”.

[0014] [Embodiment] [Imaging Device] First, an imaging device according to an embodiment of the present invention will be described. FIG. 1 is a side view showing a schematic configuration of the imaging device according to the embodiment. FIG. 2 is a schematic diagram showing the processing until the imaging device 10 shown in FIG. 1 acquires an image.

[0015] As shown in FIG. 1, the imaging device 10 according to the embodiment includes an optical element 12, an imaging element 11, and a signal processing unit 13. The imaging element 11 has a photoelectric conversion element such as a CCD or a CMOS. The signal processing unit 13 processes the photoelectric conversion signal output from the imaging element 11 to generate an image signal.

[0016] As shown in FIGS. 1 and 2, in the imaging device 10, light such as natural light or illumination light is irradiated onto an imaging target (real image), and the light transmitted / reflected / scattered by the imaging target 1 or the light emitted from the imaging target 1 forms an optical image on the imaging element 11 by the optical element 12.

[0017] The optical element 12 has a fine binary structure. In other words, the optical element 12 has a plurality of fine columnar structures 160. And the optical element 12 performs optical encoding by having a function with different imaging characteristics according to the wavelength. The optical element 12 is a lens (wavelength-dependent PSF lens) having a PSF (Point Spread Function) with clearly different shapes according to the wavelength, and has a function of generating an image obtained by performing a different convolution operation for each wavelength on a real image (subject). The optical element 12 is a wavelength-dependent PSF lens. When an object is imaged with this optical element 12, a convolution operation is performed on the real image with a different PSF for each wavelength, and the result is imaged on the imaging element.

[0018] The imaging element 11 acquires an observation image subjected to different convolution operations for each wavelength by the optical element 12 which is a wavelength-dependent PSF lens. The signal processing unit 13 generates a spectral image by performing a process of reconstructing the spatial information and spectral information of the subject from the observation image based on compressive sensing.

[0019] Note that the imaging device 10 may include known components such as an infrared cut optical filter, an electronic shutter, a viewfinder, a power source (battery), a flashlight, etc., but the descriptions thereof are omitted because they are not particularly necessary for understanding the present invention. Also, the above configuration is merely an example, and in the embodiment, known elements can be appropriately combined and used as components other than the optical element 12, the imaging element 11, and the signal processing unit 13.

[0020] [Lens and Imaging Element] Subsequently, an overview of the optical element 12 and the imaging element 11 in the embodiment will be described. The imaging element 11 has a plurality of pixels each including a photoelectric conversion element arranged in a two-dimensional array. The optical element 12 has a different PSF for each wavelength, and a plurality of fine structures having a function of imaging on the imaging element 11 an image in which a different PSF for each wavelength is convolved with the real image are arranged over the entire surface.

[0021] FIG. 3 is a diagram schematically showing a part of a cross section of an imaging device and an optical element according to an embodiment. Also, an xyz coordinate system is shown after FIG. 3. The xy plane direction corresponds to the plane direction of the imaging device 11, the transparent substrate 190, etc. described later. Hereinafter, unless otherwise specified, "plan view" refers to viewing in the z-axis direction (for example, in the negative z-axis direction). "Side view" refers to viewing in the x-axis direction or the y-axis direction (for example, in the negative y-axis direction).

[0022] As shown in FIG. 3, the optical element 12 is disposed to face the imaging device 11. The imaging device 11 and the optical element 12 are provided in this order in the positive z-axis direction.

[0023] The imaging device 11 includes a wiring layer 180 and a plurality of pixels 130 arranged in the xy plane direction. Each pixel 130 is configured to include a photoelectric conversion element. An example of the photoelectric conversion element is a photodiode (PD: Photo Diode). Each pixel corresponds to red (R), green (G), and blue (B). An example of the wavelength band of red light is 600 nm < λ0 ≦ 800 nm, where the wavelength is λ0. An example of the wavelength band of green light is 500 nm < λ0 ≦ 600 nm. An example of the wavelength band of blue light is λ0 < 500 nm. The pixels R, G, and B may be in a Bayer array. Alternatively, the pixels may be for a monochrome image.

[0024] The incident light travels along the negative z-axis direction and reaches the imaging device 11 through the optical element 12. The charges generated in each pixel of the imaging device 11 are converted into an electrical signal that forms the basis of a pixel signal by a transistor (not shown) or the like, and are output to the outside of the imaging unit 100 through the wiring layer 180. In FIG. 3, some of the wirings included in the wiring layer 180 are shown.

[0025] The optical element 12 is disposed on the side where light from the imaging object is incident. When viewed in plan view, the optical element 12 is provided so as to cover the imaging element 11. The optical element 12 is constituted by a plurality of structures 160, for example, periodically (having a periodic structure), on the bottom surface of the transparent substrate 190. The plurality of structures 160 may be arranged at equal intervals or at unequal intervals for ease of design. The plurality of structures 160 are formed in the transparent layer 150 formed on the imaging element 11 to cover a plurality of pixels.

[0026] The transparent substrate 190 is a low-refractive-index transparent substrate made of a material such as SiO2 (refractive index n = 1.45), for example. The transparent layer 150 is a low-refractive-index transparent layer made of a material such as air or SiO2. The materials of the transparent substrate 190 and the transparent layer 150 may be single or may be a laminate of a plurality of materials. The plurality of structures 160 have the same height when viewed in side view. The plurality of structures 160 are composed of a fine structure pattern formed of a material such as SiN or TiO2 having a refractive index higher than that of the transparent layer 150.

[0027] An example of the optical element 12 is a metasurface. The metasurface includes a plurality of fine structures (corresponding to the structures 160) having a width equal to or less than the wavelength of light when viewed in plan view and having the same height when viewed in side view. When each cross-sectional shape of the plurality of structures 160 is cut by a plane parallel to the xy plane, it is a four-fold rotationally symmetric shape. Note that the metasurface may have a two-dimensional structure or a three-dimensional structure. The optical element 12 can control the phase and the light intensity according to the characteristics of light (wavelength, polarization, incident angle) only by changing the parameters of the structure 160. In the case of a three-dimensional structure, the degree of design freedom is improved compared to a two-dimensional structure.

[0028] The optical element 12 has different PSFs according to the wavelength, and thus the imaging characteristics (blurring) are different according to the wavelength of the light from the imaging object 1. The light from the imaging object 1 is imaged on the imaging element 11 by the optical element 12 having a wavelength-dependent PSF function and is acquired as an image (RGB image or monochrome image).

[0029] The acquired image corresponds to the result of optically convolving the imaging target (real image) 1 for each wavelength by the wavelength-dependent PSF of the optical element 12 and integrating it along the wavelength dimension on the pixel. The optical element 12 and the imaging element 11 acquire an optically encoded and compressed image. When the imaging element 11 is a color image sensor, after the convolution operation, multiplication is performed according to the wavelength sensitivity of each pixel of R, G, and B of the imaging element 11, and then integration is performed along the wavelength dimension on the pixel.

[0030] In this way, in the imaging device 10, with a single optical element 12 (optical element 12), an optically encoded image is formed on the imaging element 11. In other words, in the imaging device 10, effective encoding in spectral image reconstruction can be performed by a single optical element 12 (optical element 12). Therefore, the components of the imaging device 10 only need to be a single optical element 12 and the imaging element 11, and a hyperspectral imaging device can be realized with a simple device.

[0031] Also, in the imaging device 10, the distance between the optical element 12 and the imaging element 11 is determined by the focal length of the lens, similar to a normal imaging device. Therefore, the size of the imaging device 10 is equivalent to that of a normal camera having the same field of view F number.

[0032] When the observation process is known (here, the PSF of the optical element 12 and the wavelength sensitivity characteristics of the sensor), the optically encoded image can restore the information of the real image by performing appropriate signal processing in the signal processing unit 13.

[0033] The imaging device 10 performs signal processing using compressive sensing, which is a method of reconstructing (restoring) the target with high precision from a small amount of information by utilizing the sparsity of natural images. Since the optical element 12 can perform different encoding for each wavelength component of the real image by the wavelength-dependent PSF it has, the signal processing unit 13 can perform image reconstruction processing based on compressive sensing to restore the spectral image.

[0034] [Image reconstruction processing] Based on the matrix defined by the imaging process of the optical element 12 and the image formed on the imaging element 11, that is, the image (observed image) in which the PSF of each wavelength is convolved, the signal processing unit 13 reconstructs the image. FIG. 4 is a diagram for explaining the image reconstruction processing by the signal processing unit 13.

[0035] As shown in FIG. 4, the reconstruction process is a process of solving an optimization problem (for example, Equation (A) in FIG. 4) that takes the observation matrix Φ defined by the optical system and the acquired encoded image g as inputs.

[0036] In Equation (A), f in the first term on the right side indicates the image that is originally desired to be restored. Since the number of data of the observed image is significantly less than the number of data of the image (reconstructed image) to be restored, there are infinitely many solutions that satisfy Φf - g = 0. However, by adding a regularization term as the second term, it becomes easier to obtain an image (reconstructed image ^f) that seems reasonable as the restored image.

[0037] Regarding the regularization term, various ones have been proposed for spectral images, and in this embodiment, any of the regularization terms can be applied. In the example of Equation (A), R corresponds to the prior probability of the signal based on prior information (image-likeness), and utilizes the sparsity generally possessed by an image, such as the difference between adjacent pixels being small. Note that τ is a balancing parameter. In this embodiment, a term called SSTV (Spatio-Spectral Total Variation) (Reference 1) is used as the regularization term, and it is optimized to minimize the difference between adjacent pixels in the spatial dimension and the wavelength dimension in image reconstruction. Reference 1: Aggarwal, H. K., & Majumdar, A. (2016). Hyperspectral image denoising using spatio-spectral total variation. IEEE Geoscience and Remote Sensing Letters, 13(3), 442 - 446.

[0038] Various methods for solving optimization problems have been proposed. In this embodiment, for example, a method called ADMM (Alternating Direction Method of Multipliers) (Reference 2) is used. And in recent years, a method has been proposed to simultaneously optimize the regularization term and the parameters of the optimization problem using machine learning and perform image reconstruction (see Non-Patent Document 2). The signal processing unit 13 can also apply this method. That is, the signal processing unit 13 may perform spectral image reconstruction using a model composed of a neural network and an optimized reconstruction algorithm. In other words, the signal processing unit 13 uses machine learning to pre-learn the form of the regularization term and various parameters of the optimization problem using various spectral images, and performs image reconstruction using the learned (optimized) regularization term and various parameters. Reference 2: S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learn- ing, vol. 3, no. 1, pp. 1-122, 2011.

[0039] In this way, in the imaging device 10, an observation matrix Φ effective for reconstruction can be realized by a simple and small optical system (optical element 12).

[0040] [Structural body] To realize the structural body 160, in this embodiment, by designing the cross-sectional shape of the fine columnar structural body 160 to design an arbitrary spatial phase distribution, an optical element 12 having a PSF function with a different shape for each wavelength of the incident light is realized.

[0041] FIG. 5 and FIG. 6 are diagrams showing an example of the schematic configuration of the structure 160. FIG. 5 is a side view of the structure 160 having a square shape when viewed from above. FIG. 6 is a bottom view of the structure 160 shown in FIG. 5.

[0042] The structure 160 is a columnar structure extending in the z-axis direction and is formed on the bottom surface of a transparent substrate 190 (for example, a SiO2 substrate (refractive index 1.45)). An example of the material of the structure 160 is SiN (refractive index n1 = 2.05). The sides and bottom of the structure 160 are air (refractive index n0 = 1.0).

[0043] Let the arrangement period of each structure 160 be P. It is desirable to set the arrangement period P as in Equation (1) so that diffracted light does not occur on the transmission side.

[0044]

Equation

[0045] λ min is the shortest wavelength in the wavelength band of the light-receiving target. n0 is the refractive index of the transparent layer on the transmission side. For example, when λ min is 420 nm, n0 is 1.0, and P = 400 nm.

[0046] The height h (the length in the z-axis direction) of the structure 160 is constant. Since it is preferable that the structure 160 has a light phase delay amount (phase value) of 2π or more with respect to the incident light, that is, the light traveling along the z-axis direction, when the desired central wavelength in the longest wavelength band of the separated wavelength region is λ r it is desirable to set it as in Equation (2).

[0047]

Equation

[0048] In Equation (2), n1 is the refractive index of the structure 160. When the structure 160 is SiN, n1 = 2.05, and the height h is, for example, 1250 nm. Also, the structure 160 may be formed of TiN (refractive index 2.40).

[0049] By designing (including dimension design) the cross-sectional shape of the structure 160, various combinations capable of giving different optical phase delay amounts to light of each wavelength can be realized. By diversifying the cross-sectional shape of the structure 160, the number of combinations increases, and the design freedom is further improved. In order not to cause polarization dependence, it is desirable that the cross-sectional shape of each of the plurality of structures 160 when cut along a plane parallel to the xy plane is a four-fold rotational symmetry shape.

[0050] [Phase delay amount] Next, the structural width of the structure 160 and the phase delay amount at each wavelength will be described. FIG. 7 is a diagram showing the relationship between the structural width of the structure 160 and the phase delay amount at each wavelength.

[0051] In FIG. 7, the phase values of each wavelength (420 to 660 nm) are shown when the structural parameters (widths) of the columnar structure 160 are set to various values with the height being constant. As shown in this FIG. 7, by appropriately designing the cross-sectional shape of the structure 160, phase values from 0 to 2π can be realized at all the design wavelengths.

[0052] Thereby, in the present embodiment, by simply designing the cross-sectional shape of the binary structure of the structure 160, an arbitrary spatial phase distribution can be designed according to the design wavelength, and an optical element 12 having a PSF with a different shape for each wavelength can be designed as will be described later. Here, although only the wavelengths from 420 to 660 nm have been described, the same design can be applied to any other wavelength band.

[0053] Note that the cross-sectional shape of the structure 160 is not limited to the shape shown in FIG. 6. FIG. 8 is a diagram showing an example of the cross-sectional shape of the structure 160. The structure 160 may have various cross-sectional shapes as exemplified in FIG. 8. The exemplified shape is a four-fold rotationally symmetric shape obtained by variously combining, for example, a square shape, a cross shape, and a circular shape.

[0054] [Lens Design Example 1] Next, a design example of the optical element 12, which is a wavelength-dependent PSF lens, will be described. In the present embodiment, a lens phase distribution having a PSF with a different shape for each wavelength is designed, and a columnar structure 160 is realized based on this phase distribution.

[0055] Here, a phase distribution was designed using the structure 160 having the SiN composition shown in FIGS. 5 and 6, and a wavelength-dependent PSF lens was realized. Note that a lens having a PSF with a different shape for each wavelength can be realized from various phase distributions.

[0056] As the simplest example, a case of designing the optical element 12 having a different PSF for each wavelength based on a phase distribution equivalent to a Fresnel lens will be described. In this case, the phase distribution φ of the lens is represented by, for example, Equation (3).

[0057] [Equation]

[0058] In Equation (3), (x, y) are spatial coordinates on the lens plane. λ d is the design wavelength. f is the focal length. n is the refractive index of the optical propagation space after passing through the lens. C is an arbitrary constant.

[0059] Figs. 9 to 11 are diagrams showing examples of the phase distribution of the structure 160 when designed to be equivalent to a Fresnel lens. Figs. 9 to 11 are examples of the phase distribution when designed to have a phase distribution equivalent to that of a Fresnel lens. Fig. 9 shows the case where the wavelength λ = 450 nm, Fig. 10 shows the case where the wavelength λ = 550 nm, and Fig. 11 shows the case where the wavelength λ = 650 nm. In the examples of Figs. 9 to 11, the lens design was performed with parameters of a lens diameter of 1.0 mm, a focal length of 5.0 mm, and a design wavelength of 520 nm. φ is converted so as to fall within the range of 0 to 2π. For example, -0.5π and 2.5π are converted to 1.5π and 0.5π, respectively.

[0060] To realize a wavelength-dependent PSF lens having a phase distribution equivalent to that of a Fresnel lens, the structure 160 of the structure (the structure with the minimum phase error) that best fits the phase distribution of Equation (2) may be selected and arranged for each position from the amount of phase delay at the design wavelength of the composition structure.

[0061] In the case of the phase distribution of a Fresnel lens, parallel light of the design wavelength is focused at one point at the focal length. That is, the PSF shape becomes a dot (specifically, a Gaussian function in the case of a circular lens and a sinc function in the case of a square lens).

[0062] For light of other wavelengths, due to the wavelength dependence of the focusing position caused by the phase pattern and the wavelength dispersion of the composition structure with respect to the phase, the size of the PSF changes depending on the wavelength. That is, chromatic aberration occurs in which the degree of image blurring varies depending on the wavelength.

[0063] In this embodiment, by utilizing this chromatic aberration, it is possible to perform different convolution operations for each wavelength on the imaging target, then acquire an image, and generate a spectral image by image reconstruction.

[0064] [Lens Design Example 2] Next, another design example of the optical element 12, which is a wavelength-dependent PSF lens, will be described. Here, a case where the design is made to have a phase distribution having a propeller-shaped PSF will be described as an example.

[0065] That is, an optical element 12 having a phase distribution such that the shape of the PSF rotates according to the wavelength is designed. In this case, the phase distribution φ of the lens is represented by, for example, Equation (4). Note that ω(θ) in Equation (4) is represented by Equation (5).

[0066]

Number

[0067]

Number

[0068] In Equations (4) and (5), r is the distance from the origin on the lens plane. θ is the angle formed by the origin and the coordinates on the lens plane. c is the speed of light in a vacuum. ω(θ) is the optical angular frequency at the position of θ. ω min is the minimum optical angular frequency in design. ω max is the maximum optical angular frequency in design. f is the focal length. n is the refractive index of the optical propagation space after passing through the lens. C is an arbitrary constant. N is the number of blades.

[0069] Figures 12 to 14 are other examples of the phase distribution of the structure 160. Figures 12 to 14 are diagrams showing examples of the phase distribution of the structure 160 when the PSF is designed to have a propeller shape. Figure 12 shows the case where the wavelength λ = 450 nm, Figure 13 shows the case where the wavelength λ = 550 nm, and Figure 14 shows the case where the wavelength λ = 650 nm. In the examples of Figures 12 to 14, the lens diameter was 1.0 mm, the focal length was 5.0 mm, the number of wings was 3, and the lens was designed with parameters of a design wavelength of 420 to 660 nm. φ is converted so as to fall within the range of 0 to 2π. For example, -0.5π and 2.5π are converted to 1.5π and 0.5π, respectively.

[0070] In order to realize a wavelength-dependent PSF lens that has a phase distribution with a propeller-shaped PSF, a structure 160 of a structure (a structure with the minimum phase error) that best fits the phase distribution of the equation may be selected and arranged for each position from the amount of phase delay at each wavelength (each angular frequency) of the composition structure.

[0071] In this case, as will be described later, the PSF shape becomes a propeller-like shape, and the number of its blades corresponds to N in Equation (5). This PSF shape rotates according to the wavelength, and its size hardly changes.

[0072] This is due to the wavelength dispersion of the phase of the phase pattern and the composition structure in which the focal length of the lens depends on the wavelength and the rotation angle θ. Only the light with the designed angular frequency ω(θ) (designed wavelength) at an arbitrary rotation angle θ is focused on the designed focal length and focal position, and the focal length of the other light changes back and forth. Since the designed angular frequency ω(θ) changes linearly according to the rotation angle θ, a PSF like a propeller shape is generated, and the PSF rotates depending on the angular frequency (wavelength).

[0073] Therefore, the optical element 12 can perform different convolution operations for each wavelength with respect to the imaging target, and after image acquisition, the imaging device 10 can generate a spectral image by image reconstruction.

[0074] Note that, as will be described later, compared with the Fresnel lens type, the propeller lens type in which the size of the PSF is almost constant and the wavelength dependence occurs in a clear form of rotation is more advantageous and more suitable in reconstruction.

[0075] Also, in Equations (4) and (5), although the designed angular frequency ω changes according to the lens position, the same effect can be obtained by replacing the angular frequency with the wavelength.

[0076] Hereinafter, an example of the optical element 12 designed based on the propeller lens type will be shown, but the same applies to other wavelength-dependent PSF lenses such as the Fresnel lens type.

[0077] [PSF Shape Example] An example of the PSF shape of the wavelength-dependent PFS lens in the embodiment is shown. FIGS. 15 to 19 are diagrams showing the PSF shapes obtained by Fresnel diffraction integration from the phase distributions shown in FIGS. 12 to 14. FIG. 15 shows the PSF shape at a wavelength λ = 450 nm. FIG. 16 shows the PSF shape at a wavelength λ = 500 nm. FIG. 17 shows the PSF shape at a wavelength λ = 550 nm. FIG. 18 shows the PSF shape at a wavelength λ = 600 nm. FIG. 19 shows the PSF shape at a wavelength λ = 650 nm.

[0078] As shown in FIGS. 15 to 19, according to the wavelength-dependent PFS lens in the embodiment, the PSF has a three-blade propeller shape and can be seen to rotate according to the wavelength. As shown in FIGS. 15 to 19, regardless of the change in wavelength, the size of the PSF itself hardly changes.

[0079] When the imaging target 1 is imaged with the optical element 12 having these PSFs, the result of convolving the image with the PSF of the corresponding wavelength is imaged on the image sensor.

[0080] [Observed Image Example] Subsequently, the simulation results of imaging a natural image with the optical element 12 having the PSFs of FIGS. 15 to 19 will be described. FIG. 20 is a diagram showing the simulation results.

[0081] The simulation was performed by convolving the publicly available spectral image (ICVL, Boaz Arad and Ohad Ben-Shahar. Sparse recovery of hyperspectral signal from natural rgb images, In European Conference on Computer Vision, pp. 19-34. Springer, 2016., [online], [searched on December 28, 2020], Internet <URL:http: / / icvl.cs.bgu.ac.il / hyperspectral / >) (wavelength 420nm - 660nm: 25 bands) with the PSFs in FIGS. 15 - 19 for each wavelength, and then integrating along the wavelength dimension considering the sensitivities of the RGB pixels of a general color image sensor. Note that FIG. 20 shows a monochrome display of an RGB color image, where the left image is the input spectral image (real image), and the right image is the image acquired on the imaging device 11 (observed image).

[0082] This corresponds to imaging using a lens with the PSFs in FIGS. 15 - 19 and a color image sensor, and simulating the RGB color image (observed image) output from the sensor. Note that the wavelength range of B light is 420 - 500nm, the wavelength range of G light is 500 - 600nm, and the wavelength range of R light is 600 - 660nm.

[0083] As shown in FIG. 20, it can be seen that the observed image is blurred by the convolution operation with the PSF of the optical element 12. Note that the observed information amount is compressed to 12% (from 25 wavelength bands to 3 colors) of the real image, and the information is restored from 3 colors to 25 wavelength bands by image reconstruction based on compressive sensing.

[0084] [Reconstructed Image] Next, an example of the reconstructed image by the imaging device 10 will be described. FIG. 21 is a diagram showing an example of the reconstructed image by the imaging device 10. In FIG. 21, an example of generating a spectral image from the observed image shown in FIG. 20 using reconstruction processing based on compressive sensing is shown.

[0085] Here, SSTV was used as the regularization term, and ADMM was used as the method for solving the optimization problem. In addition, for comparison, a real image is also shown in FIG. 21. Note that the reconstructed image and the real image shown in FIG. 21 are 25-band spectral images, but the RGB images shown for visualization are displayed in monochrome.

[0086] The reconstructed image was evaluated using evaluation indices of PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and SAM (Spectral Angle Mapping).

[0087] As shown in Equations (6) and (7), PSNR is an index for evaluating the difference per pixel, and the larger the value (dB), the higher the image quality. The PSNR of the image for each wavelength was calculated, and the average was taken over the entire wavelength to apply it to the spectral image.

[0088]

Equation

[0089]

Equation

[0090] SSIM is the structural similarity, and as shown in Equation (8), it is an index that includes the correlation with surrounding pixels. The closer SSIM is to 1, the higher the image quality. The SSIM of the image for each wavelength was calculated, and the average was taken over the entire wavelength to apply it to the spectral image.

[0091]

Equation

[0092] SAM is the wavelength spectrum similarity, and the closer it is to 0, the more similar the spectra are. The SAM of each pixel was calculated, and the average was taken over the entire image to apply it to the spectral image.

[0093] The reconstructed image had a PSNR of 29.09 dB, an SSIM of 0.9142, and a SAM of 0.1872. Therefore, it can be seen that the imaging device 10 accurately reconstructs the image.

[0094] [Reconstructed Wavelength Spectrum] Next, an example of the reconstructed wavelength spectrum will be described. FIG. 22 is a diagram showing the wavelength spectrum at the × point of the reconstructed image in FIG. 21. For comparison, FIG. 22 also shows the wavelength spectrum at the × point of the real image (Ground truth) together with the reconstructed image (Reconstructed).

[0095] As shown in FIG. 22, in the reconstructed image, a spectrum that closely matches the real image is obtained, indicating that high-precision information restoration can be achieved by image reconstruction. Note that the reconstruction accuracy varies depending on the shape of the PSF of the optical element 12, as well as the regularization term and the method of solving the optimization problem.

[0096] [Comparison of Reconstruction Accuracy According to PSF Shape] Next, the results of comparing the reconstruction accuracy according to the shape of the PSF in the optical element 12 are shown. FIG. 23 is a diagram showing the results of comparing the configuration accuracy for each shape of the PSF of the optical element 12. FIG. 24 is a diagram showing the reconstructed images respectively reconstructed based on the observed images of each shape of the PSF in FIG. 23. The reconstructed images, real images, and Fresnel lens images shown in FIG. 24 are the monochrome displays of the RGB images.

[0097] In FIG. 23, for comparison, the case of using a Fresnel lens type PSF is also shown. Further, in FIG. 24, for comparison, the real image and the reconstructed images by the Fresnel lens type are also shown. For the image by the Fresnel lens type, the reconstruction is performed using the large chromatic aberration described above.

[0098] In FIGS. 23 and 24, PSNR, SSIM, and SAM were used as evaluation indices. N in FIGS. 23 and 24 is the number of vanes. In FIG. 23, the PSF shape of the optical element 12, which is a wavelength-dependent PSF, is also shown for the case where the number of vanes N is 1 to 4. The lens parameters are a lens diameter of 1.0 mm, a focal length of 5.0 mm, and a design wavelength band of 420 to 660 nm. The sensor parameters are a pixel size of 24 μm, an image size of 128×128×25, and a patch size of 32×32×3. The ADMM parameters are a regularization term weight of λ = 0.015, a penalty term weight of ρ = 0.1, and an Iteration number of 64.

[0099] As shown in FIGS. 23 and 24, none of the evaluation indices showed a significant difference depending on the number of vanes, and showed higher accuracy than the Fresnel lens PSF. In other words, the optical element 12 showed higher accuracy than the Fresnel lens PSF regardless of the number of vanes. Therefore, it can be said that the optical element 12 according to the present embodiment is more suitable than the Fresnel lens type and constitutes an observation matrix advantageous for reconstruction.

[0100] [Effects of the Embodiment] As described above, in the imaging device 10 according to the embodiment, an optically encoded image is formed on the imaging element 11 by one optical element 12. In other words, in the imaging device 10, effective encoding can be performed in spectral image reconstruction by one optical element 12. Therefore, the components of the imaging device 10 only need to be one optical element 12, the imaging element 11, and the signal processing unit 13, and a hyperspectral imaging device can be realized with a simple device.

[0101] Also, in the imaging device 10, the distance between the optical element 12 and the imaging element 11 is determined by the focal length of the lens as in a normal imaging device. Therefore, the size of the imaging device 10 is equivalent to that of a normal camera having the same field of view F number, and the device can be miniaturized compared to the conventional one.

[0102] In addition, when the observation process is known (here, the PSF of the optical element 12 and the wavelength sensitivity characteristics of the sensor), the optically encoded image can be used in the signal processing unit 13 to accurately restore the information of the real image by performing appropriate signal processing.

[0103] In addition, in the imaging device 10, since the imaging itself (acquisition of the encoded image) can be a single shot, there is no sacrifice in the time dimension, and excluding the reconstruction processing time, a time resolution equivalent to that of a normal camera is possible.

[0104] In addition, in the imaging device 10, since the optical element 12 responsible for encoding is composed of a fine binary structure, the manufacturing man-hours can be reduced compared to general diffractive optical elements that require multi-step lithography. It is also thin in thickness, light in weight, and easy to manufacture.

[0105] In addition, the optical element 12 composed of a fine binary structure is not subject to a decrease in light transmittance and a limitation of the maximum lens aperture number (NA) due to the shadow effect (a phenomenon in which diffracted light by a diffractive optical element is reflected and scattered by its multi-step structure) that occurs in general diffractive optical elements. Therefore, a lens with a higher NA (a brighter lens with high light utilization efficiency) can be realized.

[0106] In the present embodiment, although an example based on the case where the imaging element 11 is a color image sensor is shown, the same applies to the case of a monochrome image sensor, except that 3ch (RGB: color) becomes 1cb (monochrome) when acquiring the encoded image.

[0107] [Lens Structure Example] The optical element 12 is not limited to the configurations shown in FIGS. 2 and 3, and can take various forms in terms of the number and interval of the structures 160, the structural shape, and the arrangement pattern. In addition, the structures 160 may be connected to each other, or may be embedded in a transparent material.

[0108] In FIGS. 2 and 3, the optical element 12 is formed on the bottom surface of the transparent substrate 190, but is not limited thereto. FIGS. 25 to 30 are diagrams schematically showing other examples of a part of the cross section of the optical element 12 according to the embodiment.

[0109] As shown in FIG. 25, the structure 160 of the optical element 12 may be formed on the upper surface of the transparent substrate 190A. In this case, the structure 160 is supported by the transparent substrate 190A. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the transparent layer may be single or a plurality of materials in a layered form.

[0110] Also, as shown in FIG. 26, the structure 160 of the optical element 12 may be embedded in the transparent substrate 190B. The material of the transparent substrate 190B may be single or a plurality of materials in a layered form.

[0111] Also, as shown in FIG. 27, the structure 160 of the optical element 12 may be formed on both surfaces of the transparent substrate 190C. The above-described wavelength-dependent PSF function may be realized by the structures 160 on both surfaces of the transparent substrate 190C. Also, the wavelength-dependent PSF function may be realized by the structure 160 of the transparent substrate 190C, and other optical functions such as a filter, a splitter, and a light-shielding layer may be realized on the other surface. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the transparent layer may be single or a plurality of materials in a layered form.

[0112] Also, as shown in FIG. 28, the structure 160 of the optical element 12 may be formed on the refractive lens 190D. The structure 160 is supported on the refractive lens 190D. The refractive lens 190D is beneficial in improving the light-collecting performance (such as increasing the NA) of the wavelength-dependent PSF function. The same applies to the refractive lenses 190E and 190F described later. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the refractive lens 190D may be single or a plurality of materials in a layered form.

[0113] Also, as shown in FIG. 29, the structure 160 of the optical element 12 may be embedded in the refractive lens 190E. The material of the refractive lens 190E may be single or a plurality of materials in a layered form.

[0114] Also, as shown in FIG. 30, the structure 160 of the optical element 12 may be formed on both surfaces of the refractive lens 190F. The wavelength-dependent PSF function described above may be realized by the structures 160 on both surfaces of the refractive lens 190F. Also, the wavelength-dependent PSF function may be realized by the structure 160 of the refractive lens 190F, and other optical functions such as a filter, a splitter, and a light-shielding layer may be realized on the other surface. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the refractive lens 190F may be single or a plurality of materials in a layered form.

[0115] Also, although omitted in FIGS. 25 to 30, a light-shielding film pattern or the like may be provided on the same plane or the back surface.

[0116] In the embodiment, TiO2 and SiN are exemplified as the material of the structure 160. However, the material of the structure 160 is not limited thereto. For example, for light with a wavelength of 380 nm to 1000 nm (visible light to near-infrared light), in addition to SiN, SiC, TiO2, GaN, etc. may be used as the material of the structure 6. It is suitable because it has a high refractive index and low absorption loss. When used for light with a wavelength of 800 to 1000 nm (near-infrared light), Si, SiC, SiN, TiO2, GaAs, GaN, etc. may be used as the material of the structure 6. It is suitable because it has low loss. For light in the near-infrared region of the long wavelength band (1.3 μm, 1.55 μm, etc., which are communication wavelengths), in addition to the above materials, InP, etc. can be used as the material of the structure 160.

[0117] Also, when the structure 160 is formed by pasting, coating, etc., examples of the material include polymers such as polyimide such as fluorinated polyimide, BCB (benzocyclobutene), photocurable resin, UV epoxy resin, acrylic resin such as PMMA, and resists in general.

[0118] In addition, in the embodiment, an example assuming SiO2 and an air layer as the material of the transparent layer 150 was shown, but the present invention is not limited thereto. Any material having a refractive index lower than that of the material of the structure 160 and having low loss with respect to the wavelength of incident light may be used, including general glass materials and the like. Since the transparent layer 150 only needs to have sufficiently low loss with respect to the wavelength of light that should reach the corresponding pixel, it may be made of the same material as a color filter, for example, an organic material such as resin.

[0119] In addition, in the embodiment, the three primary colors of RGB were taken as examples to describe the corresponding colors of the pixels, but the pixels may also correspond to near-infrared light and light of wavelengths other than the three primary colors (for example, infrared light, ultraviolet light, etc.).

[0120] In addition, in the embodiment, an example in which a structure having a square cross-sectional shape is used as the shape of the structure 160 was described. This shape is just an example, and one type of structure (for example, only a square shape) may be used, or two or more types of structures (for example, only a square shape and a cross shape) may be used.

[0121] As described above, the present invention has been described based on specific embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof.

Explanation of Reference Numerals

[0122] 1 Imaging object 10 Imaging device 11 Imaging element 12 Optical element 13 Signal processing unit 130 Pixel 150 Transparent layer 160 Structure 180 Wiring layer 190, 190A to 190C Transparent substrate 190D to 190F Refractive lens

Claims

1. An optical element having a transparent substrate and a plurality of structures disposed in a plane direction of the transparent substrate on or within the transparent substrate, An imaging device in which a plurality of pixels including a photoelectric conversion element are arranged, A signal processing unit that generates an image signal based on an electrical signal obtained from the imaging device, An imaging device comprising: The optical element has an imaging function, The optical element outputs light in a state having different point spread functions for each wavelength, and forms an image in which the point spread functions of each wavelength are convolved on the imaging device, The plurality of structures have the same height when viewed from the side, The signal processing unit reconstructs an image in which the point spread function of each wavelength is convolved, Each of the plurality of structures is a columnar structure having a refractive index higher than that of the transparent substrate and providing a light phase delay amount corresponding to the cross-sectional shape with respect to the incident light, The plurality of structures are arranged according to a light phase amount delay distribution for realizing imaging of an image in which the point spread function of each wavelength with respect to the pixel is convolved, and are arranged according to a light phase amount delay distribution for realizing imaging of an image in which the point spread function of each wavelength with respect to the pixel is convolved, The cross-sectional shape and arrangement of each of the plurality of structures are designed to form an image at each position on the pixel corresponding to each wavelength, The cross-sectional shape of each of the plurality of structures is any one of a plurality of types of four-fold rotationally symmetric shapes, The plurality of structures are arranged at an arrangement period represented by formula (1), The height h when the plurality of structures are viewed from the side is set as in formula (2), The imaging device outputs a multi-band image having more bands than the number of bands of the image obtained by the imaging device, An imaging device characterized by the above. 【Number 1】 【Number 2】

2. The imaging device according to claim 1, wherein the signal processing unit reconstructs an image based on a matrix defined by an imaging process of the optical element and an image in which the point spread function of each wavelength is convolved.

3. The imaging device according to claim 2, wherein the signal processing unit solves an optimization problem using a model composed of a neural network and taking as inputs a matrix defined by an imaging process of the optical element and an image in which the point spread function of each wavelength is convolved.

4. An optical element having a transparent substrate and a plurality of structures disposed in the plane direction of the transparent substrate on or within the transparent substrate, wherein the optical element has an imaging function, the optical element outputs light in a state having different point spread functions for each wavelength, and an image in which the point spread functions of each wavelength are convolved is formed on an image sensor on which a plurality of pixels including a photoelectric conversion element are disposed, when viewed from the side, the plurality of structures have the same height, each of the plurality of structures is a columnar structure having a refractive index higher than that of the transparent substrate and giving a light phase delay amount corresponding to the cross-sectional shape to the incident light, the cross-sectional shapes of the plurality of structures are set according to a light phase amount delay distribution for realizing the formation of an image in which the point spread functions of each wavelength for the pixel are convolved, and are arranged according to a light phase amount delay distribution for realizing the formation of an image in which the point spread functions of each wavelength for the pixel are convolved, the cross-sectional shape and arrangement of each of the plurality of structures are designed to form an image at each position on the pixel corresponding to each wavelength, the cross-sectional shape of each of the plurality of structures is any one of a plurality of types of four-fold rotationally symmetric shapes, the plurality of structures are arranged at an arrangement period represented by formula (3), the height h when the plurality of structures are viewed from the side is set as in formula (4) characterized optical element. 【Number 3】 【Number 4】

Images