Imaging device and optical element

Abstract

An imaging device (10) includes an optical element (12) including a transparent substrate and multiple structures arranged, on the transparent substrate or in the transparent substrate, in a surface direction of the transparent substrate; an imaging device (11) in which multiple pixels each including a photoelectric conversion device are arranged; and a signal processor (13) that generates an image signal on the basis of an electric signal obtained from the imaging device (11). The optical element (12) outputs light in a state having different point spread functions for respective wavelengths and thereby forms, according to polarization components, images convolved with the point spread functions of the wavelengths on the pixels corresponding to the polarization components. The multiple structures have the same height in a side view, and the signal processor (13) reconstructs the images convolved with the point spread functions of the wavelengths for the respective polarization components.

Description

Technical Field

[0001] The present invention relates to an imaging device and an optical element.

Background Art

[0002] Generally, an imaging device acquires a two-dimensional image of three colors, R (red), G (green), and B (blue), as optical information that can be obtained. On the other hand, in recent years, hyperspectral cameras that acquire more detailed color information (wavelength spectrum) have been put into practical use, and efforts have been made to extract new valuable information from a wider variety of optical information.

[0003] In addition, a polarization image sensor that acquires polarization information, which is optical information as important as wavelength, has also been put into practical use, and techniques for extracting newly valuable information from a wider variety of optical information have been proposed. Therefore, in recent years, the realization of a hyperspectral imaging device capable of acquiring polarization information has been desired.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Generally, commercially available hyperspectral cameras are line-scan type, and acquire spectral images by imaging multiple times using a line-scan mechanism and a spectroscopic element. Further, in addition to this, in order to simultaneously acquire polarization information, a method of combining an existing polarization image sensor with an existing hyperspectral camera is theoretically possible. However, when combining an existing polarization image sensor with an existing hyperspectral camera, there is a problem that the device becomes more complicated.

[0006] The present invention has been made in view of the above, and an object thereof is to provide a hyperspectral imaging device that has a simple device configuration and can acquire polarization information, and an optical element for realizing the hyperspectral imaging device that has a simple device configuration and can acquire polarization information.

Means for Solving the Problems

[0007] In order to solve the above-described problems and achieve the object, an imaging device according to the present invention includes an optical element having a transparent substrate and a plurality of structures arranged in the plane direction of the transparent substrate on or in the transparent substrate, an imaging element in which a plurality of pixels including a photoelectric conversion element are arranged, and a signal processing unit that generates an image signal based on an electrical signal obtained from the imaging element. The optical element outputs light in a state having a different point spread function for each wavelength, so that an image in which the condensing intensity distributions of each wavelength are convolved is imaged on a plurality of pixels corresponding to each polarization component according to the polarization component. The plurality of structures have the same height when viewed from the side, and the signal processing unit is characterized in that, for each polarization component, an image in which the point spread function of each wavelength is convolved is reconstructed.

[0008] Further, an optical element according to the present invention is an optical element having a transparent substrate and a plurality of structures arranged in the plane direction of the transparent substrate on or in the transparent substrate. The optical element outputs light in a state having a different condensing intensity distribution for each wavelength, so that an image in which the point spread function of each wavelength is convolved is imaged on a plurality of pixels corresponding to each polarization component according to the polarization component. The plurality of structures have the same height when viewed from the side.

Advantages of the Invention

[0009] According to the present invention, it is possible to realize a hyperspectral imaging device that has a simple device configuration and can acquire polarization information.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

[0011] Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings. In the following description, each drawing only schematically shows the shape, size, and positional relationship to the extent that the content of the present invention can be understood. Therefore, the present invention is not limited only to the shape, size, and positional relationship illustrated in each drawing. Also, in the description of the drawings, the same parts are denoted by the same reference numerals. Hereinafter, when it is described as “^A” with respect to A, it is considered to be equivalent to “a symbol in which ‘^’ is written immediately above ‘A’”.

[0012] [Embodiment] [Imaging Device] First, the imaging device according to the embodiment of the present invention will be described. FIG. 1 is a side view showing the schematic configuration of the imaging device according to the embodiment. FIG. 2 is a diagram for explaining the configuration of the optical element 12 shown in FIG. 1. FIG. 3 is a schematic diagram showing the processing until the imaging device 10 shown in FIG. 1 acquires an image.

[0013] As shown in FIG. 1, the imaging device 10 according to the embodiment includes an optical element 12, an imaging element 11, and a signal processing unit 13. The imaging element 11 has a photoelectric conversion element such as a CCD or a CMOS. The signal processing unit 13 processes the photoelectric conversion signal output from the imaging element 11 to generate an image signal.

[0014] As shown in FIGS. 1 and 2, in the imaging device 10, light such as natural light or illumination light is irradiated onto the imaging object (real image), and the light transmitted / reflected / scattered by the imaging object 1, or the light emitted from the imaging object 1 forms an optical image on the imaging element 11 by the optical element 12.

[0015] The optical element 12 has a function in which the imaging position varies according to the polarization information and the imaging characteristics vary according to the wavelength. The optical element 12 is composed of a fine binary structure. The optical element 12 has a plurality of fine columnar structures 160 that are arranged at a period equal to or less than the wavelength of the incident light and have a constant height when viewed from the side.

[0016] The cross-sections of the plurality of fine columnar structures 160 all have a shape that is rotationally symmetric about two axes, and this shape can realize polarization dependence. For example, the optical element 12 forms a pair with a first lens pattern region 12-1 that separates linearly polarized light of 0° (horizontal) and 90° (vertical), and a second lens pattern region 12-2 (see FIG. 2) that separates linearly polarized light of +45° (diagonal) and -45° (diagonal), and simultaneously separates the polarization components in four directions. The optical element 12 has different imaging (condensing) positions according to the polarization direction.

[0017] Furthermore, the optical element 12 performs optical encoding by having a function with imaging characteristics that vary according to wavelength. For this reason, the optical element 12 is a lens (wavelength-dependent PSF lens) that has a PSF (Point Spread Function) with clearly different shapes according to wavelength, and has a function of generating an image obtained by performing a different convolution operation for each wavelength on a real image (subject). The optical element 12 is a wavelength-dependent PSF lens and has a function of imaging an image (acquired observed image (encoded image)) in which the PSF of each wavelength is convolved, onto a plurality of pixels corresponding to each polarization direction in the imaging element 11 according to the polarization direction. That is, when an object is imaged with this optical element 12, a convolution operation is performed on the real image with a different PSF for each wavelength according to the polarization change direction, and the result is imaged on each region corresponding to each polarization direction in the imaging element 11.

[0018] The light from the imaging target 1 is imaged at different positions in a state where the polarization components are separated by the optical element 12, and the imaging characteristics (blurring degree) vary according to wavelength. The optical element 12 performs a different convolution operation for each wavelength while separating the polarization components.

[0019] The imaging element 11 acquires observed images subjected to different convolution operations for each wavelength for each polarization direction by the optical element 12, which is a polarization separation and wavelength-dependent PSF lens. For example, as shown in FIG. 3, images G1 corresponding to the 90° polarization component, G2 corresponding to the 0° polarization component, G3 corresponding to the -45° polarization component, and G4 corresponding to the 45° polarization component, and in each of these images, images G1 to G4 (see FIG. 3) in which the PSF of each wavelength is convolved are imaged on the imaging element 11.

[0020] Based on compressive sensing, the signal processing unit 13 generates a reconstructed image in which spectral information is restored by a reconstruction process of reconstructing an image in which the PSF of each wavelength is convolved for each polarization component. For example, the signal processing unit 13 generates a reconstructed image G1' corresponding to the 90° polarization component, a reconstructed image G2' corresponding to the 0° polarization component, a reconstructed image G3' corresponding to the -45° polarization component, and a reconstructed image G4' corresponding to the 45° polarization component by the reconstruction process.

[0021] Note that the imaging device 10 may include known components such as an infrared cut optical filter, an electronic shutter, a viewfinder, a power source (battery), a flashlight, etc., but the descriptions thereof are omitted because they are not particularly necessary for understanding the present invention. Also, the above configuration is merely an example, and in the embodiment, known elements can be appropriately combined and used as components other than the optical element 12, the imaging element 11, and the signal processing unit 13.

[0022] [Lens and Imaging Element] Subsequently, an outline of the optical element 12 and the imaging element 11 in the embodiment will be described. FIG. 4 is a diagram schematically showing a part of the cross section of the imaging element 11 and the optical element 12 according to the embodiment.

[0023] In FIG. 4, a part of the imaging element 11 and the optical element 12 is described as an imaging unit 100. The imaging unit 100 in FIG. 4 is a cross-sectional view when the optical element 12 shown in FIG. 2 is applied and cut along the line A-A' shown in FIG. 2. Also, from FIG. 4 onwards, an xyz coordinate system is shown. The xy plane direction corresponds to the plane direction of the imaging element 11, the transparent substrate 190 described later, etc. Hereinafter, unless otherwise specified, "planar view" refers to viewing in the z-axis direction (for example, in the negative z-axis direction). "Side view" refers to viewing in the x-axis direction or the y-axis direction (for example, in the negative y-axis direction). In the imaging unit 100, a first lens pattern region 12-1 that separates linearly polarized light of 0° (horizontal) and 90° (vertical) and a second lens pattern region 12-2 that separates linearly polarized light of +45° (diagonal) and -45° (diagonal) form a pair.

[0024] As shown in FIG. 4, in the imaging unit 100, the optical element 12 and the imaging element 11 are arranged to face each other. The imaging element 11 and the optical element 12 are provided in this order in the positive z-axis direction.

[0025] The imaging element 11 has a plurality of pixels 130 each including a photoelectric conversion element arranged in a two-dimensional array. An example of the photoelectric conversion element is a photo diode (PD: Photo Diode). Each pixel corresponds to red (R), green (G), and blue (B). An example of the wavelength band of red light is 600 nm < λ0 ≦ 800 nm, where λ0 is the wavelength. An example of the wavelength band of green light is 500 nm < λ0 ≦ 600 nm. An example of the wavelength band of blue light is λ0 < 500 nm. The pixels R, G, and B may be in a Bayer array. Alternatively, the pixels may be for a monochrome image.

[0026] The incident light travels along the negative z-axis direction and reaches the imaging element 11 through the optical element 12. The charge generated in each pixel 130 of the imaging element 11 is converted into an electrical signal that forms the basis of the pixel signal by a transistor (not shown) or the like, and is output to the outside of the imaging unit 100 through the wiring layer.

[0027] The optical element 12 is arranged on the side where light from the imaging target is incident. When viewed in plan, the optical element 12 is provided so as to cover the imaging element 11. The optical element 12 is composed of a plurality of structures 160, for example, periodically (having a periodic structure), on the bottom surface of the transparent substrate 190. The plurality of structures 160 may be arranged at equal intervals or at unequal intervals for ease of design. The plurality of structures 160 are formed in the transparent layer 150 formed on the imaging element 11 to cover a plurality of pixels.

[0028] The transparent substrate 190 is a low-refractive-index transparent substrate made of a material such as SiO2 (refractive index n = 1.45), for example. The transparent layer 150 is a low-refractive-index transparent layer made of a material such as air or SiO2. The materials of the transparent substrate 190 and the transparent layer 150 may be single or may be a plurality of materials layered. The plurality of structures 160 have the same height when viewed from the side. The plurality of structures 160 are composed of a fine structure pattern formed from a material such as SiN or TiO2 having a refractive index higher than that of the transparent layer 150.

[0029] An example of the optical element 12 is a metasurface. The metasurface includes a plurality of fine structures (corresponding to the structures 160) having a width equal to or less than the wavelength of light when viewed from above and having the same height when viewed from the side. The cross-sectional shape of each of the plurality of structures 160 when cut along a plane parallel to the xy plane is a two-fold rotational symmetry shape, and this shape can realize polarization dependence. Note that the metasurface may have a two-dimensional structure or a three-dimensional structure. The optical element 12 can control the phase and the light intensity according to the characteristics of light (wavelength, polarization, incident angle) only by changing the parameters of the structure 160. In the case of a three-dimensional structure, the degree of freedom in design is improved compared to a two-dimensional structure.

[0030] The imaging (condensing) position of the optical element 12 is different according to the polarization direction, and it has a different PSF according to the wavelength at each condensing point. Thereby, the light from the imaging object 1 is imaged at different positions on the imaging element 11 in a state where the polarization components are separated by the optical element 12 having a polarization separation and wavelength-dependent PSF function, and an image (RGB image or monochrome image) having different imaging characteristics (blurring degree) according to the wavelength is acquired.

[0031] For example, as also shown in FIG. 2, the optical element 12 is formed as a set including a first lens pattern region 12-1 that separates linearly polarized light of 0° and 90° and a second lens pattern region 12-2 that separates linearly polarized light of +45° and -45°. Images corresponding to four polarization components of 0°, 90°, +45°, and -45° are formed, and in the imaging device 11, regions where images corresponding to the respective polarization components are formed are set. FIG. 4 shows an example in which an image corresponding to the 0° polarization component is formed in the region 11-1 of the imaging device 11 and an image corresponding to the +45° polarization component is formed in the region 11-2.

[0032] Each acquired image corresponds to the result of optically convolving the imaging object (real image) 1 by the polarization separation and wavelength-dependent PSF of the optical element 12 for each wavelength and integrating along the wavelength dimension on the pixel. The imaging unit 100 acquires an optically encoded and compressed image. When the imaging device 11 is a color image sensor, after the convolution operation, multiplication according to the wavelength sensitivity of each R, G, and B pixel of the imaging device 11 is performed, and then integration is performed along the wavelength dimension on the pixel.

[0033] As described above, in the imaging device 10, with only the optical element 12, an optically encoded image is formed on the imaging device 11 for each polarization component. In other words, in the imaging device 10, the optical element 12 can perform effective encoding in spectral image reconstruction while separating polarization. Therefore, since the imaging device 10 only requires the optical element 12, the imaging device 11, and the signal processing unit 13, a hyperspectral imaging device with a simple device configuration and capable of acquiring polarization information can be realized.

[0034] In addition, in the imaging device 10, the distance between the optical element 12 and the imaging device 11 is determined by the focal length of the lens as in a normal imaging device, so the size of the imaging device 10 is equivalent to that of a normal camera having the same field of view F number.

[0035] When the optical encoding process is known (here, the PSF of the optical element 12 and the wavelength sensitivity characteristics of the sensor), the optically encoded image can be processed by the signal processing unit 13 to restore the information of the real image by performing appropriate signal processing.

[0036] The imaging device 10 performs signal processing using compressive sensing, which is a method of reconstructing (restoring) an object with high accuracy from a small amount of information by utilizing the sparsity of natural images. Since the imaging device 10 can perform different encoding on each wavelength component of the real image using the wavelength-dependent PSF of the optical element 12, the signal processing unit 13 can restore the spectral image by performing image reconstruction processing based on compressive sensing.

[0037] By applying this process to the observed images of each polarization component separated for each polarization component, a spectral image (polarization spectral image) (for example, images G1' to G4' in FIG. 3) composed of each polarization component can be generated.

[0038] In the case of this example, three of the four Stokes parameters that describe the polarization state can be derived for each wavelength from the information of the four linearly polarized components. Depending on the lens pattern and its combination, it is also possible to perform polarization separation for four or six bases capable of deriving all Stokes parameters and obtain the full Stokes parameters for each wavelength.

[0039] [Image Reconstruction Processing] The signal processing unit 13 reconstructs the image for each polarization component based on the matrix defined by the imaging process of the optical element 12 and the image formed on the imaging element 11, that is, the image (encoded image) in which the PSF of each wavelength is convolved. FIG. 5 is a diagram for explaining the image reconstruction processing by the signal processing unit 13.

[0040] As shown in FIG. 5, the reconstruction process is a process of solving an optimization problem (for example, equation (A) in FIG. 5) that takes the observation matrix Φ defined by the optical system and the acquired encoded image g as inputs.

[0041] In formula (A), f in the first term on the right side represents the image to be originally restored. Since the number of data points of the observed image is significantly less than the number of data points of the image to be restored (reconstructed image), there are infinitely many solutions that satisfy Φf - g = 0. However, by adding a regularization term as the second term, it becomes easier to obtain a plausible image (reconstructed image ^f) as the restored image.

[0042] Regarding the regularization term, various ones have been proposed for spectral images, and in this embodiment, any of the regularization terms can be applied. In the example of formula (A), R corresponds to the prior probability of the signal based on prior information (image-likeness), and utilizes the sparsity generally possessed by images, such as the difference between adjacent pixels being small. Note that τ is a balancing parameter. In this embodiment, a term called SSTV (Spatio-Spectral Total Variation) (Reference 1) is used as the regularization term, and it is optimized to minimize the differences between adjacent pixels in the spatial dimension and the wavelength dimension in image reconstruction. Reference 1: Aggarwal, H. K., & Majumdar, A. (2016). Hyperspectral image denoising using spatio-spectral total variation. IEEE Geoscience and Remote Sensing Letters, 13(3), 442 - 446.

[0043] Various methods for solving optimization problems have been proposed. In this embodiment, for example, a method called ADMM (Alternating Direction Method of Multipliers) (Reference 2) is used. In recent years, a method has been proposed in which regularization terms and parameters of an optimization problem are simultaneously optimized using machine learning or the like to perform image reconstruction (see Non-Patent Document 2). The signal processing unit 13 can also apply this method. That is, the signal processing unit 13 may reconstruct the spectral image using a model composed of a neural network and an optimized reconstruction algorithm. In other words, the signal processing unit 13 uses machine learning to pre-learn the form of the regularization term and various parameters of the optimization problem using various spectral images, and performs image reconstruction using the learned (optimized) regularization term and various parameters. Reference 2: S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1-122, 2011.

[0044] In this way, in the imaging device 10, an observation matrix Φ effective for reconstruction can be realized by a simple and small optical system (optical element 12).

[0045] [An example of a structure] To realize the structure 160, in this embodiment, by designing the cross-sectional shape of the fine columnar structure 160 to design an arbitrary spatial phase distribution, an optical element 12 which is a polarization separation / wavelength-dependent PSF lens in which the imaging position is different according to polarization information and the imaging characteristics are different according to wavelength is realized.

[0046] FIG. 6 and FIG. 7 are diagrams showing an example of the schematic configuration of the structure 160. FIG. 6 is a side view of the structure 160 having a square shape when viewed in plan. FIG. 7 is a bottom view of the structure 160 shown in FIG. 6.

[0047] The structure 160 is a columnar structure extending in the z-axis direction and is formed on the bottom surface of a transparent substrate 190 (e.g., a SiO2 substrate (refractive index 1.45)). An example of the material of the structure 160 is SiN (refractive index n1 = 2.05). The sides and bottom of the structure 160 are air (refractive index n0 = 1.0).

[0048] Let the arrangement period of each structure 160 be P. It is desirable to set the arrangement period P as in Equation (1) so that diffracted light does not occur on the transmission side.

[0049]

Equation

[0050] λ min is the shortest wavelength in the wavelength band of the light reception target. n0 is the refractive index of the transparent layer on the transmission side. For example, let λ min be 420 nm, n0 be 1.0, and P = 400 nm.

[0051] The height h (length in the z-axis direction) of the structure 160 is constant. Since it is preferable that the structure 160 has a light phase delay amount (phase value) of 2π or more with respect to the incident light, that is, the light traveling along the z-axis direction, when the desired central wavelength in the longest wavelength band of the wavelength region to be separated is λ r it is desirable to set it as in Equation (2).

[0052]

Equation

[0053] In Equation (2), n1 is the refractive index of the structure 160. When the structure 160 is SiN, n1 = 2.05, and the height h is, for example, 1250 nm. Also, the structure 160 may be formed of TiN (refractive index 2.40).

[0054] By designing (including dimension design) the cross-sectional shape of the structure 160, various combinations capable of giving different optical phase delay amounts to light of each wavelength can be realized. By diversifying the cross-sectional shape of the structure 160, the number of combinations increases, and the design freedom is further improved.

[0055] In order to cause polarization dependence, it is desirable that the cross-sectional shape of each of the plurality of structures 160 when cut along a plane parallel to the xy plane is a two-fold rotational symmetry shape. Here, the polarization dependence refers to a property capable of giving different phase delay amounts to linearly polarized lights orthogonal to each other. In the present embodiment, a lens pattern having polarization separation and wavelength-dependent PSF is designed using the phase values for each polarization component depending on the dimensional parameters of the cross-sectional shape of the structure 16.

[0056] [Principle of Phase Control] FIG. 8 is a bottom view of the structure 160. FIGS. 9 and 10 are side views of the structure. The structure 160 is formed of a material having a refractive index n1 higher than the refractive index n0 of the material or space around the structure 160, and the height h of the structure 160 when viewed from the side is constant. Also, the bottom surface and the top surface of the structure 160 are quadrilaterals.

[0057] The structure 160 can operate as an optical waveguide that confines and propagates light in the structure due to the refractive index difference from the material or space around the structure 160. Therefore, when light is incident from one side of the structure 160, the light propagates while being strongly confined in the structure. At this time, the incident light propagates while receiving a phase delay effect determined by the effective refractive index n eff of the optical waveguide, and is finally output from the other structure side.

[0058] In this case, when the phase of the light propagated through the length equal to the thickness of the material or space around the structure is used as a reference, the phase delay amount φ caused by the structure is expressed by Equation (3), where λ is the wavelength of the light in vacuum.

[0059]

Equation

[0060] n in Equation (3) eff is a function of the dimensions of the structure 160, and it is known that strong polarization dependence occurs depending on the shape of the structure 160. When the structure 160 has a rectangular structural cross-section as shown in FIG. 8, different n eff can be given independently for orthogonal incident polarizations.

[0061] Here, let the phase delay amount for the polarization component in the horizontal direction (x-axis direction) of FIG. 8 be φ x , the phase delay amount for the polarization component in the vertical direction (y-axis direction) be φ y , the effective refractive index for the polarization component in the horizontal direction be n effx , the effective refractive index for the polarization component in the vertical direction be n effy , the width of the column in the direction parallel to the horizontal direction be w1, and the width of the column in the direction parallel to the vertical direction be w2.

[0062] At this time, it is known that n effx and n effy can be controlled respectively by the combination of w1 and w2, and each takes values of n0 < n effx < n 1、 and n0 < n effy < n1.

[0063] Therefore, φ x and φ y can be arbitrarily controlled respectively by the combination of w1 and w2. That is, as exemplified in FIGS. 11 to 16 (described later), by designing w1 and w2 which are the widths of the structure 160, φ x and φ y of the phase delay amount for each polarization direction can be arbitrarily set.

[0064] From the above, in this embodiment, by arranging columnar structures 160 each having an appropriate width according to the position on the plane, an arbitrary phase delay spatial distribution can be given to each polarization direction. As a result, in this embodiment, it becomes possible to perform arbitrary wavefront control for each polarization direction at the design wavelength.

[0065] [Phase retardation amount] Next, the structural width of the structure 160 and the phase retardation amount for each polarization will be described. FIGS. 11 to 16 are diagrams showing the relationship between the phase retardation amount at each wavelength for each polarization and the structural width of the structure 160. FIGS. 11 to 16 show the phase values of the vertical polarization or horizontal polarization of wavelengths (450, 550, 660 nm) when the structural parameters (width) of the columnar structure 160 are set to various values with a constant height.

[0066] FIG. 11 shows the phase value of the horizontal polarization at a wavelength of 450 nm, and FIG. 12 shows the phase value of the vertical polarization at a wavelength of 450 nm. FIG. 13 shows the phase value of the horizontal polarization at a wavelength of 550 nm, and FIG. 14 shows the phase value of the vertical polarization at a wavelength of 550 nm. FIG. 15 shows the phase value of the horizontal polarization at a wavelength of 650 nm, and FIG. 16 shows the phase value of the vertical polarization at a wavelength of 650 nm.

[0067] As shown in FIGS. 11 to 16, by appropriately designing (including dimension design) the cross-sectional shape of the structure 160, various combinations of phase values from 0 to 2π can be realized at various design wavelengths for each polarization.

[0068] Note that in FIGS. 11 to 16, the relationship between the phase retardation amount and the structural width of the structure 160 is described for horizontal polarization and vertical polarization, but it is not limited to this. In this embodiment, the same design can be applied to any orthogonal polarization. For example, for the control of +45° and -45° polarization, the compositional structure of the structure 160 shown in FIGS. 6 to 10 may be rotated by 45° on the xy plane. Also, in FIGS. 11 to 16, the characteristics of only three wavelengths are shown, but the same characteristics can be obtained at any wavelength.

[0069] In this way, in the embodiment, only a binary structure can be designed to have different spatial phase distributions according to an arbitrary orthogonal polarization direction in accordance with a design wavelength, and as will be described later, an optical element 12 having different PSFs in shape for each wavelength can be designed while separating polarization.

[0070] Note that the cross-sectional shape of the structure 160 is not limited to the rectangular shape shown in FIGS. 7 and 8. FIG. 17 is a diagram showing an example of the cross-sectional shape of the structure 160. The structure 160 may have various cross-sectional shapes as exemplified in FIG. 17. The exemplified shapes are rotationally symmetric shapes of order two obtained by variously combining, for example, a rectangular shape, a diamond shape, a cross shape, and an elliptical shape.

[0071] [Design Example 1 of Lens] Next, a design example of the optical element 12, which is a polarization separation / wavelength-dependent PSF lens, will be described. In the present embodiment, a lens phase distribution that condenses light around different positions for each polarization and has different PSF shapes for each wavelength is designed and realized by the columnar structure 160.

[0072] Here, the phase distribution was designed using the structure 160 having the SiN composition structure shown in FIGS. 6 to 10, and the optical element 12 having a polarization separation / wavelength-dependent PSF function was realized. Note that a lens having different PSF shapes for each wavelength can be realized from various phase distributions.

[0073] As the simplest example, a case of designing the optical element 12 having different PSFs for each wavelength based on a phase distribution equivalent to that of a Fresnel lens will be described. In this case, the phase distributions φ x , φ y of the lens for each of the orthogonal polarizations (here, horizontal polarization and vertical polarization) are represented by, for example, Expressions (4) and (5).

[0074] [Equation]

[0075] [Equation]

[0076] In Equation (4) and Figure (5), (x, y) are the spatial coordinates on the lens plane. λ d is the design wavelength. x f is the focal length (eccentricity) along the x-axis. z f is the focal length along the z-axis. n is the refractive index of the optical propagation space after passing through the lens. C is an arbitrary constant.

[0077] Figures 18 to 23 are diagrams showing examples of the phase distribution for each polarization of the structure 160 designed to be equivalent to a Fresnel lens. Figure 18 shows the phase distribution of the horizontal polarization at a wavelength of 450 nm. Figure 19 shows the phase distribution of the vertical polarization at a wavelength of 450 nm. Figure 20 shows the phase distribution of the horizontal polarization at a wavelength of 550 nm. Figure 21 shows the phase distribution of the vertical polarization at a wavelength of 550 nm. Figure 22 shows the phase distribution of the horizontal polarization at a wavelength of 650 nm. Figure 23 shows the phase distribution of the vertical polarization at a wavelength of 650 nm.

[0078] In the case of the examples shown in Figures 18 to 23, the lens size is 0.5 mm × 1 mm, the focal length z f is 5.0 mm, the eccentricity x f is 0.25 mm, and the lens design was performed with parameters of a design wavelength of 520 nm. φ is converted so as to fall within the range of 0 to 2π. For example, -0.5π and 2.5π are converted to 1.5π and 0.5π, respectively.

[0079] In order to realize a polarization separation and wavelength-dependent PSF lens with a phase distribution equivalent to that of a Fresnel lens, a structure 160 of the structure that best fits the phase distributions of Equation (4) and Equation (5) (a structure with the minimum phase error for each polarization) may be selected and arranged for each position from the phase retardation amount at the design wavelength of the composition structure.

[0080] In the case of the phase distribution of the lens shown in FIGS. 18 to 23, when parallel light is incident on the optical element 12, it is separated for each polarization component in the horizontal and vertical directions, and converges around different points at the focal length. In this case, the parallel light of the design wavelength converges at one point each at the focal length. That is, the PSF shape becomes a dot (specifically, a Gaussian function in the case of a circular lens and a sinc function in the case of a square lens).

[0081] For light of other wavelengths, due to the wavelength dependence of the focusing position caused by the phase pattern and the wavelength dispersion of the phase of the composition structure, the size of the PSF changes depending on the wavelength. That is, chromatic aberration occurs in which the degree of image blurring varies according to the wavelength.

[0082] In the present embodiment, by utilizing this polarization separation function and chromatic aberration, it is possible to sort for each polarization component with respect to the imaging target, perform different convolution operations for each wavelength while forming images around different positions, acquire an image, and generate a spectral image by image reconstruction.

[0083] In this example, the separation of horizontal polarization and vertical polarization has been described, but the separation of +45° / -45° polarization may have a similar design. If the horizontal polarization / vertical polarization separation pattern and the +45° / -45° polarization separation pattern are arranged as a set, it is possible to classify and image four linearly polarized components (see FIG. 2).

[0084] [Design Example 2 of Lens] Another design example of the optical element 12, which is a wavelength-dependent PSF lens, will be described. Here, a case where it is designed to have a phase distribution with a propeller-shaped PSF will be described as an example.

[0085] That is, an optical element 12 having a phase distribution such that the shape of the PSF rotates according to the wavelength is designed. In this case, when different focusing center positions are set for each orthogonal polarization (here, horizontal polarization and vertical polarization), the phase distribution φ x , φ y is represented by, for example, Equations (6) and (7).

[0086]

Number

[0087]

Number

[0088] In Equations (6) and (7), r is the distance from the origin (designed light-gathering center position) on the lens plane. θ is the angle formed by the coordinates of the designed light-gathering center position on the lens plane. c is the speed of light in a vacuum. ω(θ) is the optical angular frequency at the position of θ. ω min is the minimum optical angular frequency in design. ω max is the maximum optical angular frequency in design. f is the focal length. n is the refractive index of the optical propagation space after passing through the lens. C is an arbitrary constant. N is the number of blades.

[0089] Figures 24 to 29 are diagrams showing examples of the phase distribution for each polarization of the structure 160 designed such that the PSF has a propeller shape. Figure 24 shows the phase distribution of the horizontal polarization at a wavelength of 450 nm. Figure 25 shows the phase distribution of the vertical polarization at a wavelength of 450 nm. Figure 26 shows the phase distribution of the horizontal polarization at a wavelength of 550 nm. Figure 27 shows the phase distribution of the vertical polarization at a wavelength of 550 nm. Figure 28 shows the phase distribution of the horizontal polarization at a wavelength of 650 nm. Figure 29 shows the phase distribution of the vertical polarization at a wavelength of 650 nm.

[0090] In the case of the examples shown in Figures 24 to 29, the lens size was 0.5 mm × 1 mm, the focal length f was 5.0 mm, the number of blades was 3, the designed wavelength was 420 to 660 nm, and the lens design was performed with the parameters of the horizontal polarization (+0.25 mm, 0 mm) and the vertical polarization (-0.25 mm, 0 mm) for the light-gathering position. φ is converted so as to fall within the range of 0 to 2π. For example, -0.5π and 2.5π are converted to 1.5π and 0.5π, respectively.

[0091] In order to realize a polarization separation and wavelength-dependent PSF lens that has a phase distribution with a propeller-shaped PSF, a structure body 160 of a structure that best matches the phase distributions of equations (6) and (7) (a structure with the minimum phase error for each polarization), from the amount of phase delay at each wavelength (each angular frequency) of the composition structure, may be selected and arranged for each position.

[0092] In the case of the phase distributions of the lenses shown in FIGS. 24 to 29, when parallel light is incident on the optical element 12, it is separated for each polarization component in the horizontal and vertical directions and converges around different points at the focal length. Further, as will be described later, the PSF shape becomes a shape like a propeller, and the number of its blades corresponds to N in equation (7). This PSF shape rotates according to the wavelength, and its size hardly changes.

[0093] This is due to the fact that the focal length of the lens is caused by the wavelength dispersion with respect to the phase of the phase pattern and the composition structure that have dependencies on the wavelength and the rotation angle θ. Only the light with the design angular frequency ω(θ) (design wavelength) at an arbitrary rotation angle θ converges at the design focal length and focal position, and the focal lengths of the other light change before and after. Since the design angular frequency ω(θ) changes linearly according to the rotation angle θ, a PSF like a propeller shape is generated, and the PSF rotates depending on the angular frequency (wavelength).

[0094] The optical element 12 can sort for each polarization component with respect to the imaging object 1 using the above-described polarization separation function and wavelength-dependent PSF, and perform different convolution operations for each wavelength while forming images around different positions. The imaging device 10 can generate a spectral image by image reconstruction after image acquisition.

[0095] Note that, as will be described later, compared with the Fresnel lens type, the propeller lens type in which the size of the PSF is almost constant and the wavelength dependence occurs in a clear form of rotation is more advantageous and more suitable in reconstruction.

[0096] Also, although the separation of horizontal polarization and vertical polarization has been described here, the same design can be applied to the separation of +45° / -45° polarization. If a set is made by arranging the horizontal polarization / vertical polarization separation pattern and the +45° / -45° polarization separation pattern side by side, classification and imaging of four linearly polarized components can be achieved (see Fig. 2).

[0097] Also, although the design angular frequency ω in formulas (6) and (7) changes according to the lens position, the same effect can be obtained by replacing the angular frequency with the wavelength.

[0098] Hereinafter, an example of the optical element 12 designed based on the propeller lens type will be shown, but the same applies to other wavelength-dependent PSF lenses such as the Fresnel lens type.

[0099] [PSF Shape Example] An example of the PSF shape of the polarization separation / wavelength-dependent PFS lens in the present embodiment is shown. FIGS. 30 to 36 are diagrams showing the PSF shapes at each polarization and each wavelength obtained by Fresnel diffraction integration from the phase distributions shown in FIGS. 24 to 29. In the case of the example shown in FIGS. 30 to 36, the lens size is 1 mm × 2 mm, the focal length f is 10 mm, the number of blades is 3, the design wavelength is 420 to 660 nm, and the lens design was performed with parameters of horizontal polarization (+0.5 mm, 0 mm) and vertical polarization (-0.5 mm, 0 mm) at the light condensing position.

[0100] FIG. 30 shows the PSF shape of horizontal polarization at a wavelength λ = 600 nm. FIG. 31 shows the PSF shape of vertical polarization at a wavelength λ = 600 nm. FIGS. 32 to 36 show the PSF shapes of horizontal polarization at horizontal polarization and near the light condensing point (+0.5 mm, 0 mm). FIG. 32 shows the PSF shape at a wavelength λ = 450 nm. FIG. 33 shows the PSF shape at a wavelength λ = 500 nm. FIG. 34 shows the PSF shape at a wavelength λ = 550 nm. FIG. 35 shows the PSF shape at a wavelength λ = 600 nm. FIG. 36 shows the PSF shape at a wavelength λ = 650 nm.

[0101] As shown in FIGS. 30 and 31, according to the optical element 12, it can be seen that the condensing position is different according to the polarization, and polarization separation can be realized. And according to the optical element 12, as shown in FIGS. 32 to 36, the blades have a three-blade propeller-shaped PSF and rotate according to the wavelength. As shown in FIGS. 32 to 36, regardless of the change in wavelength, the size of the PSF itself hardly changes.

[0102] When imaging the imaging target 1 with the optical element 12 having these PSFs, it is sorted for each polarization component, and the result of convolving the image with the PSF of the corresponding wavelength is imaged on the image sensor. In this example, the separation of horizontal polarization and vertical polarization is described, but the same result was obtained for the separation of +45° / -45° polarization.

[0103] [Observed image example] Subsequently, the simulation results of imaging a natural image with the optical element 12 having the PSFs of FIGS. 30 to 36 will be described. FIG. 37 is a diagram showing the simulation results.

[0104] The simulation was performed by convolving the publicly available spectral image (ICVL, Boaz Arad and Ohad Ben-Shahar. Sparse recovery of hyperspectral signal from natural rgb images, In European Conference on Computer Vision, pp. 19-34. Springer, 2016., [online], [searched on December 28, Reiwa 2], Internet <URL:http: / / icvl.cs.bgu.ac.il / hyperspectral / >) (wavelength 420 nm to 660 nm: 25 bands) with the PSFs of FIGS. 30 to 36 for each wavelength and integrating along the wavelength dimension considering the sensitivity of the RGB pixels of a general color image sensor. Note that FIG. 37 is a monochrome display of an RGB color image, the left image is the input spectral image (real image), and the right image is the image after convolution (observed image) imaged on the imaging device 11.

[0105] This corresponds to imaging using a lens having the PSFs of FIGS. 30 to 36 and a color image sensor, and simulating an RGB color image (observed image) output from the sensor.

[0106] Note that the wavelength range of the light of B is 420 to 500 nm, the wavelength range of the light of G is 500 to 600 nm, and the wavelength range of the light of R is 600 to 660 nm. In FIG. 37, an image of only one arbitrary polarization component (horizontal polarization) is shown as an observed image, and hereinafter, reconstruction processing corresponding to this observed image of horizontal polarization will be shown, but the same applies to other polarization components.

[0107] As shown in FIG. 37, it can be seen that the observed image is blurred by the convolution operation with the PSF of the optical element 12. Note that the observed information amount is compressed to 12% (from 25 wavelength bands to 3 colors) of the real image, and the information is restored from 3 colors to 25 wavelength bands by image reconstruction based on compressive sensing.

[0108] [Reconstructed image] Next, an example of a reconstructed image by the imaging device 10 will be described. FIG. 38 is a diagram showing an example of a reconstructed image by the imaging device 10. In FIG. 38, an example of generating a spectral image from the observed image shown in FIG. 37 using reconstruction processing based on compressive sensing is shown.

[0109] Here, SSTV is used as a regularization term, and ADMM is used as a method for solving the optimization problem. In addition, FIG. 38 also shows a real image for comparison. Note that the reconstructed image and the real image shown in FIG. 38 are spectral images of 25 bands, but are monochrome displays of those displayed as RGB images for visualization.

[0110] The reconstructed image was evaluated using evaluation indices of PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and SAM (Spectral Angle Mapping).

[0111] As shown in Expressions (8) and (9), the PSNR is an index for evaluating the difference per pixel, and the larger the value (dB), the higher the image quality. The PSNR of the image for each wavelength was calculated and averaged over all wavelengths, and then applied to the spectral image.

[0112]

Number

[0113]

Number

[0114] The SSIM is the structural similarity, and as shown in Expression (10), it is an index that includes the correlation with surrounding pixels. The closer the SSIM is to 1, the higher the image quality. The SSIM of the image for each wavelength was calculated and averaged over all wavelengths, and then applied to the spectral image.

[0115]

Number

[0116] The SAM is the wavelength spectrum similarity, and the closer it is to 0, the more similar the spectra are. The SAM for each pixel was calculated and averaged over the entire image, and then applied to the spectral image.

[0117] The reconstructed image had a PSNR of 29.10 dB, an SSIM of 0.9176, and a SAM of 0.1874. Therefore, it can be seen that the imaging device 10 accurately reconstructed the image.

[0118] [Reconstructed Wavelength Spectrum] Next, an example of the reconstructed wavelength spectrum will be described. FIG. 39 is a diagram showing the wavelength spectrum at the × point of the reconstructed image in FIG. 38. For comparison, FIG. 39 also shows the wavelength spectrum at the × point of the actual image (Ground truth) together with the reconstructed image (Reconstructed).

[0119] As shown in Fig. 39, in the reconstructed image, a spectrum that well matches the real image is obtained, and it can be seen that high-precision information restoration can be performed by image reconstruction. Note that the reconstruction accuracy varies depending on the shape of the PSF of the optical element 12, as well as the regularization term and the method of solving the optimization problem.

[0120] [Comparison of Reconstruction Accuracy According to the Shape of PSF] Next, the results of comparing the reconstruction accuracy according to the shape of the PSF in the optical element 12 are shown. Fig. 40 is a figure showing the results of comparing the reconstruction accuracy for each shape of the PSF of the optical element 12. Fig. 41 is a figure showing the reconstructed images respectively reconstructed based on the observed images of each shape of the PSF in Fig. 40. The reconstructed images, real images, and Fresnel lens images shown in Fig. 41 are the monochrome displays of those shown in RGB images.

[0121] In Fig. 40, for comparison, the case of using a Fresnel lens type PSF is also shown. Also, in Fig. 41, for comparison, the real image and the reconstructed images by the Fresnel lens type are also shown. For the image by the Fresnel lens type, reconstruction is performed using a large chromatic aberration.

[0122] In Figs. 40 and 41, PSNR, SSIM, and SAM are used as evaluation indices. N in Figs. 40 and 41 is the number of blades. Figs. 40 and 41 are the results of calculation and evaluation assuming only one polarization component. Note that the lens parameters are such that the lens size is 0.5 mm × 1.0 mm, the focal length z f is 5.0 mm, the eccentricity x f is 0.25 mm, and the designed wavelength band is 420 - 660 nm.

[0123] As shown in FIGS. 40 and 41, none of the evaluation indices showed a significant difference depending on the number of vanes, and showed higher accuracy than the Fresnel lens PSF. In other words, the optical element 12 showed higher accuracy than the Fresnel lens PSF regardless of the number of vanes. Therefore, it can be said that the optical element 12 according to the present embodiment is more suitable than the Fresnel lens type and constitutes an observation matrix advantageous for reconstruction.

[0124] [Effects of the Embodiment] As described above, in the imaging device 10 according to the present embodiment, with only the optical element 12, an optically encoded image is formed on the imaging element 11 for each polarization component. In the present embodiment, the hyperspectral camera optical system and the polarization information acquisition optical system are realized as an integrated optical element by the optical element 12 which is a metasurface. In other words, in the imaging device 10, with only the optical element 12, effective encoding can be performed in spectral image reconstruction while separating polarization. Therefore, the components of the imaging device 10 only need to be the optical element 12, the imaging element 11, and the signal processing unit 13, and a hyperspectral imaging device with a simple device configuration and capable of acquiring polarization information can be realized.

[0125] Further, in the imaging device 10, the distance between the optical element 12 and the imaging element 11 is determined by the focal length of the lens as in a normal imaging device, so the size of the imaging device 10 is equivalent to that of a normal camera having the same field of view F number.

[0126] When the observation process is known (here, the PSF of the optical element 12 and the wavelength sensitivity characteristics of the sensor), the optically encoded image can restore the information of the real image by performing appropriate signal processing in the signal processing unit 13.

[0127] In addition, in the imaging device 10, imaging itself (acquisition of an image separated and encoded into each polarization component) may be a single shot, so there is no sacrifice in the time dimension, and time resolution equivalent to that of a normal camera is possible except for the reconstruction process.

[0128] In addition, in the imaging device 10, since the optical element 12 responsible for polarization separation and encoding is composed of a fine binary structure, the number of manufacturing steps can be reduced compared to general diffractive optical elements that require multi-step lithography. It also has a thinner thickness, lighter weight, and is easier to manufacture.

[0129] Furthermore, the optical element 12 with a fine binary structure is not subject to a decrease in light transmittance and a limitation of the maximum lens aperture number (NA) caused by the shadow effect (a phenomenon in which diffracted light by a diffractive optical element is reflected and scattered by its multi-step structure) that occurs in general diffractive optical elements. Therefore, a lens with a higher NA (a brighter lens with high light utilization efficiency) can be realized.

[0130] In addition, in the present embodiment, although an example based on the case where the imaging element 11 is a color image sensor is shown, even in the case of a monochrome image sensor, it is the same except that 3ch (RGB: color) becomes 1cb (monochrome) when acquiring an encoded image.

[0131] [Modification Example 1 of the Embodiment] In this modification example 1, a modification example capable of reducing the image overlap (crosstalk) on the pixels of the imaging element 11 will be described.

[0132] In this modification example 1, a part of the imaging element 11 and the optical element 12 will be described as an imaging unit. FIG. 42 is a plan view of the imaging unit to which the optical element 12 shown in FIG. 1 is applied. Note that the transparent substrate 190 is omitted. FIG. 43 is a cross-sectional view when the imaging unit is cut along the line A-A' in FIG. 42.

[0133] As shown in FIGS. 42 and 43, in each adjacent lens pattern, there is a concern that image overlap (crosstalk) may occur near the boundary of each image of the imaging element 11 (near the boundary between region 11-1 and region 11-2). This image overlap may lead to deterioration of the reconstructed image and deterioration of the polarization extinction ratio (desired polarization component / other polarization components).

[0134] FIG. 44 is a plan view of the imaging unit according to Modification 1 of the embodiment. Note that the transparent substrate 190 is omitted. FIG. 45 is a cross-sectional view of the imaging unit taken along line B-B' in FIG. 44.

[0135] As shown in FIGS. 44 and 45, in the imaging unit 200, in order to avoid image overlap, a barrier 240 is provided directly below the boundary between the first lens pattern region 12-1 and the second lens pattern region 12-2 (polarization separation region) of the optical element 12.

[0136] This barrier 240 is preferably made of a material that absorbs light and does not generate stray light, or is subjected to surface processing that adds a similar function.

[0137] This barrier 240 is provided between the optical element 12, which is a polarization separation / wavelength-dependent PSF lens, and the imaging element 11. If the influence between the first lens pattern region 12-1 and the second lens pattern region 12-2 is completely blocked by the barrier 240, image overlap can be completely removed.

[0138] Note that even a partial barrier has the effect of reducing the influence of image overlap, and the barrier height and position may be determined according to the application, manufacturing, and mounting processes.

[0139] [Modification 2 of the Embodiment] In this Modification 2, a modification that can reduce image overlap (crosstalk) on the pixels of the imaging element 11 and can improve the polarization extinction ratio will be described.

[0140] In this Modification 2, a part of the imaging element 11 and the optical element 12 will be described as an imaging unit. FIG. 46 is a plan view of the imaging unit to which the optical element 12 shown in FIG. 1 is applied. Note that the transparent substrate 190 is omitted. FIG. 47 is a cross-sectional view of the imaging unit taken along line C-C' in FIG. 46.

[0141] As shown in FIGS. 46 and 47, in the same lens pattern (for example, the first lens pattern region 12-1), after polarization separation, there is a concern that crosstalk may occur between two images formed in the region 11-1a where an image corresponding to the 0° polarization component of the imaging device 11 is formed and the region 11-1b where an image corresponding to the 90° polarization component is formed. This image overlap may lead to degradation of the reconstructed image and degradation of the polarization extinction ratio (desired polarization component / other polarization components).

[0142] FIG. 48 is a plan view of an imaging unit according to Modification 2 of the embodiment. The transparent substrate 190 is omitted. FIG. 49 is a cross-sectional view when the imaging unit is cut along the line B-B' in FIG. 48.

[0143] As shown in FIGS. 48 and 49, in the imaging unit 200A, in order to avoid image overlap, a plurality of polarization filters 250 are provided between the optical element 12 and the imaging device 11, and each corresponds to the imaging position of light spatially polarization-separated by the optical element 12.

[0144] Therefore, the light composed of each polarization component separated by the lens always passes through the corresponding polarization filter. Then, the respective lights are imaged on the imaging device 11. At this time, the polarization direction of the separated light is made to coincide with the polarization transmission axis of the corresponding polarization filter 250.

[0145] In this way, in the imaging unit 200A, a polarization filter is provided between the optical element 12 and the imaging device 11, in which the polarization direction corresponding to the pixel directly below coincides with the polarization transmission axis. As a result, in the imaging unit 200A, light can be guided onto the pixel 130 of the imaging device 11 in a state where components other than the desired polarization component are cut, so that image overlap near the boundary of images composed of different polarization components can be completely removed. Therefore, according to the imaging unit 200A, image crosstalk can be greatly reduced.

[0146] Furthermore, in the imaging unit 200A, double polarization filtering is performed by the optical element 12 and the polarization filter 250. Since this leads to an improvement in the polarization extinction ratio, the imaging unit 200A can also achieve an improvement in the quality of the polarization image.

[0147] Note that in the imaging unit 200A that uses the polarization filter 250 in combination, the above effects can be added while maintaining high light utilization efficiency. This is because polarization filtering is performed after polarization separation, so the total amount of light reaching the pixel array is hardly reduced.

[0148] Note that the imaging unit 200A can further be provided with the barrier 240 shown in FIGS. 44 and 45. By using the polarization filter 250 and the barrier 240 in combination, crosstalk between each polarization image can be almost eliminated, and a higher-quality polarization spectrum image can be generated.

[0149] [Lens Structure Example] The optical element 12 is not limited to the configuration shown in FIGS. 3 and 4, and can take various forms in terms of the number and spacing of the structures 160, the structural shape, and the arrangement pattern. Also, the structures 160 may be connected to each other, or may be embedded in a transparent material.

[0150] In FIGS. 3 and 4, the optical element 12 is formed on the bottom surface of the transparent substrate 190, but it is not limited to this. FIGS. 50 to 55 are diagrams schematically showing other examples of a part of the cross-section of the optical element 12 according to the embodiment.

[0151] As shown in FIG. 50, the structure 160 of the optical element 12 may be formed on the upper surface of the transparent substrate 190A. In this case, the structure 160 is supported by the transparent substrate 190A. The transparent layer above the structure 160 may be air or a protective layer such as resin, and the material of the transparent layer may be single or a plurality of materials in a layered form.

[0152] Also, as shown in FIG. 51, the structure 160 of the optical element 12 may be embedded in the transparent substrate 190B. The material of the transparent substrate 190B may be single or may be a laminate of a plurality of materials.

[0153] Also, as shown in FIG. 52, the structure 160 of the optical element 12 may be formed on both surfaces of the transparent substrate 190C. The above-described polarization separation / wavelength-dependent PSF function may be realized by the structures 160 on both surfaces of the transparent substrate 190C. Also, the wavelength-dependent PSF function may be realized by the structure 160 of the transparent substrate 190C, and other optical functions such as a filter, a splitter, and a light-shielding layer may be realized on the other surface. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the transparent layer may be single or may be a laminate of a plurality of materials.

[0154] Also, as shown in FIG. 53, the structure 160 of the optical element 12 may be formed on the refractive lens 190D. The structure 160 is supported on the refractive lens 190D. The refractive lens 190D is beneficial in improving the light-collecting performance of the wavelength-dependent light-collecting characteristics (such as increasing the NA). The same applies to the refractive lenses 190E and 190F described later. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the refractive lens 190D may be single or may be a laminate of a plurality of materials.

[0155] Also, as shown in FIG. 54, the structure 160 of the optical element 12 may be embedded in the refractive lens 190E. The material of the refractive lens 190E may be single or may be a laminate of a plurality of materials.

[0156] Also, as shown in FIG. 55, the structure 160 of the optical element 12 may be formed on both surfaces of the refractive lens 190F. The wavelength-dependent PSF function described above may be realized by the structures 160 on both surfaces of the refractive lens 190F. Further, the wavelength-dependent PSF function may be realized by the structure 160 of the refractive lens 190F, and other optical functions such as a filter, a splitter, and a light-shielding layer may be realized on the other surface. The transparent layer above the structure 160 may be air or a protective layer such as resin. The material of the refractive lens 190F may be single or may be a plurality of materials layered.

[0157] Also, although omitted in FIGS. 50 to 55, a light-shielding film pattern or the like may be provided on the same plane or the back surface.

[0158] In the embodiment, TiO2 and SiN are exemplified as the material of the structure 160. However, the material of the structure 160 is not limited thereto. For example, for light with a wavelength of 380 nm to 1000 nm (visible light to near-infrared light), in addition to SiN, SiC, TiO2, GaN, etc. may be used as the material of the structure 6. It is suitable because it has a high refractive index and low absorption loss. When used for light with a wavelength of 800 to 1000 nm (near-infrared light), Si, SiC, SiN, TiO2, GaAs, GaN, etc. may be used as the material of the structure 6. It is suitable because it has low loss. For light in the near-infrared region of the long-wavelength band (such as 1.3 μm and 1.55 μm which are communication wavelengths), in addition to the above materials, InP, etc. can be used as the material of the structure 160.

[0159] Also, when the structure 160 is formed by pasting, coating, etc., polymers such as polyimide such as fluorinated polyimide, BCB (benzocyclobutene), photocurable resin, UV epoxy resin, acrylic resin such as PMMA, and resist in general can be mentioned as materials.

[0160] In addition, in the embodiment, an example assuming SiO2 and an air layer as the material of the transparent layer 150 is shown, but the present invention is not limited thereto. Any material having a refractive index lower than that of the material of the structure 160 and having low loss with respect to the wavelength of incident light may be used, including general glass materials and the like. Since the transparent layer 150 only needs to have sufficiently low loss with respect to the wavelength of light to reach the corresponding pixel, it may be made of the same material as a color filter, for example, an organic material such as resin. In this case, not only is the transparent layer 150 made of the same material as the color filter, but it may also have the same structure as the color filter and be designed to have absorption characteristics corresponding to the wavelength of light to be guided to the corresponding pixel.

[0161] In addition, in the embodiment, the three primary colors of RGB are taken as examples to explain the corresponding colors of the pixels. However, the pixels may also correspond to near-infrared light and light with wavelengths other than the three primary colors (for example, infrared light, ultraviolet light, etc.).

[0162] In addition, in the embodiment, an example in which a structure having a rectangular cross-sectional shape, a diamond cross-sectional shape, a cross shape, and an elliptical cross-sectional shape is used as the shape of the structure 160 has been described. This shape is just an example. One type of structure (for example, only a rectangular shape) may be used, or two or more types of structures (for example, only a rectangular shape and a cross shape) may be used.

[0163] As described above, the present invention has been described based on specific embodiments. However, it goes without saying that the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof.

Explanation of Reference Numerals

[0164] 1 Imaging object 10 Imaging device 11 Imaging element 12 Optical element 13 Signal processing unit 130 Pixel 150 Transparent layer 160 Structure 190, 190A~190C Transparent substrate 190D~190F folding lens

Claims

1. An optical element including a transparent substrate and a plurality of structures disposed in a plane direction of the transparent substrate on or within the transparent substrate, An imaging device in which a plurality of pixels including a photoelectric conversion element are disposed, A signal processing unit configured to generate an image signal based on an electrical signal obtained from the imaging device, An imaging device having: The optical element having an imaging function, The optical element outputs light in a state having different point spread functions for each wavelength, and an image in which the point spread functions of each wavelength are convolved is imaged on a plurality of pixels corresponding to each polarization component according to the polarization component, The plurality of structures have the same height when viewed from the side, The signal processing unit reconstructs an image for each polarization component, in which the point spread function of each wavelength is convolved, Each of the plurality of structures is a columnar structure having a refractive index higher than that of the transparent substrate and providing a light phase delay amount corresponding to a cross-sectional shape with respect to incident light, The cross-sectional shapes of the plurality of structures are set according to a light phase amount delay distribution for imaging an image in which the point spread function of each wavelength with respect to the pixel is convolved on a plurality of pixels corresponding to each polarization component according to the polarization component, and are arranged according to the light phase amount delay distribution for imaging an image in which the point spread function of each wavelength with respect to the pixel is convolved on a plurality of pixels corresponding to each polarization component according to the polarization component, The cross-sectional shape and arrangement of each of the plurality of structures are designed to be imaged at each position on a pixel corresponding to each wavelength and each polarization component, The cross-sectional shape of each of the plurality of structures is any one of a plurality of types of two-fold rotational symmetric shapes, The plurality of structures are arranged at an arrangement period represented by formula (1), The height h of the plurality of structures when viewed from the side is set as in formula (2), The imaging device outputs a multi-band image having more bands than the number of bands of the image obtained by the imaging device, An imaging device characterized by the above. 【Number 1】 【Number 2】

2. The signal processing unit reconstructs an image for each polarization component based on a matrix defined by an imaging process of the optical element and an image in which the point spread function of each wavelength is convolved, according to claim 1,

3. The imaging device according to claim 2, wherein the signal processing unit solves an optimization problem using a model composed of a neural network, with an input being a matrix defined by the imaging process of the optical element and an image in which the point spread function of each wavelength is convolved.

4. The imaging device according to any one of claims 1 to 3, wherein a barrier for absorbing light is provided directly below the boundary of the polarization separation region in the optical element.

5. The imaging device according to any one of claims 1 to 4, further comprising a polarization filter provided between the optical element and the imaging element, in which the polarization direction corresponding to the pixel located directly below coincides with the polarization transmission axis.

6. An optical element having a transparent substrate and a plurality of structures arranged in the plane direction of the transparent substrate on or within the transparent substrate, wherein the optical element has an imaging function, the optical element outputs light with different point spread functions for each wavelength, and forms an image in which the point spread functions of each wavelength are convolved, on a plurality of pixels of the imaging element corresponding to each polarization component according to the polarization component, the plurality of structures have the same height when viewed from the side, each of the plurality of structures is a columnar structure having a refractive index higher than that of the transparent substrate and giving a light phase delay amount corresponding to the cross-sectional shape to the incident light, the cross-sectional shape of each of the plurality of structures is set according to the light phase amount delay distribution for forming an image in which the point spread function of each wavelength for the pixel is convolved, on a plurality of pixels corresponding to each polarization component according to the polarization component, and the plurality of structures are arranged according to the light phase amount delay distribution for forming an image in which the point spread function of each wavelength for the pixel is convolved, on a plurality of pixels corresponding to each polarization component according to the polarization component, the cross-sectional shape and arrangement of each of the plurality of structures are designed to form an image at each position on the pixel corresponding to each wavelength and each polarization component, the cross-sectional shape of each of the plurality of structures is any one of a plurality of types of two-fold rotational symmetric shapes, the plurality of structures are arranged at an arrangement period represented by formula (3), the height h of the plurality of structures when viewed from the side is set as in formula (4) characterizing the optical element. 【Number 3】 【Number 4】

Images