Imaging device

Abstract

An imaging device according to an embodiment includes an image sensor (13), an optical element (12), and a computer. The optical element (12) includes a surface provided with recesses and protrusions. The optical element (12) guides, to the image sensor (13) provided facing the surface, light obtained by focusing incident light using a focusing pattern that varies for each component. The structure pattern of the recesses and protrusions is four-fold rotationally symmetric with respect to an axis perpendicular to the surface.

Description

Imaging device

[0001] The present invention relates to an imaging device.

[0002] Conventionally, a compressive sensing type spectral camera is known (see, for example, Non-Patent Document 1). Further, Non-Patent Document 1 describes a diffractive optical element (DOE: diffractive optical element) that condenses incident light due to a fine structure.

[0003] Daniel S. Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H. Kim. 2019. Compact snapshot hyperspectral imaging with diffracted rotation. ACM Trans. Graph. 38, 4, Article 117 (August 2019), 13 pages., [online], [Retrieved on June 4, 2024], Internet <https: / / doi.org / 10.1145 / 3306346.<3322946>

[0004] However, in the conventional technology, there is a problem that the resolution in the vertical direction and the horizontal direction of the obtained image may be different. For example, in the DOE having the structural pattern shown in Fig. 2.(b) described in Non-Patent Document 1, since the light condensing patterns in the vertical direction and the horizontal direction are different, finally, images with different resolutions in the vertical direction and the horizontal direction are output. The light condensing pattern is represented by, for example, a point spread function (PSF: Point spread function).

[0005] An object of the present invention is to obtain an image with equal resolution in the vertical direction and the horizontal direction by a sensor and reconstruct a spectral image with equivalent resolution in the vertical direction and the horizontal direction.

[0006] ​The imaging device of the present invention is an imaging device having an optical element including a surface with concave and convex portions, and an image sensor provided opposite the surface, and is an imaging device that generates a spectral image using a computer having more bands than the number of bands in the image obtained by the image sensor, and is characterized in that the optical element focuses incident light with different focusing patterns for each component and guides the light to the image sensor, and the structural pattern of the concave and convex portions is four-fold rotationally symmetric with respect to an axis perpendicular to the surface.

[0007] According to the present invention, it is possible to obtain an image with equal vertical and horizontal resolutions using a sensor, and to reconstruct a spectral image with equal vertical and horizontal resolutions.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of an imaging device. FIG. 2 is a diagram illustrating a process for reconstructing a spectral image. FIG. 3 is a flowchart illustrating the flow of a process for reconstructing a spectral image. FIG. 4 is a diagram illustrating an example of the configuration of an imaging system. FIG. 5 is a diagram illustrating a conventional light-collecting pattern. FIG. 6 is a diagram illustrating a light-collecting pattern of an embodiment. FIG. 7 is a diagram illustrating an example of a structural pattern of an optical element. FIG. 8 is a diagram illustrating an example of a structural pattern of an optical element. FIG. 9 is a diagram illustrating an example of a structural pattern of an optical element. FIG. 10 is a diagram illustrating an example of a structural pattern of an optical element. FIG. 11 is a diagram illustrating an example of a column structure with four-fold rotational symmetry. FIG. 12 is a diagram illustrating an example of the configuration of an imaging system. FIG. 13 is a diagram illustrating an example of the configuration of an imaging system. FIG. 14 is a diagram illustrating an example of a computer that executes a program.

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The shapes, sizes, positional relationships, etc. shown in the drawings are merely schematic and do not limit the present invention. The same parts are designated by the same reference numerals, and duplicated explanations will be omitted.

[0010] First Embodiment An imaging device that reconstructs a spectral image will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the system.

[0011] As shown in Fig. 1, the imaging device 1 includes an imaging system 10 and a computer 20. The imaging system 10 outputs an image 32 of an object captured by the imaging system 10. The object is represented as spectral images 31 corresponding to a plurality of wavelengths. However, the imaging system 10 outputs a single-shot image 32.

[0012] The computer 20 performs image processing based on the image 32 output by the imaging system 10, and outputs a spectral image 33. That is, the imaging system 10 estimates a spectral image of the subject.

[0013] An outline of the process of reconstructing a spectral image by the computer 20 will be described with reference to Fig. 2. Fig. 2 is a diagram for explaining the process of reconstructing a spectral image.

[0014] As shown in FIG. 2, an image 32 output by the imaging system 10 is regarded as data obtained by combining a spectral image 31 of the subject and a PSF (Point Spread Function) 35 .

[0015] The computer 20 inputs the image 32 to a reconstruction process 37 to obtain a spectral image 33. The reconstruction process 37 may use a technique such as a deep neural network (e.g., U-Net) or a mathematical algorithm (e.g., ADMM (Alternating Direction Method of Multipliers)). The reconstruction process 37 may also be compressed sensing. Alternatively, the reconstruction process 37 may use the network architecture described in Non-Patent Document 1. Furthermore, the reconstruction process 37 uses a PSF 35.

[0016] The processing flow of the imaging system 10 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the processing flow for reconstructing a spectral image. As shown in Fig. 3, first, the computer 20 receives input of a single-shot image 32 captured by the imaging system 10 equipped with an optical element (step S101).

[0017] Next, the computer 20 acquires a light-collecting pattern when a point light source is collected using an optical element mounted on the imaging system 10 (step S102). The light-collecting pattern may be the PSF 35.

[0018] Next, the computer 20 constructs a spectral image 33 corresponding to each wavelength based on the light-collecting pattern and the image 32 (step S103). Then, the computer 20 outputs the spectral image (step S104). In other words, the imaging device 1 generates the spectral image by using the computer 20.

[0019] The configuration of the imaging system 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the imaging system. As shown in Fig. 4, the imaging system 10 has a transparent substrate 11, an optical element 12, and an image sensor (imaging element) 13.

[0020] The optical element 12 includes a surface with concave and convex portions. For example, the optical element 12 has a plurality of minute columnar structures. The optical element 12 performs optical encoding by having imaging characteristics that vary depending on wavelength. The optical element 12 may also be a DOE.

[0021] The optical element 12 focuses light incident through the transparent substrate 11 with different focusing patterns for each component, and guides the focused light to the image sensor 13 provided opposite the surface. In other words, the optical element 12 is a lens with a PSF that clearly differs depending on the wavelength (wavelength-dependent PSF lens). When an object is captured by the optical element 12, a convolution operation is performed on the actual image with a PSF that differs for each wavelength, and the result is imaged on the image sensor 13. Note that the convolution operation referred to here does not mean a convolution operation performed by information processing, but an optical convolution operation.

[0022] The image sensor 13 has photoelectric conversion elements such as a CCD or a CMOS. For example, the image sensor 13 has a plurality of photoelectric conversion elements arranged in a two-dimensional array. For example, each of the plurality of photoelectric conversion elements corresponds to a pixel that constitutes the image 32.

[0023] The image sensor 13 includes a wiring layer and a plurality of photoelectric conversion elements arranged in the xy plane. The photoelectric conversion elements are, for example, photodiodes (PD).

[0024] For example, each photoelectric conversion element corresponds to red (R), green (G), and blue (B). An example of the wavelength band of red light is 600 nm < λ ≦ 800 nm, where λ is the wavelength. An example of the wavelength band of green light is 500 nm < λ ≦ 600 nm. An example of the wavelength band of blue light is λ ≦ 500 nm.

[0025] The incident light travels along the positive direction of the z-axis and reaches the image sensor 13 via the optical element 12. The electric charges generated in each photoelectric element of the image sensor 13 are converted into electrical signals that form the basis of pixel signals by transistors or the like and transmitted via the wiring layer. The imaging system 10 generates and outputs an image 32 based on the electrical signals.

[0026] The image sensor 13 outputs an image 32 in which a different convolution operation is performed for each wavelength using the optical element 12, which is a wavelength-dependent PSF lens. The computer 20 reconstructs the spectral information of the image 32 input from the imaging system 10 to generate a spectral image 33. The spectral image 33 has more bands than the number of bands in the image 32 obtained by the image sensor 13.

[0027] The imaging system 10 may include known components such as an infrared light blocking optical filter, an electronic shutter, a viewfinder, a power source (battery), a flashlight, etc. The above configuration is merely an example, and in the embodiment, known components other than the optical element 12 may be used in appropriate combination.

[0028] 4, the optical element 12 is disposed on the side where light is incident (negative direction of the z axis) as viewed from the image sensor 13. For example, when the imaging system 10 is viewed from the negative direction of the z axis, the transparent substrate 11 and the optical element 12 are provided so as to cover the image sensor 13. The optical element 12 is provided on the surface of the transparent substrate 11 facing the image sensor (positive direction of the z axis).

[0029] The transparent substrate 11 is a substrate with a low refractive index made of a material such as SiO2 (refractive index n = 1.45). The layer between the optical element 12 and the image sensor 13, i.e., the transparent layer, is a layer with a low refractive index made of air or a material such as SiO2. The transparent substrate 11 and the transparent layer may be made of a single material, or may be made of a layer of multiple materials. The optical element 12 is made of a material such as SiN or TiO2 that has a refractive index higher than that of the transparent layer.

[0030] An example of the optical element 12 is a metasurface. In a metasurface, at least some of the spacing between the convex and concave structures is smaller than the wavelength of a specific light. The metasurface may have a two-dimensional structure or a three-dimensional structure. The optical element 12 can control the phase and light intensity according to the characteristics of the light (wavelength, polarization, and incident angle) simply by changing the parameters of the convex and concave portions. A three-dimensional structure offers greater design freedom than a two-dimensional structure.

[0031] As described above, the optical element 12 focuses light incident through the transparent substrate 11 in a different focusing pattern for each component (e.g., wavelength), and guides the resulting light to the image sensor 13 provided opposite the surface. This causes the imaging characteristics (e.g., the degree of blur) of the image sensor 13 to change depending on the wavelength of the light from the subject. The light from the subject is focused on the image sensor 13 by the optical element 12, which has a wavelength-dependent PSF function, and is acquired as an image 32.

[0032] For example, image 32 corresponds to the result of optically convolving the actual image (spectral image 31 of the subject) for each wavelength using the wavelength-dependent PSF of optical element 12 and integrating the result along the wavelength dimension on the pixels. In this way, imaging system 10 acquires image 32 in an optically encoded and compressed state. Note that if image sensor 13 is a color image sensor, after the convolution, multiplication is performed according to the wavelength sensitivity of each of the R, G, and B pixels of image sensor 13, and then integration is performed along the wavelength dimension on the pixels.

[0033] Thus, in the imaging system 10, the optical element 12 focuses the optically encoded image 32 onto the image sensor 13. In other words, the optical element 12 can provide effective encoding for spectral image reconstruction.

[0034] Then, when the PSF is known, the optically encoded image 32 can be subjected to appropriate signal processing in the computer 20 to reconstruct information on the actual image, that is, the spectral image 33 .

[0035] The imaging system 10 can perform different encoding for each wavelength component of the actual image using the wavelength-dependent PSF possessed by the optical element 12, and therefore can restore the spectral image by performing image reconstruction processing based on compressed sensing.

[0036] Here, four-fold rotational symmetry will be described. Four-fold rotational symmetry means that when a structure pattern is rotated 90 degrees about an axis, it overlaps with the structure pattern before rotation. Note that the four-fold rotational symmetry in this embodiment includes the overlapping with the structure pattern before rotation when the structure pattern is rotated by 90 degrees ÷ N (where N is a positive integer including infinity) about an axis, that is, 4N-fold rotational symmetry. For example, the four-fold rotational symmetry in the following description may be replaced with 4N-fold rotational symmetry as appropriate.

[0037] [Structural Pattern of Optical Element] A description will be given of the structural pattern of the recesses and protrusions of the optical element 12. The structural pattern of the recesses and protrusions of the optical element 12 of this embodiment has four-fold rotational symmetry with respect to an axis perpendicular to the surface on which the recesses and protrusions are provided (an axis parallel to the z-axis in FIG. 4).

[0038] FIG. 5 is a diagram illustrating a conventional light-collecting pattern. As shown in FIG. 5, the conventional structure pattern is not four-fold rotationally symmetric with respect to an axis (optical center axis) parallel to the z-axis. Note that the curved portions (circumferences of each ellipse) in the structure pattern in FIG. 5 represent convex portions. Therefore, the conventional light-collecting pattern is not four-fold rotationally symmetric. As a result, the image output by the conventional imaging device may have different resolutions in the vertical and horizontal directions, and different degrees of blur in the vertical and horizontal directions.

[0039] On the other hand, the structure pattern of this embodiment has four-fold rotational symmetry with respect to an axis (optical center axis) parallel to the z-axis, as shown in FIG. 6 . FIG. 6 is a diagram illustrating the light-collecting pattern of this embodiment. Note that the curved portions (circumferences of each perfect circle) in the structure pattern of FIG. 6 represent convex portions. Therefore, the light-collecting pattern of this embodiment has four-fold rotational symmetry. As a result, in an image output by a conventional imaging device, the resolution is equal in the vertical and horizontal directions, and the degree of blur is equal in the vertical and horizontal directions.

[0040] 7 to 10 show the structural pattern of the optical element 12. Fig. 7 to Fig. 10 are diagrams showing examples of the structural pattern of the optical element. Fig. 7 to Fig. 10 show the structural pattern of the optical element 12 when viewed from the image sensor 13 side. Furthermore, hatched portions in Fig. 7 to Fig. 10 represent convex portions.

[0041] In the example of Figure 7, the convex portion includes a columnar structure having a base formed of a ring arranged concentrically on the surface (the surface facing the transparent substrate 11) centered on the intersection of the axis and the surface, and a columnar structure having a base formed of each of a plurality of rectangles arranged on the surface.

[0042] In the example of Fig. 8, the convex portion has a columnar structure with a bottom surface formed of a ring arranged concentrically on a surface (the surface on the transparent substrate 11 side) with the intersection of the axis and the surface as the center. The examples of Figs. 7 and 8 include a portion where the structure is continuously connected. A portion where the structure is continuously connected has improved mechanical strength and is easier to fabricate than a portion where the structure is isolated. Note that the shape of the bottom surface is not limited to a ring, and may be any shape with four-fold rotational symmetry.

[0043] 9 and 10, the structural pattern of the recesses and protrusions may be such that the recesses and protrusions are arranged periodically. In this case, the spacing between the recesses and protrusions is smaller than the wavelength of a specific light. The spacing between the recesses and protrusions includes the width of the protrusions and the spacing between the protrusions (the width of the recesses).

[0044] In particular, Figure 9 shows a columnar structure in which the protrusions or protrusions have a base made up of a plurality of quadrilaterals arranged on a surface of minute width. In this case, the columnar structures constituting the structural pattern themselves have four-fold rotational symmetry. In this way, the optical element 12 may have a four-fold rotational symmetry structural pattern as a whole by arranging a plurality of four-fold rotationally symmetric columnar structures. Note that the shape of the base is not limited to a quadrilateral.

[0045] A four-fold rotationally symmetric pillar structure is shown in Fig. 11. Fig. 11 is a diagram showing an example of a four-fold rotationally symmetric pillar structure. Note that the pillar structure in the optical element 12 may be replaced with a member embedded in the transparent substrate 11, or a hole or groove provided in the transparent substrate 11.

[0046] [Other Configuration Examples] As shown in Fig. 12, the height of the convex portions relative to the surface of the convex portions may be constant. In the example of Fig. 12, the height (length in the positive direction of the z-axis) of the convex portions of the optical element 12a in the imaging system 10a is constant at h. This simplifies the manufacturing process compared to when the heights of the convex portions are different. Note that the heights of the convex portions of the imaging system 10a and the optical element 12a are different from those of the imaging system 10 and the optical element 12.

[0047] As shown in FIG. 13 , an imaging element 14 having a curved surface may be provided between the optical element 12a and the image sensor 13. The imaging element 14 is, for example, a convex lens. In this case, instead of the optical element 12a, the imaging element 14 performs the light-collecting function, while the optical element 12a performs the wavelength dispersion control function, thereby separating the functions into the respective elements. This reduces the structural complexity of the optical element 12a and makes it easier to manufacture. Note that the imaging element 14 may be provided closer to the subject (in the negative direction of the z-axis) than the optical element 12a and the image sensor 13.

[0048] [Effects of the embodiment] As described above, the imaging device 1 of this embodiment can acquire images with equal vertical and horizontal resolutions using a sensor, and reconstruct spectral images with equal vertical and horizontal resolutions. Furthermore, as a secondary effect, the amount of PSF data required is approximately 1 / (4N) compared to conventional methods, thereby reducing the amount of inverse matrix calculations required in image reconstruction processing. Note that N is the rotation angle at which the structure pattern is rotationally symmetric divided by 90 degrees.

[0049] In the example of Fig. 7, the structure pattern is rotationally symmetric when the rotation angle is 90 degrees, but is not rotationally symmetric when the rotation angle is less than 90 degrees, so N is 1. In this case, the amount of PSF data is about 1 / 4 of that in the conventional case.

[0050] 8, the structure pattern is rotationally symmetric regardless of the rotation angle, so N is infinite. In this case, the amount of data in the PSF is extremely small.

[0051] 8 is replaced with a regular dodecagon, the structure pattern becomes rotationally symmetric at rotation angles of 30 degrees, 60 degrees, 90 degrees, ..., 330 degrees, and 360 degrees, i.e., at intervals of 30 degrees, so N is 3. In this case, the amount of PSF data is about 1 / 12 of that in the past.

[0052] The configuration of the imaging system 10 of this embodiment may be the same as the configuration of the imaging device disclosed in Reference 1 or Reference 2, except for the configuration of the optical element 12. Reference 1: WO 2022 / 162801 Reference 2: WO 2022 / 162800

[0053] Here, the calculator 20 may be a computer provided in the imaging system 10, or may be a device such as a server or PC external to the imaging system 10.

[0054] When the computer 20 is an external device, a communication unit (e.g., a communication module) of the imaging system 10 transmits data of the image 32 to the computer 20 via a communication network. The computer 20 then performs image reconstruction processing based on the data of the image 32 received from the imaging system 10. Furthermore, the computer 20 transmits data of the spectral image 33 obtained by the image reconstruction processing to the imaging system 10. The communication unit receives the data of the spectral image 33.

[0055] The image reconstruction process imposes a large processing load. For this reason, the computing resources of the imaging system 10, which is, for example, a portable camera, may not be sufficient to efficiently execute the image reconstruction process. In contrast, if the computer 20 is a device with abundant computing resources, such as a cloud server, the image reconstruction process can be efficiently executed using the above method.

[0056] [Program] In one embodiment, the computer 20 can be implemented by installing a program that executes the above-described processing as package software or online software on a desired computer. The computer 20 includes a desktop or notebook personal computer. Other examples of the computer include a smartphone, a tablet terminal, and the like.

[0057] 14 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0058] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0059] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the computer 20 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the computer 20 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD.

[0060] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0061] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0062] REFERENCE SIGNS LIST 1 imaging device 10 imaging system 11 transparent substrate 12 optical element 13 image sensor 20 computer 31, 33 spectral image 35 PSF 37 reconstruction processing

Claims

1. An imaging device having an optical element including a surface with concave and convex portions, and an image sensor provided opposite said surface, wherein the imaging device generates, by computer, a spectral image having more bands than the number of bands of an image obtained by said image sensor, wherein said optical element focuses incident light in a different focusing pattern for each component, and guides the resulting light to said image sensor, and the structural pattern of said concave and convex portions is four-fold rotationally symmetric with respect to an axis perpendicular to said surface.

2. The imaging device according to claim 1, characterized in that the convex portion includes a columnar structure having a base in the form of a ring arranged concentrically on the surface with the intersection of the axis and the surface as its center.

3. The imaging device according to claim 1, characterized in that the structural pattern of the recesses and protrusions is a relief structure arranged periodically, and the spacing between the relief structures is smaller than the wavelength of a specific light.

4. The imaging device according to claim 3, wherein each of the concave-convex structures has four-fold rotational symmetry with respect to an axis perpendicular to the surface.

5. The imaging device according to claim 1, characterized in that the height of the convex portion or the concave portion relative to the surface is constant.

6. The imaging device according to claim 1, further comprising an imaging element having a curved surface, which is provided between the optical element and the image sensor or on the subject side of the optical element and the image sensor.

7. An imaging device according to any one of claims 1 to 6, characterized in that image data obtained by the image sensor of said imaging device is transmitted to an external computer, and a spectral image having more bands than the number of bands of the image obtained by said image sensor is generated.

Images