Filter array and imaging system

Abstract

This filter array comprises a plurality of two-dimensionally arranged optical filters. The plurality of optical filters include a first filter and second filter. The first filter comprises: a first multimode filter having, within a given wavelength band, a plurality of first peak wavelengths at which light transmittance is at the maximum value; and a first band limiting filter having, as the limited band, a first sub wavelength band that is a part of the given wavelength band. The second filter comprises: a second multimode filter having, within the given wavelength band, a plurality of second peak wavelengths at which light transmittance is at the maximum value, at least one of the plurality of second peak wavelengths being different from the plurality of first peak wavelengths; and a second band limiting filter that limits the transmission of light in a second sub wavelength band that is a part of the given wavelength band and is different from the first sub wavelength band.

Description

[Technical Field]

[0001] This disclosure relates to a filter array and an imaging system. [Background technology]

[0002] By utilizing spectral information from numerous narrow-band frequencies, such as tens of bands, it becomes possible to understand the detailed physical properties of an object in a way that was impossible with conventional RGB images. Cameras that acquire such multi-wavelength information are called "hyperspectral cameras." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Document 1 discloses an example of a hyperspectral imaging device utilizing compressed sensing. The imaging device comprises an encoding element which is an array of optical filters whose light transmittances have different wavelength dependencies, an image sensor which detects light transmitted through the encoding element, and a signal processing circuit. The encoding element is placed on the optical path connecting the subject and the image sensor. The image sensor acquires a single wavelength-division multiplexed image by simultaneously detecting light with superimposed components of multiple wavelength bands for each pixel. The signal processing circuit generates image data for each of the multiple wavelength bands by applying compressed sensing to the acquired wavelength-division multiplexed image using information on the spatial distribution of the spectral transmittance of the encoding element. In the imaging device disclosed in Patent Document 1, a filter array having two or more transmittance peaks (i.e., maximum values) within the target wavelength range is used as the encoding element.

[0004] Patent Document 2 discloses an example of a filter array equipped with a Fabry-Perot resonator using dielectric multilayer films for each reflective layer. Patent Document 3 discloses the structure of an optical filter using a diffraction grating. Patent Document 4 discloses the structure of an optical filter utilizing surface plasmon resonance. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Patent No. 9,599,511 [Patent Document 2] U.S. Patent No. 9,466,628 [Patent Document 3] Japanese Patent Application Laid-Open No. 2008-070427 [Patent Document 4] Japanese Patent Application Laid-Open No. 2010-008990 [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] The present disclosure provides a technique for acquiring hyperspectral information with less error over a wider wavelength range than in the prior art. [Means for Solving the Problems]

[0007] A filter array according to one aspect of the present disclosure includes a plurality of optical filters arranged two-dimensionally. The plurality of optical filters includes a first filter and a second filter. The first filter includes a first multimode filter having a plurality of first peak wavelengths at which the light transmittance exhibits a maximum value within a target wavelength range, and a first band-limiting filter that restricts the transmission of light in a first sub-wavelength range that is part of the target wavelength range. The second filter includes a second multimode filter having a plurality of second peak wavelengths at which the light transmittance exhibits a maximum value within the target wavelength range, at least one of the plurality of second peak wavelengths being different from the plurality of first peak wavelengths, and a second band-limiting filter that restricts the transmission of light in a second sub-wavelength range that is part of the target wavelength range and different from the first sub-wavelength range.

[0008] The comprehensive or specific embodiments of this disclosure may be implemented as systems, apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable discs, or as any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. Computer-readable recording media may include, for example, non-volatile recording media such as CD-ROMs (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. If an apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. In this specification and in the claims, “apparatus” may mean not only a single device but also a system consisting of multiple devices. [Effects of the Invention]

[0009] According to one aspect of this disclosure, it becomes possible to acquire hyperspectral information with smaller errors over a wider wavelength range than conventional methods. [Brief explanation of the drawing]

[0010] [Figure 1A] Figure 1A is a schematic diagram illustrating an exemplary imaging system. [Figure 1B] Figure 1B shows another example of an imaging system. [Figure 1C] Figure 1C shows yet another example of an imaging system. [Figure 1D] Figure 1D shows yet another example of an imaging system. [Figure 2A] Figure 2A is a schematic diagram showing an example of a filter array. [Figure 2B] Figure 2B shows an example of the spatial distribution of light transmittance for each of the multiple wavelength bands included in the target wavelength range in a filter array. [Figure 2C] Figure 2C shows an example of the spectral transmittance of region A1 included in the filter array shown in Figure 2A. [Figure 2D]Figure 2D shows an example of the spectral transmittance of region A2 included in the filter array shown in Figure 2A. [Figure 3A] Figure 3A is a diagram illustrating the relationship between the target wavelength range W and the multiple wavelength bands contained within it. [Figure 3B] Figure 3B is a diagram illustrating the relationship between the target wavelength range W and the multiple wavelength bands contained within it. [Figure 4A] Figure 4A is a diagram illustrating the spectral transmittance characteristics in a certain region of the filter array. [Figure 4B] Figure 4B shows the spectral transmittances shown in Figure 4A, averaged for each wavelength band. [Figure 5] Figure 5 is a schematic diagram showing a portion of the cross-section of an imaging device in one example. [Figure 6] Figure 6 schematically shows an example of transmission spectra in multiple pixels. [Figure 7] Figure 7 is a schematic cross-sectional view showing an example of a filter structure in which each reflective layer is formed from a dielectric multilayer film. [Figure 8] Figure 8 schematically shows another example of a filter in which each of the reflective layers is formed from a dielectric multilayer film. [Figure 9A] Figure 9A shows examples of transmission spectra for eight different filters included in the filter array. [Figure 9B] Figure 9B is a magnified view of the transmission spectrum in the wavelength range of 400 nm to 600 nm shown in Figure 9A. [Figure 10] Figure 10 shows an example of the error in the estimation calculation of hyperspectral images. [Figure 11] Figure 11 shows a schematic configuration of the filter array. [Figure 12] Figure 12 shows examples of transmission spectra for the first and second multimode filters. [Figure 13] Figure 13 shows examples of transmission spectra for the first and second band-limiting filters. [Figure 14] Figure 14 is a schematic cross-sectional view showing another example of a filter array. [Figure 15] Figure 15 shows examples of the limited bandwidths for each of the four types of bandwidth limiting filters. [Figure 16A] Figure 16A shows an example of the arrangement of a bandwidth limiting filter. [Figure 16B] Figure 16B shows another example of the arrangement of a bandwidth-limiting filter. [Figure 17] Figure 17 is a cross-sectional view showing an example of a more detailed structure of the filter array. [Figure 18A] Figure 18A shows an example of the transmission spectrum of a band-limiting filter. [Figure 18B] Figure 18B shows another example of the transmission spectrum of a band-limiting filter. [Figure 18C] Figure 18C shows another example of the transmission spectrum of a band-limiting filter. [Figure 19A] Figure 19A shows an example of the transmission spectrum of a filter array without a band-limiting filter. [Figure 19B] Figure 19B shows an example of the transmission spectrum of a filter array equipped with one type of band-limiting filter. [Figure 19C] Figure 19C shows an example of the transmission spectrum of a filter array equipped with three types of band-limiting filters. [Figure 19D] Figure 19D is a graph showing the standard deviation of the transmittance of the filter array 110 at each wavelength. [Figure 20A] Figure 20A shows the calculation results of the least squares error (MSE) of the reconstructed images for each band. [Figure 20B] Figure 20B shows the calculation results of the least squares error (MSE) of the reconstructed images for each band. [Figure 21] Figure 21 is a schematic cross-sectional view showing another example of a filter array. [Figure 22] Figure 22 is a schematic cross-sectional view showing yet another example of a filter array. [Figure 23] Figure 23 is a schematic cross-sectional view showing yet another example of a filter array. [Figure 24] Figure 24 is a schematic cross-sectional view showing yet another example of a filter array. [Figure 25] Figure 25 is a schematic cross-sectional view showing yet another example of a filter array. [Figure 26A] Figure 26A shows an example of a transmission spectrum of a filter array equipped with a band-limiting filter having a rectangular transmission spectrum with 0% transmittance in the limited band. [Figure 26B] Figure 26B shows an example of a transmission spectrum of a filter array equipped with a band-limiting filter having a rectangular transmission spectrum with a transmittance of 60% in the limited band. [Figure 26C] Figure 26C shows an example of the transmission spectrum of a filter array without a band-limiting filter. [Figure 27] Figure 27 is a graph showing an example of the relationship between the mean squared error (MSE) between the estimated image and the ground truth image, the average value for multiple bands, and the transmittance in the limiting band. [Modes for carrying out the invention]

[0011] The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement, position and connection configurations of components, steps, and order of steps shown in the embodiments below are examples and are not intended to limit the art of this disclosure. Components in the embodiments below that are not described in the independent claim representing the highest-level concept are described as optional components. The figures are schematic and not necessarily strictly illustrative. Furthermore, in each figure, substantially identical or similar components are denoted by the same reference numerals. Duplication of explanation may be omitted or simplified.

[0012] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large-scale integration (LSI). The LSI or IC may be integrated on a single chip or may be composed of multiple chips combined. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSIs or ICs, but the name may change depending on the degree of integration, and they may also be called system LSIs, VLSIs (very large-scale integrations), or ULSIs (ultra-large-scale integrations). Field-programmable gate arrays (FPGAs) that are programmed after the manufacture of the LSI, or reconfigurable logic devices that allow for the reconfiguration of junction relationships within the LSI or the setup of circuit compartments within the LSI, can also be used for the same purpose.

[0013] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROMs, optical disks, or hard disk drives, and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0014] (Embodiment) The following describes an example configuration of the imaging system according to the embodiment of this disclosure.

[0015] <Imaging System> Figure 1A is a schematic diagram illustrating an exemplary hyperspectral imaging system. This system comprises an imaging device 100 and a signal processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device 100 comprises an optical system 140, a filter array 110, and an image sensor 160. The filter array 110 has a structure and function similar to the "encoding element" disclosed in Patent Document 1. The optical system 140 and the filter array 110 are arranged on the optical path of light incident from the object 70, which is the subject. The filter array 110 is positioned between the optical system 140 and the image sensor 160.

[0016] Figure 1A shows an apple as an example of the object 70. The object 70 is not limited to an apple; it can be any object. The image sensor 160 generates compressed image 120 data, in which information from multiple wavelength bands is compressed as a two-dimensional monochrome image. The signal processing device 200 generates image data for each of the multiple wavelength bands included in a predetermined wavelength range (hereinafter referred to as the "target wavelength range") based on the compressed image 120 data generated by the image sensor 160. This generated image data of multiple wavelength bands is referred to as "hyperspectral image data" in this specification. Here, the number of wavelength bands included in the target wavelength range is N (N is an integer of 4 or more). In the following description, the generated image data of multiple wavelength bands will be referred to as hyperspectral images 220W1, 220W2, 220W3, ..., 220W N These are sometimes referred to as hyperspectral images 220. In this specification, the signals that represent an image, that is, the set of signals that represent the pixel value of each pixel, may be simply referred to as an "image".

[0017] The filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include several types of filters whose spectral transmittance, i.e., wavelength dependence of light transmittance, differs from one another. The filter array 110 modulates the intensity of the incident light for each wavelength and outputs it. This process by the filter array 110 is referred to as "coding" in this specification.

[0018] As shown in Figure 1A, the filter array 110 may be positioned near or directly above the image sensor 160. Here, "nearby" means that it is close enough that an image of light from the optical system 140 is formed on the surface of the filter array 110 in a reasonably clear state. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0019] The optical system 140 includes at least one lens. In Figure 1A, the optical system 140 is shown as a single lens, but the optical system 140 may be a combination of multiple lenses. The optical system 140 forms an image on the imaging plane of the image sensor 160 via the filter array 110.

[0020] The filter array 110 may be positioned away from the image sensor 160. Figures 1B to 1D show examples of the configuration of an imaging device 100 in which the filter array 110 is positioned away from the image sensor 160. In the example in Figure 1B, the filter array 110 is positioned between the optical system 140 and the image sensor 160, but away from the image sensor 160. In the example in Figure 1C, the filter array 110 is positioned between the object 70 and the optical system 140. In the example in Figure 1D, the imaging device 100 comprises two optical systems 140A and 140B, with the filter array 110 positioned between them. As in these examples, an optical system including one or more lenses may be positioned between the filter array 110 and the image sensor 160.

[0021] The image sensor 160 is a monochrome type photodetector having a plurality of photodetectors (also referred to herein as "pixels") arranged in two dimensions. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. Depending on the application, a terahertz array sensor or a millimeter-wave array sensor may be used. The photodetectors include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may be used. By using a color type sensor, the amount of information regarding wavelength can be increased, and the accuracy of reconstructing the hyperspectral image 220 can be improved. The wavelength range to be acquired can be arbitrarily determined and is not limited to the visible wavelength range, but may also include ultraviolet, near-infrared, mid-infrared, far-infrared, microwave, and radio wave wavelength ranges.

[0022] The signal processing device 200 is a computer comprising a processor and a storage medium such as memory. The signal processing device 200 may be connected to the imaging device 100 by wire or wireless. Based on the compressed image 120 acquired by the image sensor 160, the signal processing device 200 generates multiple hyperspectral images 220W1, 220W2, 220W3, ... 220W, each containing information for multiple wavelength bands. N The data is generated. Details of the processing by the signal processing device 200 will be described later. The signal processing device 200 may be incorporated into the imaging device 100. The signal processing device 200 may be connected to the imaging device 100 via a network such as the Internet.

[0023] <Filter Array> Figure 2A is a schematic diagram showing an example of a filter array 110. The filter array 110 has multiple regions arranged in two dimensions. In this specification, these regions may be referred to as "cells". Each region is equipped with an optical filter having an individually set spectral transmittance. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take values ​​between 0 and 1, inclusive.

[0024] In the example shown in Figure 2A, the filter array 110 has 48 rectangular regions arranged in a 6x8 grid. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be, for example, roughly equivalent to the number of pixels in the image sensor 160. The number of filters included in the filter array 110 is determined according to the application, for example, ranging from tens of thousands to tens of millions.

[0025] Figure 2B shows multiple wavelength bands W1, W2, ..., W included in the target wavelength range in the filter array 110. N This figure shows an example of the spatial distribution of light transmittance for each wavelength. In the example shown in Figure 2B, the difference in intensity in each region represents the difference in transmittance. Lighter regions have higher transmittance, and darker regions have lower transmittance. As shown in Figure 2B, the spatial distribution of transmittance differs depending on the wavelength band.

[0026] Figures 2C and 2D show examples of spectral transmittances for regions A1 and A2, respectively, included in the filter array 110 shown in Figure 2A. The spectral transmittances of region A1 and region A2 are different from each other. Thus, the spectral transmittance of the filter array 110 differs by region. However, it is not necessary for the spectral transmittances of all regions to be different. In the filter array 110, the spectral transmittances of at least some of the regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns for multiple regions included in the filter array 110 may be equal to or greater than the number of wavelength bands N included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0027] Figures 3A and 3B show the target wavelength range W and the multiple wavelength bands W1, W2, ..., W contained within it. N This diagram illustrates the relationship. The target wavelength range W can be set to various ranges depending on the application. For example, the target wavelength range W may be the visible light wavelength range from about 400 nm to about 700 nm, the near-infrared wavelength range from about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range from about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be the radio wave range, such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. Thus, the wavelength range used is not limited to the visible light range. In this specification, not only visible light but also non-invisible light such as near-ultraviolet, near-infrared, and radio waves will be referred to as "light" for convenience. In the following description, the shortest wavelength in the target wavelength range W will be λ1 and the longest wavelength will be λ2.

[0028] In the example shown in Figure 3A, N is any integer greater than or equal to 4, and the target wavelength range W is divided into N equal parts, with each wavelength range being designated as wavelength bands W1, W2, ..., W NIt is assumed that. However, it is not limited to such examples. The plurality of wavelength bands included in the target wavelength range W may be arbitrarily set. For example, the bandwidth may be made non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidths are different depending on the wavelength bands, and there is a gap between two adjacent wavelength bands. Thus, the plurality of wavelength bands only need to be different from each other, and the determination method is arbitrary.

[0029] FIG. 4A is a diagram for explaining the spectral transmittance characteristics in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has a plurality of maximum values P1 to P5 and a plurality of minimum values with respect to the wavelengths within the target wavelength range W. In the example shown in FIG. 4A, it is normalized so that the maximum value of the light transmittance is 1 and the minimum value is 0 within the target wavelength range W. In the example shown in FIG. 4A, in wavelength ranges such as the wavelength band W2 and the wavelength band W N-1 etc., the spectral transmittance has a maximum value. Thus, the spectral transmittance of each region has a maximum value in at least two of the plurality of wavelength ranges W1 to W N among them. In the example of FIG. 4A, the maximum values P1, P3, P4, and P5 are 0.5 or more.

[0030] As described above, the light transmittance of each region varies depending on the wavelength. Therefore, each region in the filter array 110 transmits more components of a certain wavelength range of the incident light and does not transmit components of other wavelength ranges so much. For example, for the light of k wavelength bands out of N wavelength bands, the transmittance may be greater than 0.5, and for the light of the remaining N - k wavelength ranges, the transmittance may be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that evenly contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of discrete intensity peaks with respect to the wavelength for each region, and superimposes these multi-wavelength lights and makes them incident on the image sensor 160.

[0031] Figure 4B shows, as an example, the spectral transmittance shown in Figure 4A, for wavelength bands W1, W2, ..., W N This figure shows the results after averaging for each wavelength band. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the transmittance value averaged for each wavelength band in this way is referred to as the transmittance for that wavelength band. In this example, the transmittance is exceptionally high in three wavelength bands where the maximum values ​​are P1, P3, and P5. In particular, the transmittance exceeds 0.8 in the two wavelength bands where the maximum values ​​are P3 and P5.

[0032] The chromatic resolution of the transmission spectrum of each filter in the filter array 110 can be set to approximately the bandwidth of a desired wavelength band. For example, the width of the range within the wavelength range containing a single maximum value in the transmission spectrum curve that takes a value greater than or equal to the average value of the minimum value closest to that maximum value and that maximum value can be set to approximately the bandwidth of a desired wavelength band. In this case, if the transmission spectrum is decomposed into frequency components, for example by a Fourier transform, the values ​​of the frequency components corresponding to that wavelength range will be relatively large.

[0033] The filter array 110 typically has multiple filters divided into a grid, as shown in Figure 2A. Some or all of these filters have different transmission spectra. The wavelength and spatial distribution of the light transmittance of the multiple filters included in the filter array 110 may be, for example, a random distribution or a quasi-random distribution.

[0034] The concepts of random and quasi-random distributions are as follows. First, each filter in the filter array 110 can be considered as a vector element having a value from, for example, 0 to 1, depending on its light transmittance. Here, when the transmittance is 0, the value of the vector element is 0, and when the transmittance is 1, the value of the vector element is 1. In other words, a set of filters arranged in a row or column can be considered as a multidimensional vector having a value from 0 to 1. Therefore, it can be said that the filter array 110 has multiple multidimensional vectors in the column or row direction. In this case, a random distribution means that any two multidimensional vectors are independent, that is, they are not parallel. A quasi-random distribution means that some of the multidimensional vectors are not independent of each other. Therefore, in random and quasi-random distributions, a vector whose elements are the transmittance values ​​of light in a certain wavelength band for each filter belonging to a set of filters arranged in one row or column that is included in a set of filters, and a vector whose elements are the transmittance values ​​of light in the same wavelength band for each filter belonging to another set of filters arranged in another row or column, are independent of each other. Similarly, for wavelength bands other than those mentioned above, the vector whose elements are the transmittance values ​​of light in those other wavelength bands for each filter belonging to a set of filters arranged in one row or column within a set of filters, and the vector whose elements are the transmittance values ​​of light in those other wavelength bands for each filter belonging to another set of filters arranged in another row or column, are independent of each other.

[0035] When the filter array 110 is placed near or directly above the image sensor 160, the spacing between the multiple filters included in the filter array 110 may be approximately equal to the pixel pitch of the image sensor 160. In this way, the resolution of the encoded light image emitted from the filter array 110 will approximately match the resolution of the pixels. By ensuring that the light transmitted through each filter is incident on only one corresponding pixel, the calculations described later can be made easier. When the filter array 110 is placed far away from the image sensor 160, the filter pitch may be made finer depending on the distance.

[0036] In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value from 0 to 1. However, it is not necessarily a grayscale transmittance distribution. For example, a binary scale transmittance distribution in which the transmittance of each region can take either a value close to 0 or a value close to 1 may be adopted. In the binary scale transmittance distribution, each region transmits most of the light in at least two wavelength regions out of a plurality of wavelength regions included in the target wavelength region, and does not transmit most of the light in the remaining wavelength regions. Here, "most" means generally 80% or more.

[0037] A part of the entire filter, for example, half of the filter, may be replaced with a transparent region. Such a transparent region transmits light from all wavelength bands W1 to W included in the target wavelength region W with a similarly high transmittance, for example, a transmittance of 80% or more. In such a configuration, the plurality of transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the plurality of regions in the filter array 110, regions with different light transmittances depending on the wavelength and transparent regions may be alternately arranged N . Data indicating the spatial distribution of the spectral transmittance of such a filter array 110 can be acquired in advance based on design data or measured calibration and stored in a storage medium provided in the signal processing device 200. This data is used for the arithmetic processing described later.

[0038]

[0039] ​Each filter in the filter array 110 has multiple peak wavelengths that exhibit maximum values ​​within the target wavelength range W. In the following description, such filters may be referred to as "multimode filters." Multimode filters can be constructed using microstructures that include, for example, a multilayer film, an organic material, a diffraction grating structure, and a metal. When using multilayer films, for example, a dielectric multilayer film or a multilayer film containing a metal layer may be used. In this case, each filter is formed so that at least one of the thickness, material, and stacking order of each multilayer film is different. This allows different spectral characteristics to be achieved depending on the filter. By using multilayer films, a sharp rise and fall in spectral transmittance can be achieved. Configurations using organic materials can be achieved by having different pigments or dyes contained in each filter, or by stacking different materials. Configurations using diffraction grating structures can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each filter. When using microstructures containing metals, they can be fabricated using spectroscopy by the plasmon effect.

[0040] <Example of signal processing> Next, an example of signal processing by the signal processing device 200 will be described. The signal processing device 200 reconstructs a multi-wavelength hyperspectral image 220 based on the compressed image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, "multi-wavelength" means more wavelength ranges than, for example, the three RGB wavelength ranges acquired by a normal color camera. The number of these wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is called the number of bands. Depending on the application, the number of bands may exceed 100.

[0041] The data we want is the data from the hyperspectral image 220, and we will call this data f. If the number of bands is N, then f is the image data for each band f1, f2, ..., f NThis is integrated data. Here, as shown in Figure 1A, the horizontal direction of the image is the x-direction, and the vertical direction of the image is the y-direction. If the number of pixels in the x-direction of the image data to be obtained is n, and the number of pixels in the y-direction is m, then the image data f1, f2, ..., f N Each of these is two-dimensional data with n × m pixels. Therefore, data f is three-dimensional data with n × m × N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube". On the other hand, the number of elements in data g of the compressed image 120, which is obtained by encoding and multiplexing by the filter array 110, is n × m. Data g can be expressed by the following equation (1).

number

[0042] Here, f1, f2, ..., f N Each of these is data with n × m elements. Therefore, the vector on the right side is strictly an n × m × N x 1 1 dimensional vector. Vector g is transformed into an n × m x 1 1 dimensional vector and represented and computed. Matrix H is the vector f, with each component f1, f2, ..., f N This represents a transformation that encodes and intensity modulates the signal using different encoding information (also called "mask information") for each wavelength band, and then adds them together. Therefore, H is an n×m row n×m×N column matrix.

[0043] Given a vector g and a matrix H, it appears that f can be calculated by solving the inverse problem of equation (1). However, since the number of elements n × m × N of the data f to be obtained is greater than the number of elements n × m of the acquired data g, this problem is poorly set up and cannot be solved as is. Therefore, the signal processing device 200 uses the redundancy of the image contained in the data f and employs a compressed sensing technique to find the solution. Specifically, the data f to be obtained is estimated by solving the following equation (2).

number

[0044] Here, f' represents the estimated data of f. The first term in parentheses in the above equation represents the difference between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares may also be used as the residual term. The second term in parentheses is the regularization term or stabilization term. Equation (2) means finding the f that minimizes the sum of the first and second terms. The signal processing device 200 can converge the solution through recursive iterative operations and calculate the final solution f'.

[0045] The first term in parentheses in equation (2) represents the operation of calculating the sum of squares of the differences between the acquired data g and Hf, which is obtained by transforming the estimation process f by matrix H. The second term, Φ(f), is a constraint in the regularization of f and is a function that reflects the sparse information of the estimated data. This function has the effect of making the estimated data smoother or more stable. The regularization term can be represented by, for example, the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation allows for obtaining stable estimated data that suppresses the effects of noise in the observed data g. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term may be selected that makes the texture of the object 70 more sparse in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the reduction of redundant data and the higher the compression rate. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to a moderate value that allows f to converge to a certain extent without overcompression.

[0046] In the configurations shown in Figures 1B and 1C, the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, by storing this blur information in advance and reflecting it in the matrix H mentioned above, the hyperspectral image 220 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree to which a point image spreads to surrounding pixels. For example, if a point image corresponding to one pixel on the image spreads to a k×k pixel region around that pixel due to blurring, the PSF can be defined as a set of coefficients, i.e., a matrix, that shows the influence on the brightness of each pixel in that region. By reflecting the effect of blurring of the encoding pattern due to the PSF in the matrix H, the hyperspectral image 220 can be reconstructed. The position where the filter array 110 is placed is arbitrary, but a position can be selected where the encoding pattern of the filter array 110 does not spread too much and disappear.

[0047] Although an example calculation using compressed sensing as shown in equation (2) is presented here, other methods may also be used to solve the problem. For example, other statistical methods such as maximum likelihood estimation or Bayesian estimation can be used. Also, the number of hyperspectral images 220 is arbitrary, and each wavelength range may be set arbitrarily. Details of the reconstruction method are disclosed in Patent Document 1. The entirety of the disclosure in Patent Document 1 is incorporated herein by reference.

[0048] <Filter array equipped with Fabry-Perot filters> Next, with reference to Figure 5, an example of the specific structure of the filter array 110 will be described. Figure 5 is a schematic diagram showing a part of the cross-section of an imaging device 100 in one example. The imaging device 100 comprises a filter array 110 and an image sensor 160. The filter array 110 includes a plurality of filters 112 arranged in two dimensions. The plurality of filters 112 may be arranged in rows and columns. Figure 5 schematically shows a part of the cross-sectional structure of one row.

[0049] In the example shown in Figure 5, the filter array 110 is located on the surface of the image sensor 160. Each of the multiple photodetectors 162 included in the image sensor 160 is located directly beneath the corresponding filter 112 among the multiple filters 112 included in the filter array 110. The filter array 110 and the image sensor 160 may be separated. Even in this case, each of the multiple photodetectors 162 may be positioned to receive light transmitted through one of the multiple filters 112. The components may be arranged such that light transmitted through the multiple filters 112 is incident on the corresponding multiple photodetectors 162 via mirrors (not shown).

[0050] Each filter 112 in the filter array 110 shown in Figure 5 is a Fabry-Perot (FP) filter. An FP filter comprises two opposing first reflective layers 28a and second reflective layers 28b, and an interference layer 26 (also referred to as an "intermediate layer") between them. The interference layer 26 has a thickness and refractive index that forms a resonant structure having one or more resonant modes. The transmittance of light of wavelengths corresponding to the resonant modes becomes high, and the transmittance of light of other wavelengths becomes low. By changing the refractive index or thickness of the interference layer 26 for each filter, different transmission spectra can be achieved for each filter.

[0051] Each of the multiple filters 112 has a resonant structure. A resonant structure is a structure in which light of a certain wavelength exists stably by forming a standing wave inside. This state of light is called a "resonant mode". In the example shown in Figure 5, the resonant structure is composed of a first reflective layer 28a, a second reflective layer 28b, and an interference layer 26. The first reflective layer 28a and the second reflective layer 28b can each be formed from, for example, a dielectric multilayer film or a metal thin film. The interference layer 26 can be formed from a dielectric or semiconductor that is substantially transparent in the target wavelength range. The interference layer 26 can be formed from, for example, at least one selected from the group consisting of Si, Si3N4, TiO2, Nb2O5, and Ta2O5. At least one of the refractive index and thickness of the interference layer 26 differs depending on the filter 112. The transmission spectrum of each of the multiple filters 112 has a maximum transmittance value at multiple wavelengths. These multiple wavelengths correspond to multiple resonant modes of different orders in the resonant structure described above.

[0052] All filters 112 in the filter array 110 may have the above-described resonant structure, or the filter array 110 may include filters that do not have a resonant structure. For example, the filter array 110 may include filters that do not have wavelength dependence of light transmittance, such as transparent filters or ND filters (Neutral Density Filters).

[0053] Each of the multiple photodetectors 162 in the image sensor 160 is sensitive to light in a preset target wavelength range W. In this disclosure, "sensitive to" light in a certain wavelength range means having substantial sensitivity necessary to detect light in that wavelength range. For example, it means that the external quantum efficiency in that wavelength range is 1% or more. The external quantum efficiency of the photodetector 162 in that wavelength range may be 10% or more, or 20% or more. The multiple wavelengths in which the light transmittance of each filter 112 takes its maximum value are all included in the target wavelength range W.

[0054] In this specification, the filter 112 having the resonant structure described above is referred to as a "Fabry-Perot filter." A Fabry-Perot filter is a type of interference filter. Other types of interference filters, such as color separation filters composed of diffraction gratings, may be used instead of a Fabry-Perot filter. In this specification, the portion of the transmission spectrum having a maximum value is referred to as a "peak," and the wavelength at which the light transmittance shows a maximum value within the target wavelength range W is referred to as the "peak wavelength."

[0055] In filter 112, the thickness of the interference layer 26 is L, the refractive index is n, and the angle of incidence of light incident on filter 112 is θ. i Let m be the mode order of the resonant mode. m is an integer greater than or equal to 1. At this time, the peak wavelength λ of the transmission spectrum of filter 112 is m This is expressed by the following equation (3).

number

[0056] Within the target wavelength range W, the shortest wavelength is defined as λ1 and the longest wavelength as λ2. In this specification, λ1 ≤ λ m A filter 112 for which there is exactly one m satisfying ≤ λ2 is called a "single-mode filter", where λ1 ≤ λ m A filter 112 for which there are two or more m values ​​satisfying ≤λ2 is called a "multimode filter".

[0057] The following describes an example of a structure where the shortest wavelength in the target wavelength range W is λ1 = 400 nm and the longest wavelength is λ2 = 700 nm. The thickness of the interference layer 26 is L = 300 nm, the refractive index of the interference layer 26 is n = 1.0, and θ i When m=0° (i.e., normal incidence), the peak wavelength is 600 nm when m=1, and the peak wavelength is 300 nm or less when m≧2. Therefore, in this case, filter 112 is a single-mode filter in which one peak wavelength is contained within the target wavelength range W.

[0058] On the other hand, if the thickness L is further increased, multiple peak wavelengths will be included within the target wavelength range W. For example, L=3000nm, n=1.0, θ i When = 0, the peak wavelength is 750 nm or greater when 1 ≤ m ≤ 8, the peak wavelength is between 400 nm and 700 nm when 9 ≤ m ≤ 15, and the peak wavelength is 375 nm or less when m ≥ 16. Therefore, in this case, filter 112 is a multimode filter that contains seven peak wavelengths within the target wavelength range W.

[0059] As described above, a multimode filter can be realized by appropriately designing the thickness of the interference layer 26 of the filter 112. Instead of adjusting the thickness of the interference layer 26, the refractive index of the interference layer 26 of the filter 112 may be adjusted. Alternatively, both the thickness and refractive index of the interference layer 26 of the filter 112 may be adjusted.

[0060] Figure 6 schematically shows an example of the transmission spectrum at each pixel when multiple multimode filters, each with a different transmission spectrum, are placed on multiple pixels. Figure 6 illustrates several peaks in the transmission spectra at pixels A, B, and C, and their mode orders m, m+1, and m+2. As illustrated, multiple multimode filters can be designed so that each pixel has a slightly different peak wavelength. Such a design can be achieved by slightly varying the thickness L and / or refractive index n in equation (3). The imaging device 100 can simultaneously detect light with multiple peak wavelengths that differ from pixel to pixel.

[0061] Each of the first reflective layer 28a and the second reflective layer 28b may have, for example, a distributed Bragg reflector (DBR) structure. In that case, each of the first reflective layer 28a and the second reflective layer 28b may be formed from a dielectric multilayer film. Several examples of such filter 112 structures are described below.

[0062] Figure 7 is a schematic cross-sectional view showing an example of the structure of a filter 112 in which the first reflective layer 28a and the second reflective layer 28b are each formed from a dielectric multilayer film. In this example, the filter 112 is provided on a substrate 80. Each of the first reflective layer 28a and the second reflective layer 28b has a structure in which a plurality of low refractive index layers 27l and a plurality of high refractive index layers 27h are alternately stacked. Each of the plurality of low refractive index layers 27l has a refractive index n l It has a plurality of high refractive index layers 27h, each of which has a refractive index n l Higher refractive index n h The low refractive index layer 27l in the first reflective layer 28a and the low refractive index layer 27l in the second reflective layer 28b may have the same refractive index or different refractive indices. Similarly, the high refractive index layer 27h in the first reflective layer 28a and the high refractive index layer 27h in the second reflective layer 28b may have the same refractive index or different refractive indices.

[0063] The dielectric multilayer film shown comprises multiple paired layers. Each paired layer includes one low refractive index layer 27l and one high refractive index layer 27h. In the example shown in Figure 7, each of the first reflective layer 28a and the second reflective layer 28b comprises five paired layers. To obtain a high reflectivity for a specific wavelength λ0 within the target wavelength range W, the thickness of the high refractive index layer 27h is t h =λ0 / (4n h The thickness of the low refractive index layer 27l is set to t l =λ0 / (4n l ) can be set to ). In other words, the thickness t of the high refractive index layer 27h h The optical length and the thickness t of the low refractive index layer 27l. l The optical length can be set to λ0 / 4, where the optical length is the value obtained by multiplying the thickness by the refractive index.

[0064] Figure 8 schematically shows another example of a filter 112 in which the first reflective layer 28a and the second reflective layer 28b are each formed from a dielectric multilayer film. In the example shown in Figure 8, unlike the example shown in Figure 7, the thickness of the high refractive index layer 27h and the low refractive index layer 27l are not uniform in each of the first reflective layer 28a and the second reflective layer 28b. In each of the first reflective layer 28a and the second reflective layer 28b, at least two of the low refractive index layers 27l have different thicknesses, and at least two of the high refractive index layers 27h have different thicknesses. In each of the first reflective layer 28a and the second reflective layer 28b, the optical length of each of the multiple low refractive index layers 27l is equal to the optical length of the adjacent high refractive index layer 27h that forms a pair with that low refractive index layer 27l. The dielectric multilayer film shown in Figure 8 is, for example, included in the wavelength λ of the target wavelength range W. s from λ l It can be designed to reflect light in the wavelength range up to λ. s The lower limit wavelength λ1 of the target wavelength range W may be the same as or different from it. Similarly, wavelength λ l This may be the same as or different from the upper limit wavelength λ2 of the target wavelength range W.

[0065] Here, in each of the first reflective layer 28a and the second reflective layer 28b, multiple pair layers are numbered from n=0 to n=4 in order of their distance from the interference layer 26. The thickness of the high refractive index layer 27h is, for example, t h (n) = [λ s +n(λ l -λ s ) / 4] / (4n h ) can be set to ). Also, the thickness of the low refractive index layer 27l can be, for example, t l (n) = [λ s +n(λ l -λ s ) / 4] / (4n l ) can be set to ). Thus, in each of the first reflective layer 28a and the second reflective layer 28b, the thickness t of the high refractive index layer 27h h (n), and the thickness t of the low refractive index layer 27l l Each of (n) is λ s / 4 to λ l It can be designed to be linearly modulated up to λ.s =350nm and λ l If =700nm, the optical length of each layer thickness is λ s / 4 = 87.5 nm l The change is linear up to / 4 = 175 nm.

[0066] In the example shown in Figure 8, the thin line representing loop 29a and the thick line representing loop 29b represent the wavelength λ confined within the filter 112, respectively. s and wavelength λ l This schematically represents light with wavelength λ. s The light is reflected by the pair of layers on the incident surface side (i.e., the upper side in Figure 8) in the first reflective layer 28a and the pair of layers on the substrate 80 side in the second reflective layer 28b. Meanwhile, the wavelength λ l The light is reflected by the pair layer on the interference layer 26 side in the first reflective layer 28a and the pair layer on the interference layer 26 side in the second reflective layer 28b. In this way, the incident light is reflected by the pair layer corresponding to its wavelength. With this structure, the non-uniformity of reflectivity within the target wavelength range W in the dielectric multilayer film can be suppressed.

[0067] Each of the high refractive index layer 27h and the low refractive index layer 27l may be formed from a material that has low absorption for light within the target wavelength range W, for example. If the target wavelength range W is in the visible light region, the material may be, for example, SiO2, Al2O3, SiO2 x N y It may be at least one selected from the group consisting of Si3N4, Ta2O5, and TiO2. When the target wavelength range W is in the infrared region, the material may be, for example, the above SiO2, Al2O3, SiO x N y In addition to Si3N4, Ta2O5, and TiO2, it may be at least one selected from the group consisting of single-crystal Si, polycrystalline Si, and amorphous Si.

[0068] The interference layer 26 may also be formed from any of the above materials. However, the interference layer 26 has a different refractive index from the low refractive index layer 27l or high refractive index layer 27h that is in contact with the interference layer 26. The interference layer 26 is not limited to a single layer, but may include multiple stacked layers. These multiple layers may be formed from different materials. The refractive indices of these multiple layers may differ to such an extent that they do not substantially affect the transmission spectrum of the filter. Reflection may occur at the interface of layers with different refractive indices. However, as long as it does not substantially affect the transmission spectrum, each of these multiple layers can be considered as part of a substantially uniform interference layer 26. The interference layer 26 may also be a layer of gas such as air.

[0069] The structures shown in Figures 7 and 8 are merely examples, and the structure of each filter 112 in the filter array 110 is designed appropriately according to the required performance. For example, the number of paired layers in each reflective layer is not limited to 5 and is determined according to the required performance. In the example shown in Figure 8, the thickness of the layers decreases as you move away from the interference layer 26, but conversely, a structure in which the thickness of the layers increases as you move away from the interference layer 26 may also be adopted. Furthermore, the thickness of the layers does not necessarily have to increase or decrease monotonically as you move away from the interference layer 26.

[0070] In the examples shown in Figures 7 and 8, a first reflective layer 28a and a second reflective layer 28b are provided on both sides of the interference layer 26, but a reflective layer may be provided on only one side of the interference layer 26. In that case, one surface of the interference layer 26 may be exposed to an external medium such as air. Even with such a configuration, reflection occurs at the interface between the interference layer 26 and the external medium, so resonance can be generated within the interference layer 26. With a configuration in which a reflective layer is provided on only one side of the interference layer 26, the loss of light can be suppressed.

[0071] <Structure equipped with a bandwidth limiting filter> As described above, in an imaging system that generates hyperspectral images using methods such as compressed sensing, a filter array 110 including multiple filters with different spectral transmittances is used. As the wavelength distribution and spatial distribution of the optical transmittances of the multiple filters, a highly random distribution such as the random distribution or quasi-random distribution mentioned above may be selected. A hyperspectral image can be estimated using a matrix H determined by the spatial distribution of spectral transmittances. By increasing the randomness of the wavelength distribution and spatial distribution of the optical transmittances of the multiple filters, a more accurate hyperspectral image can be generated.

[0072] The filter array 110 can be realized using multiple interference filters that change the reflectivity of light for each wavelength by utilizing the phenomenon of light interference, as described above. As interference filters, for example, filters with a Fabry-Perot resonator structure as described above can be used. In addition, filters with a structure using a diffraction grating, such as that disclosed in Patent Document 3, or a structure utilizing surface plasmon resonance, such as that disclosed in Patent Document 4, can also be used.

[0073] When a filter array 110 is constructed using multiple interference filters, the error in the reconstructed hyperspectral image increases as the target wavelength range, i.e., the operating wavelength range of the imaging system, widens. This is because the lower-order modes on the longer wavelength side and the higher-order modes on the shorter wavelength side of the target wavelength range occur with the same structure and interfere with each other. When the lower-order modes on the longer wavelength side and the higher-order modes on the shorter wavelength side occur with the same structure and interfere with each other in an interference filter, the spatial and wavelength-related randomness of the optical transmittance of the filter array 110 decreases. As a result, the randomness of the matrix H values ​​used in the reconstruction calculation decreases, and the error in the reconstructed hyperspectral image increases. The above problem will be explained in more detail below using a filter with a Fabry-Perot resonator structure as an example.

[0074] As mentioned above, a Fabry-Perot resonator has a structure in which an interference layer that interferes with light is sandwiched on both sides by reflective layers that reflect light in a specific wavelength range at a fixed rate. A Fabry-Perot resonator has the property of preferentially transmitting light of a wavelength corresponding to the thickness of the interference layer. Each reflective layer can be formed from a dielectric multilayer film that functions as a distributed Bragg reflector (DBR), for example. In that case, the wavelength of the reflected light is determined by the refractive index and thickness of each layer in the DBR. If the thickness of one layer of the DBR is t and the refractive index of that layer is n, then light of wavelength λi that satisfies λi / 4 = n × t will exhibit the highest reflectivity in that layer. On the other hand, each layer of the DBR may also function as an interference layer. The conditions under which a mode of light is generated in a layer of the DBR depend on the thickness t and refractive index n of that layer. If the condition λj / 2 ≤ n × t is satisfied, that layer functions as an interference layer, and a mode of light of wavelength λj is formed inside that layer.

[0075] Here, as shown in Figures 3A and 3B, let λ1 be the wavelength at the short-wavelength end of the target wavelength range W, and λ2 be the wavelength at the long-wavelength end. When λ2 is greater than twice λ1, i.e., when λ2 > 2λ1, then within the layer with optical length λ2 / 4 that reflects light of the longest wavelength λ2, λ1 ≤ λ k Wavelength λ that satisfies ≤ λ² / 2 k A mode of light is generated. The generation of a mode of light during DBR means that a lower-order mode on the longer wavelength side of the operating wavelength range and a higher-order mode on the shorter wavelength side of the operating wavelength range are generated with the same structure and interfere with each other. In that case, the resulting transmission spectrum will have wavelengths with low transmittance randomness on the shorter wavelength side of the target wavelength range W.

[0076] Figure 9A shows examples of transmission spectra for eight types of filters included in the filter array 110. Figure 9B shows a magnified view of the transmission spectra in the wavelength range of 400 nm to 600 nm in Figure 9A. In this example, the target wavelength range W, i.e., the operating wavelength range, is set to a relatively wide range of 400 nm to 1600 nm. Each filter has a Fabry-Perot resonator structure using the aforementioned DBR in each reflective layer. By adjusting the structure of the interference layer and DBR for each filter, the transmission spectra shown were obtained. When the target wavelength range W is wide, as in this example, optical modes are generated within the DBR of each filter, and the difference in transmittance between filters becomes smaller on the shorter wavelength side of the target wavelength range W. As a result, the randomness of transmittance is greatly reduced, especially on the shorter wavelength side. For example, at the wavelength indicated by the arrow in Figure 9B, all eight types of filters have approximately the same transmittance. This reduction in the randomness of transmittance of the filter array 110 directly leads to an increase in the estimation error of the hyperspectral image.

[0077] Figure 10 shows an example of the error in the estimation calculation of a hyperspectral image performed using a filter array containing multiple filters having the transmission spectra shown in Figure 9A. The spectrum shown in part (a) of Figure 10 is the same as the spectrum shown in Figure 9A. Part (b) of Figure 10 shows examples of estimation errors for each of the 40 wavelength bands obtained by dividing the 400nm to 600nm wavelength range and the 1400nm to 1600nm wavelength range in part (a) of Figure 10 into 20 equal parts. Here, the wavelength bands in the 400nm to 600nm wavelength range and the 1400nm to 1600nm wavelength range are numbered from 1 to 20, starting with the shortest wavelength. The estimation error for each wavelength band was evaluated using the Mean Squared Error (MSE) between the estimated image and the ground truth image. MSE is the sum of the squares of the errors of each pixel between the estimated image and the ground truth image, divided by the number of pixels. The MSE calculation result is affected by the number of bits in the acquired and restored image. In this embodiment, the image is acquired with 8 bits, i.e., 256 gradations, and the estimated image is also generated with 8 bits. As shown in Figure 10, it was confirmed that the estimation error of the hyperspectral image is large on the short wavelength side where randomness is low. On the other hand, it was confirmed that the hyperspectral image can be estimated with a small error on the long wavelength side where randomness is high. When the hyperspectral image is represented with 256 gradations, the estimation error of the hyperspectral image is,

number

[0078] Thus, when the operating wavelength range is wide, the estimation error of the hyperspectral image tends to increase, especially in the shorter wavelength range. This problem can also occur when using multimode filters with structures different from the Fabry-Perot resonator structure that uses a DBR as the reflective layer. For example, the above problem can also occur when using other types of multimode filters that utilize optical interference, such as filters using diffraction gratings or filters using the surface plasmon effect. In other words, in multimode filters, when λ2 > 2λ1 is satisfied, the randomness of transmittance decreases on the shorter wavelength side of the target wavelength range W, and the estimation error of the hyperspectral image increases.

[0079] The inventors identified the above problems and investigated a novel filter array configuration to solve them. As a result, they found that the above problems can be solved by adopting a filter configuration in which a band-limiting filter that restricts the transmission of light in a certain wavelength range is superimposed on a multimode filter. With such a filter configuration, it is possible to maintain a high degree of randomness in transmittance over a wide wavelength range and expand the operating wavelength range. An example of such a filter array configuration will be specifically described below.

[0080] Figure 11 is a schematic diagram showing part of the configuration of a filter array 110 in one embodiment. The filter array 110 is arranged two-dimensionally and includes a plurality of optical filters having different transmission spectra. Figure 11 schematically shows cross-sections of two adjacent filters 112. Hereinafter, these two filters will be referred to as the first filter 112a and the second filter 112b.

[0081] The first filter 112a comprises a first multimode filter 113a and a first band-limiting filter 114a. The first multimode filter 113a has a plurality of first peak wavelengths within the target wavelength range W where the light transmittance shows a maximum value. The first band-limiting filter 114a is positioned to overlap the first multimode filter 113a. Here, "overlapping" of the two filters means that they are in a positional relationship where they overlap at least partially when viewed from the direction of the normal of those filters. Other components may be interposed between the two filters. The limiting band of the first band-limiting filter 114a is a first subwavelength range which is part of the target wavelength range W.

[0082] In the example shown in Figure 11, the first multimode filter 113a is positioned on the optical path of the transmitted light that has passed through the first band-limiting filter 114a. Conversely, the first band-limiting filter 114a may be positioned on the optical path of the transmitted light that has passed through the first multimode filter 113a.

[0083] The second filter 112b comprises a second multimode filter 113b and a second band-limiting filter 114b. The second multimode filter 113b has a plurality of second peak wavelengths within the target wavelength range W where the light transmittance shows a maximum value. Here, the combination of the plurality of second peak wavelengths is different from the combination of the plurality of first peak wavelengths. That is, at least one of the plurality of second peak wavelengths is different from the plurality of first peak wavelengths. The second band-limiting filter 114b is positioned to overlap the second multimode filter 113b. The second band-limiting filter 114b limits the transmission of light in a second subwavelength range, which is part of the target wavelength range W. The second subwavelength range is different from the first subwavelength range. Note that the second subwavelength range and the first subwavelength range may partially overlap.

[0084] In the example shown in Figure 11, the second multimode filter 113b is positioned on the optical path of the transmitted light that has passed through the second band-limiting filter 114b. Conversely, the second band-limiting filter 114b may be positioned on the optical path of the transmitted light that has passed through the second multimode filter 113b.

[0085] Figure 12 shows examples of transmission spectra for the first multimode filter 113a and the second multimode filter 113b. Part (a) of Figure 12 shows an example of the transmission spectrum of the first multimode filter 113a. Part (b) of Figure 12 shows an example of the transmission spectrum of the second multimode filter 113b. As shown in Figure 12, each of the first multimode filter 113a and the second multimode filter 113b has multiple peak wavelengths in which the transmittance is maximized in multiple wavelength bands included in the target wavelength range W. Some of the peak wavelengths of the first multimode filter 113a may overlap with the peak wavelengths of the second multimode filter 113b, but they do not completely coincide.

[0086] Each of the first multimode filter 113a and the second multimode filter 113b may be an interference filter having, for example, the structure of the Fabry-Perot resonator described above. By appropriately designing the structure of the interference layer and each reflection layer, a transmission spectrum having multiple peak wavelengths within the target wavelength range W can be realized.

[0087] In this embodiment, in addition to the first multimode filter 113a and the second multimode filter 113b, a first band-limiting filter 114a and a second band-limiting filter 114b are provided. The first band-limiting filter 114a and the second band-limiting filter 114b limit the transmission of light in a portion of the target wavelength range W. Here, "limiting the transmission of light" for a certain wavelength range does not mean completely limiting the light in that wavelength range. In this specification, "limiting the transmission of light" for a certain wavelength range means limiting the transmittance of light in that wavelength range to at least 80% or less. In this specification, for each filter, the band in which the transmittance of light is 80% or less is referred to as the "limiting band".

[0088] Figure 13 shows examples of transmission spectra for the first band-limiting filter 114a and the second band-limiting filter 114b. Part (a) of Figure 13 shows an example of the transmission spectrum of the first band-limiting filter 114a. Part (b) of Figure 13 shows an example of the transmission spectrum of the second band-limiting filter 114b. Each of the first band-limiting filter 114a and the second band-limiting filter 114b has a limiting band Ws with relatively low transmittance and a pass-band with relatively high transmittance within the target wavelength range W. The pass-band is the wavelength range within the target wavelength range W other than the limiting band Ws.

[0089] Although only the first filter 112a and the second filter 112b are shown in Figure 11, the filter array 110 may include other filters having a similar structure. For example, the optical filters in the filter array 110 may include a third filter in addition to the first filter 112a and the second filter 112b described above. The third filter may include a third multimode filter and a third band-limiting filter. The third multimode filter has a plurality of third peak wavelengths within the target wavelength range at which the light transmittance is maximized. Here, the combination of the plurality of third peak wavelengths may be designed to be different from any of the combinations of the plurality of first peak wavelengths and the plurality of second peak wavelengths described above. That is, at least one of the plurality of third peak wavelengths is different from the plurality of first peak wavelengths, and at least one of the plurality of third peak wavelengths is different from the plurality of second peak wavelengths. For example, the plurality of third peak wavelengths may include peak wavelengths λ1 and λ2, with peak wavelength λ1 being different from the plurality of first peak wavelengths and peak wavelength λ2 being different from the plurality of second peak wavelengths. The third band-limiting filter may be positioned to overlap the third multimode filter. The limiting band of the third band-limiting filter is a third subwavelength range, which is part of the target wavelength range. The third subwavelength range may be designed to be different from both the first and second subwavelength ranges. The third subwavelength range and the first subwavelength range may partially overlap. Similarly, the third subwavelength range and the second subwavelength range may partially overlap.

[0090] Thus, the filter array 110 may include three or more band-limiting filters with different limiting bands. Alternatively, multiple band-limiting filters having equivalent characteristics may be arranged to overlap multiple multimode filters having different transmission spectra. All filters in the filter array 110 may have a stacked structure of multimode filters and band-limiting filters, or only some filters may have such a stacked structure. The filter array 110 may also include filters that do not have band-limiting filters, i.e., filters that consist only of multimode filters. For example, multiple filters in the filter array 110 may further include a fourth filter having a different structure from the first to third filters described above. The fourth filter has multiple fourth peak wavelengths within the target wavelength range where the light transmittance shows a maximum value. The combination of multiple fourth peak wavelengths may be designed to be different from any of the combinations of the first to third peak wavelengths described above. That is, at least one of the multiple fourth peak wavelengths is different from the multiple first peak wavelengths, at least one of the multiple fourth peak wavelengths is different from the multiple second peak wavelengths, and at least one of the multiple fourth peak wavelengths is different from the multiple third peak wavelengths. For example, multiple fourth peak wavelengths may include peak wavelengths λ1, λ2, and λ3, with peak wavelength λ1 being different from multiple first peak wavelengths, peak wavelength λ2 being different from multiple second peak wavelengths, and peak wavelength λ3 being different from multiple third peak wavelengths. By mixing filters without band-limiting filters with filters that include band-limiting filters, the overall transmission spectrum of the filter array 110 can be further diversified.

[0091] In the following explanation, individual multimode filters may be referred to as "multimode filter 113" without distinction. Similarly, individual band-limiting filters may be referred to as "band-limiting filter 114" without distinction.

[0092] Each band-limiting filter 114 may be constructed using a microstructure that includes, for example, a dielectric multilayer film, an organic material, a diffraction grating structure, and at least one of a metal. The constituent materials of the multimode filter 113 and the band-limiting filter 114 may be the same or different. Each band-limiting filter 114 may be, for example, a color filter that selectively transmits light of a specific color. Alternatively, it may be a filter made of a dielectric multilayer film such as a DBR that selectively transmits light in a specific wavelength range.

[0093] In the example shown in Figure 11, each multimode filter 113 and its corresponding band-limiting filter 114 are stacked on top of each other without any other components in between. The structure is not limited to this configuration; other layers or components may be interposed between the multimode filter 113 and the band-limiting filter 114. For example, an optical component that causes light reflection or refraction may be placed between the multimode filter 113 and the band-limiting filter 114. In that case, it is also possible to arrange the multimode filter 113 and the band-limiting filter 114 so that they do not overlap each other.

[0094] Figure 14 is a schematic cross-sectional view showing another example of the filter array 110. This filter array 110 comprises a substrate 80, a plurality of multimode filters 113 stacked on the substrate 80, and a plurality of band-limiting filters 114. The filter array 110 includes four types of multimode filters 113 with different combinations of peak wavelengths and four types of band-limiting filters 114 with different limiting bands. In this example, incident light enters from the top of Figure 14, passes through both the band-limiting filters 114 and the multimode filters 113, and then passes further through the substrate 80 to be received by the photodetector element of an image sensor located on the same optical axis.

[0095] Figure 15 shows examples of the limiting bands W1, W2, W3, and W4 for four types of band-limiting filters 114. As shown in Figure 15, the limiting band and passband may differ depending on the band-limiting filter 114. The limiting bands of multiple band-limiting filters 114 may be separated from each other or overlap. In this example, each band-limiting filter 114 has one limiting band within the target wavelength range, but it may have multiple limiting bands.

[0096] Figures 16A and 16B show examples of the arrangement of the band-limiting filters 114. In the example shown in Figure 16A, four types of band-limiting filters 114 are arranged regularly in two dimensions. On the other hand, in the example shown in Figure 16B, four types of band-limiting filters 114 are arranged randomly in two dimensions. In this way, multiple types of band-limiting filters 114 with different limiting bandwidths can be arranged in any manner.

[0097] In the examples shown in Figures 11 and 14, each band-limiting filter 114 covers the entire corresponding multimode filter 113. However, the configuration is not limited to this; each band-limiting filter 114 may cover only a portion of the corresponding multimode filter 113. Each band-limiting filter 114 may have the same dimensions as the multimode filter 113, or different dimensions, with respect to the direction perpendicular to the optical path of the incident light. In one example, the pair of multimode filter 113 and band-limiting filter 114 may be arranged so that the light passing through their filters incidents on only one corresponding photodetector in the image sensor. In another example, each pair of multimode filter 113 and band-limiting filter 114 may be arranged so that the light passing through their filters incidents on multiple corresponding photodetectors in the image sensor.

[0098] Figure 17 is a cross-sectional view showing a more detailed example of the structure of the filter array 110. Figure 17 shows a portion of the cross-sectional structure of one row or column in the filter array 110. In this example, an array of multimode filters 113 and an array of band-limiting filters 114 are stacked on a substrate 80. The multimode filter 113 includes a reflective layer 28 having a DBR structure and an interference layer 26. The band-limiting filter 114 in this example has a DBR structure similar to the reflective layer 28. The reflective layer 28, interference layer 26, and band-limiting filter 114 are stacked on the substrate 80 in this order. The stacking order of the multimode filters 113 and band-limiting filters 114 may be reversed from the order shown. That is, the band-limiting filters 114 may be placed on the substrate 80, and the multimode filters 113 may be placed on top of them. Also, the stacking order of the multimode filters 113 and band-limiting filters 114 may differ depending on the filter. In the example shown in Figure 17, the multimode filter 113 and the band-limiting filter 114 have the same size in the direction perpendicular to the optical path of the incident light, but their sizes may be different.

[0099] In the example shown in Figure 17, the thickness of the interference layer 26 varies depending on the filter. This realizes a structure in which the combination of peak wavelengths differs depending on the filter. The array of band-limiting filters 114 includes three types of band-limiting filters 114 with different limiting bandwidths and a portion where no band-limiting filters 114 are placed. That is, this filter array 110 comprises three types of filters, each containing one of the three types of band-limiting filters 114 with different limiting bandwidths, and one type of filter that does not contain a band-limiting filter 114. These four types of filters can be arranged two-dimensionally in a manner similar to the four types of filters shown as W1 to W4 in Figure 16A or Figure 16B, for example.

[0100] Unlike the Fabry-Perot filters shown in Figures 7 and 8, each multimode filter 113 in this example has a DBR, or reflective layer 28, on only one side. Of the two surfaces of the interference layer 26, the surface without the reflective layer 28 is in contact with the band-limiting filter 114 or air. Even with this structure, the multimode filter 113 functions as a Fabry-Perot filter and can obtain a good transmission spectrum.

[0101] Figures 18A, 18B, and 18C show examples of transmission spectra for the first band-limiting filter 114a, the second band-limiting filter 114b, and the third band-limiting filter 114c shown in Figure 17. The transmission spectrum shown in Figure 18A has a minimum transmittance at approximately 500 nm. The transmission spectrum shown in Figure 18B has a minimum transmittance at approximately 650 nm. The transmission spectrum shown in Figure 18C has the lowest minimum at approximately 675 nm. In this example, the first band-limiting filter 114a, the second band-limiting filter 114b, and the third band-limiting filter 114c all strongly limit the bandwidth in the relatively short wavelength range. By providing these three types of band-limiting filters, the randomness of transmittance, especially in the short wavelength range, can be increased, and the estimation error of the hyperspectral image can be reduced.

[0102] The effects of providing the bandwidth limiting filter 114 will be explained below with reference to Figures 19A to 19D.

[0103] Figure 19A shows an example of the transmission spectrum of a filter array without a band-limiting filter 114. In this filter array, eight different multimode filters 113 with different transmission spectra are arranged two-dimensionally, and no band-limiting filter 114 is provided. Figure 19A shows a magnified view of the spectrum shown in Figure 9A, specifically the portion in the range from 400 nm to 800 nm.

[0104] Figure 19B shows an example of the transmission spectrum of a filter array 110 in which, in addition to the eight types of multimode filters 113 described above, one type of band-limiting filter 114 is partially arranged. In this example, for each of the eight types of multimode filters 113, there are areas where the band-limiting filter 114 is provided and areas where it is not. Therefore, Figure 19B shows a total of 8 × 2 = 16 different transmission spectra. In this example, the first band-limiting filter 114a having the transmission spectrum shown in Figure 18A was used. This first band-limiting filter 114a strongly limits light in the wavelength range around 500 nm. As can be seen from Figure 19B, simply adding one type of band-limiting filter 114 diversifies the transmission spectrum and improves the randomness of the transmittance. Note that in the configuration with the band-limiting filter 114, the peak wavelength shifts compared to the configuration without the band-limiting filter 114. This is because providing the band-limiting filter 114 has the same effect as substantially increasing the thickness of the interference layer 26.

[0105] Figure 19C shows an example of the transmission spectrum of a filter array 110 in which three types of band-limiting filters 114 are arranged in addition to the eight types of multimode filters 113 described above. In this example, for each of the eight types of multimode filters 113, there are locations where a first band-limiting filter 114a is provided, locations where a second band-limiting filter 114b is provided, locations where a third band-limiting filter 114c is provided, and locations where no band-limiting filter 114 is provided. Therefore, Figure 19C shows a total of 8 × 4 = 32 different transmission spectra. The three types of band-limiting filters 114 used in this example have the transmission spectra shown in Figures 18A to 18C, respectively. These three types of band-limiting filters 114 strongly limit light in the wavelength range around 500 nm, 650 nm, and 675 nm, respectively. As can be seen from Figure 19C, increasing the number of band-limiting filters 114 to three further diversifies the transmission spectrum and further improves the randomness of the transmittance.

[0106] Figure 19D is a graph showing the standard deviation of transmittance at each wavelength in the filter array 110 for each case from Figures 19A to 19C. A larger standard deviation of transmittance indicates higher randomness. Compared to the filter array in Figure 19A, which does not have a band-limiting filter 114, the filter array 110 in Figure 19B shows improved transmittance randomness, particularly around 500 nm. However, in the configuration of Figure 19B, which adds a band-limiting filter 114 with a limiting band of around 500 nm, the improvement in transmittance randomness in some wavelength ranges (e.g., around 530 nm, 590 nm, 610 nm, and 700-800 nm) is not very significant. In contrast, as in the example in Figure 19C, when multiple types of band-limiting filters 114 are added, it becomes possible to maintain high randomness over a wider wavelength range. From this, it can be seen that the more types of band-limiting filters 114 are added, the greater the effect of expanding the operating wavelength range.

[0107] Figure 20A shows that the reconstruction error of hyperspectral images is reduced by providing the band-limiting filter 114. Figure 20A shows the calculation results of the least squares error (MSE) of the reconstructed images generated for each of the 20 wavelength bands included in the wavelength range from 400 nm to 600 nm, for both the case where the band-limiting filter 114 is not provided (Figure 19A) and the case where the band-limiting filter 114 is provided (Figure 19C). As shown in Figure 20A, the MSE is significantly reduced in the shorter wavelength range by providing the band-limiting filter 114. In other words, it can be seen that by providing the band-limiting filter 114, the randomness of transmittance in the shorter wavelength range is improved, and the estimation error of the hyperspectral image is reduced.

[0108] Figure 20B shows the calculated MSE of the reconstructed images for each of the 20 wavelength bands included in the wavelength range from 1400 nm to 1600 nm, for both the case where the band-limiting filter 114 is not provided (Figure 19A) and the case where the band-limiting filter 114 is provided (Figure 19C). As shown in Figure 20B, in the longer wavelength range, the estimation error of the hyperspectral image is kept small even when the band-limiting filter 114 is provided. From this, it can be seen that by providing the band-limiting filter 114, it is possible to obtain hyperspectral images with small errors over a wide wavelength range from the short wavelength side to the long wavelength side. In other words, it can be seen that the operating wavelength range of the hyperspectral imaging device can be expanded by configuring the filter array 110 using a filter in which the band-limiting filter 114 is placed in superimposed on the multimode filter 113.

[0109] The structure of the filter array 110 described above is merely illustrative, and various modifications to the structure of the filter array 110 are possible. Several modifications are described below.

[0110] Figure 21 is a schematic cross-sectional view showing a modified example of the filter array 110. In this example, the filter array 110 comprises a multimode filter 113 with an interference layer 26 and a first reflective layer 28a and a second reflective layer 28b located on either side thereof. The first reflective layer 28a and the second reflective layer 28b are DBRs. That is, the multimode filter 113 in this example has a structure in which the interference layer 26 is sandwiched between the two DBRs on either side thereof. In this structure, the DBRs constituting the band-limiting filter 114 are stacked on top of the upper DBR (i.e., the first reflective layer 28a) of the multimode filter 113. Each of these DBRs has a structure in which low refractive index layers with relatively low refractive indexes and high refractive index layers with relatively high refractive indexes are alternately stacked. In the DBR of the first reflective layer 28a, the thicknesses of the multiple low refractive index layers are not uniform, and the thicknesses of the multiple high refractive index layers are also not uniform. In contrast, in the DBR of the band-limiting filter 114, the thickness of the multiple low-refractive-index layers is uniform, and the thickness of the multiple high-refractive-index layers is also uniform. Therefore, the DBR of the first reflective layer 28a and the DBR of the band-limiting filter 114 can be distinguished based on whether or not the thickness of the low-refractive-index layers and the high-refractive-index layers are uniform. Furthermore, in each filter, the structure of the DBR of the first reflective layer 28a and the structure of the second reflective layer 28b are in a symmetrical relationship. Therefore, the DBR of the upper first reflective layer 28a in the multimode filter 113 and the DBR of the band-limiting filter 114 can also be distinguished based on whether or not they have a structure symmetrical with respect to the DBR of the second reflective layer 28b. In the example shown in Figure 21, the number of types of band-limiting filters 114 is less than the number of types of multimode filters 113, and multiple band-limiting filters 114 having the same structure are each placed on multiple pixels located within a relatively narrow range in the image sensor 160. In such cases, the first reflection layer 28a and the band-limiting filter 114 can also be distinguished based on their identity with the DBR structure in filters on other neighboring pixels.

[0111] Figure 22 is a schematic cross-sectional view showing an example of the structure of a filter array 110 having a spacer layer 90 between a multimode filter 113 and a band-limiting filter 114. The spacer layer 90 may be formed from a material that is transparent in the target wavelength range W. By providing the spacer layer 90, the steps on the surface of the multimode filter 113 are filled, making it easier to form the band-limiting filter 114. Although Figure 22 depicts a relatively thin spacer layer 90, in reality, a spacer layer 90 with a thickness that does not cause resonance inside the spacer layer 90 may be used. In addition to the spacer layer 90 shown, a separate spacer layer may be provided to fill the steps on the surface of the band-limiting filter 114. On such a spacer layer, for example, a microlens array that efficiently focuses incident light to each pixel may be arranged.

[0112] In the filter array 110 shown in Figure 22, a first reflective layer 28a and a second reflective layer 28b are provided on both sides of the interference layer 26, but a reflective layer may be provided on only one side of the interference layer 26. Alternatively, filters with reflective layers on both sides of the interference layer 26 and filters with reflective layers on only one side of the interference layer 26 may be mixed. Thus, in each multimode filter 113, a reflective layer may be provided on only one side of the interference layer 26, or reflective layers may be provided on both sides of the interference layer 26.

[0113] In each of the above examples, the relative positions of the multimode filter 113 and the band-limiting filter 114 may be reversed. That is, the band-limiting filter 114 may be positioned relatively close to the image sensor, and the multimode filter 113 may be positioned relatively far from the image sensor.

[0114] The bandwidth limiting filter 114 is not limited to DBR; for example, it may be a color filter. Below, we will describe an example of a configuration in which the bandwidth limiting filter 114 is a color filter.

[0115] Figure 23 is a schematic cross-sectional view showing an example of the structure of a filter array 110 equipped with color filters as band-limiting filters 114. Color filters can be used as band-limiting filters 114 because they have the characteristic of transmitting or absorbing only light in a specific wavelength range. In the example in Figure 23, three types of color filters 114R, 114G, and 114B are arranged, each selectively transmitting light in the red, green, and blue wavelength ranges, respectively. The filter array 110 may also include filters that do not contain color filters. In addition to red, green, and blue color filters, color filters that selectively transmit other colors of light, such as cyan, magenta, and yellow, may also be used. As shown in the example in Figure 23, the color filters do not necessarily need to cover the entire multimode filter 113. That is, a part of the multimode filter 113 may not be covered by color filters.

[0116] In the example shown in Figure 23, one side of the interference layer 26 is in contact with the DBR reflection layer 28, but both sides of the interference layer 26 may be in contact with the DBR. In the structure shown in Figure 23, the positional relationship between the interference layer 26 and the reflection layer 28 may be reversed. That is, the interference layer 26, the reflection layer 28, and the band-limiting filter 114 (i.e., the color filter) may be stacked on the substrate 80 in this order. Also, from the configuration shown in Figure 23, the positional relationship between the multimode filter 113 and the band-limiting filter 114 may be reversed. That is, the band-limiting filter 114 and the multimode filter 113 may be stacked on the substrate 80 in this order.

[0117] Figure 24 shows an example of a structure in which, in the configuration shown in Figure 23, DBRs, i.e., the first reflective layer 28a and the second reflective layer 28b, are arranged on both sides of the interference layer 26. In this structure as well, the positional relationship between the multimode filter 113 and the band-limiting filter 114 may be reversed.

[0118] Figure 25 shows an example of a structure in which, in addition to the structure shown in Figure 24, a spacer layer 90 is further added between the multimode filter 113 and the band-limiting filter 114, i.e., the color filter. The spacer layer 90 may be made of a material that is substantially transparent in the target wavelength range W. The multimode filter 113 comprises a first reflective layer 28a and a second reflective layer 28b on both sides of the interference layer 26. One of the first reflective layer 28a and the second reflective layer 28b may be removed from the structure shown in Figure 25.

[0119] Next, we will describe an example of the characteristics of the bandwidth limiting filter 114.

[0120] The inventors conducted a simulation to investigate how the error in the reconstructed hyperspectral image changes when the light transmittance in the limiting band of each band-limiting filter 114 is changed, using a filter array 110 equipped with multiple types of band-limiting filters 114 with different limiting bands. Figures 26A to 26C show examples of transmission spectra of the filter array 110 used in the simulation. These figures show examples of transmission spectra obtained when each of the four types of band-limiting filters 114 with different limiting bands is superimposed on eight types of multimode filters 113. Here, each multimode filter 113 is assumed to have an ideal transmission spectrum represented by a Lorentz function. The transmittance in the passband of each band-limiting filter 114 was set to 100%, and the reconstruction error was evaluated under multiple conditions in which the transmittance in the limiting band was changed.

[0121] Figure 26A shows an example of the transmission spectrum of a filter array 110 equipped with a band-limiting filter 114 having a rectangular transmission spectrum with a transmittance of 0% in the limiting band. Figure 26B shows an example of the transmission spectrum of a filter array 110 equipped with a band-limiting filter 114 having a rectangular transmission spectrum with a transmittance of 60% in the limiting band. Figure 26C shows an example of the transmission spectrum of a filter array 110 without a band-limiting filter 114, i.e., with a transmittance of 100% in the limiting band. These figures show, from bottom to top, the transmission spectra of filter 1 with a band-limiting filter 114 with a passband of 450-500 nm superimposed, filter 2 with a band-limiting filter 114 with a passband of 500-550 nm superimposed, filter 3 with a band-limiting filter 114 with a passband of 550-600 nm superimposed, and filter 4 with a band-limiting filter 114 with a passband of 600-650 nm superimposed.

[0122] As shown in Figures 26B and 26C, the filter 112, which includes a multimode filter 113 and a band-limiting filter 114, may have multiple peak wavelengths that exhibit maximum values ​​within the target wavelength range W.

[0123] In other words, the transmitted light that has passed through the multimode filter 113 and the band-limiting filter 114 has at least two peak wavelengths, and each of the photodetectors 162 included in the image sensor 160 can detect the transmitted light having at least two peak wavelengths.

[0124] The reconstruction error was evaluated when estimating hyperspectral images for each of the 20 wavelength bands included in the wavelength range from 450 nm to 650 nm using a filter array 110 that combines the four types of band-limiting filters 114 described above. Figure 27 is a graph showing the relationship between the average mean squared error (MSE) between the estimated image and the ground truth image for each of the 20 wavelength bands and the transmittance in the limiting band. As shown in Figure 27, the estimation error is large when the transmittance in the limiting band is 100%. This is because it is equivalent to the absence of a band-limiting filter 114. On the other hand, when the transmittance in the limiting band is 0%, the estimation error also increases because luminance information in the limiting band is lost. From the results shown in Figure 27, it is desirable that the transmittance in the limiting band of the band-limiting filter 114 be between 10% and 80% of the transmittance in the passband. In this example, for simplicity, it is assumed that the transmittance in the limiting band of each band-limiting filter 114 is constant regardless of wavelength, but generally, the transmittance within the limiting band can vary with wavelength. Even in that case, the reconstruction error can be reduced by setting the average value of the transmittance in the restricted band to within 10 to 80% of the maximum transmittance in the target wavelength range W. Here, the average value of the transmittance in the restricted band can be calculated, for example, by averaging the transmittances of multiple wavelength bands included in that restricted band. From the above results, by setting the average value of the transmittance for each wavelength band in the restricted band of each band-limiting filter 114 to 10% to 80% of the maximum transmittance of the band-limiting filter in the target wavelength range W, the estimation error of the hyperspectral image can be greatly reduced. Furthermore, if the average value of the transmittance for each wavelength band in the restricted band of each band-limiting filter 114 is set to 15% to 70% of the maximum transmittance of the band-limiting filter in the target wavelength range W, the estimation error of the hyperspectral image can be further reduced.

[0125] Here, we consider the case where the band-limiting filter 114 has a limiting band with a transmittance of 0%, as shown in Figure 26A or Figure 27. In this case, the light transmitted through both the multimode filter 113 and the band-limiting filter 114 may have only one peak in the target wavelength range. In this case, the signal output from the image sensor corresponds to the signal output when a single-mode filter is used. This situation can be avoided by either the multimode filter 113 having multiple peaks in the target wavelength range other than the limiting band, or the transmittance of the band-limiting filter 114 in the limiting band being a significant value (for example, a transmittance of 10% or more) rather than 0%. In other words, each of the photodetectors 162 included in the image sensor 160 can output a signal corresponding to light having multiple peak wavelengths.

[0126] As described above, the filter array according to an exemplary embodiment of the present disclosure includes a plurality of optical filters arranged in two dimensions. The plurality of optical filters includes a first filter and a second filter. The first filter includes a first multimode filter having a plurality of first peak wavelengths in which the light transmittance is maximum within a target wavelength range, and a first band-limiting filter that limits the transmission of light in a first subwavelength range which is part of the target wavelength range. The second filter is a second multimode filter having a plurality of second peak wavelengths in which the light transmittance is maximum within the target wavelength range, wherein at least one of the plurality of second peak wavelengths is different from the combination of the plurality of first peak wavelengths, and a second band-limiting filter that limits the transmission of light in a second subwavelength range which is part of the target wavelength range but different from the first subwavelength range.

[0127] The filter array can be used, for example, in an imaging system that generates image data (i.e., hyperspectral data) for each of the multiple wavelength bands included in the target wavelength range. With the above configuration, it becomes easier to improve the randomness of the wavelength distribution and spatial distribution of the transmittance of the multiple optical filters included in the filter array. This makes it easier to improve the accuracy of the hyperspectral data generated by hyperspectral imaging and reconstruction calculations using the filter array.

[0128] At least a portion of the first multimode filter and the first band-limiting filter may be arranged on the first optical axis, and at least a portion of the second multimode filter and the second band-limiting filter may be arranged on the second optical axis.

[0129] In a plan view, at least a portion of the first multimode filter may overlap with the first band-limiting filter, and in a plan view, at least a portion of the second multimode filter may overlap with the second band-limiting filter.

[0130] Each of the first and second multimode filters may be, for example, interference filters. By using interference filters, it becomes easier to realize a multimode filter having multiple peak wavelengths within the target wavelength range.

[0131] In one example, each of the first multimode filter and the second multimode filter may include an interference layer and a first reflective layer, which is a dielectric multilayer film and is in contact with the interference layer.

[0132] In one example, each of the first multimode filter and the second multimode filter may further include a second reflective layer which is a dielectric multilayer film and is in contact with the interference layer, and in each of the first multimode filter and the second multimode filter, the interference layer may be located between the first reflective layer and the second reflective layer.

[0133] This structure makes it easier to realize multimode filters that have multiple peak wavelengths within the target wavelength range.

[0134] Each of the first and second band-limiting filters may be, for example, a color filter. Alternatively, each of the first and second band-limiting filters may be a filter containing a dielectric multilayer film.

[0135] The average value of the light transmittance of the first band-limiting filter in the first subwavelength range may be between 10% and 80% of the maximum light transmittance of the first band-limiting filter in the target wavelength range. Similarly, the average value of the light transmittance of the second band-limiting filter in the second subwavelength range may be between 10% and 80% of the maximum light transmittance of the second band-limiting filter in the target wavelength range. Here, the "average value of light transmittance" in a certain wavelength range can be calculated, for example, by averaging the light transmittance measured for each of the multiple wavelength bands included in that wavelength range. By providing a band-limiting filter that satisfies the above conditions, the error in the generated hyperspectral data can be reduced.

[0136] The average value of the light transmittance of the first band-limiting filter in the first subwavelength range may be between 15% and 70% of the maximum light transmittance of the first band-limiting filter in the target wavelength range. Similarly, the average value of the light transmittance of the second band-limiting filter in the second subwavelength range may be between 15% and 70% of the maximum light transmittance of the second band-limiting filter in the target wavelength range. By providing band-limiting filters that satisfy these conditions, the error in the generated hyperspectral data can be further reduced.

[0137] The plurality of optical filters may further include a third filter. The third filter is a third multimode filter having a plurality of third peak wavelengths within the target wavelength range in which the light transmittance is maximum, wherein at least one of the plurality of third peak wavelengths is different from the plurality of first peak wavelengths, and at least one of the plurality of third peak wavelengths is different from the plurality of second peak wavelengths, and may include a third band-limiting filter that limits the transmission of light in a third subwavelength range that is part of the target wavelength range and is different from both the first subwavelength range and the second subwavelength range. By providing such a third filter, the wavelength distribution and spatial distribution of the transmittance of the filter array can be further diversified, and the reconstruction error can be further reduced.

[0138] The plurality of filters may further include a fourth filter. The fourth filter is a fourth multimode filter having a plurality of fourth peak wavelengths within the target wavelength range in which the light transmittance shows a maximum value, wherein at least one of the plurality of fourth peak wavelengths is different from the plurality of first peak wavelengths, and at least one of the plurality of fourth peak wavelengths is different from the plurality of second peak wavelengths. The fourth filter may or may not include a band-limiting filter. By providing such a fourth filter, the wavelength distribution and spatial distribution of the transmittance of the filter array can be further diversified, and the reconstruction error can be further reduced.

[0139] The aforementioned plurality of filters may include four or more types of filters with different wavelength dependences of transmittance. At least some of these filters may include band-limited filters in addition to multimode filters. The more types of filters with different transmission spectra of multimode filters and different limiting bands of band-limited filters are mixed, the greater the diversity of the transmission spectrum of the filter array and the reduction of reconstruction errors can be.

[0140] The first transmitted light that has passed through the first multimode filter and the first band-limiting filter may have at least two peak wavelengths in the target wavelength range, and the second transmitted light that has passed through the second multimode filter and the second band-limiting filter may also have at least two peak wavelengths in the target wavelength range.

[0141] At least two of the plurality of first peak wavelengths may be in a wavelength range different from the first subwavelength range, and at least two of the plurality of second peak wavelengths may be in a wavelength range different from the second subwavelength range.

[0142] An imaging system according to an exemplary embodiment of the present disclosure may include an image sensor that receives light passing through any of the above-described filter arrays and outputs an image signal based on the light, and a signal processing device that generates a plurality of image data relating to corresponding wavelength bands among a plurality of wavelength bands included in the target wavelength range based on the image signal. Such an imaging system can generate good hyperspectral data. [Industrial applicability]

[0143] The technology disclosed herein is useful, for example, in cameras and measuring instruments that acquire multi-wavelength images. The technology disclosed herein can also be applied, for example, to sensing for biological, medical, and cosmetic applications, foreign object and pesticide residue detection systems for food, remote sensing systems, and in-vehicle sensing systems. [Explanation of Symbols]

[0144] 26 Interference layer 27h High refractive index layer 27l low refractive index layer 28a, 28b reflective layer 70 Objects 80 circuit boards 90 Spacer layer 100 Imaging device 110 filter array 112 filters 113 Multimode Filter 114 Bandwidth Limiting Filter 120 compressed images 140 Optical system 160 Image Sensors 162 Photodetector 200 Signal Processing Equipment 220 Hyperspectral Images

Claims

1. Equipped with multiple optical filters arranged in a two-dimensional plane, The plurality of optical filters include a first filter and a second filter, The first filter is, A first multimode filter having multiple first peak wavelengths within the target wavelength range where the light transmittance shows a maximum value, A first band-limiting filter that limits the transmission of light in a first subwavelength range, which is a part of the target wavelength range, Includes, The second filter described above is A second multimode filter having a plurality of second peak wavelengths within the target wavelength range where the light transmittance shows a maximum value, wherein at least one of the plurality of second peak wavelengths is different from the plurality of first peak wavelengths, A second band-limiting filter that limits the transmission of light in a second subwavelength region that is part of the target wavelength region but different from the first subwavelength region, Includes, The average value of the light transmittance of the first band-limiting filter in the first subwavelength range is 10% or more and 80% or less of the maximum light transmittance of the first band-limiting filter in wavelength ranges other than the first subwavelength range included in the target wavelength range. The average value of the light transmittance of the second band-limiting filter in the second subwavelength range is 10% or more and 80% or less of the maximum light transmittance of the second band-limiting filter in wavelength ranges other than the second subwavelength range included in the target wavelength range. Filter array.

2. At least a portion of the first multimode filter and the first band-limiting filter are arranged on the first optical axis. At least a portion of the second multimode filter and the second band-limiting filter are arranged on the second optical axis. The filter array according to claim 1.

3. In a plan view, at least a portion of the first multimode filter overlaps with the first band-limiting filter, In a plan view, at least a portion of the second multimode filter overlaps with the second band-limiting filter. The filter array according to claim 1 or 2.

4. Each of the first multimode filter and the second multimode filter is an interference filter. A filter array according to any one of claims 1 to 3.

5. Each of the first multimode filter and the second multimode filter is, Interference layer and, The interference layer is in contact with a first reflective layer which is a dielectric multilayer film, A filter array according to any one of claims 1 to 4.

6. Each of the first multimode filter and the second multimode filter further includes a second reflective layer which is a dielectric multilayer film and is in contact with the interference layer. In each of the first multimode filter and the second multimode filter, the interference layer is located between the first reflective layer and the second reflective layer. The filter array according to claim 5.

7. Each of the first and second band-limiting filters is a color filter. A filter array according to any one of claims 1 to 6.

8. Each of the first and second band-limiting filters includes a dielectric multilayer film. A filter array according to any one of claims 1 to 6.

9. The average value of the light transmittance of the first band-limiting filter in the first subwavelength range is 15% or more and 70% or less of the maximum light transmittance of the first band-limiting filter in wavelength ranges other than the first subwavelength range included in the target wavelength range. The average value of the light transmittance of the second band-limiting filter in the second subwavelength range is 15% or more and 70% or less of the maximum light transmittance of the second band-limiting filter in wavelength ranges other than the second subwavelength range included in the target wavelength range. A filter array according to any one of claims 1 to 8.

10. The aforementioned plurality of optical filters further include a third filter, The third filter is, A third multimode filter having a plurality of third peak wavelengths within the target wavelength range where the light transmittance is maximized, wherein at least one of the plurality of third peak wavelengths is different from the plurality of first peak wavelengths, and at least one of the plurality of third peak wavelengths is different from the plurality of second peak wavelengths, A third band-limiting filter that limits the transmission of light in a third subwavelength region which is a part of the target wavelength region and is different from both the first subwavelength region and the second subwavelength region, including, A filter array according to any one of claims 1 to 9.

11. The aforementioned plurality of filters further include a fourth filter, The fourth filter includes a fourth multimode filter having a plurality of fourth peak wavelengths within the target wavelength range where the light transmittance is maximum, wherein at least one of the plurality of fourth peak wavelengths is different from the plurality of first peak wavelengths, and at least one of the plurality of fourth peak wavelengths is different from the plurality of second peak wavelengths. A filter array according to any one of claims 1 to 10.

12. The first transmitted light that has passed through the first multimode filter and the first band-limiting filter has at least two peak wavelengths in the target wavelength range. The second transmitted light that has passed through the second multimode filter and the second band-limiting filter has at least two peak wavelengths in the target wavelength range. A filter array according to any one of claims 1 to 11.

13. At least two of the plurality of first peak wavelengths are located in a wavelength range different from the first subwavelength range. At least two of the plurality of second peak wavelengths are included in a wavelength range different from the second subwavelength range. A filter array according to any one of claims 1 to 12.

14. A filter array according to any one of claims 1 to 13, An image sensor that receives light that has passed through the filter array and outputs an image signal based on the light, A signal processing device that generates, based on the aforementioned image signal, a plurality of image data relating to a corresponding wavelength band among a plurality of wavelength bands included in the target wavelength range, Equipped with, Imaging system.

Images