Filter array, light detection device, and light detection system

Abstract

A filter array (10) is used in a light detection system that generates image data of each of N (N is an integer of 4 or more) wavelength bands. The filter array includes a plurality of optical filters having light transmittances different from each other at the respective N wavelength bands, and when the mean value of a plurality of transmittances that correspond one-to-one with the plurality of optical filters regarding light at an i-th wavelength band (i is an integer of 1 to N inclusive) among the plurality of N wavelength bands is μi, and the standard deviation of the plurality of transmittances that correspond one-to-one with the plurality of optical filters regarding the light at the i-th wavelength band is σi, satisfies (σ1 / μ1) ≥ 0.1, …, ((σN / μN) ≥ 0.1.

Description

FILTER ARRAY, LIGHT DETECTION DEVICE, AND LIGHT DETECTION SYSTEM

[0001] The present disclosure relates to filter arrays, photodetection devices, and photodetection systems.

[0002] By utilizing spectral information from multiple narrow bands, such as several dozen bands, it is possible to obtain detailed information about the physical properties of an object that was previously impossible with conventional RGB images. A camera that captures such multi-wavelength information is called a "hyperspectral camera." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral analysis.

[0003] Patent Documents 1 and 2 disclose examples of hyperspectral cameras that utilize compressed sensing. For example, Patent Document 1 discloses an imaging device that includes an encoding element, which is an array of multiple optical filters with different wavelength dependencies of light transmittance, and an image sensor that detects light that has passed through the encoding element. The image sensor simultaneously detects light of multiple wavelength bands for each pixel to acquire a single wavelength-multiplexed image. By applying compressed sensing to the acquired wavelength-multiplexed image, images for each of the multiple wavelength bands are reconstructed.

[0004] US Patent Application Publication No. 2016 / 138975 JP 2016-100703 A

[0005] The present disclosure provides techniques for reducing errors associated with the reconstruction of images of multiple wavelength bands.

[0006] A filter array according to one aspect of the present disclosure is a filter array used in a light detection system that generates image data for each of N wavelength bands (N is an integer of 4 or more), the filter array including a plurality of optical filters with different light transmittances in each of the N wavelength bands, and a filter array that calculates an average value of a plurality of transmittances that correspond one-to-one to the plurality of optical filters for light in an i-th wavelength band (i is an integer of 1 to N) among the N wavelength bands by μ ｉand the standard deviation of the plurality of transmittances corresponding one-to-one to the plurality of optical filters for the light of the i wavelength band is σ ｉ In this case, (σ1 / μ1)≧0.1, ..., (σ Ｎ / μ Ｎ ) ≧0.1.

[0007] A comprehensive or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. When an apparatus consists of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices. The multiple devices included in a "system" may include devices installed in remote locations away from the other devices and connected via a communication network.

[0008] According to one aspect of the present disclosure, errors associated with reconstructing images of multiple wavelength bands can be reduced.

[0009] 5A is a diagram for explaining the optical characteristics of a filter array; FIG. 5B is a diagram showing an example of a histogram of transmittance; FIG. 5C is a diagram showing a schematic diagram of a light detection system according to an exemplary embodiment of the present disclosure; FIG. 5D is a diagram showing a modified example of a light detection system according to an exemplary embodiment of the present disclosure; FIG. 5E is a diagram showing another modified example of a light detection system according to an exemplary embodiment of the present disclosure; FIG. 5F is a diagram showing an example of a filter array; FIG. 5G is a diagram showing an example of the spatial distribution of light transmittance of a filter array; FIG. 5H is a diagram showing an example of the transmission spectrum of a filter; FIG. 5I is a diagram showing another example of the transmission spectrum of a filter; FIG. 5I is a diagram showing the relationship between a target wavelength range W and a plurality of wavelength bands W1, W2, ..., WN included therein; FIG. 5I is a diagram showing the relationship between a target wavelength range W and a plurality of wavelength bands W1, W2, ..., WN included therein; FIG. 5I is a diagram showing the characteristics of the transmission spectrum in a certain region of a filter array; １ , W ２ , ..., W Ｎ 6 is a diagram showing the results of averaging for each band. FIG. 7 is a diagram schematically showing a portion of a cross section of a photodetector in an exemplary embodiment. FIG. 8 is a diagram schematically showing an example of a Fabry-Perot filter. FIG. 9 is a diagram schematically showing the transmission spectrum of a filter array included in the photodetector shown in FIG. 6. FIG. 10 is a diagram schematically showing the sensitivity spectrum of an image sensor included in the photodetector shown in FIG. 6. FIG. 11 is a diagram schematically showing the output spectrum of the photodetector shown in FIG. 10. FIG. 11 is a diagram schematically showing a histogram of pixel values ​​for a certain band when the wavelength dependency of the sensitivity of the image sensor is taken into account and when it is not taken into account. FIG. 12 is a graph showing the relationship between σ / μ for all bands and the average value of MSE for all bands. FIG. 13 is a diagram for explaining restoration characteristics when an ideal filter array having similar transmission characteristics for all wavelength bands is used.

[0010] In the present disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the connection relationships within the LSI or set up circuit sections within the LSI, can also be used for the same purpose.

[0011] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices, such as interfaces.

[0012] Exemplary embodiments of the present disclosure will be described below. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts will be described as optional components. Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0013] Before describing the embodiments of the present disclosure, the findings of the present inventors will be described.

[0014] In a hyperspectral camera using compressed sensing, the optical properties of the encoding element, i.e., the optical filter array, determine the quality of the reconstructed image. In this specification, the optical filter array is simply referred to as the "filter array." Inappropriate filter array characteristics result in large errors in the reconstructed image, making it impossible to obtain a high-quality reconstructed image. Mathematically, an ideal filter array may be one that performs random sampling in space and frequency (i.e., wavelength). However, it is not easy to actually create such an ideally random filter array. Furthermore, as described below, it is necessary to design a filter array that takes into account the sensitivity characteristics of the image sensor 60. Therefore, there is room for improvement in the specific configuration of a filter array that can reduce errors associated with the reconstruction of images of multiple wavelength bands.

[0015] An outline of an embodiment of the present disclosure will be described below.

[0016] FIG. 1A is a diagram illustrating the optical characteristics of a filter array 10 according to an embodiment of the present disclosure. The filter array 10 shown in FIG. 1A includes multiple optical filters. The multiple optical filters are arranged two-dimensionally. The multiple optical filters include multiple types of optical filters with different light transmission characteristics. The multiple optical filters can be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure including a metal. The filter array 10 is used in a photodetector that generates image data for each of multiple wavelength bands. The number of wavelength bands is N (N is an integer greater than or equal to 4). The distribution of light transmittance of the filter array 10 differs for each wavelength band. In FIG. 1, the spatial pattern of light transmittance for each wavelength band is expressed as a mosaic pattern.

[0017] Here, consider a histogram of the transmittance of the multiple optical filters in the filter array 10 for the i-th wavelength band (i is an integer between 1 and N). FIG. 1B shows an example of a histogram of the transmittance of the filter array 10 according to an embodiment of the present disclosure. This histogram represents a distribution with the horizontal axis representing the transmittance and the vertical axis representing the number of filters having that transmittance. From this histogram, the average transmittance μ for light in the i-th wavelength band can be calculated. ｉ and standard deviation σ ｉ The histogram of the transmittance of the filter array 10 in the embodiment of the present disclosure has a finite standard deviation σ ｉ The average transmittance μ ｉ and standard deviation σ ｉ can be derived as follows:

[0018] For light in the i-th wavelength band (i is an integer between 1 and N) among the N wavelength bands, the average transmittance of the multiple optical filters included in the filter array 10 is μ ｉ The filter array 10 includes M filters (M is an integer of 4 or more), and the transmittance of the jth filter (j is an integer of 1 to M) among the M filters for light in the i-th wavelength band is defined as T ｉｊ Then, the average transmittance μ ｉ is expressed by the following equation (1).

[0019]

[0020] Average value μ of transmittance for N wavelength bands ｉ The standard deviation of σ μ Then, σ μ is expressed by the following equation (2).

[0021]

[0022] A histogram of the number of filters included in the filter array 10 versus the transmittance of light in the i-th wavelength band can be obtained by measuring the transmittance of each optical filter in the filter array 10 using a photodetector that detects light intensity at a predetermined number of gradations. For example, the histogram can be obtained using a photodetector such as an image sensor that can detect a two-dimensional distribution of light intensity at a predetermined number of gradations, such as 8 or 16 bits. Specifically, the transmittance of light in the i-th wavelength band of each filter in the filter array 10 can be calculated from the ratio of the light intensity of the i-th wavelength band detected with the filter array 10 in place to the light intensity of the i-th wavelength band detected without the filter array 10 in place. A histogram such as that illustrated in FIG. 1B can be obtained from the transmittance data of each filter obtained by the above method. Furthermore, if it is difficult to obtain filter transmittance data, a histogram that takes into account the wavelength dependency of the sensitivity of the image sensor can be obtained based on pixel values ​​output from the image sensor that detects light passing through the filter array 10. The histogram obtained based on the pixels output from the image sensor reflects the transmittance characteristics of the filters. For simplicity, Figure 1B illustrates a histogram close to a normal distribution. In an actual filter array 10, a histogram with a different shape than that shown in Figure 1B may be obtained. Because the wavelength dependence of transmittance differs depending on the filter, the shape of the histogram differs for each wavelength band. Therefore, the average value and standard deviation of the transmittance of multiple filters also differ for each wavelength band.

[0023] In a hyperspectral camera using compressed sensing, images of multiple wavelength bands are estimated and acquired by using a compressed sensing technique to solve an ill-posed problem in which the optical properties of the filter array 10 are used as parameters. As will be described in detail below, the inventors have found that in the case of the recursive iterative calculation used in compressed sensing, the more uniform the average value of the transmittance for each wavelength band of the multiple filters in the filter array 10 is, and the larger the standard deviation of the transmittance is, the better the convergence of the solution and the smaller the error in the reconstructed image.

[0024] That is, the inventors have discovered that, in order to reduce errors associated with the reconstruction of images of multiple wavelength bands, it is preferable to design the filter array 10 so that the average transmittance for each wavelength band is uniform and the standard deviation of the transmittance is equal to or greater than a certain value. However, because light that passes through the filter array 10 is detected by an image sensor whose sensitivity is wavelength-dependent, the average value and standard deviation of the output pixel values ​​in a photodetector including the filter array 10 and the image sensor differ from the average value and standard deviation of the transmittance of the filter array 10, respectively. Therefore, in practice, the filter array 10 must be designed taking into account the wavelength-dependence of the sensitivity of the image sensor.

[0025] The inventors have identified the above-mentioned problems and investigated the configuration of a filter array 10 to solve these problems. According to an embodiment of the present disclosure, the filter array 10 is designed so that the value obtained by dividing the standard deviation of transmittance by the average transmittance for all bands is equal to or greater than a certain value. This design reduces image restoration errors for each band, regardless of the wavelength dependence of the sensitivity of the image sensor. Below, a filter array, a photodetector, and a photodetection system according to an embodiment of the present disclosure are described.

[0026] The filter array according to the first item is a filter array used in a light detection system that generates image data for each of N wavelength bands (N is an integer of 4 or more), and includes a plurality of optical filters with different light transmittances in each of the N wavelength bands, and an average value of a plurality of transmittances that correspond one-to-one to the plurality of optical filters for light in an i-th wavelength band (i is an integer of 1 to N) among the N wavelength bands is calculated as μ ｉ and the standard deviation of the plurality of transmittances corresponding one-to-one to the plurality of optical filters for the light of the i wavelength band is σ ｉ In this case, (σ1 / μ1)≧0.1, ..., (σ Ｎ / μ Ｎ ) ≧0.1.

[0027] This filter array can reduce errors associated with reconstructing images of multiple wavelength bands.

[0028] A filter array according to a second item is the filter array according to the first item, wherein at least one of the plurality of optical filters is a Fabry-Perot filter.

[0029] In this filter array, Fabry-Perot filters are used to reduce σ ｉ / μ ｉ It is possible to realize that the value is 0.1 or more.

[0030] The filter array according to the third item is the filter array according to the first or second item, wherein at least one of the plurality of optical filters has a resonant structure including a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer, and having a plurality of resonant modes of different orders.

[0031] In this filter array, different transmission spectra can be achieved for each filter by changing the refractive index or thickness of the intermediate layer for each filter.

[0032] A photodetector according to a fourth aspect is a photodetector used in a photodetection system that generates image data for each of N wavelength bands (N is an integer equal to or greater than 4). The photodetector includes a plurality of optical filters having different light transmittances in the N wavelength bands, and an image sensor that detects light that has passed through the plurality of optical filters. The image sensor detects only light corresponding to an i-th wavelength band (i is an integer equal to or greater than 1 and equal to or less than N) of the N wavelength bands, and outputs data indicating a pixel value distribution corresponding to the i-th wavelength band. The average pixel value of the pixel value distribution corresponding to the i-th wavelength band is expressed as μ ｉ and the standard deviation of the pixel values ​​of the pixel value distribution corresponding to the i wavelength band is σ ｉ In this case, (σ1 / μ1)≧0.1, ..., (σ Ｎ / μ Ｎ ) ≧0.1.

[0033] This photodetector can reduce errors associated with reconstructing images of multiple wavelength bands.

[0034] A photodetector according to a fifth aspect is the photodetector according to the fourth aspect, wherein at least one of the plurality of optical filters is a Fabry-Perot filter.

[0035] In this photodetector, a Fabry-Perot filter is used to reduce σ ｉ / μ ｉ It is possible to realize that the value is 0.1 or more.

[0036] A photodetector according to a sixth item is the photodetector according to the fourth or fifth item, wherein at least one of the plurality of optical filters includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer, and has a resonant structure having a plurality of resonant modes of different orders.

[0037] In this filter array, different transmission spectra can be achieved for each filter by changing the refractive index or thickness of the intermediate layer for each filter.

[0038] A seventh aspect of the present invention relates to a photodetector device according to any one of the fourth to sixth aspects, wherein the transmission spectrum of each of the plurality of optical filters has a maximum value of transmittance at each of a plurality of wavelengths included in a target wavelength range, and the image sensor includes a plurality of photodetector elements, each of which is disposed at a position to receive transmitted light that has passed through at least one of the plurality of optical filters and detects light of the plurality of wavelengths included in the transmitted light.

[0039] In this light detection device, by processing the signals output by the image sensor that detects the light, it is possible to reconstruct images in a plurality of wavelength bands.

[0040] An optical detection system according to an eighth item includes an optical detection device according to any one of the fourth to seventh items, and a signal processing circuit that generates the image data for each of the N wavelength bands based on a signal output from the image sensor.

[0041] This optical detection system can reduce errors associated with reconstructing images of multiple wavelength bands.

[0042] A light detection system according to a ninth item is the light detection system according to the eighth item, wherein the signal processing circuit generates the image data by performing a calculation using compressed sensing.

[0043] This light detection system can generate image data for each of the N wavelength bands with high accuracy.

[0044] In this specification, a signal representing an image (that is, a set of signals representing the pixel values ​​of each pixel) may be simply referred to as an “image.” In the following description, xyz coordinates shown in the drawings will be used.

[0045] 2A is a diagram schematically illustrating a light detection system 400 according to an exemplary embodiment of the present disclosure. The light detection system 400 includes an optical system 40, a filter array 10, an image sensor 60, and a signal processing circuit 200. The filter array 10 has a function similar to that of the "encoding element" disclosed in Patent Document 1. For this reason, the filter array 10 can also be referred to as the "encoding element." The optical system 40 and the filter array 10 are disposed on the optical path of light incident from an object 70.

[0046] The filter array 10 has a plurality of light-transmitting regions arranged in rows and columns. The filter array 10 is an optical element in which the light transmission spectrum, i.e., the wavelength dependency of light transmittance, varies from region to region. The filter array 10 modulates the intensity of incident light before transmitting it. The configuration of the filter array 10 will be described in detail below.

[0047] The filter array 10 may be disposed near or directly above the image sensor 60. Here, "near" means close enough that a light image from the optical system 40 is formed on the surface of the filter array 10 with a certain degree of clarity. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 10 and the image sensor 60 may be integrated. An apparatus including the filter array 10 and the image sensor 60 is referred to as a "photodetection apparatus 300."

[0048] The filter array 10 may be located away from the image sensor 60. FIGS. 2B and 2C show examples of configurations in which the filter array 10 is located away from the image sensor 60. In the example of FIG. 2B , the filter array 10 is located between the optical system 40 and the image sensor 60. In the example of FIG. 2C , the filter array 10 is located between the object 70 and the optical system 40. In these examples, the image encoded by the filter array 10 is acquired in a blurred state on the imaging surface of the image sensor 60. Therefore, by storing this blur information in advance and reflecting the blur information in the system matrix H used in the calculation process described below, the separated image 220 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to one pixel on an image spreads due to blurring to a region of k × k pixels around that pixel, the PSF can be defined as a set of coefficients, i.e., a matrix, that indicates the influence on the pixel values ​​of each pixel within that region. The separated image 220 can be reconstructed by reflecting the influence of blurring of the coding pattern due to the PSF in a system matrix H, which will be described later. The filter array 10 may be positioned at any position, but a position can be selected that prevents the coding pattern of the filter array 10 from being lost due to excessive diffusion.

[0049] The optical system 40 includes at least one lens. Although shown as a single lens in FIG. 1, the optical system 40 may be a combination of multiple lenses. The optical system 40 forms an image on the imaging surface of the image sensor 60 via the filter array 10.

[0050] The image sensor 60 is a monochrome photodetector having a plurality of photodetection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 60 may be, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor, an infrared array sensor, a terahertz array sensor, or a millimeter-wave array sensor. The photodetection elements include, for example, photodiodes. The image sensor 60 does not necessarily have to be a monochrome sensor. For example, the image sensor 60 may be a color sensor. Color sensors include sensors having filters that transmit red light, green light, and blue light; sensors having filters that transmit red light, green light, blue light, and infrared light; or sensors having filters that transmit red light, green light, blue light, and white light. By using a color type sensor, the amount of information related to wavelength can be increased, and the accuracy of reconstructing the separated image 220 can be improved. However, when a color type sensor is used, the amount of information related to wavelength decreases in the spatial direction (x, y directions), so there is a trade-off between the amount of information related to wavelength and the resolution. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, and may also be the wavelength range of ultraviolet, near-infrared, mid-infrared, far-infrared, microwave, or radio wave.

[0051] The signal processing circuit 200 reconstructs a plurality of separated images 220 including information of multiple wavelengths based on the image 120 acquired by the image sensor 60. Details of the plurality of separated images 220 and the method of processing the image signals by the signal processing circuit 200 will be described later. The signal processing circuit 200 may be incorporated into the photodetector 300, or may be a component of a signal processing device electrically connected to the photodetector 300 by wire or wirelessly.

[0052] The filter array 10 in this embodiment will be described below. The filter array 10 is disposed on the optical path of light incident from the object 70, and modulates the intensity of the incident light for each wavelength and outputs the modulated light. This process performed by the filter array is referred to as "encoding" in this specification.

[0053] 3A is a diagram schematically illustrating an example of a filter array 10. The filter array 10 has a plurality of regions arranged two-dimensionally. In this specification, each of the plurality of regions may be referred to as a "cell." An optical filter having an individually set transmission spectrum is disposed in each region. The transmission spectrum is expressed by a function T(λ), where λ is the wavelength of incident light. The transmission spectrum T(λ) can take a value between 0 and 1.

[0054] In the example of FIG. 3A , the filter array 10 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in a typical photodetector, such as an image sensor. The number of pixels may be, for example, hundreds of thousands to tens of millions. In the example of FIG. 2A , the filter array 10 is disposed directly above the image sensor 60, and each region is disposed so as to correspond to one pixel of the photodetector. Each region faces, for example, one pixel of the image sensor 60.

[0055] FIG. 3B shows the wavelength band W included in the target wavelength range. １ , wavelength band W ２ , ..., wavelength band W Ｎ 3B is a diagram showing an example of the spatial distribution of light transmittance of each of the wavelength bands. In the example of FIG. 3B, the difference in shading of each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 3B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0056] 3C and 3D are diagrams showing examples of the transmission spectra of region A1 and region A2 included in the multiple regions of the filter array 10 shown in FIG. 3A . The transmission spectrum of region A1 and the transmission spectrum of region A2 are different from each other. Thus, the transmission spectra of the filter array 10 vary from region to region. However, the transmission spectra of all regions do not necessarily need to be different. In the filter array 10, the transmission spectra of at least some of the multiple regions are different from each other. The filter array 10 includes two or more filters with different transmission spectra. In some examples, the number of transmission spectrum patterns of the multiple regions included in the filter array 10 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 10 may be designed so that half or more of the regions have different transmission spectra. Furthermore, as shown in FIGS. 3C and 3D , each filter may be designed to transmit light of wavelengths corresponding to multiple peaks in the target wavelength range.

[0057] 4A and 4B show the target wavelength range W and the wavelength bands W included therein. １ , wavelength band W ２ , ..., wavelength band W Ｎ 1 is a diagram for explaining the relationship between the wavelengths of the visible light and the near-infrared light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, the near-ultraviolet wavelength range of about 10 nm to about 400 nm, or radio wave ranges such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. Thus, the wavelength range used is not limited to the visible light range. In this specification, for convenience, "light" refers not only to visible light but also to non-visible light such as near-ultraviolet light, near-infrared light, and radio waves.

[0058] In the example shown in FIG. 4A , N is an arbitrary integer equal to or greater than 4, and each wavelength range obtained by dividing the target wavelength range W into N equal parts is called a wavelength band W １ , wavelength band W ２ , ..., wavelength band W ＮHowever, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 4B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the multiple wavelength bands only need to be different from each other, and the method for determining this may be arbitrary. The number of wavelength divisions N may be 3 or less.

[0059] 5A is a diagram illustrating the characteristics of the transmission spectrum in a certain region of the filter array 10. In the example shown in FIG. 5A, the transmission spectrum has multiple maximum values ​​(i.e., maximum values ​​P1 to P5) and multiple minimum values ​​for wavelengths within the target wavelength band W. In the example shown in FIG. 5A, the optical transmittance within the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 5A, the wavelength band W ２ , and the wavelength band W Ｎ－１ In this embodiment, the transmission spectrum of each region has a maximum value in a wavelength range such as a plurality of wavelength bands W １ From W Ｎ As a result, the transmission spectrum of each region has a maximum value of transmittance at each of the multiple wavelengths included in the target wavelength range W. As can be seen from FIG. 5A , the maximum values ​​P1, P3, P4, and P5 are all 0.5 or greater.

[0060] As described above, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 10 transmits a large amount of components in a certain wavelength range among the incident light, while not transmitting components in other wavelength ranges as much. For example, the transmittance of light in k wavelength bands out of N wavelength bands may be greater than 0.5, while the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light is white light that contains all visible light wavelength components equally, the filter array 10 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light in a superimposed form.

[0061] FIG. 5B shows an example of the transmission spectrum shown in FIG. 5A in a wavelength range W １ , wavelength range W ２ ,..., wavelength range W Ｎ 1 shows the results of averaging for each wavelength band. The averaged transmittance is obtained by integrating the transmission spectrum T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the transmittance value averaged for each wavelength band in this manner is referred to as the transmittance for that wavelength band. In this example, the transmittance is remarkably high in the three wavelength ranges with maximum values ​​P1, P3, and P5. In particular, the transmittance exceeds 0.8 in the two wavelength ranges with maximum values ​​P3 and P5.

[0062] When the filter array 10 is disposed near or directly above the photodetector, the cell pitch, which is the spacing between multiple regions in the filter array 10, may be approximately equal to the pixel pitch of the photodetector. In this way, the resolution of the encoded light image emitted from the filter array 10 approximately matches the pixel resolution. By ensuring that light transmitted through each cell is incident on only one corresponding pixel, the calculation described below can be facilitated. When the filter array 10 is disposed away from the photodetector, the cell pitch may be made finer depending on the distance.

[0063] In the examples shown in Figures 3A to 3D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary scale transmittance distribution may be employed in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two wavelength ranges out of multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most of the region" refers to approximately 80% or more.

[0064] A portion of all cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength bands W included in the wavelength band of interest. １ From W Ｎ The filter array 10 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the plurality of transparent regions may be arranged, for example, in a checkerboard pattern. That is, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately in two arrangement directions of the plurality of regions in the filter array 10.

[0065] <Example of Signal Processing> Next, an example of processing by the signal processing circuit 200 will be described. The signal processing circuit 200 reconstructs a multi-wavelength separated image 220 based on the image 120 output from the image sensor 60 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 10. Here, "multiple wavelengths" means a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a typical color camera, for example. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the number of bands. Depending on the application, the number of bands may exceed 100.

[0066] The data to be obtained is the separated image 220, and this data is denoted as f. If the number of spectral bands is N, then f is the wavelength band W １ Image data f corresponding to １ , wavelength band W ２ Image data f corresponding to ２ , ..., wavelength band W Ｎ Image data f corresponding to ＮIf the number of pixels in the x direction of the image data to be obtained is n and the number of pixels in the y direction is m, then the image data f １ , image data f ２ , ..., image data f Ｎ Each of the elements f is two-dimensional data of n×m pixels. Therefore, the data f is three-dimensional data of n×m×N elements. On the other hand, the number of elements of the data g of the image 120 obtained by encoding and multiplexing by the filter array 10 is n×m. The data g of the image 120 in this embodiment can be expressed by the following equation (3).

[0067]

[0068] where f １ , f ２ , ..., f Ｎ Each of the elements in the matrix H is data having n×m elements. Therefore, the vector on the right side is a one-dimensional vector with n×m×N rows and one column. The data g of the image 120 in the matrix H is a one-dimensional vector with n×m rows and one column. １ , f ２ , ..., f Ｎ represents a transformation in which different coded information is used for each wavelength band, and the resulting signals are intensity-modulated and encoded with different coded information, and then added together. Therefore, H is a matrix with n×m rows and n×m×N columns. In this specification, the matrix H may be referred to as a "system matrix." The function of the system matrix H in equation (3) corresponds to the image sensor 60 acquiring an image that has been coded and intensity-modulated through the filter array 10. The elements of the matrix H depend on the transmittance of each optical filter in the filter array 10 and the wavelength dependency of the sensitivity of the image sensor 60.

[0069] Now, if vector g and matrix H are given, it seems possible to calculate f by solving the inverse problem of equation (3). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem becomes an ill-posed problem and cannot be solved as is. Therefore, the signal processing circuit Pr of this embodiment utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (4).

[0070]

[0071] Here, f' represents the estimated data of f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term, which will be described later. Equation (4) means that f that minimizes the sum of the first and second terms is found. The signal processing circuit 200 can converge the solution by recursive iterative calculations and calculate the final solution f'.

[0072] The first term in the parentheses in Equation (4) represents a calculation to obtain the sum of squares of the difference between the acquired data g and Hf, which is the system transformation of f in the estimation process using the matrix H. The second term, Φ(f), is a constraint for the regularization of f and is a function that reflects the sparsity information of the estimation data. Its function is to smooth or stabilize the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, when total variation is used, stable estimation data that suppresses the influence of noise in the observation data g can be obtained. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the calculation. τ is a weighting coefficient. The larger the weighting factor τ, the more redundant data is reduced, and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent, but does not result in over-compression.

[0073] Included in formula (3) and formula (4)

[0074]

[0075] is sometimes denoted as g in descriptions relating to formulas (3) and (4).

[0076] Here, we qualitatively explain the results of the inventors' study on the influence of the optical properties of the filter array 10 on the convergence of the solution of Equation (4). First, the standard deviation of the transmittance for each wavelength band of multiple filters corresponds to the randomness in the encoding for each wavelength band. Therefore, it is believed that increasing the standard deviation can improve encoding performance and convergence of the solution. Next, if the average values ​​of the transmittances of multiple filters are not uniform for each wavelength band, the data f in the estimation process will also not be uniform for each wavelength band due to the effect of transformation by the system matrix H. As a result, the effect of the regularization term in Equation (4) in smoothing or stabilizing the estimated data differs for each wavelength band, which may reduce the convergence of the solution. From another perspective, it is believed that the convergence of the solution can be improved by uniforming the average values ​​of the transmittances of multiple filters for each wavelength band.

[0077] To summarize the above, the inventors have discovered that in order to reduce errors associated with the reconstruction of images of multiple wavelength bands, it is preferable to design the filter array 10 so that the average transmittance for each wavelength band is uniform and the standard deviation of the transmittance is equal to or greater than a certain value.

[0078] Although an example of calculation using compressed sensing shown in Equation (4) has been shown here, other methods may be used for the solution. For example, other statistical methods such as maximum likelihood estimation or Bayesian estimation may be used. Furthermore, the number of separated images 220 may be any number, and each wavelength band may be set arbitrarily. Details of the reconstruction method are disclosed in Patent Document 1. The entire disclosure of Patent Document 1 is incorporated herein by reference.

[0079] <Detailed Configuration of Filter Array> Next, a specific configuration example of the filter array 10 that reduces errors in the reconstructed image will be described.

[0080] In the following description, each filter in the filter array 10 is assumed to be a Fabry-Perot (FP) filter. The FP filter includes a first reflective layer, a second reflective layer, and an intermediate layer between the first and second reflective layers. Each reflective layer can be formed from either a dielectric multilayer film or a metal thin film. The intermediate layer has a thickness and refractive index that form a resonant structure with at least one resonant mode. A resonant structure is a structure in which light of a certain wavelength exists stably by forming a standing wave therein. This state of light is called a "resonant mode." The transmittance of light of a wavelength corresponding to the resonant mode is high, while the transmittance of light of other wavelengths is low. By varying the refractive index or thickness of the intermediate layer for each filter, different transmission spectra can be achieved for each filter.

[0081] FIG. 6 is a diagram schematically illustrating a portion of a cross section of a photodetector 300 according to this embodiment. The photodetector 300 includes a filter array 10 and an image sensor 60. The filter array 10 includes a plurality of filters 100 arranged two-dimensionally. The plurality of filters 100 are arranged in rows and columns. FIG. 6 schematically illustrates the cross-sectional structure of one row. Each of the plurality of filters 100 has a resonant structure. The resonant structure illustrated in FIG. 6 includes a first reflective layer 28a, a second reflective layer 28b, and an intermediate layer 26 between the first reflective layer 28a and the second reflective layer 28b. Each of the first reflective layer 28a and the second reflective layer 28b may be formed from a dielectric multilayer film or a metal thin film. The intermediate layer 26 may be formed from a dielectric or semiconductor that is transparent in a specific wavelength range. The intermediate layer 26 may be formed from, for example, Si, Si ３ N ４ , TiO ２ , Nb ２ O ５ , Ta ２ O ５The filters 100 may be formed from at least one selected from the group consisting of: At least one of the refractive index and thickness of the intermediate layer 26 of each of the filters 100 varies depending on the filter. In the example shown in FIG. 6 , the transmission spectrum of each of the filters 100 has a maximum transmittance value at each of a plurality of wavelengths. The plurality of wavelengths correspond to a plurality of resonance modes of different orders in the above-described resonant structure. In this embodiment, all of the filters 100 in the filter array 10 have the above-described resonant structure. The filter array 10 may include filters other than FP filters as long as it includes at least one FP filter. For example, the filter array 10 may include filters whose light transmittance does not depend on wavelength, such as transparent filters or ND (Neutral Density) filters.

[0082] The image sensor 60 includes a plurality of photodetector elements 60a. Each of the photodetector elements 60a may be disposed, for example, at a position where it receives light transmitted through at least one of the filters 100. Each of the photodetector elements 60a detects light of a plurality of wavelengths at which the transmission spectrum of each filter 100 exhibits a maximum value. In the example of Fig. 6, each of the photodetector elements 60a is disposed opposite one of the filters.

[0083] Each of the multiple photodetector elements 60a is sensitive to light in a specific wavelength range. This specific wavelength range corresponds to the target wavelength range W described above. In this disclosure, "sensitive to light in a certain wavelength range" refers to having the substantial sensitivity required to detect light in that wavelength range. For example, this refers to an external quantum efficiency of 1% or more in that wavelength range. The external quantum efficiency of the photodetector element 60a may be 10% or more. The external quantum efficiency of the photodetector element 60a may be 20% or more. All of the multiple wavelengths at which the light transmittance of each filter 100 has a maximum value are included in the target wavelength range W. In other words, the photodetector device 300 in this embodiment can simultaneously detect light of multiple wavelengths that have passed through at least one filter, which is different for each pixel 60a.

[0084] Next, a configuration example in which each of the first reflective layer 28a and the second reflective layer 28b is formed from a dielectric multilayer film will be described.

[0085] 7 is a diagram schematically illustrating an example of a filter 100 in which each reflective layer is formed from a dielectric multilayer film. The filter 100 is provided on a substrate 80. Each of the first reflective layer 28a and the second reflective layer 28b is formed from a dielectric multilayer film. That is, each of the first reflective layer 28a and the second reflective layer 28b has a structure in which a plurality of low-refractive-index layers 27l and a plurality of high-refractive-index layers 27h are alternately positioned. Each of the plurality of low-refractive-index layers 27l has a refractive index n ｌ Each of the plurality of high refractive index layers 27h has a refractive index n ｌ A refractive index n ｈ The low refractive index layer 27l in the first reflective layer 28a and the low refractive index layer 27l in the second reflective layer 28b may have the same refractive index or different refractive indexes. The high refractive index layer 27h in the first reflective layer 28a and the high refractive index layer 27h in the second reflective layer 28b may have the same refractive index or different refractive indexes.

[0086] The dielectric multilayer film includes a plurality of pair layers. Each pair layer includes one low-refractive-index layer 27l and one high-refractive-index layer 27h. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include four pair layers including eight refractive index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include four pair layers including eight refractive index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include eight low-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include eight high-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include four pair layers including eight high-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include ... four pair layers including eight high-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include eight low-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include eight high-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second reflective layer 28b each include eight low-refractive-index layers. In the example of Fig. 7, the first reflective layer 28a and the second ０ In order to obtain a high reflectance at t ｈ = λ ０ / (4n ｈ ), and the thickness of the low refractive index layer 27l is set to t ｌ = λ ０ / (4n ｌ In other words, the thickness t ｈ and the thickness t of the low refractive index layer 27l ｌ The optical length of ０ Here, the optical length means the value obtained by multiplying the thickness by the refractive index. ０ is, for example, the central wavelength of the target wavelength range W (λ ｉ +λ ｅ) / 2.

[0087] <Configuration Example of Filter Array for Reducing Reconstruction Error> Next, a configuration example of the filter array 10 for reducing reconstruction error will be described.

[0088] First, we will explain the impact of using a filter array 10 composed of multiple FP filters in a hyperspectral camera that performs restoration processing using compressed sensing. According to the above discussion, by appropriately designing each reflective layer and intermediate layer of the FP filter, it is possible to design the filter array 10 so that the average transmittance for each wavelength band is uniform and the standard deviation of the transmittance is equal to or greater than a certain value. As a result, it is possible to improve the convergence of the solution to ill-posed problems in compressed sensing and reduce errors associated with reconstructing images of multiple wavelength bands.

[0089] However, in general, the sensitivity of the image sensor 60 is wavelength-dependent. In a hyperspectral camera using compressed sensing, the image sensor 60 detects light passing through the filter array 10, and a multi-wavelength separated image 220 is reconstructed based on pixel value signals output from the image sensor 60. Therefore, when reconstructing the multi-wavelength separated image 220, it is necessary to consider the sensitivity characteristics of the image sensor 60 in addition to the transmission characteristics of the filter array 10.

[0090] Therefore, when considering the wavelength dependency of the sensitivity of the image sensor, in order to reduce the restoration error, it is necessary to make the average value of the pixel values ​​for each band output from the image sensor uniform across all bands, rather than making the average value of the transmittance of the filter array 10 uniform across all bands. Similarly, it is necessary to make the standard deviation of the pixel values ​​for each band output from the image sensor uniform across all bands, rather than making the standard deviation of the transmittance of the filter array 10 equal to or greater than a certain value across all bands.

[0091] In this specification, the pixel value for a certain band may refer to a signal value output by detecting only light corresponding to that band via the filter array 10. A method of causing the photodetector 300 to detect only light corresponding to a certain band includes, for example, a method using a tunable laser and an integrating sphere. Specifically, light corresponding to a certain wavelength band emitted by a tunable laser is incident on an integrating sphere, and the light spatially homogenized by the integrating sphere is detected by the photodetector 300. Furthermore, when causing the photodetector 300 to detect only light corresponding to a certain wavelength band, light having a wavelength shifted by several nanometers from the wavelength range corresponding to the certain wavelength band may be incident. In other words, light having a wavelength several nanometers smaller than the lower limit of the wavelength range corresponding to the certain wavelength band, or light having a wavelength several nanometers larger than the upper limit of the wavelength range corresponding to the certain wavelength band, may be incident on the photodetector 300.

[0092] Next, the wavelength dependency of the signal output from the photodetector 300 shown in Fig. 6 will be described with reference to Figs. 8A to 8C. Fig. 8A is a diagram schematically showing the transmission spectrum of the filter array 10 included in the photodetector 300 shown in Fig. 6. Fig. 8B is a diagram schematically showing the sensitivity spectrum of the image sensor 60 included in the photodetector 300 shown in Fig. 6. Fig. 8C is a diagram schematically showing the output spectrum of the photodetector 300 shown in Fig. 6. The output signal from the photodetector 300 includes pixel value information.

[0093] As shown in FIG. 8A , the transmission spectrum of the filter array 10 has multiple transmittance maxima in the target wavelength range W. As shown in FIG. 8B , the sensitivity of the image sensor 60 is maximized at a certain wavelength within the target wavelength range W and decreases toward the longer and shorter wavelength ends of the target wavelength range W. For example, a silicon sensor used for detecting light in the visible light range has the highest sensitivity near a wavelength of 500 nm and decreases toward the shorter and longer wavelength ends of the visible light range. Sensitivity of any image sensor, not just silicon sensors, exhibits some wavelength dependence due to the photoelectric conversion material and / or sensor structure. As shown in FIG. 8C , the output spectrum of the photodetector device 300 differs from the transmission spectrum of the filter array 10 by the amount multiplied by the sensitivity spectrum of the image sensor 60 shown in FIG. 8B . The dashed line in FIG. 8C represents the sensitivity spectrum shown in FIG. 8B .

[0094] Suppose that improvements in FP filter design and processing technology have made it possible to fabricate a filter array 10 with an ideal transmittance distribution that is spatially and wavelength-random. Even in such a case, because the sensitivity of the image sensor 60 is wavelength-dependent, the pixel value distribution obtained from the output signal of a photodetector device 300 in which the filter array 10 and the image sensor 60 are integrally formed may not be spatially and wavelength-random. To obtain an ideal pixel value distribution that is spatially and wavelength-random, it is conceivable to design the filter array 10 by taking into account the sensitivity spectrum of the image sensor 60. However, if the sensitivity spectrum of the image sensor 60 changes over time, such a filter array 10 design would be ineffective.

[0095] Here, we consider the case where appropriate signal processing is performed on the signal output from the image sensor to cancel out the wavelength dependency of the image sensor's sensitivity. Specifically, we consider the case where gain adjustment is performed on pixel values ​​for a certain wavelength band included in the wavelength range where the image sensor's sensitivity is reduced, thereby making the average value of the pixel values ​​for each band output from the image sensor uniform across all bands. Let μ be the average pixel value before gain adjustment, and μ' be the average pixel value after gain adjustment. Similarly, let σ be the standard deviation of the pixel values ​​before gain adjustment, and σ' be the standard deviation of the pixel values ​​after gain adjustment. Figure 9 is a schematic diagram showing histograms of pixel values ​​for a certain band before and after gain adjustment. As shown in Figure 9, assume that gain adjustment is performed for a certain wavelength band so that the average pixel value changes from μ = 40 to μ' = 80. In this case, the standard deviation of the pixel values ​​changes from σ = 50 - 30 = 20 to σ' = 100 - 60 = 40. This shows that gain adjustment changes the standard deviation along with the average pixel value.

[0096] Although it is possible to uniformly set the average pixel values ​​output from the image sensor by adjusting the gain, the gain adjustment does not necessarily result in a standard deviation of the pixel values ​​across all bands that is sufficient to reduce the restoration error. Therefore, the inventors have discovered the need to design a filter array 10 that takes into account the wavelength dependency of the sensitivity of the image sensor and gain adjustment to cancel out the wavelength dependency of the sensitivity of the image sensor. The inventors have discovered that by using a filter array 10 in which the value σ / μ obtained by dividing the standard deviation σ of the transmittance by the average transmittance μ for all bands is equal to or greater than a certain value, the error between the correct image and the restored image for each band can be reduced, regardless of the wavelength dependency of the sensitivity of the image sensor 60.

[0097] As described above, by performing gain adjustment on pixel values ​​for each band, it is possible to make the average value of pixel values ​​uniform across all bands. σ / μ can be a physical quantity obtained by normalizing the standard deviation σ with the average value μ. In other words, σ / μ being equal to or greater than a certain value can mean that when the average values ​​of pixel values ​​across all bands are made uniform by gain adjustment or the like, the standard deviation of pixel values ​​across all bands is equal to or greater than a certain value.

[0098] Furthermore, σ / μ is independent of the wavelength dependency of the sensitivity of the image sensor and signal processing such as gain adjustment. In other words, if the value obtained by dividing the average transmittance of the filter array 10 by the standard deviation of the transmittance in a certain band is σ1 / μ1, and the value obtained by dividing the average pixel value output by the image sensor by the standard deviation of the pixel value is σ2 / μ2, then σ1 / μ1 = σ2 / μ2 holds. Furthermore, as explained with reference to FIG. 9, the index σ / μ does not change even if the pixel values ​​of all pixels in each band are multiplied by a constant.

[0099] In general, it is possible to reduce the restoration error by designing the filter array 10 using σ / μ as an index, which is a value that can specify the standard deviation of the pixel values ​​output by the image sensor and is independent of the wavelength dependency of the sensitivity of the image sensor and signal processing such as gain adjustment.

[0100] Next, referring to Fig. 10, we will explain how the restoration characteristics depend on σ / μ for all bands. In the following example, mean square error (MSE) is used as the error. MSE is calculated by the following equation (5).

[0101]

[0102] Here, n and m represent the number of pixels in the vertical and horizontal directions, respectively. ｉ、ｊ I′ represents the pixel value of the ground truth image at the pixel at position (i, j). ｉ、ｊrepresents the pixel value of the reconstructed image of each band at the pixel at position (i, j). FIG. 10 is a graph showing the relationship between σ / μ for all bands and the average value of MSE for all bands. In the example shown in FIG. 10, the average value and standard deviation of pixel values ​​in all bands are uniform. As shown in FIG. 10, the average value of MSE for all bands decreases exponentially as σ / μ for all bands increases. When σ / μ for all bands is 0.1 or more, the average value of MSE for all bands is less than 30. When σ / μ for all bands is 0.2 or more, the average value of MSE for all bands is less than 10. When σ / μ for all bands is 0.3 or more, the average value of MSE for all bands is less than 6. In this embodiment, σ of any i-th band among N bands is ｉ / μ ｉ may be set to be, for example, 0.1 or greater, 0.2 or greater, or 0.3 or greater.

[0103] A filter array 10 that satisfies such conditions can be realized by appropriately designing the filters 100 included in the filter array 10. Specifically, in the filter 100 shown in Figure 7, the number of layers, refractive index, and thickness of the high-refractive-index layers 27h and low-refractive-index layers 27l included in each of the first reflective layer 28a and the second reflective layer 28b, as well as the refractive index and thickness of the intermediate layer 26, are appropriately designed. The filter array 10 can be formed integrally on the image sensor 60 by a known semiconductor process.

[0104] FIG. 11 is a diagram illustrating the restoration characteristics when a filter array 10 with a σ / μ of 0.2 in each wavelength band is used. In the example of FIG. 11, ten bands 1 to 10 are assumed. The transmittance distribution of each band is given by random numbers ranging from 0.0 to 1.0, following a normal distribution with an average transmittance of 0.5 and a standard deviation of 0.1. In this example, the subject is a color chart containing 24 color samples arranged in a matrix. The bottom panel of FIG. 11 shows an example of an image for each band restored by performing the aforementioned compressed sensing process on an image acquired by a hyperspectral camera. The middle panel of FIG. 11 shows the correct image. In this example, a 640 x 480 two-dimensional filter array 10 is used. An image sensor 60 is also used that expresses pixel values ​​in 8-bit gradations (i.e., 0 to 255).

[0105] In this example, since σ / μ > 0.1 is satisfied for all ten bands 1 to 10, the MSE is kept low for all bands, and the average MSE for all bands is 8.88. This value is converted to a pixel value of approximately 3 on the image sensor 60, which corresponds to an error of approximately 1% relative to the maximum pixel value of 255. In this way, when σ / μ for each band of the transmittance of the filter array 10 exceeds 0.1, the image of each band can be restored with high accuracy.

[0106] In the filter array 10 according to this embodiment, as long as the mean value μ and standard deviation σ of the transmittance across all bands satisfy σ / μ≧0.1 for the transmittance distribution, the mean value μ of the transmittance does not need to be uniform across all bands. Similarly, in the photodetector device 300 according to this embodiment, as long as the mean value μ and standard deviation σ of the pixel values ​​across all bands satisfy σ / μ≧0.1 for the pixel value distribution, the mean value μ of the pixel values ​​does not need to be uniform across all bands. Even if the mean value μ of the transmittance or the mean value μ of the pixel values ​​is not uniform across all bands, the signal processing circuit 200 can correct it by multiplying the mean value μ of the pixel values ​​by a constant through signal processing so that it becomes uniform across all bands. Even with this correction, σ / μ across all bands remains unchanged before and after the correction, and σ / μ≧0.1 is satisfied. Therefore, this embodiment can reduce errors associated with the reconstruction of images of multiple wavelength bands.

[0107] The optical properties of the filter array 10 discussed above, i.e., the average transmittance and standard deviation of transmittance for each wavelength band, can be determined by measuring and analyzing histograms in any region containing approximately 8 vertical pixels by 8 horizontal pixels. If measuring the transmission spectrum of the filter array 10 is technically difficult, histograms can also be measured and analyzed by measuring the reflection spectrum for each wavelength band. Furthermore, if the filter array 10 is integrated on an image sensor 60, histograms can also be measured and analyzed, including the sensitivity characteristics of the image sensor 60 itself. Furthermore, when the filters constituting the filter array 10 are FP filters, there is generally a correlation between the thicknesses of the filters, which are composed of a first reflective layer, a second reflective layer, and an intermediate layer disposed between the first and second reflective layers, and the histogram. Therefore, similar information can be obtained by measuring the thickness distribution in any region containing approximately 8 vertical pixels by 8 horizontal pixels.

[0108] Although the above embodiment has been described mainly with reference to the filter array 10 in which a plurality of filters are arranged two-dimensionally, the plurality of filters may be arranged one-dimensionally. In this case, a one-dimensional image sensor may be used as the photodetector. When the measurement target is a one-dimensional area, such a configuration may also be adopted.

[0109] The following cases are also included in this disclosure:

[0110] The photodetector 300 may use a coding element other than a filter array. For example, the photodetector 300 may be configured to detect light passing through an optical element such as a metalens. Such an optical element may be incorporated into at least a portion of the optical system 40. Alternatively, the photodetector 300 may include an optical element instead of the filter array 10. The optical element spatially and wavelength-wise changes incident light that has entered a light incident surface of the optical element, and causes the light to exit from a light exit surface of the optical element. The image sensor 60 detects the light exiting from the light exit surface of the optical element.

[0111] That is, a photodetection device used in a photodetection system that generates image data for each of N wavelength bands (N is an integer of 4 or more), comprising: an encoding element in which light transmittance in each of the N wavelength bands varies depending on the position; and an image sensor that detects light that has passed through the encoding element, wherein the image sensor detects only light corresponding to an i-th wavelength band (i is an integer of 1 or more and N or less) of the N wavelength bands, thereby outputting data indicating a pixel value distribution corresponding to the i-th wavelength band, and calculating an average value of the pixel values ​​of the pixel value distribution corresponding to the i-th wavelength band as μ ｉ The standard deviation of the pixel values ​​of the pixel value distribution corresponding to the i wavelength band is σ ｉ In this case, (σ1 / μ1)≧0.1, ..., (σ Ｎ / μ Ｎ ) ≧0.1.

[0112] The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength images. The technology disclosed herein can also be applied to, for example, biomedical, cosmetic, and other sensing applications, food foreign matter and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

[0113] 10 filter array 40 optical system 60 image sensor 70 object 80 substrate 100 filter 120 image 200 signal processing circuit 220 reconstructed image 300 photodetector

Claims

1. A filter array used in an optical detection system for generating image data of each of N (N is an integer of 4 or more) wavelength bands, comprising a plurality of optical filters, Let μ be the average value of the transmittances of the plurality of optical filters that correspond one-to-one to the light of the i-th wavelength band (where i is an integer from 1 to N) among the N wavelength bands. ｉ And Let σ be the standard deviation of the plurality of transmittances that correspond one-to-one to the plurality of optical filters for the light of the i-th wavelength band. ｉ When it is set as such, (σ 1 / μ 1 ) ≥ 0.1,..., (σ Ｎ / μ Ｎ ) ≥ 0.

1. filter array.

2. At least one of the plurality of optical filters is a Fabry - Perot filter, The filter array according to Claim 1.

3. At least one of the plurality of optical filters includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer, and has a resonant structure having a plurality of resonant modes with different orders, The filter array according to Claim 1 or 2.

4. An optical detection device used in an optical detection system for generating image data of each of N (N is an integer of 4 or more) wavelength bands, comprising a plurality of optical filters, an image sensor for detecting light passing through the plurality of optical filters, and comprising, The image sensor outputs data indicating a pixel value distribution corresponding to the i - th wavelength band (i is an integer from 1 to N) among the N wavelength bands by detecting light corresponding to the i - th wavelength band, Let μ be the average value of the pixel values of the pixel value distribution corresponding to the i-th wavelength band ｉ and Let the standard deviation of the pixel values of the pixel value distribution corresponding to the i-th wavelength band be σ ｉ When this is the case, (σ 1 / μ 1 ) ≥ 0.1, ···, (σ Ｎ / μ Ｎ ) ≥ 0.1 optical detection device.

5. At least one of the plurality of optical filters is a Fabry - Perot filter, The optical detection device according to Claim 4.

6. At least one of the plurality of optical filters includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer, and has a resonant structure having a plurality of resonant modes with different orders, The optical detection device according to Claim 4 or 5.

7. The transmission spectrum of each of the plurality of optical filters has a maximum value of transmittance at each of a plurality of wavelengths included in the target wavelength range, The image sensor includes a plurality of light - detecting elements, Each of the plurality of light - detecting elements is arranged at a position to receive transmitted light that has passed through at least one of the plurality of optical filters and detects light of the plurality of wavelengths included in the transmitted light, The optical detection device according to Claim 4 or 5.

8. The optical detection device according to Claim 4 or 5, and a signal processing circuit for generating the image data for each of the N wavelength bands based on a signal output from the image sensor, and comprising, optical detection system.

9. The signal processing circuit generates the image data by performing an operation using compressive sensing. The photodetection system according to claim 8.

10. A photodetection device used in a photodetection system that generates image data for each of N (N is an integer of 4 or more) wavelength bands, an encoding element, an image sensor that detects light passing through the encoding element, comprising: the image sensor outputs data indicating a pixel value distribution corresponding to the i-th wavelength band (i is an integer from 1 to N) among the N wavelength bands by detecting light corresponding to the i-th wavelength band, Let μ be the average value of the pixel values of the pixel value distribution corresponding to the i-th wavelength band. ｉ and Let the standard deviation of the pixel values of the pixel value distribution corresponding to the i-th wavelength band be σ ｉ When it is, (σ 1 / μ 1 ) ≥ 0.1,..., (σ Ｎ / μ Ｎ ) ≥ 0.1 A photodetection device.