Device used in system for generating images, and filter array

Abstract

To provide a device capable of improving the restoration accuracy of a restored image generated from an image in which spectral information is compressed.SOLUTION: A device is used in a system that generates N images, each corresponding to one of N (N is an integer equal to or greater than 4) wavelength bands. The device includes: an optical element that has multiple areas with spectral transmittances different from each other; and an image sensor that detects light via the optical element. When the central wavelengths are numbered in an ascending or descending order in the N wavelength bands, the image sensor detects only the light corresponding to each waveband to output the corresponding mask data. In the correlation coefficient between the i-th mask data corresponding to the i-th waveband and the j-th mask data corresponding to the j-th waveband, in the case that |i-j|=1 and i<j, the correlation coefficient of (i, j)=(1, 2), (N-1, N) is larger than any of the remaining correlation coefficients.SELECTED DRAWING: Figure 7B

Description

[Technical field]

[0001] The present disclosure relates to devices and filter arrays for use in systems for producing images. [Background technology]

[0002] Compressed sensing is a technique for recovering more data than observed data by assuming that the data distribution of an observed object is sparse in a certain space such as a frequency space. Compressed sensing can be applied to, for example, an imaging device that recovers an image containing more information from a small amount of observed data. An imaging device to which compressed sensing is applied generates a recovered image by calculation from an image in which the spectral information of the object is compressed. As a result, it is possible to obtain various effects such as high image resolution, multi-wavelength, shortened imaging time, or high sensitivity.

[0003] Patent Documents 1 and 2 disclose examples of applying compressed sensing technology to a hyperspectral camera that acquires images of multiple narrow wavelength bands. The technology disclosed in Patent Document 1 makes it possible to realize a hyperspectral camera that generates high-resolution, multi-wavelength images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,599,511 Summary of the Invention [Problem to be solved by the invention]

[0005] Provided is an apparatus capable of improving the restoration accuracy of a restored image generated from an image whose spectral information is compressed. [Means for solving the problem]

[0006] An apparatus according to an aspect of the present disclosure is an apparatus used in a system that generates N (where N is an integer of 4 or more) images respectively corresponding to N wavelength bands, the apparatus including: an optical element having a plurality of regions with different spectral transmittances; and an image sensor that detects light passing through the optical element, the image sensor outputs first mask data corresponding to a pixel value distribution corresponding to the i-th wavelength band (where i is an integer from 1 to N) by detecting only light corresponding to the i-th wavelength band among the N wavelength bands, outputs second mask data corresponding to a pixel value distribution corresponding to the j-th wavelength band (where j is an integer from 1 to N) by detecting only light corresponding to the j-th wavelength band among the N wavelength bands, and a correlation coefficient r between the first mask data and the second mask data ij is

Number

[0007] A comprehensive or specific aspect of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be arranged in one device, or may be arranged separately in two or more separate devices. In this specification and the claims, "apparatus" may mean not only one device, but also a system consisting of multiple devices. Effect of the Invention

[0008] According to one aspect of the present disclosure, it is possible to realize an apparatus capable of improving the restoration accuracy of a restored image generated from an image with compressed spectral information. [Brief description of the drawings]

[0009] [Figure 1A] FIG. 1A is a diagram illustrating a schematic configuration example of an imaging system. [Figure 1B] FIG. 1B is a diagram illustrating a schematic configuration example of an imaging system. [Figure 1C] FIG. 1C is a diagram illustrating a schematic configuration example of still another imaging system. [Figure 1D] FIG. 1D is a diagram illustrating a schematic configuration example of still another imaging system. [Figure 2A] FIG. 2A is a schematic diagram illustrating an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of a spatial distribution of the light transmittance of each of wavelength bands W1, W2, . . . , WN included in the target wavelength range. [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of the region A1 included in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of the area A2 included in the filter array shown in FIG. 2A. [Diagram 3] FIG. 3 is a diagram for explaining an example of the relationship between a target wavelength range W and wavelength bands W1, W2, . . . , WN included therein. [Figure 4A] FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. [Figure 4B] FIG. 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W1, W2, . . . , WN. [Figure 5A] FIG. 5A is a cross-sectional view that illustrates a schematic example of an imaging device included in the imaging system according to this embodiment. [Figure 5B] FIG. 5B is a cross-sectional view that illustrates a schematic diagram of another example of the imaging device included in the imaging system according to this embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a calculation result of a transmission spectrum of a filter. [Figure 7A] FIG. 7A is a diagram showing a correlation matrix R in an encoding mask of a comparative example. [Figure 7B] FIG. 7B is a diagram illustrating a correlation matrix R in the encoding mask according to the first embodiment. [Figure 8A] FIG. 8A is a graph showing (MSEcorr-MSE0) / MSEcorr in each wavelength band calculated for 35 types of objects using the coded mask of Example 1. [Figure 8B] FIG. 8B is an enlarged graph showing the average of (MSEcorr-MSE0) / MSEcorr in each wavelength band shown in FIG. 8A. [Figure 9] FIG. 9 is a graph showing the average of (MSEcorr-MSE0.25) / MSEcorr in each wavelength band calculated for 35 types of objects 70 using the coded mask of Example 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components. Each figure is a schematic diagram and is not necessarily illustrated strictly. Furthermore, in each figure, substantially the same or similar components are given the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0011] In the present disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, although it is called an LSI or an IC, the name may change depending on the degree of integration, and it may be called a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device, which can reconfigure the junction relationship inside the LSI or set up the circuit partition inside the LSI, can also be used for the same purpose.

[0012] Furthermore, all or part of the functions or operations of a circuit, unit, device, member, or part can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0013] (Explanation of terms used in this specification) Before describing the embodiments of the present disclosure, the terms used in this specification will be described. In the imaging device according to this embodiment, a compressed image in which spectral information is compressed is obtained by capturing light reflected from an object through a filter array having a plurality of optical filters arranged in a two-dimensional plane. In the imaging device according to this embodiment, N restored images corresponding to N wavelength bands (N is an integer equal to or greater than 4) in the target wavelength range are generated from the captured compressed image by calculation based on mask data of the filter array.

[0014] <Target wavelength range> The target wavelength range is a wavelength range determined based on the upper and lower limits of the wavelength of light incident on an image sensor used for imaging. The target wavelength range may be, for example, any range from the upper limit to the lower limit of the wavelength to which the image sensor is sensitive, i.e., within the sensitivity wavelength range. When an object that absorbs and / or reflects light in the sensitivity wavelength range is placed on the optical axis of the image sensor, the target wavelength range may be a part of the sensitivity wavelength range of the image sensor. The target wavelength range may correspond to the wavelength range of data output from the image sensor, i.e., the output wavelength range. In this specification, the sensitivity wavelength range is also simply referred to as the "sensitivity range."

[0015] <Wavelength resolution> The wavelength resolution is the width of the wavelength band when a restored image is generated for each wavelength band by restoration. For example, when a restored image corresponding to a wavelength band with a width of 5 nm is generated, the wavelength resolution is 5 nm. Similarly, when a restored image corresponding to a wavelength band with a width of 20 nm is generated, the wavelength resolution is 20 nm.

[0016] <Mask data> The mask data is data indicating an arrangement based on the spatial distribution of the transmittance of the filter array for each wavelength band in the light receiving band of the imaging system. Data indicating the spatial distribution of the transmittance of the filter array itself may be used as the mask data, or data obtained by performing reversible calculations on the transmittance of the filter array may be used as the mask data. Here, reversible calculations refer to, for example, addition, subtraction, multiplication, and division of a fixed value, power calculations, exponential calculations, logarithmic calculations, and gamma correction. Reversible calculations may be performed uniformly within the target wavelength band, or may be performed for each wavelength band described later.

[0017] When data showing the spatial distribution of the transmittance of the filter array for each wavelength range is used as mask data, the intensity of light passing through the filter array in a wavelength range having a finite width within the target wavelength range is observed as a two-dimensionally arranged matrix. The target wavelength range is, for example, 400 nm to 700 nm, and the wavelength range having a finite width can be, for example, 400 nm to 450 nm. By performing the above observation so as to cover the entire range within the target wavelength range, multiple matrices are generated. Each of the multiple matrices is data arranged two-dimensionally in the spatial direction. The collective term for data acquired in multiple wavelength ranges and arranged two-dimensionally in the spatial direction is mask data.

[0018] In the above example, the wavelength range between 400nm and 450nm inclusive was defined as a "wavelength range with a finite width," but in the calculations, no distinction is made between wavelengths within this wavelength range. In other words, only intensity information is recorded and used in the calculations, so whether 420nm light or 430nm light is incident, only the intensity is recorded and no wavelength information is stored. For this reason, in the calculations, all wavelengths within this wavelength range are treated as the same wavelength.

[0019] The spatial distribution of the transmittance of the filter array can be observed, for example, by using a light source that outputs only a specific wavelength and an integrating sphere. In the above example, only light with a wavelength of 400 nm to 450 nm is output from the light source, and the output light is uniformly diffused by the integrating sphere and then detected through the filter array. As a result, an image is obtained in which, for example, the sensitivity of the image sensor and / or the aberration of the lens are superimposed on the spatial distribution of the transmittance of the filter array in the wavelength range of 400 nm to 450 nm. The obtained image can be treated as a matrix. If the sensitivity of the image sensor and / or the aberration of the lens are known, the spatial distribution of the transmittance of the filter array can be obtained by performing correction on the obtained image. The obtained image can be interpreted as an image in which reversible calculations such as the sensitivity of the image sensor and / or the aberration of the lens are performed on the spatial distribution of the transmittance of the filter array. Therefore, it is not necessary to perform correction on the obtained image.

[0020] In reality, the transmittance cannot vary discontinuously around a wavelength, but varies with finite rise and fall angles. Thus, the upper and lower limits of a wavelength range can be defined by the wavelengths at which the transmittance decays at a certain percentage from its peak intensity. The certain percentage can be, for example, 90%, 50%, or 10% of the peak intensity.

[0021] If the mask data is stored in memory, for example, it may be compressed in a lossless format, such as Portable Network Graphics (PNG) or Graphics Interchange Format (GIF).

[0022] <Wavelength band> A wavelength band is a part of a wavelength range within a target wavelength range, and is a range of wavelengths that are treated as the same wavelength in mask data. A wavelength band may be a wavelength range having a certain width, as it is called a "band." A wavelength band may be, for example, a wavelength range having a width of 50 nm, which is equal to or greater than 500 nm and equal to or less than 550 nm. In this specification, a collection of wavelength ranges having a certain width is also referred to as a "wavelength band."

[0023] <Restored image> A restored image is a two-dimensional image output for each wavelength band as a result of the restoration calculation. Since a restored image is generated for each wavelength band, one restored image corresponding to a certain wavelength band is determined. The restored image may be output as a monochrome image. A plurality of restored images corresponding to a plurality of wavelength bands respectively may be output as three-dimensional array data in the spatial direction and the wavelength direction. Alternatively, the plurality of restored images may be output as data in which a plurality of pixel values ​​are arranged one-dimensionally. Each of the plurality of pixel values ​​corresponds to a combination of a wavelength band and a pixel. Alternatively, the plurality of restored images may be output with header information including meta-information such as spatial resolution and the number of wavelength bands.

[0024] <Restoration accuracy> The restoration accuracy is the degree of discrepancy between the restored image and the correct image. The restoration accuracy can be expressed using various indices such as MSE (Mean Squared Error) or PSNR (Peak Signal-to-Noise Ratio).

[0025] (Findings on which this disclosure is based) Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

[0026] In the field of imaging, the process of classifying multiple objects captured in an image into types is carried out in fields such as factory automation (FA) and medicine. Examples of features used in classification processing include the shape of the object and the spectral information of the object. Since hyperspectral cameras can acquire multi-wavelength images that contain a lot of spectral information for each pixel, the use of hyperspectral cameras is expected in the future. While research and development of hyperspectral cameras has been conducted around the world for many years, their use has been limited for the following reasons. For example, line-scan hyperspectral cameras have high spatial and wavelength resolution, but the shooting time is long due to line scanning. Although snapshot hyperspectral cameras can take images in one shot, their sensitivity and spatial resolution are often insufficient.

[0027] In response to these problems, it has been reported in recent years that the sensitivity and spatial resolution of hyperspectral cameras can be improved by restoring images based on sparsity. Sparsity is the property that elements that characterize an observation target are sparsely present in a certain space, such as frequency space. Sparsity is widely seen in the natural world. By utilizing sparsity, it becomes possible to observe necessary information efficiently. Sensing technology that utilizes sparsity is called compressed sensing technology. By utilizing compressed sensing technology, it is possible to build highly efficient devices and systems.

[0028] As a specific application example of the compressed sensing technology, for example, a hyperspectral camera with improved wavelength resolution as disclosed in Patent Document 1 has been proposed. The hyperspectral camera includes, for example, an optical filter having irregular light transmission characteristics with respect to space and / or wavelength. Such an optical filter is also called an "encoding mask." The encoding mask is placed on the optical path of light incident on the image sensor, and transmits light incident from an object with different light transmission characteristics depending on the region. This process by the encoding mask is called "encoding." In the image of the object acquired through the encoding mask, the spectral information of the object is compressed. The image is called a "compressed image." Mask information indicating the light transmission of the encoding mask is stored in advance in a storage device as a restoration table.

[0029] The processing device of the imaging device performs restoration processing based on the compressed image and the restoration table. The restoration processing can obtain more information than the compressed image, such as image information with higher resolution or image information with more wavelengths. The restoration table can be, for example, data indicating the spatial distribution of the optical response characteristics of the coding mask. The restoration processing based on such a restoration table can generate multiple restored images corresponding to multiple wavelength bands included in the target wavelength range from one compressed image. In the following description, the "multiple restored images" are also simply referred to as "restored images".

[0030] The inventors have investigated the restoration accuracy of a plurality of restored images, and found that the restoration accuracy of two restored images corresponding to the wavelength bands at both ends of the target wavelength range is lower than that of the remaining restored images. The "wavelength bands at both ends" refers to the wavelength band with the longest central wavelength and the wavelength band with the shortest central wavelength. The inventors have found this problem and have come up with an imaging device according to an embodiment of the present disclosure that can improve the restoration accuracy of two restored images corresponding to the wavelength bands at both ends.

[0031] As will be explained in more detail later, it is believed that the above problem arises because, while there are many wavelength bands with strong correlation in the light transmission characteristics of the coding mask near the center of the target wavelength range, there are few wavelength bands with strong correlation at both ends of the target wavelength range.

[0032] Therefore, in the imaging device according to the present embodiment, the coding mask is appropriately designed so that the correlation between each of the two adjacent wavelength bands is stronger than the remaining correlation between each of the two wavelength bands at both ends. As a result, the restoration accuracy of the restored image, particularly the restoration accuracy of the two restored images corresponding to the two wavelength bands at both ends, can be improved.

[0033] Hereinafter, more specific embodiments of the present disclosure will be described with reference to the drawings.

[0034] (Embodiment) In the following, first, an imaging system that generates a restored image from a compressed image will be described, followed by a method for improving the restoration accuracy of two restored images corresponding to both end wavelength bands.

[0035] [1. Imaging system] FIG. 1A is a diagram showing a schematic configuration example of an imaging system. The system shown in FIG. 1A includes an imaging device 100 and an image processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject. The filter array 110 in the example of FIG. 1A is disposed between the optical system 140 and the image sensor 160.

[0036] FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple, and may be any object. The image sensor 160 generates data of a compressed image 10 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 200 generates data indicating a plurality of images corresponding one-to-one to a plurality of wavelength bands included in a predetermined target wavelength range, based on the data of the compressed image 10 generated by the image sensor 160. Here, the number of wavelength bands included in the target wavelength range is set to N (N is an integer equal to or greater than 4). In the following description, the N images generated based on the compressed image 10 are referred to as restored images 20W1, 20W2, . . . , 20W N These are sometimes collectively referred to as "hyperspectral images 20."

[0037] In this embodiment, the filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittances. The filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process by the filter array 110 is called "encoding," and the filter array 110 is also called an "encoding mask."

[0038] 1A, filter array 110 is disposed near or directly above image sensor 160. Here, "near" means close enough that an image of light from optical system 140 is formed on the surface of filter array 110 with a certain degree of clarity. "Directly above" means close enough that there is almost no gap between them. Filter array 110 and image sensor 160 may be integrated.

[0039] Optical system 140 includes at least one lens. Although optical system 140 is shown as a single lens in FIG. 1A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image through filter array 110 onto the imaging surface of image sensor 160.

[0040] The filter array 110 may be disposed away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is disposed away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position distant from the image sensor 160. In the example of FIG. 1C, the filter array 110 is disposed between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.

[0041] The image sensor 160 is a monochrome type light detection device having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor. The light detection elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor may be used. The color type sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light. The color type sensor may further include a plurality of IR filters that transmit infrared light. The color type sensor may also include a plurality of transparent filters that transmit all red, green, and blue light. By using a color type sensor, the amount of information regarding wavelengths can be increased, and the accuracy of reconstruction of the hyperspectral image 20 can be improved. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, but may be an ultraviolet, near infrared, mid infrared, or far infrared wavelength range.

[0042] The image processing device 200 may be a computer including one or more processors and one or more storage media such as a memory. The image processing device 200 generates decompressed images 20W1, 20W2, . . . , 20W3 based on the compressed image 10 acquired by the image sensor 160. N Generate data.

[0043] 2A is a diagram showing a schematic example of a filter array 110. The filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, the regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0044] 2A, the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels of the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application and may range from several tens to several tens of millions.

[0045] FIG. 2B shows the wavelength bands W1, W2, . . . , W N 2B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands. In the example shown in FIG. 2B, the difference in the shading of each region represents the difference in the transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of the light transmittance differs depending on the wavelength band.

[0046] 2C and 2D are diagrams showing examples of the spectral transmittance of the region A1 and the region A2 included in the filter array 110 shown in FIG. 2A, respectively. The spectral transmittance of the region A1 and the spectral transmittance of the region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary that the spectral transmittances of all the regions are different. In the filter array 110, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 110 includes two or more filters having different spectral transmittances from each other. In an example, the number of patterns of the spectral transmittances of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0047] Figure 3 shows the target wavelength range W and the wavelength bands W1, W2, ..., W N FIG. 1 is a diagram for explaining the relationship between the wavelength range W and the visible light wavelength range. The target wavelength range W may be set to various ranges depending on the application. The target wavelength range W may be, for example, a visible light wavelength range of about 400 nm to about 700 nm, a near-infrared wavelength range of about 700 nm to about 2500 nm, or a near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. In this manner, the wavelength range used is not limited to the visible light range. In this specification, the term "light" refers to radiation in general, including not only visible light but also infrared and ultraviolet light.

[0048] In the example shown in FIG. 3, N is an arbitrary integer equal to or greater than 4, and the wavelength bands obtained by dividing the target wavelength range W into N equal parts are called wavelength bands W1, W2, . . . , W N However, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be made non-uniform depending on the wavelength band. In this way, the method of determining the multiple wavelength bands is arbitrary.

[0049] FIG. 4A is a diagram for explaining the spectral transmittance characteristics in a region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has a plurality of maxima P1 to P5 and a plurality of minima with respect to the wavelengths within the target wavelength range W. In the example shown in FIG. 4A, the spectral transmittance is normalized such that the maximum value of the light transmittance within the target wavelength range W is 1 and the minimum value is 0. In the example shown in FIG. 4A, the spectral transmittance has maxima in wavelength ranges such as the wavelength band W2 and the wavelength band W N-1 and so on. Thus, the spectral transmittance of each region can be designed to have maxima in at least two of the plurality of wavelength ranges of the wavelength bands W1, W2, ···, W N . In the example of FIG. 4A, the maxima P1, P3, P4, and P5 are 0.5 or more.

[0050] Thus, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 110 transmits a large amount of components in a certain wavelength range of the incident light and does not transmit components in other wavelength ranges so much. For example, for the light in k of the N wavelength bands, the transmittance can be greater than 0.5, and for the light in the remaining N - k wavelength ranges, the transmittance can be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that equally contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of discrete intensity peaks with respect to the wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0051] FIG. 4B is a diagram showing, as an example, the result of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W1, W2, ···, W N . The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is taken as the transmittance in that wavelength band. In this example, the transmittance is prominently high in three wavelength ranges taking the maxima P1, P3, and P5. In particular, in the two wavelength ranges taking the maxima P3 and P5, the transmittance exceeds 0.8.

[0052] In the examples shown in Figures 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessary to use a grayscale transmittance distribution. For example, a binary scale transmittance distribution may be adopted in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two of the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.

[0053] A portion of all the cells, for example half of the cells, may be replaced with transparent regions. Such transparent regions are arranged in the wavelength bands W1, W2, . . . , W3, W4, W5, W6, W7, W8, W9, W10, W11, W12, W13, W14, W15, W16, W17, W18, W19, W20, W21, W22, W23, W24, W25, W26, W27, W28, W29, W30, W31, W32, W33, W34, W35, W36, W37, W38, W39, W40, W41, W42, W43, W44, W45, W46, W47, W48, W49, W50, W51, W52, W53, W54, W55, W56, W N Each of the light beams is transmitted with a similarly high transmittance, for example, 80% or more. In such a configuration, the multiple transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the multiple regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0054] Such data indicating the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the image processing device 200. This data is used in the calculation processing described later.

[0055] The filter array 110 may be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, a microstructure including a metal, or a metasurface. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film including a metal layer may be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is formed so that it differs for each cell. This allows different spectral characteristics to be realized for each cell. By using a multilayer film, a sharp rise and fall in the spectral transmittance can be realized. A configuration using an organic material can be realized by making the pigment or dye contained different for each cell, or by stacking different materials. A configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. A microstructure including a metal can be fabricated by utilizing the spectrum due to the plasmon effect. A metasurface can be fabricated by microfabricating a dielectric material in a size smaller than the wavelength of the incident light. In this structure, the refractive index for the incident light is spatially modulated. Alternatively, the incident light may be encoded by directly processing a plurality of pixels included in the image sensor 160 without using the filter array 110.

[0056] From the above, it can be said that the imaging device 100 has a plurality of light receiving regions with different photoresponse characteristics. When the imaging device 100 includes a filter array 110 including a plurality of filters, and the plurality of filters have irregularly different light transmission characteristics, the plurality of light receiving regions can be realized by an image sensor 160 disposed adjacent to or directly above the filter array 110. In this case, the photoresponse characteristics of the plurality of light receiving regions are determined based on the light transmission characteristics of the plurality of filters included in the filter array 110.

[0057] Alternatively, when the imaging device 100 does not include the filter array 110, the multiple light receiving regions may be realized by, for example, the image sensor 160 in which multiple pixels are directly processed so that their photoresponse characteristics are irregularly different from each other. In this case, the photoresponse characteristics of the multiple light receiving regions are determined based on the photoresponse characteristics of the multiple pixels included in the image sensor 160.

[0058] The above multilayer film, organic material, diffraction grating structure, microstructure containing metal, or metasurface can encode incident light as long as it is configured so that the spectral transmittance is modulated to vary depending on the position within a two-dimensional plane. Therefore, the above multilayer film, organic material, diffraction grating structure, microstructure containing metal, or metasurface does not need to be configured with multiple filters arranged in an array.

[0059] Next, an example of signal processing by the image processing device 200 will be described. The image processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, multi-wavelength means more wavelength ranges than the wavelength ranges of three colors of RGB acquired by a normal color camera, for example. The number of wavelength ranges can be, for example, about 4 to 100. The number of wavelength ranges is referred to as the "number of bands." Depending on the application, the number of bands may exceed 100.

[0060] The data to be obtained is the data of the hyperspectral image 20, and the data is denoted as f. If the number of bands is N, f is the data of N image bands f1, f2, . . . , f N Here, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is u and the number of pixels in the y direction is v, then the image data f1, f2, ..., f N Each of has u×v luminance values. Therefore, the data f is data with the number of elements u×v×N. On the other hand, the data g of the compressed image 10 obtained by encoding and multiplexing by the filter array 110 is two-dimensional data including u×v pixel values ​​corresponding to u×v pixels. The data g can be expressed by the following formula (1).

number

[0061] In equation (1), f represents the hyperspectral image data expressed as a one-dimensional vector. f1, f2, . . . , f N has u × v elements. Therefore, the vector on the right side is strictly a one-dimensional vector of u × v × N rows and 1 column. In formula (1), data g of compressed image 10 is converted and expressed as a one-dimensional vector of u × v rows and 1 column. Matrix H is expressed as each of components f1, f2, ..., f of vector f. N This represents a transformation in which each wavelength band is encoded with different encoding information, intensity-modulated, and then added together. Therefore, H is a matrix with u × v rows and u × v × N columns. Equation (1) can also be expressed as follows. g=(pg 11 pg 1u pg v1 pg vu ) T =H(f1 f N ) T Here, pg ij represents the luminance value of the i-th row and j-th column of the compressed image 10.

[0062] Given a vector g and a matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, since the number of elements u×v×N of the desired data f is greater than the number of elements u×v of the acquired data g, this problem is an ill-posed problem and cannot be solved as is. Therefore, the image processing device 200 utilizes the sparsity of the image contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

number

[0063] Here, f' represents the estimated f data. The first term in the parentheses in the above formula represents the amount of deviation between the estimated result Hf and the acquired data g, that is, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc. may be used as the residual term. The second term in the parentheses is a regularization term or a stabilization term. Formula (2) means to obtain f that minimizes the sum of the first and second terms. The function in the parentheses in formula (2) is called the evaluation function. The image processing device 200 can converge the solution by recursive iterative calculation and calculate f that minimizes the evaluation function as the final solution f'.

[0064] The first term in the parentheses in formula (2) means an operation to obtain the sum of squares of the difference between the acquired data g and Hf obtained by transforming f in the estimation process by the matrix H. The second term Φ(f) is a constraint condition in the regularization of f, and is a function reflecting the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT) of f, the wavelet transform, the Fourier transform, or the total variation (TV). For example, when the total variation is used, stable estimated data that suppresses the influence of noise in the observed data g can be obtained. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the more redundant data is reduced, and the higher the compression ratio. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent without being over-compressed.

[0065] In the configurations of FIG. 1B and FIG. 1C, the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, the hyperspectral image 20 can be reconstructed by holding this blur information in advance and reflecting the blur information in the above-mentioned matrix H. Here, the blur information is expressed by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, when a point image corresponding to one pixel on an image spreads to a region of k×k pixels around the pixel due to blurring, the PSF can be defined as a group of coefficients that indicate the influence on the luminance value of each pixel in the region, that is, a matrix. The hyperspectral image 20 can be reconstructed by reflecting the influence of blurring of the encoding pattern by the PSF in the matrix H. The position where the filter array 110 is arranged is arbitrary, but a position where the encoding pattern of the filter array 110 does not disappear due to excessive diffusion can be selected.

[0066] Through the above processing, the hyperspectral image 220 can be restored based on the compressed image 120 acquired by the image sensor 160. The image processing device 200 applies a compressed sensing algorithm to all wavelength bands included in the target wavelength range to generate and output the hyperspectral image 220. Specifically, the image processing device 200 causes the image sensor 160 to detect light reflected by the object 70 via the filter array 110, thereby generating and outputting an image signal. The image processing device 200 further generates and outputs the hyperspectral image 20 based on the image signal and N pieces of mask data corresponding to the N wavelength bands obtained from the filter array 110.

[0067] Each of the N mask data may be stored in a storage medium as, for example, two-dimensional data indicating the spatial distribution of the transmittance of the filter array 110 for the corresponding wavelength band. A certain component included in the two-dimensional data is the transmittance of a filter at a position corresponding to the component. Each mask data can be acquired by detecting light of the corresponding wavelength band through the filter array 110 with the image sensor 160. Therefore, by replacing the transmittance with a pixel value, the above two-dimensional data can be said to indicate a pixel value distribution reflecting the spatial distribution of the transmittance of the filter array 110.

[0068] Alternatively, each mask data may be stored in a storage medium as the following data used for H in equation (1): H=(H1···H i H j H N ), and the first mask data H1, . . . , the i-th mask data H i , , the jth mask data H j , . . . , Nth mask data H N Each of the i-th mask data H i and the jth mask data H j is illustrated in formula (3) and formula (4).

number

number

[0069] The i-th mask data H, i.e., the matrix H i can be, for example, a diagonal matrix with zero off-diagonal elements. i is the transmittance of the part of the filter array 110 corresponding to the first pixel of the image sensor 160 for the light of the i-th wavelength band, and the matrix H iis the transmittance of the part of the filter array 110 corresponding to the second pixel of the image sensor 160 for the light of the i-th wavelength band, . . . , the matrix H i i is the (u×v, u×v) component in u×v is determined based on the transmittance of a portion of filter array 110 that corresponds to the (u×v)th pixel of image sensor 160 for light in the i-th wavelength band.

[0070] Similarly, the jth mask data H j , i.e., the matrix H j can be, for example, a diagonal matrix with zero off-diagonal elements. j j1, which is the (1,1) component included in the matrix H j j2, which is the (2,2) component included in the matrix H j j is the (u×v, u×v) component in u×v is determined based on the transmittance of the portion of the filter array 110 that corresponds to the (u×v)th pixel of the image sensor 160 for light in the jth wavelength band.

[0071] Details of the method for restoring the hyperspectral image 20 are disclosed in U.S. Patent No. 6,333,636, the disclosure of which is incorporated herein by reference in its entirety.

[0072] [2. Example of the configuration of the filter array 110] A more specific example of the structure of filter array 110 will be described below with reference to FIGS. 5A and 5B.

[0073] Fig. 5A is a cross-sectional view showing a schematic example of an imaging device 100 included in the imaging system according to this embodiment. The imaging device 100 includes a filter array 110 and an image sensor 60. In Fig. 5A, the optical system 140 is omitted.

[0074] The filter array 110 includes a plurality of filters 112 arranged two-dimensionally. The plurality of filters 112 are arranged in rows and columns, for example, as shown in FIG. 2A. FIG. 5A shows a schematic cross-sectional structure of one row shown in FIG. 2A. Each of the plurality of filters 112 is a Fabry-Perot filter and includes a resonant structure. The resonant structure means a structure in which light of a certain wavelength exists stably by forming a standing wave therein. The state of the light is sometimes called a "resonant mode."

[0075] The resonant structure shown in FIG. 5A includes a first reflective layer 28a, a second reflective layer 28b, and an intermediate layer 26 between the first reflective layer 28a and the second reflective layer 28b. The first reflective layer 28a and / or the second reflective layer 28b may be formed of a dielectric multilayer film or a metal thin film. The intermediate layer 26 may be formed of a dielectric or a semiconductor that is transparent in a specific wavelength range. The intermediate layer 26 may be formed of at least one selected from the group consisting of Si, Si3N4, TiO2, Nb2O5, and Ta2O5, for example. The refractive index and / or thickness of the intermediate layer 26 of the multiple filters 112 differs depending on the filter. The transmission spectrum of each of the multiple filters 112 has a maximum value of transmittance at multiple wavelengths. The multiple wavelengths correspond to multiple resonance modes of different orders in the above-mentioned resonant structure. In this embodiment, all the filters 112 in the filter array 110 have the above-mentioned resonant structure. However, the filter array 110 may include a filter that does not have the above-mentioned resonant structure. For example, a filter having no wavelength dependency of light transmittance, such as a transparent filter or an ND filter (Neutral Density Filter), may be included in the filter array 110. In this embodiment, each of two or more filters 112 among the multiple filters 112 includes the above-mentioned resonant structure.

[0076] The image sensor 160 includes a plurality of photodetection elements 160a. Each of the plurality of photodetection elements 160a is disposed opposite one of the plurality of filters. The aforementioned target wavelength range W may be, for example, an overlapping portion of the sensitivity range of each photodetection element 160a and the transmission range of the optical system 140. The sensitivity range of each photodetection element 160a is a wavelength range in which each photodetection element 160a is sensitive to light. The transmission range of the optical system 140 is a wavelength range in which the optical system 140 transmits light with a transmittance of, for example, 60%, 80%, or 90% or more. When the transmission range of the optical system 140 includes all of the sensitivity ranges of each photodetection element 160a, the target wavelength range W corresponds to the sensitivity range of each photodetection element 160a.

[0077] In this specification, the term "wavelength range having sensitivity to light" refers to a wavelength range having substantial sensitivity required for detecting light. For example, it refers to an external quantum efficiency of 1% or more in the wavelength range. The external quantum efficiency of the light detection element 160a may be 10% or more, or may be 20% or more. In the following description, the light detection element 160a is also referred to as a "pixel."

[0078] In the example shown in FIG. 5A, the filter array 110 and the image sensor 160 are integrally formed, but this is not limiting. The filter array 110 and the image sensor 160 may be separated. Even in this case, each of the multiple photodetection elements 160a is disposed at a position where it receives light transmitted through one of the multiple filters 112. Each component may be disposed so that the light transmitted through the multiple filters 112 is incident on each of the multiple photodetection elements 160a via a mirror. In this case, each of the multiple photodetection elements 160a is not disposed directly below one of the multiple filters.

[0079] 5A, the multiple photodetection elements 160a correspond one-to-one to the multiple filters 112, but this is not limited to the example. The multiple photodetection elements 160a do not have to correspond one-to-one to the multiple filters 112. For example, light transmitted through two or more filters 112 may be incident on one photodetection element 160a.

[0080] FIG. 5B is a cross-sectional view showing a schematic diagram of another example of the imaging device 100 included in the imaging system according to the present embodiment. The imaging device 100 further includes a bandpass filter 29 in addition to the filter array 110 and the image sensor 160. In the example shown in FIG. 5B, the bandpass filter 29 is disposed between the filter array 110 and the image sensor 160, as compared with the example shown in FIG. 5A. The bandpass filter 29 has a transmission band, which is a wavelength band that transmits light, and a blocking band, which is a wavelength band that is on the shorter wavelength side and longer wavelength side than the transmission band and suppresses the transmission of light. The transmittance in the transmission band of the bandpass filter 29 may be, for example, 60% or more, 80% or more, or 90% or more. The transmittance in the blocking band of the bandpass filter 29 may be, for example, 20% or less, 10% or less, or 5% or less.

[0081] When the imaging device 100 includes the bandpass filter 29, the target wavelength range W described above may be, for example, an overlapping portion among the sensitivity ranges of the photodetection elements 160a, the transmission range of the optical system 140, and the transmission range of the bandpass filter 29. When the transmission range of the optical system 140 includes all of the sensitivity ranges of the photodetection elements 160a, and the sensitivity range of the photodetection elements 160a includes all of the transmission ranges of the bandpass filter 29, the target wavelength range W corresponds to the transmission range of the bandpass filter 29.

[0082] Next, the transmission spectrum of the filter 112 will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the calculation result of the transmission spectrum of the filter 112. In this example, the first reflection layer 28a in the filter 112 is formed of a dielectric multilayer film in which TiO2 layers and SiO2 layers are alternately stacked. The same is true for the second reflection layer 28b. The intermediate layer 26 in the filter 112 is formed of a TiO2 layer. The solid line and the dashed line shown in FIG. 6 represent the transmission spectrum when the intermediate layer 26 has different thicknesses. DiffractMOD based on Rigorous Coupled-Wave Analysis (RCWA) by RSoft was used to calculate the transmission spectrum.

[0083] As shown in Fig. 6, the transmission spectrum varies depending on the thickness of the intermediate layer 26. In this manner, a filter array 110 having a plurality of filters 112 with different transmission spectra can be realized. 3 10 pieces or more 7 In a filter array 110 including up to five filters 112, three or more or four or more types of filters 112 having different transmission spectra are irregularly distributed. The transmission spectrum of each filter 112 has multiple peaks in a target wavelength range W. Each peak has a maximum value of transmittance and minimum values ​​on both sides of the maximum value. The difference between the maximum value and each minimum value rate may be, for example, 10% or more, 20% or more, or 30% or more. The filter array 110 is an example of an optical element having multiple regions having different transmission spectra. The multiple filters 112 are an example of multiple regions.

[0084] [3. Details of restoration process using evaluation function and issues regarding restoration accuracy] The restoration process using the evaluation function will be described in more detail below. As expressed by the following formula (5), the evaluation function in parentheses is minimized.

number

[0085] In formula (5), the first term is a DF (Data Fidelity) term, and the second term is a TV (Total Variation) term. The DF term and the TV term are examples of terms included in the evaluation function.

[0086] f that minimizes the evaluation function can be estimated by, for example, the TwIST (Two-step Iterative Shrinkage / Thresholding) method using the following equations (6) and (7). The TwIST method is an example of a method for estimating f that minimizes the evaluation function.

[0087] In the TwIST method, f is updated a certain number of times to minimize the evaluation function. The updated f at the tth time is called f tBy setting the initial value f0, f1 can be obtained using equation (6). TV is the noise reduction function.

number

[0088] f t (t≧2) is obtained using equation (7), where α and β are preset values.

number

[0089] Updating f using equation (7) includes the following steps (A) and (B). (A) Update f by using the steepest descent method to reduce the DF term. (B) By minimizing the Chambolle TV norm, f in (A) is used as the initial value and f is updated so that the TV term is reduced.

[0090] Therefore, by repeatedly updating f using equation (7), processes (A) and (B) are repeatedly executed.

[0091] In process (B), only the initial value is important, and the coding mask does not affect the update of the initial value. On the other hand, in process (A), the coding mask affects the update of f. In process (A), the steepest descent method is performed by calculating the error of the DF term with f at that time, and subtracting the value obtained by proportionally allocating the error based on the transmittance of each region of the coding mask from f at that time.

[0092] The coding mask can be realized by a Fabry-Perot filter, such as the filter array 110 shown in FIG. 5A and FIG. 5B. The transmission characteristics of such a coding mask are strongly correlated between neighboring wavelength bands. There are many wavelength bands with strong correlation near the center of the target wavelength range W, while there are few wavelength bands with strong correlation at both ends of the target wavelength range W. For this reason, in process (A), the convergence characteristics by the TwIST method are worse in the wavelength bands at both ends compared to the wavelength bands near the center. As a result, among the multiple restored images corresponding to the multiple wavelength bands included in the target wavelength range W, the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends is lower than the restoration accuracy of the remaining restored images.

[0093] It is believed that the degradation of the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends occurs when the following two conditions are satisfied. In the restoration process, the optical transmission properties of the coding mask are exploited, such as minimizing the DF term. The correlation of the optical transmission characteristics of the coding masks used is stronger between nearby wavelength bands than between distant wavelength bands.

[0094] As described above, the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends is reduced due to the light transmission characteristics of the coding mask. The present inventors have found this problem and have come up with an imaging device according to the present embodiment that solves the problem. In the imaging device according to the present embodiment that generates a restored image using a coding mask, the coding mask is appropriately designed so that the correlation between each of the wavelength bands at both ends and the wavelength band adjacent thereto is stronger than any of the remaining correlations regarding the above correlation between two adjacent wavelength bands. As a result, the restoration accuracy of the restored image, particularly the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends, can be improved.

[0095] Below, we explain how to evaluate the correlation of the optical transmission characteristics of a coding mask between two wavelength bands, and then explain how to design a coding mask that improves the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends.

[0096] [4. Method for evaluating the correlation of the optical transmission characteristics of a code mask between two wavelength bands] A method for evaluating the correlation of the optical transmission characteristics of a code mask between two wavelength bands will be described below. Here, for ease of understanding, the N wavelength bands included in the target wavelength range W are numbered in ascending order of center wavelength. In other words, the shorter the center wavelength, the smaller the number. The center wavelength may be numbered in ascending order instead of in descending order.

[0097] The correlation of the optical transmission characteristics of the code mask between two wavelength bands is evaluated using a correlation coefficient of the mask data between the two wavelength bands. Therefore, the correlation of the optical transmission characteristics of the code mask between two wavelength bands may be rephrased as the correlation of the mask data between the two wavelength bands. The correlation coefficient of the mask data between two wavelength bands is, for example, the correlation coefficient between the i-th mask data corresponding to the i-th wavelength band and the j-th mask data corresponding to the j-th wavelength band. i and j are integers between 1 and N.

[0098] Image sensor 160 detects only light corresponding to a certain wavelength band among the N wavelength bands, and outputs mask data according to the pixel value distribution corresponding to that wavelength band. In this manner, N mask data including the i-th and j-th mask data can be obtained. In this specification, light corresponding to a wavelength band is also simply referred to as "wavelength band light."

[0099] When only light in a certain wavelength band is detected by the image sensor 160, light having a wavelength shifted by several nm from the wavelength range corresponding to the certain wavelength band may be incident on the image sensor 160. In other words, light having a wavelength shorter by several nm than the lower limit of the wavelength range corresponding to the certain wavelength band, or light having a wavelength longer by several nm than the upper limit of the wavelength range corresponding to the certain wavelength band may be incident on the image sensor 160.

[0100] The correlation coefficient r between the i-th mask data and the j-th mask data ij is expressed as the second-order correlation coefficient by the following equation (8).

number

[0101] The correlation coefficient r shown in equation (8) ij is an index showing the degree of similarity between the i-th mask data and the j-th mask data. The higher the similarity, the higher the correlation coefficient r ij The correlation coefficient r approaches 1, and is 1 when the similarity is perfect. Conversely, the lower the similarity, the lower the correlation coefficient r ij approaches 0, reaching zero when there is complete correlation.

[0102] As described above, when each mask data is stored in a storage medium as a diagonal matrix, the formula (8) is calculated based on the diagonal components of the i-th mask data and the j-th mask data. In this case, in the formula (8), i m and j m and m respectively represent the pixel value at the mth pixel. Here, all pixels included in the image sensor 160 are numbered, and m is an integer between 1 and u×v. i0 is the average value obtained by adding up all diagonal components of the ith mask data and dividing the sum by the total number of pixels. Similarly, j0 is the average value obtained by adding up all diagonal components of the jth mask data and dividing the sum by the total number of pixels.

[0103] r ij can be expressed in a matrix form. The following equation (9) is ij represents the correlation matrix R containing the matrix elements

number

[0104] When multiple wavelength bands are numbered in ascending order of center wavelength, r in the correlation matrix R ij are arranged from left to right and top to bottom in ascending order of center wavelength. In the correlation matrix R, r 11 =1, r 22 = 1, , r NN = 1. The correlation matrix R is r ij =rji The off-diagonal elements of the symmetric matrix are ij teeth 、 It represents the similarity of mask data between two different wavelength bands and contributes to the wavelength resolution and restoration accuracy of hyperspectral images.

[0105] The i-th and j-th mask data used in formula (8) may be replaced with the transmittance distribution of the filter array 110 for the light of the i-th and j-th wavelength bands, respectively. In that case, the pixel value of the m-th pixel in the i-th and j-th mask data may be replaced with the transmittance of the m-th filter in the i-th and j-th transmittance distributions, respectively. Here, all filters included in the filter array 110 are numbered. The average values ​​of the pixel values ​​of the i-th and j-th mask data may be replaced with the average values ​​of the transmittance of the i-th and j-th transmittance distributions, respectively. In this specification, the transmittance distribution of the filter array 110 for the light of the i-th and j-th wavelength bands is also referred to as the "i-th transmittance distribution" and the "j-th transmittance distribution", respectively.

[0106] In reality, the image sensor 160 has wavelength dependency of detection sensitivity in each photodetector element 160a. In many cases, the wavelength dependency of detection sensitivity is almost the same regardless of the photodetector element 160a. In the target wavelength range W, the wavelength dependency of detection sensitivity may be negligible in some cases, but may not be negligible in other cases.

[0107] Each photodetector element 160a detects light of a certain wavelength band through a corresponding portion of the filter array 110 and outputs the following signal. The signal indicates the transmittance of the corresponding portion of the filter array 110 for the light and an effective sensitivity corresponding to the detection sensitivity of each photodetector element 160a for the light. More specifically, the effective sensitivity is a value obtained by multiplying the two. In this specification, the distributions of effective sensitivities obtained by detecting light of the i-th and j-th wavelength bands by the imaging device 100 are also referred to as the "i-th effective sensitivity distribution" and the "j-th effective sensitivity distribution", respectively.

[0108] From the above, considering the wavelength dependence of the detection sensitivity, the i-th mask data used in Equation (8) may be rewritten as the i-th mask data corresponding to the distribution of the i-th effective sensitivity. The distribution of the i-th effective sensitivity is obtained based on the i-th transmittance distribution of the plurality of filters 112 included in the filter array 110 for the light in the i-th wavelength band, and the detection sensitivity of the image sensor 160 for the light in the i-th wavelength band. The same applies to the j-th mask data used in Equation (8). When considering the wavelength dependence of the detection sensitivity, the pixel values in the i-th and j-th mask data may be rewritten as the effective sensitivity.

[0109] Correlation coefficient r ij The details are disclosed in International Publication No. 2023 / 106143. The entire disclosure content of International Publication No. 2023 / 106143 is incorporated herein by reference.

[0110] [5. Encoding Mask for Improving the Restoration Accuracy of the Restored Image] Next, an encoding mask for improving the restoration accuracy of the restored image, particularly the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends, will be described. The reason for the decrease in the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends is that, as described above, in the vicinity of the center of the target wavelength range W, there are many wavelength bands with strong correlation in the transmission characteristics of the encoding mask, while at both ends of the target wavelength range W, there are few such wavelength bands with strong correlation.

[0111] Therefore, as the correlation coefficient of the mask data between two adjacent wavelength bands, when |i - j| = 1 and i < j are satisfied for r ij and for r where (i, j) = (1, 2), (N - 1, N) ij is larger than any of the remaining r ij the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends can be improved.

[0112] Hereinafter, the restoration errors of the restored images generated using the encoding masks of Examples 1 and 2 will be described.

[0113] <Encoding mask of Example 1> Figures 7A and 7B are diagrams showing the correlation matrix R in the encoding masks of the comparative example and Example 1, respectively. The correlation matrix R represents the correlation coefficient of the mask data between two wavelength bands. In the examples shown in Figures 7A and 7B, the wavelength bands W1, W2, ···, W 19 , W 20 are 20 wavelength bands obtained by equally dividing the target wavelength range W of 450 nm or more and 650 nm or less into 20 parts. The wavelength width of each wavelength band is 10 nm.

[0114] Regarding the correlation matrix R of the comparative example shown in Figure 7A, it is as follows. The correlation coefficients between two identical wavelength bands, that is, r ij where i = j are all 1.0. Regarding the correlation coefficients between two different wavelength bands, when the two wavelength bands are wavelength bands other than the two ends (2 ≤ i ≤ 19 and 2 ≤ j ≤ 19), r ij between two adjacent wavelength bands (|i - j| = 1) are all 0.7, r ij between two second-adjacent wavelength bands (|i - j| = 2) are all 0.28, and the remaining correlation coefficients r ij are all zero.

[0115] Regarding r ij where i < j as the correlation coefficient between two different wavelength bands, r ij between each of the wavelength bands at both ends and the wavelength band adjacent to it, that is, r ij for (i, j) = (1, 2), (19, 20) is 0.69, r ij between each of the wavelength bands at both ends and the wavelength band second-adjacent to it, that is, r ij for (i, j) = (1, 3), (18, 20) is 0.15, and the remaining correlation coefficients r ij are all zero.

[0116] In the correlation matrix R of the comparative example, for r ij where |i - j| = 1 and i < j are satisfied, r ij for (i, j) = (1, 2), (19, 20) is, compared with any of the remaining rij is smaller than.

[0117] Regarding the correlation matrix R of Example 1 shown in FIG. 7B, it is as follows. The underline shown in FIG. 7B indicates the location where the component of the correlation matrix R shown in FIG. 7B is different from the component of the correlation matrix R shown in FIG. 7A. For r where (i, j) = (1, 2), (19, 20) ij increased to 0.76, and for r where (i, j) = (1, 3), (18, 20) ij increased to 0.23.

[0118] In the correlation matrix R of Example 1, for r where |i - j| = 1 and i < j are satisfied ij for which, for r where (i, j) = (1, 2), (19, 20) ij is larger than any of the remaining r ij .

[0119] When the encoding masks of Example 1 and the comparative example are, for example, the filter array 110 shown in FIGS. 5A and 5B, the encoding masks of Example 1 and the comparative example that satisfy the above correlation matrix R can be realized, for example, by appropriately designing the thickness and / or refractive index of the intermediate layer 26 included in each filter 112 in the filter array 110.

[0120] Next, the restoration error of the restored image generated using the encoding mask of Example 1 is compared with the restoration error of the restored image generated using the encoding mask of the comparative example. As the restoration error, the mean squared error MSE between the pixel value of the pixel in the restored image and the pixel value of the pixel in the correct image is used. MSE is calculated using the following formula (10).

Equation

[0121] u and v are the number of pixels in the horizontal and vertical directions, respectively. I i、j is the pixel value of the pixel at the position (i, j) in the correct image. I’ i、j is the pixel value of the pixel at the position (i, j) in the restored image.

[0122] The restoration errors of the restored images generated using the coding masks of the first embodiment and the comparative example are expressed as MSE corr and MSE0, (MSE corr -MSE0) / MSE corr Using the index, we will explain how much the restoration error of the restored image is reduced by the coding mask of the first embodiment. (MSE corr -MSE0) / MSE corr The more negative the value is and the further away from zero it is, the more effectively the coding mask in the first embodiment reduces the restoration error of the restored image.

[0123] FIG. 8A shows the (MSE corr -MSE0) / MSE corr The 35 types of objects are, for example, butterflies, cloth, flowers, and color charts. The correct images of these objects were obtained from the following URL:

[0124] http: / / www.ok.sc.e.titech.ac.jp / res / MSI / MSIdata31.html The numbers on the horizontal axis in FIG. 8A represent the center wavelength of each wavelength band. The 35 thin lines in FIG. 8A with different shading represent the (MSE corr -MSE0) / MSE corr The thick black lines represent the (MSE corr -MSE0) / MSE corr Represents the average of.

[0125] FIG. 8B shows the (MSE corr -MSE0) / MSE corrIt is a graph showing the expansion of the average. As shown in FIG. 8B, by using the mask data of Example 1, the restoration errors of the two restored images corresponding to the wavelength bands at both ends were reduced. The same applies to the restoration errors of the four restored images corresponding to the wavelength bands that are one and two away from both ends. The restoration errors of the remaining restored images corresponding to the wavelength bands that are a certain distance away from both ends increased.

[0126] In the conventional restoration method, the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends decreases. In contrast, in Example 1, even if the restoration accuracy of the restored images corresponding to the wavelength bands other than both ends decreases, the restoration accuracy of these two restored images improves. Therefore, overall, the restoration accuracy of the hyperspectral image can be improved.

[0127] <Example 2> The 20 mask data included in the encoding mask of Example 2 are obtained by correcting the 20 mask data included in the encoding mask of the comparative example as represented by the following formula (11). In formula (11), α = 0.1, 0.25, 0.4, 0.6, 0.7. However, this correction is an example.

Equation

[0128] In the correlation matrix R of Example 2, for r where |i - j| = 1 and i < j are satisfied ij it is as follows. For r where (i, j) = (1, 2), (19, 20) ij is 0.366 when α = 0.1, 0.491 when α = 0.25, 0.598 when α = 0.4, 0.71 when α = 0.6, and 0.754 when α = 0.7. For r other than (i, j) = (1, 2), (19, 20) ij is all 0.508 regardless of the value of α.

[0129] However, for r where (i, j) = (1, 2) ij and for r where (i, j) = (19, 20) ijAlthough they are actually almost equal, they are different from each other. Therefore, the average of these two correlation coefficients is r for (i, j) = (1, 2), (19, 20). ij was used as such.

[0130] In the correlation matrix R of Example 2, for α = 0.1, 0.25, the r satisfying |i - j| = 1 and i < j ij for (i, j) = (1, 2), (19, 20) is r ij is smaller than any of the remaining r ij On the other hand, for α = 0.4, 0.6, 0.7, the r satisfying |i - j| = 1 ij for (i, j) = (1, 2), (19, 20) is r ij is larger than any of the remaining r ij .

[0131] The encoding mask of Example 2 that satisfies the above correlation matrix R can be realized, similar to the encoding masks of Example 1 and the comparative example described above, by appropriately designing, for example, the thickness and / or refractive index of the intermediate layer 26 included in each filter 112 in the filter array 110.

[0132] FIG. 9 is a graph showing the average of (MSE corr - MSE 0.25 ) / MSE corr for each wavelength band calculated for 35 types of objects 70 using the encoding mask of Example 2. Here, MSE 0.25 is the MSE at α = 0.25. The reason for using MSE 0.25 as a reference is that at α = 0.25, for the r satisfying |i - j| = 1 and i < j ij for (i, j) = (1, 2), (19, 20), the r ij is almost equal to any of the remaining r ij . The five types of symbols shown in FIG. 9 represent the calculation results at α = 0.1, 0.25, 0.4, 0.6, 0.7. At α = 0.25, since MSE corr coincides with MSE 0.25 , (MSE corr - MSE 0.25 ) / MSEcorr is zero in all wavelength bands.

[0133] As shown in Fig. 9, when α = 0.1, the restoration errors of the two restored images corresponding to the wavelength bands at both ends increased. The same was true for the restoration errors of the four restored images corresponding to the wavelength bands one and two away from both ends. The restoration errors of the remaining restored images corresponding to the wavelength bands that were a certain distance away from both ends did not change much. In contrast, when α = 0.4, 0.6, 0.7, the restoration errors of the two restored images corresponding to the wavelength bands at both ends decreased. The same was true for the restoration errors of the four restored images corresponding to the wavelength bands one and two away from both ends. As α increased from 0.4 to 0.7, the restoration errors were further reduced. The restoration errors of the remaining restored images corresponding to the wavelength bands that were a certain distance away from both ends did not change much.

[0134] As calculated using the encoding masks of Examples 1 and 2 as described above, from the results of examining the average of (MSE corr - MSE 0.25 ) / MSE corr in each wavelength band, the following can be said. In the combinations of the i-th mask data and the j-th mask data where |i - j| = 1 and i < j selected from N mask data, when r ij in (i, j) = (1, 2), (N - 1, N) is larger than any other r ij , the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends can be improved.

[0135] [6. Appendix] From the description of the above embodiments, the following technologies are disclosed.

[0136] (Technology 1) An apparatus used in a system for generating N images respectively corresponding to N (N is an integer of 4 or more) wavelength bands, an optical element having a plurality of regions with different spectral transmittances, an image sensor for detecting light passing through the optical element, comprising wherein the image sensor detects only the light corresponding to the i-th wavelength band (where i is an integer from 1 to N) among the N wavelength bands, and outputs the i-th mask data corresponding to the pixel value distribution corresponding to the i-th wavelength band; detects only the light corresponding to the j-th wavelength band (where j is an integer from 1 to N) among the N wavelength bands, and outputs the j-th mask data corresponding to the pixel value distribution corresponding to the j-th wavelength band; the correlation coefficient r between the i-th mask data and the j-th mask data ij is [Equation] where i m and j m are respectively the pixel values at the m-th pixel among the i-th and j-th mask data, i0 and j0 are respectively the average values of the pixel values of the i-th and j-th mask data, and when the N wavelength bands are numbered in ascending or descending order of the central wavelength, in the combination of the i-th mask data and the j-th mask data where |i - j| = 1 and i < j, and (i, j) = (1, 2), (N - 1, N), the r ij is larger than any other r ij ; device.

[0137] In this device, the restoration accuracy of the restored image generated from the image in which the spectral information is compressed can be improved.

[0138] (Technology 2) The N images are generated based on N mask data including the i-th mask data and the j-th mask data, and include the image corresponding to the i-th wavelength band and the image corresponding to the j-th wavelength band. The device according to Technology 1.

[0139] This device can generate an image corresponding to the i-th wavelength band and an image corresponding to the j-th wavelength band.

[0140] (Technology 3) the optical element is a filter array including a plurality of filters having different spectral transmittances, the plurality of regions respectively corresponding to the plurality of filters; The apparatus according to technique 1 or 2.

[0141] In this device, compressed sensing technology can be used to generate an image corresponding to the i-th wavelength band and an image corresponding to the j-th wavelength band.

[0142] (Technology 4) An apparatus for use in a system for generating N images corresponding to N wavelength bands (N is an integer equal to or greater than 4), an optical element having a plurality of regions with different spectral transmittances; an image sensor that detects light passing through the optical element; Equipped with The image sensor includes: detecting only light corresponding to an ith wavelength band (i is an integer between 1 and N) among the N wavelength bands, and outputting ith mask data corresponding to an ith transmittance distribution of the plurality of regions for the light corresponding to the ith wavelength band and an ith effective sensitivity distribution based on the detection sensitivity of the image sensor for the light corresponding to the ith wavelength band; detecting only light corresponding to a jth wavelength band (j is an integer between 1 and N) among the N wavelength bands, and outputting jth mask data corresponding to a jth transmittance distribution of the plurality of regions for light of the jth wavelength band and a jth effective sensitivity distribution based on the detection sensitivity of the image sensor to light of the jth wavelength band; A correlation coefficient r between the i-th mask data and the j-th mask data ij of

number

[0143] In this device, the restoration accuracy of the restored image generated from the image with compressed spectral information can be improved.

[0144] (Technology 5) A filter array used in a system for generating N images respectively corresponding to N (N is an integer of 4 or more) wavelength bands, comprising a plurality of optical filters with different spectral transmittances, Among the N wavelength bands, for the light corresponding to the i-th wavelength band (i is an integer from 1 to N), the correlation coefficient r between the i-th transmittance distribution of the plurality of optical filters and the j-th transmittance distribution of the plurality of optical filters for the light corresponding to the j-th wavelength band (i is an integer from 1 to N) ij is

Equation

[0145] With this filter array, the restoration accuracy of the restored image generated from the image with compressed spectral information can be improved.

Industrial Applicability

[0146] The technology of the present disclosure is useful, for example, in cameras and measuring devices that acquire multi-wavelength or high-resolution images. The technology of the present disclosure can also be applied to, for example, sensing for living bodies, medical and beauty purposes, foreign matter and residual pesticide inspection systems for foods, remote sensing systems, and in-vehicle sensing systems.

Explanation of Signs

[0147] 10 Compressed image 20 Hyperspectral image 20W1~20W N Restored image 70 Object 100 Imaging device 110 Filter array 112 Filter 140, 140A, 140B Optical system 160 Image sensor 160a Photo-detection element 200 Image processing device

Claims

1. An apparatus for use in a system for generating N images corresponding to N wavelength bands, respectively (N is an integer equal to or greater than 4), comprising: an optical element having a plurality of regions with different spectral transmittances; an image sensor that detects light passing through the optical element; Equipped with The image sensor includes: outputting i-th mask data according to a pixel value distribution corresponding to the i-th wavelength band (i is an integer between 1 and N) by detecting only light corresponding to an i-th wavelength band (i is an integer between 1 and N) among the N wavelength bands; outputting a jth mask data corresponding to a pixel value distribution corresponding to the jth wavelength band by detecting only light corresponding to a jth wavelength band (j is an integer between 1 and N) among the N wavelength bands; A correlation coefficient r between the i-th mask data and the j-th mask data ｉｊ of [0010] Let i ｍ and j ｍ are the pixel values ​​of the m-th pixel in the i-th and j-th mask data, respectively, and i ０ and j ０ are the average pixel values ​​of the i-th and j-th mask data, respectively, and the N wavelength bands are numbered in ascending or descending order of center wavelength, In the combination of the i-th mask data and the j-th mask data where |i-j|=1 and i<j, (i, j)=(1, 2), (N-1, N) ｉｊ Any of the remaining r ｉｊ Greater than Device.

2. The N images are generated based on N mask data including the i mask data and the j mask data, and include an image corresponding to the i wavelength band and an image corresponding to the j wavelength band.

2. The apparatus of claim 1.

3. the optical element is a filter array including a plurality of filters having different spectral transmittances, the plurality of regions respectively corresponding to the plurality of filters; 3. Apparatus according to claim 1 or 2.

4. An apparatus for use in a system for generating N images corresponding to N wavelength bands, respectively (N is an integer equal to or greater than 4), comprising: an optical element having a plurality of regions with different spectral transmittances; an image sensor that detects light passing through the optical element; Equipped with The image sensor includes: detecting only light corresponding to an ith wavelength band (i is an integer between 1 and N) among the N wavelength bands, and outputting ith mask data corresponding to an ith transmittance distribution of the plurality of regions for the light corresponding to the ith wavelength band and an ith effective sensitivity distribution based on the detection sensitivity of the image sensor for the light corresponding to the ith wavelength band; detecting only light corresponding to a jth wavelength band (j is an integer equal to or greater than 1 and equal to or less than N) among the N wavelength bands, and outputting jth mask data corresponding to a jth transmittance distribution of the plurality of regions for light of the jth wavelength band and a jth effective sensitivity distribution based on the detection sensitivity of the image sensor to light of the jth wavelength band; A correlation coefficient r between the i-th mask data and the j-th mask data ｉｊ of [0025] Let i ｍ and j ｍ are the effective sensitivities at the m-th pixel of the i-th and j-th mask data, respectively, and i ０ and j ０ are the average values ​​of the effective sensitivities of the i-th and j-th mask data, respectively, and the N wavelength bands are numbered in ascending or descending order of center wavelengths, In the combination of the i-th mask data and the j-th mask data where |i-j|=1 and i<j, (i, j)=(1, 2), (N-1, N) ｉｊ Any of the remaining r ｉｊ Greater than Device.

5. 1. A filter array for use in a system for generating N images corresponding to N wavelength bands, where N is an integer equal to or greater than 3, comprising: A plurality of optical filters having different spectral transmittances are provided, A correlation coefficient r between the i-th transmittance distribution of the plurality of optical filters for light corresponding to the i-th wavelength band (i is an integer greater than or equal to 1 and less than or equal to N) among the N wavelength bands and the j-th transmittance distribution of the plurality of optical filters for light corresponding to the j-th wavelength band (i is an integer greater than or equal to 1 and less than or equal to N) ｉｊ of [0030] Let i ｍ and j ｍ are the transmittances of the m-th optical filter in the i-th and j-th transmittance distributions, respectively, and i ０ and j ０ are the average values ​​of the transmittance of the i-th and j-th transmittance distributions, respectively, and the N wavelength bands are numbered in ascending or descending order of center wavelength, In a combination of the i-th transmittance distribution and the j-th transmittance distribution such that |i-j|=1 and i<j, (i, j)=(1, 2), (N-1, N) ｉｊ Any of the remaining r ｉｊ Greater than Filter array.

Images