Spectroscopic imaging system, measurement system

Abstract

To provide a spectroscopic imaging technique of accurately and rapidly acquiring image information with a large number of bands.SOLUTION: A spectroscopic imaging system according to the present invention has a plurality of optical filters with different optical transmittances according to optical wavelengths. The spectroscopic imaging system determines whether or not incident lights entering the optical filters have the same spectrum, and calculates the spectrum of the incident lights by using the optical filters for which incident lights are determined to have the same spectrum.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a spectroscopic imaging system that disperses and detects incident light. [Background technology]

[0002] When observing materials or objects using a camera or microscope, the reflected or transmitted light from the material or object is detected and imaged using the camera's or microscope's detector (e.g., image sensor). The detector captures all light of wavelengths to which it is sensitive. For example, a Si CMOS image sensor is sensitive to light of wavelengths of approximately 0.4 to 1 μm, integrating and detecting light of that wavelength. In contrast, spectroscopic imaging obtains images at specific wavelengths by dispersing reflected or transmitted light and detecting light of specific wavelengths with a detector. Multispectral imaging, which involves several to a dozen wavelengths (bands), i.e., several to a dozen corresponding images, is sometimes called multispectral imaging, while hyperspectral imaging, which involves more than that, is sometimes called hyperspectral imaging. Generally, different types of materials or objects absorb different wavelengths, so images obtained using spectroscopic imaging can be used to classify the type of material or object and determine its moisture content.

[0003] Multispectral imaging can be achieved by arranging multiple types of optical filters on an image sensor that only allow light of specific wavelengths to pass through. For example, a color image sensor can be considered a type of multispectral imaging because it has color filters that pass red, green, and blue, respectively. On the other hand, when using a similar method in hyperspectral imaging, increasing the number of optical filters reduces spatial resolution, so a method is sometimes used in which light dispersed by a spectroscopic element such as a diffraction grating is irradiated onto the image sensor. However, because light is dispersed in the x or y direction and irradiated onto an image sensor that consists of a two-dimensional array, the remaining dimension is one, i.e., it becomes a line sensor, and the entire image cannot be captured at once.

[0004] In contrast, a compressed sensing technique is known that uses optical filters but significantly reduces the number of filters to less than the number of bands desired. This technique uses multiple types of optical filters that pass light of various wavelengths but have different transmittances depending on the wavelength, rather than optical filters that only pass light of specific wavelengths. The intensity of the light that passes through each filter is detected by an image sensor, and the original light spectrum is reconstructed from this intensity information. To increase the accuracy of the reconstructed spectrum, it is best for the wavelength dependence of the transmittance to differ as much as possible between different types of filters. Although this technique also uses multiple types of filters, while the spatial resolution is inferior to images that do not use filters, it can significantly improve spatial resolution compared to using filters for the number of bands desired.

[0005] The following Patent Document 1 addresses the issue of "providing a spectroscopic device capable of detecting the intensity of light for each wavelength to be detected with high accuracy," and describes the following technology (see abstract): "Spectroscopic device 1 detects the intensity of light for each of a plurality of wavelengths to be detected. Spectroscopic device 1 includes a plurality of optical filters having a plurality of transmission wavelength bands whose transmittance is equal to or greater than a threshold transmittance in a wavelength band of interest that includes the plurality of wavelengths to be detected. Spectroscopic device 1 includes detection units 11 and 17 that detect the intensity of light for each of the plurality of wavelengths to be detected based on light that has passed through the plurality of optical filters. The plurality of optical filters have a plurality of transmission wavelength bands that are different from one another." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-138670 Summary of the Invention [Problem to be solved by the invention]

[0007] When acquiring images of moving objects or substances, or moving bodies, using spectral imaging, if the subject moves relatively significantly between the time the detector acquires one piece of information (image information or line information) and the time the next piece of information is acquired, it is preferable to acquire image information at high speed. Therefore, a method using an optical filter is more suitable than a method that disperses light using a diffraction grating, etc. Furthermore, when observing objects or substances of a specific type, a large number of bands is generally required, so hyperspectral imaging is preferable to multispectral imaging.

[0008] When performing hyperspectral imaging using filters with compressed sensing techniques, a spectrum is reconstructed from the light intensity transmitted through multiple types of filters, but the spectrum of the light before passing through the filters used for reconstruction must be approximately identical. However, with conventional technologies such as Patent Document 1, it is difficult to determine whether the spectrum of the light before passing through the filters used for reconstruction is approximately identical.

[0009] The present invention has been made in view of the above-mentioned problems, and has an object to provide a spectral imaging technique that can acquire image information with a large number of bands accurately and quickly. [Means for solving the problem]

[0010] The spectroscopic imaging system of the present invention has a plurality of optical filters with different light transmittances for each light wavelength, determines whether the incident light entering each of the optical filters has the same spectrum, and calculates the spectrum of the incident light using the optical filters for which it has been determined that the incident light has the same spectrum. [Effects of the Invention]

[0011] The spectral imaging system according to the present invention makes it possible to acquire image information with a large number of bands accurately and quickly. Other objects, configurations, advantages, etc. of the present invention will become apparent from the description of the following embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a configuration diagram illustrating an outline of a spectral imaging system according to a first embodiment. [Figure 2] 1 is a bird's-eye view showing an enlarged view of a part of the chip 2. FIG. [Figure 3] FIG. 2 is an enlarged top view of a part of the chip 2. [Figure 4] The relationship between the spectrum I of the incident light 1, the effective transmittance T, and the output P of the image sensor 7 is shown. [Figure 5] 10A and 10B are schematic diagrams showing examples in which the spectrum of light incident on each pattern is different. [Figure 6] The following shows a modification of equation (1) when the spectrum differs for each pattern. [Figure 7] The results of decomposing equation (2) into two independent equations are shown below. [Figure 8] FIG. 10 is a top view showing the configuration of a macrocell in the second embodiment. [Figure 9] FIG. 10 is a configuration diagram of a spectral imaging system according to a third embodiment. [Figure 10] FIG. 10 is a configuration diagram of a measurement system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] <Embodiment 1> Figure 1 is a schematic diagram illustrating the configuration of a spectroscopic imaging system according to a first embodiment of the present invention. The system includes: a chip 2 that converts incident light 1 into an electrical signal representing the light intensity; a computing device 3 that specifies a driving method for the chip 2 and receives and performs calculations on the output signal from the chip 2; and a storage device 4 that stores the calculation results. A display that displays the calculation results may be connected as needed. The computing device 3 may be composed of multiple devices, and the incident light 1 may be focused by a lens and irradiated onto the chip 2.

[0014] Figure 2 is a bird's-eye view showing an enlarged portion of the chip 2. Nanostructures 5, each with dimensions similar to or shorter than the wavelength of the incident light 1, are formed in an array to form an optical filter 6. An image sensor 7 is formed below the optical filter 6. Although the figure shows only a small number of holes making up the nanostructure 5, in reality, many holes are formed. The shape of the holes is not limited to being round or square.

[0015] A substrate used in fabricating the optical filter 6 or an anti-reflection film may be formed on the upper layer of the optical filter 6. If the substrate used in fabricating the optical filter 6 is left, the substrate must be able to transmit the incident light 1 well. Therefore, if the incident light 1 is visible light, the substrate is made of, for example, quartz, and if the incident light 1 is infrared light, the substrate is made of, for example, quartz or silicon. Since the optical filter 6 must partially transmit the incident light 1, it is formed thin, with a film thickness of, for example, 100 nm to 500 nm, and made of, for example, silicon nitride, silicon, or silicon carbide. The optical filter 6 may be fabricated separately from the image sensor 7 and then bonded to the image sensor 7, or it may be formed by forming a film on the image sensor 7 and processing the nanostructure 5. The nanostructure 5 is formed so that, for example, circular or square-shaped holes penetrate the optical filter 6, and the pattern dimensions and spacing are approximately the same as or shorter than the wavelength of the incident light 1.

[0016] FIG. 3 is a top view showing an enlarged portion of the chip 2. The nanostructure 5 is configured with nine different patterns repeatedly arranged. A set of these nine different patterns is called a macrocell. Each pattern may correspond to one pixel of the image sensor 7 directly below it, or may be formed on multiple pixels. The dimensions of each pattern of the nanostructure 5 are formed to be an integer multiple of the pixel size of the image sensor 7. In FIG. 3, the nanostructure 5 is formed with nine different patterns, but any number of patterns may be used. In this embodiment, nine types will be described.

[0017] First, let us assume that the incident light 1 is spatially uniform. In this case, the spectrum of the incident light 1 is reconstructed using the outputs of the image sensor 7 directly below the nine types of nanostructures 5. If each pattern of the nanostructures 5 is arranged on multiple pixels of the image sensor 7, the output of a representative pixel may be used, or the average output of multiple pixels may be used. The spectrum I of the incident light 1, the effective transmittance T, and the output P of the image sensor 7 have the relationship shown in Equation (1) in Figure 4.

[0018] The effective transmittance is not the transmittance of the optical filter 6 alone, but the transmittance of the entire material existing between the incident light 1 and the image sensor 7 multiplied by the sensitivity of the image sensor 7 (i.e., the efficiency of converting incident light into output). n is the number of bands when the wavelength range to which the image sensor 7 responds is expressed discretely. Intensity I m is the intensity of incident light 1 at the mth wavelength, and transmittance T lm is the effective transmittance of light at the mth wavelength through the lth optical filter, P l is the output of the image sensor 7 corresponding to the l-th optical filter. The output P is a measured value, and the transmittance T is measured in advance. This information is used to find the intensity I of the incident light 1, but since the number of bands n is greater than the number of types of optical filters 6, it cannot be found analytically, so I that satisfies equation (1) is estimated by search, i.e., the spectrum I is reconstructed.

[0019] Figure 5 is a schematic diagram showing an example in which the incident light spectrum for each pattern is different. Correct reconstruction is not possible unless the spectrum of the light incident on the optical filters (nine types in this case) used for reconstruction is approximately the same, but as shown in Figure 5, the spectrum may differ for each pattern. In this case, equation (1) can be rewritten as equation (2) in Figure 6. The incident light spectrum is I or J, and spectrum I is irradiated onto the 1st to Kth filters, and spectrum J is irradiated onto the K+1th to 9th filters. In Figure 5, K=4.

[0020] Equation (2) can be decomposed into two independent equations, equations (3) and (4), shown in Figure 7. Both equations (3) and (4) have the same form as equation (1). In the case of equation (3), the outputs P1 to P2 of the image sensor corresponding to the K types of optical filters are K The important point here is that the value of K is known, that is, the boundary where the spectrum changes is known. Therefore, the problem according to the present invention can be solved by recognizing the boundary and dynamically selecting the optical filter to be used for reconstruction according to the boundary.

[0021] In the first embodiment, the outputs (pixel values) of the detection elements of the image sensor corresponding to the same type of optical filter are compared between adjacent macrocells in FIG. 3. If the comparison reveals that the outputs are substantially the same, it is assumed that incident light with the same spectrum is being irradiated. On the other hand, if the comparison reveals a difference in output, it is assumed that incident light with different spectra is being irradiated. In other words, there is a boundary somewhere between the two types of optical filters being compared. By comparing all optical filters in this manner, it is possible to determine the boundary, thereby providing spectral imaging that shows a correct spectrum. Note that when comparing the outputs of the detection elements of the image sensor, outputs that have been corrected for individual differences between the detection elements are used.

[0022] According to this embodiment, spectral information and spatial information can be acquired at the same time. That is, the computing device 3 creates data to which spectral information is attached for each pixel of the image sensor 7 (for each pixel, if one optical filter 6 is arranged across multiple pixels), thereby creating data that describes both the spatial information represented by the image and the spectral information for each pixel. This makes it possible to obtain information seamlessly by loading it onto a fast-moving vehicle. Therefore, by combining it with a drone or the like that knows its own coordinates by a method such as satellite communication, it is possible to provide the spectral information embedded in a map.

[0023] In this embodiment, the amount of data acquired is compressed. In other words, the amount of information acquired is small compared to the spectral information and spatial information obtained by reconstruction, so the load on storage is small. Therefore, the acquired data is stored in storage on the mobile object, and the spectra can be later reconstructed together to obtain the spectral information and spatial information. In this case, the computing device on the mobile object can be small, making it suitable for use in drones and other devices that require lightweight design.

[0024] <Embodiment 2> Embodiment 2 of the present invention describes a configuration example in which the accuracy of the reconstructed spectrum is improved by changing the internal structure of the macrocell from embodiment 1, while using the same device configuration (Figure 1) and chip structure (Figure 2) as embodiment 1.

[0025] FIG. 8 is a top view showing the configuration of a macrocell in the second embodiment. In reconstructing a spectrum by compressed sensing, the outputs of image sensors corresponding to different optical filters in the macrocell are used. In this case, the more different types of optical filters are installed in the macrocell, the higher the accuracy of the reconstructed spectrum. On the other hand, the more different types of optical filters are installed in the macrocell, the larger the area occupied by the macrocell, and therefore the higher the probability that different spectra will be irradiated within the macrocell. Therefore, in the second embodiment, the same type of pattern is arranged within the macrocell. FIG. 8 shows an example.

[0026] By comparing the image sensor outputs corresponding to optical filters of the same type of pattern within the macrocell, it is possible to detect whether a boundary between different spectra exists within the macrocell, and dynamically select the type of optical filter to be used for reconstruction according to the boundary. When identifying the boundary, the outputs of the same type of filters are compared, but different types of optical filters are selected to be used for reconstructing the spectrum. In the second embodiment, even if there are multiple patterns of the same type within the macrocell and the spectra of the incident light irradiated onto them are approximately the same, one of the same type of patterns is selected or the average output of the same type of filters is taken and used to reconstruct the spectrum.

[0027] <Embodiment 3> Fig. 9 is a configuration diagram of a spectral imaging system according to embodiment 3 of the present invention. The spectral imaging system shown in Fig. 9 includes an image sensor 7 having an optical filter 6 made of nanostructures 5 directly thereon, and an image sensor 8 that does not have an optical filter 6 made of nanostructures 5.

[0028] The image sensor 8 has high resolution because it does not have a macrocell configured by the optical filter 6. The image sensor 8 identifies the pattern boundary where the spectra differ as described in the first embodiment, and dynamically selects the optical filter 6 to be used for spectral reconstruction based on the boundary, taking into account the parallax between the image sensors 7 and 8. It is desirable that the image sensors 7 and 8 be of the same type, but they may be of different types. Even if an expensive near-infrared image sensor that is sensitive to longer wavelengths than a Si CMOS image sensor is used as the image sensor 7, it is possible to prevent an increase in system costs by using an inexpensive Si CMOS image sensor (i.e., a sensor that can detect a narrower range of light wavelengths than the image sensor 7) as the image sensor 8.

[0029] <Embodiment 4> 10 is a configuration diagram of a measurement system according to a third embodiment of the present invention. The measurement system is configured by mounting a spectral imaging system 110 and a measuring instrument 120 on a mobile object 100. The spectral imaging system 110 is the spectral imaging system described in the first to third embodiments. The measuring instrument is a device that measures the geographical coordinates of the mobile object 100, and can be configured by, for example, a GPS (Global Positioning System).

[0030] While moving, the mobile object 100 acquires a spectrum for each pixel using the spectroscopic imaging system 110, and stores the spectrum in the storage 4 in association with the geographical coordinates for each position of the mobile object 100. This makes it possible to generate a map describing the spectrum for each position of the mobile object 100. This process may be performed by the calculation device 3 or by a calculation device provided in the mobile object 100.

[0031] <Modifications of the present invention> The present invention is not limited to the above-described embodiments and includes various modifications. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment, without departing from the spirit of the invention. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with the configuration of another embodiment.

[0032] For ease of understanding, the position, size, range, etc. of each component shown in the drawings etc. of this specification may not represent the actual position, size, range, etc. Therefore, the present invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings etc.

[0033] In the above-described embodiment, the light wavelength range that can be detected by the image sensor 7 does not necessarily completely include the light wavelength range that can be detected by the image sensor 8. In other words, it is sufficient if the image sensor 8 can be used to identify the boundary line of a filter into which light of the same spectrum is incident.

[0034] In the above embodiment, the optical filters 6 having the same optical transmittance for each optical wavelength means that the optical filters 6 can be considered to have the same dependency on optical wavelength, assuming that there are minute differences due to manufacturing variations. For example, if the calculation device 3 receives results of the image sensor detecting light of the same spectrum that has passed through different optical filters 6 and can handle the results as having the same signal level, it can be said that the optical filters 6 have the same optical transmittance for each optical wavelength.

[0035] In the above-described embodiments, if there is variation in the detection signal for each detection element of the image sensor, the arithmetic unit 3 may correct the variation before performing the processes described in the above-described embodiments. If the variation has been corrected at the time the image sensor outputs the detection signal, the detection signal may be used as is.

[0036] The present invention relates to a spectroscopic imaging device and a spectroscopic imaging system using the same, which can be used to classify the types of objects or substances, or to identify the amount of moisture contained, in places such as factory production lines, farms, and forests. [Explanation of symbols]

[0037] 1...Incoming light 2 chips 3...Arithmetic unit 4. Storage 5. Nanostructures 6. Optical Filter 7. Image sensor 8. Image sensor

Claims

1. A spectroscopic imaging system that detects incident light by dispersing it, a first image sensor for detecting the intensity of the incident light; a macrocell disposed on the first image sensor; a computing unit that calculates the spectrum of the incident light using the intensity detected by the first image sensor; Equipped with the macrocell has a plurality of optical filters each having a different light transmittance for each light wavelength; the arithmetic unit is configured to calculate a spectrum of the incident light for each of one or more pixels of the first image sensor, using the light transmittance for each light wavelength of each of the optical filters and a result of detection by the first image sensor of the intensity of the incident light that has passed through each of the optical filters; the calculation device determines whether the incident light beams incident on the optical filters have the same spectrum; The calculation device calculates the spectrum of the incident light using the optical filters that have been determined to have the same spectrum as each other. A spectroscopic imaging system comprising:

2. the arithmetic unit identifies a boundary between the optical filter onto which the incident light having the same spectrum is incident and the optical filter onto which the incident light having a different spectrum is incident by comparing the intensities of the incident light that has passed through two or more of the optical filters having the same light transmittance for each light wavelength; The calculation device determines the optical filter having the same spectrum as the incident light based on the identified boundary.

2. The spectroscopic imaging system according to claim 1.

3. The computing device divides the optical filters into one or more groups into which the incident light having the same spectrum is incident based on the identified boundary; The calculation device calculates the spectrum of the incident light for each spectrum possessed by the incident light by using the optical filters belonging to each of the groups.

3. The spectroscopic imaging system according to claim 2.

4. The arithmetic unit calculates a spectrum of the incident light using a result of detection by the first image sensor of the intensity of the incident light that has passed through each of the plurality of optical filters having different light transmittances for each light wavelength.

2. The spectroscopic imaging system according to claim 1.

5. the spectroscopic imaging system comprises a plurality of the macrocells; At least one of the plurality of macrocells has, as the optical filter, a plurality of different optical filters having different optical transmittances for each optical wavelength; a plurality of homogeneous optical filters having the same light transmittance for each light wavelength; and the calculation device calculates a spectrum of the incident light using a plurality of the different optical filters belonging to the same macrocell; The computing device identifies the boundary by comparing the intensities of the incident light that has passed through a plurality of the same optical filters that belong to the same macrocell.

3. The spectroscopic imaging system according to claim 2.

6. When the macrocell has a plurality of the heterogeneous optical filters but does not have a plurality of the homogeneous optical filters, the calculation device identifies the boundary by using the optical filters in each of two adjacent macrocells, the optical filters having the same optical transmittance for each optical wavelength.

6. The spectroscopic imaging system according to claim 5.

7. the calculation device selects one of the intensities of the incident light that have passed through the plurality of optical filters of the same type that belong to the same macrocell, or calculates an average of the intensities; The computing device calculates the spectrum of the incident light using the selected intensity or the average as the intensity of the incident light that has passed through the homogeneous optical filter.

7. The spectroscopic imaging system according to claim 6.

8. the spectroscopic imaging system further comprises a second image sensor; the second image sensor does not have the macrocell; The computing device determines the boundary using the incident light incident on the second image sensor.

3. The spectroscopic imaging system according to claim 2.

9. The range of light wavelengths that the second image sensor can detect is narrower than the range of light wavelengths that the first image sensor can detect.

9. The spectroscopic imaging system according to claim 8.

10. the computing device creates an image of the surrounding space of the spectroscopic imaging system using the result of the first image sensor detecting the incident light; The computing device generates data describing the calculated spectrum for one or more pixels of the image.

2. The spectroscopic imaging system according to claim 1.

11. A moving object equipped with the spectral imaging system according to any one of claims 1 to 10. a measuring instrument for measuring the geographic coordinates of the moving object; Equipped with The mobile object acquires the spectrum while moving, and stores the spectrum in association with the geographical coordinates of each location, thereby generating a map describing the spectrum for each location. A measurement system characterized by:

Images