Imaging device, imaging method, and program

Abstract

One aspect of the present invention is an imaging device that measures hyper-spectral images by means of compressive sensing, the imaging device comprising: a CASSI observation system that has an encoding unit that encodes an input signal by activating an encoding aperture mask on the input signal and outputs the encoded signal, a dispersion unit that performs wavelength dispersion on the input signal encoded by the encoding unit and outputs the dispersed input signal, and a measurement unit that images the input signal subjected to wavelength dispersion by the dispersion unit; and a modulation unit that modulates, with a multiplication mask, the output signal from the CASSI observation system.

Description

Technical Field

[0005]

[0001] The present invention relates to the technology of imaging devices, imaging methods, and programs.

Background Art

[0002] Conventionally, there is a technique for measuring hyperspectral images based on the compressive sensing theory of compressive spectral imaging. As one of the implementation methods, there is a technique called CASSI (Coded Aperture Snapshot Spectral Imaging) that combines a coded aperture mask and a dispersive optical element (see, for example, Non-Patent Document 1).

[0003] Also, regarding such CASSI, as a method for improving the measurement accuracy of hyperspectral images, a design method for the coded aperture mask in the CASSI optical system has been proposed (see, for example, Non-Patent Document 2).

[0004] On the other hand, a technique has been proposed for improving the measurement accuracy by devising the formulation of the optimization problem in the design of the coded aperture mask in CASSI and the reconstruction process from the compressed signal to the original signal, and limiting the information in the wavelength range to be acquired (see, for example, Non-Patent Document 3).

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0006] In general, the CASSI described above is formulated as shown in equation (1).

[0007]

number

[0008] However, the only design freedom in the CASSI observation process is the design of the coded aperture mask. Therefore, it is difficult to design an ideal observation matrix for reconstructing the original signal. The CASSI observation process (Φ) involves coding → variance → integration, but the variance and integration processes are specific to the optical element and therefore have no design freedom.

[0009] Furthermore, conventional techniques that limit the wavelength range to be acquired offer only two options: acquire each wavelength range or not, making it impossible to consider the relative importance of each range. In addition, since the wavelength range limitation is applied uniformly to the input signal, it was not possible to acquire information from different wavelength ranges in spatially different regions.

[0010] Furthermore, in conventional techniques, in order to limit the wavelength range to be acquired, pixels containing only information from the limited wavelength range are extracted from the compressed signal to solve the reconstruction problem. This reduces the number of usable elements in the compressed signal, which is expected to decrease the reconstruction accuracy. Therefore, conventional approaches have been taken to suppress the decrease in reconstruction accuracy by supplementing the information in the compressed signal through multiple imaging, but a disadvantage of this is that the measurement costs increase.

[0011] In view of the above circumstances, the present invention aims to provide a technology that can improve the measurement accuracy of hyperspectral images by CASSI. [Means for solving the problem]

[0012] One aspect of the present invention is an imaging device for measuring hyperspectral images by compressed sensing, comprising: an encoding unit that encodes and outputs an input signal by applying an encoding aperture mask to the input signal; a dispersion unit that outputs the input signal encoded by the encoding unit after wavelength dispersion; a measurement unit that images the input signal wavelength dispersed by the dispersion unit; and a modulation unit that modulates the output signal of the CASSI observation system with a multiplication mask.

[0013] One aspect of the present invention is an imaging method for measuring a hyperspectral image by compressed sensing, comprising: an imaging step of imaging an input signal using a CASSI observation system comprising: an encoding unit that encodes and outputs an input signal by applying an encoding aperture mask to the input signal; a dispersion unit that outputs the input signal encoded by the encoding unit after wavelength dispersion; and a measurement unit that images the input signal wavelength dispersed by the dispersion unit; and a modulation step of modulating the output signal of the CASSI observation system with a multiplication mask.

[0014] One aspect of the present invention is a program for causing a computer to function as the above-mentioned imaging device. [Effects of the Invention]

[0015] According to the present invention, it becomes possible to improve the measurement accuracy of hyperspectral images by CASSI.

Brief Description of Drawings

[0016] [Figure 1] It is a block diagram showing the configuration of the imaging device 1A of the first embodiment. [Figure 2] It is an image diagram showing the flow of processing by the imaging device 1A of the first embodiment. [Figure 3] It is a diagram showing an example of the effects exhibited by the imaging device 1A of the first embodiment. [Figure 4] It is a block diagram showing the configuration of the imaging device 1B of the second embodiment. [Figure 5] It is an image diagram showing the flow of processing by the imaging device 1B of the second embodiment. [Figure 6] It is an image diagram showing the flow of processing when the imaging device 1B of the second embodiment generates an encoding aperture mask and a multiplication mask. [Figure 7] It is an image diagram showing the flow in which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model in the imaging device 1B of the second embodiment.

Modes for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. <First Embodiment> FIG. 1 is a block diagram showing the configuration of the imaging device 1A of the first embodiment. The imaging device 1A includes an optical system 110, a CASSI observation system 120, a modulation unit 130, and a reconstruction processing unit 140.

[0018] The imaging device 1A is composed of a processor such as a CPU (Central Processing Unit) and memory. The imaging device 1A functions as a device that takes light as an input signal and outputs an estimated signal of a hyperspectral image by having the processor execute a program. Of the various parts of the imaging device 1A, a part of the CASSI observation system 120, the modulation unit 130, and the reconstruction processing unit 140 are realized by the processor executing a program. Some or all of the functions of the imaging device 1A that perform electrical signal processing may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor memory devices (e.g., SSDs: Solid State Drives), and memory devices such as hard disks and semiconductor memory devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0019] The optical system 110 consists of lenses that form an image. The optical system 110 may include multiple lenses.

[0020] The CASSI observation system 120 has the function of encoding and measuring input signals using CASSI. More specifically, the CASSI observation system 120 comprises, for example, an encoding unit 121, a distribution unit 122, and a measurement unit 123. The configuration of the CASSI observation system 120 will be described below, with the intention that the CASSI observation system 120 of the first and second embodiments may be the same as that of a conventional CASSI observation system.

[0021] The encoding unit 121 outputs an encoded input signal by applying an encoding aperture mask to the input signal. The encoding aperture mask may be an LCOS (Liquid Crystal On Silicon) or a DMD (Digital Mirror Device). The encoding aperture mask may consist of discrete values ​​of 0 and 1 (sometimes referred to as {0,1} below) or continuous values ​​from 0 to 1 (sometimes referred to as [0,1] below). For example, the encoding aperture mask can be generated by a data-driven approach in which the encoding aperture is modeled as a variable parameter and determined by end-to-end input-output optimization including the reconstruction model. On the other hand, contrary to such a data-driven approach, the encoding aperture mask may also be generated by a theoretical approach based on the theory of compressed sensing that utilizes the incoherence to the basis during sparse transformation of images.

[0022] The dispersion unit 122 is, for example, a prism. The dispersion unit 122 receives the output signal from the encoding unit 121 and outputs the input signal after wavelength dispersion.

[0023] The measurement unit 123 is, for example, an FPA (Focal Plane Array) array sensor. Each sensor for each pixel in the measurement unit 123 receives the output signal from the dispersion unit 122, integrates it in the wavelength direction, and outputs it. The measurement unit 123 outputs a compressed signal (image) as the measurement result.

[0024] The modulation unit 130 receives the output signal from the CASSI observation system 120, modulates it using a multiplication mask, and outputs the result. For example, the multiplication mask can be modeled as a variable parameter, similar to the coded aperture mask described above, and generated by a data-driven approach that optimizes end-to-end input and output, including the coded aperture mask and the reconstruction model.

[0025] The reconstruction processing unit 140 receives the output signal from the modulation unit 130, performs reconstruction processing on it, and outputs the estimation result (estimated signal) of the hyperspectral image.

[0026] Figure 2 is an illustrative diagram showing the processing flow by the imaging device 1A. Figure 2 shows the parameterized (modeled) signal conversion at each step. In the signal conversion in Figure 2, solid lines represent conversions based on the intrinsic (fixed) parameter values ​​of optical elements (prisms, sensors, etc.), while dashed lines represent conversions based on designable variable parameters.

[0027] First, the imaging device 1A inputs wavelength data of light in the imaging target space (Spectral Data Cube) D0 to the CASSI observation system 120 via the optical system 110 (step S1).

[0028] Next, in the CASSI observation system 120, the encoding unit 121 applies the encoding aperture mask M1 to the input wavelength data (input signal), thereby outputting the encoded wavelength data D1 to the dispersion unit 122 (step S2). This step means that each pixel of the 3D (spatial and wavelength) input signal is covered with the encoding aperture mask.

[0029] Next, in the CASSI observation system 120, the dispersion unit 122 wavelength-disperses the encoded wavelength data D1 and outputs it to the measurement unit 123 (step S3). This step means tilting the wavelength axis using a dispersion optical element (prism).

[0030] Next, in the CASSI observation system 120, the measurement unit 123 receives wavelength-dispersed wavelength data D2, and each pixel integrates the wavelength data D2 in the wavelength direction to generate compressed signal data D3 as the measurement result of the target space, which is then output to the modulation unit 130 (step S4). This process means that each tilted wavelength signal is integrated for each pixel. Through these steps, a 2D compressed signal is obtained.

[0031] Next, the modulation unit 130 generates modulated compressed signal data D4 by applying modulation using a multiplication mask to the 2D compressed signal data D3 input from the CASSI observation system 120 (step S5). The modulation unit 130 outputs the generated modulated compressed signal data D4 to the reconstruction processing unit 140.

[0032] Next, the reconstruction processing unit 140 performs reconstruction processing on the modulated compressed signal data D4 input from the modulation unit 130 to generate and output a hyperspectral image IMG as an estimation result (step S6). For example, the reconstruction processing unit 140 performs reconstruction processing by inputting the modulated compressed signal data D4 into a reconstruction model.

[0033] Here, the reconstruction model is generated, for example, by pre-training the correlation (i.e., observation matrix) between the desired signal data and the modulated compressed signal data using a machine learning technique such as a neural network. For example, the reconstruction model can be constructed using a DUN (Deep Unrolled Network), which implements iterative optimization algorithms such as ADMM (Alternating Direction Method of Multiplier) and ISTA (Iterative Shrinkage Thresholding Algorithm) through deep learning, and can be learned by optimizing the variable parameters, including the coding unit 121 and the modulation unit 130, end-to-end.

[0034] Figure 3 shows an example of the effect achieved by the imaging device 1A of the first embodiment. Figure 3 shows a comparison of the image estimation accuracy with a conventional configuration that does not use a multiplication mask (only an encoded aperture mask) and the image estimation accuracy with the configuration of this embodiment that uses the multiplication mask (a combination of an encoded aperture mask and a multiplication mask). The vertical axis of the graph shown in Figure 3 represents the PSNR (Peak Signal to-Noise Ratio) intensity. As is clear from Figure 3, the estimation result of this embodiment has a higher PSNR value than the conventional configuration, indicating improved image quality.

[0035] According to the imaging device 1A of the first embodiment configured in this way, the measurement accuracy of hyperspectral images by CASSI can be improved by performing modulation processing using a multiplication mask on the measured signal in the CASSI observation system that measures hyperspectral images.

[0036] More specifically, in the CASSI observation system, each pixel of the coded aperture mask either blocks or attenuates the corresponding pixel information of the input signal (attenuation in the case of [0,1]). Therefore, in conventional configurations, signal conversion for each wavelength signal within the same pixel in the input signal was limited to a common signal conversion for the pixel (conversion to scalar multiplication of {0,1} or [0,1]). In contrast, the imaging device 1A of the first embodiment has a coding aperture mask that directly acts on the input signal before dispersion processing, as well as a modulation unit 130 that modulates the measurement signal obtained by multiplying it by the mask after dispersion processing and integration processing. This enables different signal conversions for each wavelength signal within the same pixel in the input signal.

[0037] Therefore, for hyperspectral image measurement, it becomes possible to design observation matrices with a high degree of freedom, and by appropriately designing two masks, the coded aperture mask and the multiplication mask, it is possible to improve the accuracy of the final obtained original signal.

[0038] Furthermore, the design of the CASSI observation matrix involves the use of various optical elements and is implemented in hardware. In contrast, the modulation unit 130 of this embodiment is a signal processing unit after signal compression and can be implemented in software. Therefore, there are fewer physical constraints in the implementation of the modulation unit 130, and it has advantages such as ease of handling in terms of conversion speed and representation of continuous values.

[0039] Furthermore, in the imaging device 1A of the first embodiment, the modulation unit 130 does not add to or modify the conventional imaging process using CASSI. Therefore, the imaging device 1A of this embodiment makes it possible to easily improve the estimation accuracy of the original signal without complicating the device.

[0040] <Second Embodiment> Figure 4 is a block diagram showing the configuration of the imaging device 1B of the second embodiment. For Figure 4, components similar to those in Figure 1 are denoted by the same reference numerals as in Figure 1, and their explanation is omitted here. The imaging device 1B differs from the imaging device 1A of the first embodiment in that it does not include a modulation unit 130, but includes a focus weight setting unit 150 and a mask generation unit 160.

[0041] The attention weight setting unit 150 sets a weight (hereinafter referred to as "attention weight") for each pixel captured by the CASSI observation system 120, indicating the degree to which the observed data should be considered noteworthy. For example, the attention weight setting unit 150 receives input for setting the attention weight from an operation input unit (not shown), and based on this input information, generates attention weight information indicating the attention weight for each pixel and outputs it to the mask generation unit 160.

[0042] More specifically, the attention weight is a value within the continuous range [0,1], and the attention weight setting unit 150 sets the 3D (spatial and wavelength) attention weight for each pixel.

[0043] The method for setting the attention weight may be any method other than those described above. For example, the attention weight information may be received from another device via communication, or the attention weight information may be read from a storage unit (not shown) that has been stored in advance.

[0044] The mask generation unit 160 generates an encoded aperture mask based on the attention weight information input from the attention weight setting unit 150. The mask generation unit 160 sets the generated encoded aperture mask in the encoding unit 121 of the subsequent CASSI observation system 120.

[0045] Figure 5 is an illustrative diagram showing the processing flow by the imaging device 1B. First, in the imaging device 1B, the focus weight setting unit 150 performs the process of setting the focus weight for each pixel (step S201), and outputs the focus weight information indicating the setting to the mask generation unit 160. For example, the focus weight setting unit 150 may accept an operation to select a focus region in the image using a rectangle, or it may accept an operation to input an annotation map that associates information on whether or not each region in the image is a focus region.

[0046] Next, the mask generation unit 160 receives attention weight information from the attention weight setting unit 150 and generates an encoded aperture mask based on that attention weight information (step S202). For example, the mask generation unit 160 can generate an encoded aperture mask using a DNN (Deep Neural Network) based image processing model such as CNN (Convolutional Neural Network) or U-NET. The mask generation unit 160 sets the generated encoded aperture mask in the encoding unit 121 of the subsequent CASSI observation system 120.

[0047] The subsequent processing is the same as the processing flow in the first embodiment when a multiplication mask is not applied. Specifically, the CASSI observation system 120 acquires observation data (images) using the coded aperture mask set in step S202, and the reconstruction processing unit 140 performs reconstruction processing on the acquired observation data to output an image which is the estimated result of the original signal. In Figure 5, the same reference numerals as in Figure 2 are used for processing that is the same as the processing performed by the imaging device 1A in the first embodiment.

[0048] In Figure 5, the case in which the mask generation unit 160 generates only an encoded aperture mask by setting the weights of interest was described. However, the mask generation unit 160 may also generate a multiplication mask in addition to the encoded aperture mask, as in the first embodiment. In this case, the function of the mask generation unit 160 to generate a multiplication mask can be realized by machine learning using a deep learning model.

[0049] Figure 6 is an illustrative diagram showing the processing flow in this case. In this case, the imaging device 1B may further include the modulation unit 130 of the first embodiment and be configured to modulate the observation data of the CASSI observation system 120 before performing reconstruction processing.

[0050] Figure 7 is an illustrative diagram showing the flow in which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model. As described above, the function of the mask generation unit 160 to generate a multiplication mask and the reconstruction processing unit 140 can be realized by a deep learning model. For example, the mask generation unit 160 and the reconstruction processing unit 140 can each construct a multiplication mask generation model and a reconstruction model specialized for the reconstruction of the region of interest by learning the Mean Squared Error (MSE) weighted by the weight of interest set on a pixel-by-pixel basis as the loss. In other words, by representing the weight of interest as a real number [0,1] and designing the loss using that value, it becomes possible to measure hyperspectral images that take into account the magnitude of importance. For example, the loss is defined as shown in equation (2) below.

[0051]

number

[0052] As described above, the imaging device 1B of the second embodiment can construct a reconstruction model specialized for the region of interest by setting a weight indicating the degree of interest for each pixel in the captured image for the CASSI observation system that measures hyperspectral images, and generating an encoded aperture mask based on the set weight information of interest. For this reason, the imaging device 1B of the second embodiment can improve the measurement accuracy of hyperspectral images by CASSI.

[0053] Furthermore, the imaging device 1B of the second embodiment includes the modulation unit 130 of the first embodiment, and the mask generation unit 160 generates a multiplication mask based on a generative model (DNN) that has been trained to be optimized end-to-end, including an encoded aperture mask based on the attention weight information set by the attention weight setting unit 150, and the modulation unit 130 modulates the observed data using the generated multiplication mask, thereby further improving the measurement accuracy of hyperspectral images by CASSI.

[0054] More specifically, the imaging device 1B of the second embodiment takes the coded aperture mask used in the CASSI observation system 120, assigns weights (set of focus weights) to the wavelength range of interest on a pixel-by-pixel basis, and then generates a coded aperture mask using a DNN (Deep Neural Network) with the focus weights for each pixel and wavelength range as input. In other words, by designing pixel-wise weights and learning the coding aperture mask generation model and reconstruction model that take these weights into account, it becomes possible to acquire measurement images with spatially varied wavelength regions of interest.

[0055] This allows the importance (weight) of each wavelength range to be set as a value within a continuous range of [0,1] rather than a discrete value of {0,1}, enabling imaging with different importance values ​​for different wavelength ranges for each pixel. According to the imaging device 1B of the second embodiment, unlike conventional methods, all pixel information of the acquired compressed signal can be used, so the reconstruction result from a single scan is expected to have sufficient performance, thus reducing measurement costs.

[0056] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0057] This invention is applicable to an imaging device that measures hyperspectral images using the CASSI observation system. [Explanation of Symbols]

[0058] 1A, 1B…Imaging device, 110…Optical system, 120…CASSI observation system, 121…Encoding unit, 122…Dispersion unit, 123…Measurement unit, 130…Modulation unit, 140…Reconstruction processing unit, 150…Weight setting unit, 160…Mask generation unit

Claims

1. An imaging device that measures hyperspectral images using compressed sensing, A CASSI observation system comprising: an encoding unit that encodes and outputs an input signal by applying an encoding aperture mask to an optical input signal; a dispersion unit that outputs the input signal encoded by the encoding unit after wavelength dispersion; and a measurement unit that images the input signal wavelength dispersed by the dispersion unit. A modulation unit that modulates the output signal of the CASSI observation system using a multiplication mask, A reconstruction processing unit, which has been pre-trained as a reconstruction model a neural network constructed to optimize the input and output of the imaging device, including the encoding aperture mask and multiplication mask, Equipped with, The reconstruction processing unit outputs the estimation result of the hyperspectral image by performing reconstruction processing on the output signal of the modulation unit using the reconstruction model. Imaging device.

2. The modulation unit has pre-learned the input and output of the imaging device as a multiplication mask optimized including the encoding aperture mask and the reconstruction process. The imaging apparatus according to claim 1.

3. An imaging method in which an imaging device measures a hyperspectral image by compressed sensing, An imaging step of imaging an input signal using a CASSI observation system comprising: an encoding unit that encodes and outputs an input signal by applying an encoding aperture mask to an optical input signal; a dispersion unit that outputs the input signal encoded by the encoding unit after wavelength dispersion; and a measurement unit that images the input signal that has been wavelength dispersed by the dispersion unit; A modulation step in which the output signal of the CASSI observation system is modulated by a multiplication mask, A reconstruction processing step that outputs the estimation result of a hyperspectral image by applying reconstruction processing to the output signal of the modulation step using a pre-trained reconstruction model, It has, The reconstruction model is a neural network constructed to optimize the input and output of the imaging device, including the encoding aperture mask and multiplication mask. Imaging method.

4. An imaging device for measuring hyperspectral images by compressed sensing, A CASSI observation system comprising: an encoding unit that encodes and outputs an input signal by applying an encoding aperture mask to an optical input signal; a dispersion unit that outputs the input signal encoded by the encoding unit after wavelength dispersion; and a measurement unit that images the input signal wavelength dispersed by the dispersion unit. A modulation unit that modulates the output signal of the CASSI observation system using a multiplication mask, A reconstruction processing unit, which has been pre-trained as a reconstruction model a neural network constructed to optimize the input and output of the imaging device, including the encoding aperture mask and multiplication mask, Equipped with, The reconstruction processing unit outputs the estimation result of the hyperspectral image by performing reconstruction processing on the output signal of the modulation unit using the reconstruction model. In the aforementioned imaging device, A program for causing a computer mounted on the imaging device to perform the processing of the encoding unit, the modulation unit, and the reconstruction processing unit.

Images