Imaging device, imaging method, and program

Abstract

One aspect of the present invention relates to an imaging device for measuring a hyperspectral image using compressive sensing, the imaging device comprising: a CASSI observation system comprising an encoding unit that encodes an input signal by applying a coded-aperture mask to the input signal and outputs the encoded signal, a dispersing unit that disperses the input signal encoded by the encoding unit into wavelength components and outputs the dispersed wavelength components, and a measuring unit that captures an image of the input signal subjected to the wavelength dispersion by the dispersing unit; an attention weight setting unit that sets a weight, which indicates the degree of attention, for each pixel used for imaging by the CASSI observation system; and a mask generating unit that generates the coded-aperture mask on the basis of the attention weight set for each pixel by the attention weight setting unit.

Description

[Technical Field]

[0001] The present invention relates to an imaging device, an imaging method, and a program technology. [Background technology]

[0002] Conventionally, there is a hyperspectral image measurement technology based on the compressed sensing theory called compressed spectral imaging. One of the implementation methods is a technology called CASSI (Coded Aperture Snapshot Spectral Imaging), which combines a coded aperture mask and a dispersive optical element (see, for example, Non-Patent Document 1).

[0003] Regarding CASSI, a method for designing a coded aperture mask in a CASSI optical system has been proposed as a technique for improving the measurement accuracy of hyperspectral images (see, for example, Non-Patent Document 2).

[0004] On the other hand, a technology has been proposed to improve measurement accuracy by limiting the wavelength range information to be acquired through the design of the coded aperture mask in CASSI and the formulation of the optimization problem in the reconstruction process from the compressed signal to the original signal (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Compressive Coded Aperture Spectral Imaging: An Introduction [Arce et al., 2013] [Non-patent document 2] Compressive spectral imaging approach using adaptive coded apertures [Zhang et al., 2020] [Non-patent document 3] Rank minimization code aperture design for spectrally selective compressive imaging[Arguello et al., 2013] Summary of the Invention [Problem to be solved by the invention]

[0006] Generally, the above-mentioned CASSI is formulated as in equation (1).

[0007]

number

[0008] However, the only degree of freedom in the design of the CASSI observation process is the design of the coded aperture mask. Therefore, it is difficult to design an ideal observation matrix for reconstructing the original signal. The CASSI observation process (Φ) is a process of encoding → dispersion → accumulation, but the dispersion and accumulation processes are inherent to the optical elements and there is no degree of freedom in the design.

[0009] Furthermore, conventional technologies that limit the wavelength range to be acquired only allow for the choice of whether or not to acquire each wavelength range, and are unable to consider the importance of each.In addition, because the wavelength range is limited uniformly for the input signal, it is not possible to acquire information on different wavelength ranges for spatially different regions.

[0010] Furthermore, in conventional technology, in order to limit the wavelength range to be acquired, pixels containing only information from the limited wavelength range are extracted from the compressed signal to solve the reconstruction problem, which reduces the number of available compressed signal elements and is expected to reduce reconstruction accuracy. Therefore, a conventional approach has been to suppress the reduction in reconstruction accuracy by capturing compressed signal information multiple times, but the disadvantage is that the cost of measurement increases.

[0011] In view of the above circumstances, an object of the present invention is to provide a technique that can improve the measurement accuracy of hyperspectral images using CASSI. [Means for solving the problem]

[0012] One aspect of the present invention is an imaging device that measures hyperspectral images using compressed sensing, and includes a CASSI observation system that includes: an encoding unit that encodes and outputs an input signal by applying a coded aperture mask to the input signal; a dispersing unit that wavelength-disperses and outputs the input signal coded by the encoding unit; and a measurement unit that captures the input signal wavelength-dispersed by the dispersing unit; an attention weight setting unit that sets an attention weight for each pixel captured by the CASSI observation system; and a mask generation unit that generates the coded aperture mask based on the attention weight for each pixel set by the attention weight setting unit.

[0013] One aspect of the present invention is an imaging method for measuring hyperspectral images using compressed sensing, comprising: an imaging step of imaging an input signal using a CASSI observation system that includes: an encoding unit that encodes the input signal by applying a coded aperture mask to the input signal and outputs the encoded signal; a dispersing unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measurement unit that images the input signal wavelength-dispersed by the dispersing unit; and a modulation step of modulating the output signal of the CASSI observation system using a multiplication mask.

[0014] One aspect of the present invention is a program for causing a computer to function as the imaging device described above. [Effects of the Invention]

[0015] The present invention makes it possible to improve the measurement accuracy of hyperspectral images using CASSI. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing the configuration of an imaging device 1A according to a first embodiment. [Figure 2] FIG. 2 is a conceptual diagram showing the flow of processing by the imaging device 1A of the first embodiment. [Figure 3] 10A and 10B are diagrams illustrating an example of an effect achieved by the imaging device 1A of the first embodiment. [Figure 4] FIG. 10 is a block diagram showing the configuration of an imaging device 1B according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing the flow of processing by an imaging device 1B of a second embodiment. [Figure 6] FIG. 10 is an image diagram showing the flow of processing when an imaging device 1B of the second embodiment generates a coded aperture mask and a multiplication mask. [Figure 7] FIG. 10 is a conceptual diagram showing a flow in which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model in the imaging device 1B of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First Embodiment 1 is a block diagram showing the configuration of an image pickup apparatus 1A according to the first embodiment. The image pickup apparatus 1A includes an optical system 110, a CASSI observation system 120, a modulation unit 130, and a reconstruction processing unit 140.

[0018] The imaging device 1A is configured using a processor such as a CPU (Central Processing Unit) and a memory. The imaging device 1A functions as a device that receives light as an input signal and outputs an estimated signal of a hyperspectral image by having the processor execute a program. Of the components of the imaging device 1A, part of the CASSI observation system 120, the modulation unit 130, and the reconstruction processing unit 140 are implemented by having the processor execute a program. Note that some or all of the functions of the imaging device 1A that perform electrical signal processing may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The program may be transmitted via a telecommunications line.

[0019] The optical system 110 is composed of a lens that forms an image, and may include multiple lenses.

[0020] CASSI observation system 120 has the function of encoding and measuring an input signal using CASSI. More specifically, CASSI observation system 120 includes, for example, an encoding unit 121, a dispersing unit 122, and a measuring unit 123. The configuration of CASSI observation system 120 will be described below, but this is intended to mean that CASSI observation system 120 of the first and second embodiments may be similar to a conventional CASSI observation system.

[0021] The encoding unit 121 applies a coded aperture mask to the input signal to output a coded input signal. The coded aperture mask may be a liquid crystal on silicon (LCOS) or a digital mirror device (DMD). The coded aperture mask may be composed of discrete values ​​of 0 and 1 (hereinafter sometimes represented as {0,1}) or continuous values ​​from 0 to 1 (hereinafter sometimes represented as [0,1]). For example, the coded aperture mask may be generated using a data-driven approach in which the coded aperture is modeled as a variable parameter and determined by end-to-end input / output optimization including a reconstruction model. Alternatively, the coded aperture mask may be generated using a theoretical approach based on the theory of compressed sensing, which utilizes incoherence with respect to the basis during sparse image transformation.

[0022] The dispersing unit 122 is, for example, a prism, and receives the output signal of the encoding unit 121, disperses the input signal according to wavelength, and outputs the result.

[0023] The measurement unit 123 is, for example, an FPA (Focal Plane Array) array sensor. The measurement unit 123 receives the output signal of the dispersion unit 122 from each sensor for each pixel, integrates the signal in the wavelength direction, and outputs the result. The measurement unit 123 outputs a compressed signal (image) as the measurement result.

[0024] The modulator 130 receives the output signal of the CASSI observation system 120, modulates it with a multiplication mask, and outputs the modulated signal. For example, the multiplication mask can be generated by a data-driven approach in which it is modeled as a variable parameter, similar to the coded aperture mask described above, and determined by end-to-end input / output optimization including the coded aperture mask and the reconstruction model.

[0025] The reconstruction processing unit 140 receives the output signal of the modulation unit 130, performs reconstruction processing on it, and outputs an estimation result (estimated signal) of the hyperspectral image.

[0026] Fig. 2 is an image diagram showing the flow of processing by the imaging device 1A. Fig. 2 shows how the signal conversion at each step is parameterized (modeled). In the signal conversion in Fig. 2, the solid lines represent conversion based on the intrinsic parameter values ​​(fixed) of optical elements (prisms, sensors, etc.), and the dashed lines represent conversion based on designable variable parameters.

[0027] First, the imaging device 1A inputs wavelength data (Spectral Data Cube) D0 of light in the imaging target space to the CASSI observation system 120 via the optical system 110 (step S1).

[0028] Next, in the CASSI observation system 120, the encoding unit 121 applies a coded aperture mask M1 to the input wavelength data (input signal) and outputs coded wavelength data D1 to the dispersing unit 122 (step S2). This process means that each pixel of the 3D (spatial and wavelength) input signal is covered with a coded aperture mask.

[0029] Next, in the CASSI observation system 120, the dispersion unit 122 wavelength-disperses the encoded wavelength data D1 and outputs it to the measurement unit 123 (step S3). This step means that the wavelength axis is tilted by a dispersion optical element (prism).

[0030] Next, in the CASSI observation system 120, the measurement unit 123 inputs the wavelength-dispersed wavelength data D2, and each pixel accumulates the wavelength data D2 in the wavelength direction to generate compressed signal data D3 as the measurement result of the target space, which is output to the modulation unit 130 (step S4). This process means accumulating each tilted wavelength signal for each pixel. Through the process up to this point, a 2D compressed signal is acquired.

[0031] Next, the modulation unit 130 generates modulated compressed signal data D4 by modulating the 2D compressed signal data D3 input from the CASSI observation system 120 using a multiplication mask (step S5). The modulation unit 130 outputs the generated modulated compressed signal data D4 to the reconstruction processing unit 140.

[0032] Next, the reconstruction processing unit 140 performs reconstruction processing on the modulated compressed signal data D4 input from the modulation unit 130, thereby generating and outputting a hyperspectral image IMG as an estimation result (step S6). For example, the reconstruction processing unit 140 performs reconstruction processing by inputting the modulated compressed signal data D4 to a reconstruction model.

[0033] Here, the reconstruction model is generated by, for example, learning in advance the correlation between desired signal data and modulated compressed signal data (i.e., the observation matrix) using a machine learning method such as a neural network. For example, the reconstruction model can be constructed using a deep unrolled network (DUN), which implements an iterative optimization algorithm, such as the alternating direction method of multiplier (ADMM) or the iterative shrinkage thresholding algorithm (ISTA), using deep learning, and can be learned by end-to-end optimization of variable parameters including those of the encoding unit 121 and the modulation unit 130.

[0034] Fig. 3 is a diagram showing an example of the effect achieved by the imaging device 1A of the first embodiment. Fig. 3 shows a comparison between the image estimation accuracy of a conventional configuration that does not use a multiplication mask (only a coded aperture mask) and the image estimation accuracy of a configuration of this embodiment that uses the above-mentioned multiplication mask (a combination of a coded aperture mask and a multiplication mask). The vertical axis of the graph shown in Fig. 3 represents the intensity of PSNR (Peak Signal-to-Noise Ratio). As is clear from Fig. 3, the estimation result of this embodiment has a larger PSNR value than the conventional configuration, indicating improved image quality.

[0035] According to the imaging device 1A of the first embodiment configured in this manner, the accuracy of measuring hyperspectral images by CASSI can be improved by performing modulation processing using a multiplication mask on the measured signal in the CASSI observation system that measures hyperspectral images.

[0036] More specifically, because each pixel of the coded aperture mask used in the CASSI observation system blocks or blocks the corresponding pixel information of the input signal (attenuation in the case of [0,1]), in conventional configurations, signal conversion for each wavelength signal within the same pixel in the input signal was limited to a signal conversion common to the pixel (conversion of scalar multiplication of {0,1} or [0,1]). In contrast, the imaging device 1A of the first embodiment has, in addition to a coded aperture mask that acts directly on the input signal before dispersion processing, a modulation unit 130 that multiplies the measurement signal resulting from dispersion processing and integration processing by the mask, thereby enabling different signal conversion for each wavelength signal within the same pixel of the input signal.

[0037] This allows for the design of observation matrices with a high degree of freedom when measuring hyperspectral images, and by appropriately designing the two masks, the coded aperture mask and the multiplication mask, it is possible to improve the accuracy of the original signal that is ultimately obtained.

[0038] Furthermore, the design of the CASSI observation matrix uses various optical elements and is implemented in hardware. In contrast, the modulation unit 130 of this embodiment processes signals after signal compression and can be implemented in software. Therefore, there are fewer physical constraints on the implementation of the modulation unit 130, and advantages include ease of handling in terms of conversion speed and representation of continuous values.

[0039] Furthermore, in the imaging device 1A of the first embodiment, the modulation unit 130 does not add or change anything to the conventional imaging process using CASSI. Therefore, according to the imaging device 1A of the present embodiment, it is possible to easily improve the estimation accuracy of the original signal without complicating the device.

[0040] Second Embodiment Fig. 4 is a block diagram showing the configuration of an image pickup device 1B according to the second embodiment. In Fig. 4, the same components as those in Fig. 1 are assigned the same reference numerals as in Fig. 1, and descriptions thereof will be omitted here. The image pickup device 1B differs from the image pickup device 1A according to the first embodiment in that it does not include a modulation section 130, but includes a focus weight setting section 150 and a mask generation section 160.

[0041] Attention weight setting unit 150 sets a weight (hereinafter referred to as "attention weight") indicating the degree to which observation data should be focused on for each pixel captured by CASSI observation system 120. For example, attention weight setting unit 150 receives an input for setting an attention weight from an operation input unit (not shown), and generates attention weight information indicating an attention weight for each pixel based on the input information and outputs the generated weight information to mask generation unit 160.

[0042] More specifically, the weight of interest is a value within the range of continuous values ​​[0, 1], and the weight of interest setting unit 150 sets a 3D (spatial and wavelength) weight of interest for each pixel.

[0043] The attention weight may be set by any method other than the above-described method. For example, attention weight information may be received from another device via communication, or attention weight information stored in advance in a storage unit (not shown) may be read out.

[0044] The mask generation unit 160 generates a coded aperture mask based on the attention weight information input from the attention weight setting unit 150. The mask generation unit 160 sets the generated coded aperture mask in the coding unit 121 of the CASSI observation system 120 at the subsequent stage.

[0045] 5 is a conceptual diagram showing the flow of processing by imaging device 1B. First, in imaging device 1B, attention weight setting section 150 performs processing to set an attention weight for each pixel (step S201), and outputs attention weight information indicating the settings to mask generation section 160. For example, attention weight setting section 150 may accept an operation to select an attention area in an image with a rectangle, or may accept an operation to input an annotation map that associates each area in an image with information as to whether it is an attention area or not.

[0046] Next, the mask generation unit 160 receives the attention weight information from the attention weight setting unit 150 and generates a coded aperture mask based on the attention weight information (step S202). For example, the mask generation unit 160 can generate a coded aperture mask using an image processing model based on a DNN (Deep Neural Network) such as a CNN (Convolutional Neural Network) or U-NET. The mask generation unit 160 sets the generated coded aperture mask in the coding unit 121 of the downstream CASSI observation system 120.

[0047] The subsequent processing is the same as the processing flow when the multiplication mask is not applied in the first embodiment. Specifically, the CASSI observation system 120 acquires (images) observation data using the coded aperture mask set in step S202, and the reconstruction processing unit 140 performs reconstruction processing on the acquired observation data to output an image that is an estimation result of the original signal. In Fig. 5, the same processes as those executed by the imaging device 1A of the first embodiment are denoted by the same reference numerals as in Fig. 2.

[0048] 5, the case where the mask generation unit 160 generates only the coded aperture mask by setting the attention weight has been described, but the mask generation unit 160 may generate the multiplication mask of the first embodiment in addition to the coded aperture mask. In this case, the function of the mask generation unit 160 to generate the multiplication mask can be realized by machine learning using a deep learning model.

[0049] 6 is a conceptual diagram showing the processing flow in this case. In this case, the imaging device 1B may further include the modulation unit 130 of the first embodiment, and may be configured to modulate the observation data of the CASSI observation system 120 and then perform reconstruction processing.

[0050] FIG. 7 is a conceptual diagram illustrating the process by which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model. As described above, the function of the mask generation unit 160 to generate a multiplication mask and the reconstruction processing unit 140 can be realized by a deep learning model. For example, the mask generation unit 160 and the reconstruction processing unit 140 can construct a multiplication mask generation model and a reconstruction model specialized for reconstructing the region of interest by learning the MSE (Mean Squared Error) weighted by the attention weight set on a pixel-by-pixel basis as a loss. In other words, by expressing the attention weight as a real value [0, 1] and designing a loss using this value, it becomes possible to measure hyperspectral images taking into account the magnitude of importance. For example, the loss is defined as shown in the following equation (2):

[0051]

number

[0052] As described above, the image capture device 1B of the second embodiment can configure a reconstruction model specialized for a region of interest by setting an attention weight indicating the weight of the degree of attention for each pixel in an image captured by a CASSI observation system that measures a hyperspectral image and generating a coded aperture mask based on the set attention weight information. Therefore, the image capture device 1B of the second embodiment can improve the measurement accuracy of a hyperspectral image by CASSI.

[0053] Furthermore, the imaging device 1B of the second embodiment includes the modulation unit 130 of the first embodiment, and the mask generation unit 160 generates a multiplication mask based on a generative model (DNN) trained to perform end-to-end optimization, including a coded aperture mask based on the attention weight information set by the attention weight setting unit 150, and the modulation unit 130 modulates the observation data using the generated multiplication mask, thereby further improving the measurement accuracy of hyperspectral images using CASSI.

[0054] More specifically, the imaging device 1B of the second embodiment weights (sets attention weights) the wavelength range of interest on a pixel-by-pixel basis for the coded aperture mask used in the CASSI observation system 120, and then generates the coded aperture mask using a DNN (Deep Neural Network) with the attention weights for each pixel and wavelength range as input. This means that by designing pixel-wise weights and learning a generation model and a reconstruction model for the coded aperture mask that take this into account, it becomes possible to acquire a measurement image in which the wavelength area of ​​interest is spatially varied.

[0055] As a result, the importance (weight) for each wavelength range is set as a value within a continuous range of [0,1] instead of a discrete value of {0,1}, and imaging can be performed by setting a different importance of the wavelength range for each pixel. According to the imaging device 1B of the second embodiment, unlike conventional methods, all pixel information of the acquired compressed signal can be used, so the reconstruction result from a single imaging can be expected to have sufficient performance, and measurement costs can be reduced.

[0056] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0057] The present invention is applicable to an imaging device that measures hyperspectral images using the CASSI observation system. [Explanation of symbols]

[0058] 1A, 1B... imaging device, 110... optical system, 120... CASSI observation system, 121... encoding unit, 122... dispersion unit, 123... measurement unit, 130... modulation unit, 140... reconstruction processing unit, 150... weight setting unit, 160... mask generation unit

Claims

1. An imaging device that measures hyperspectral images using compressed sensing, a CASSI observation system including: an encoding unit that applies a coded aperture mask to an input signal to encode the input signal and output the encoded signal; a dispersing unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measuring unit that captures the input signal wavelength-dispersed by the dispersing unit; an attention weight setting unit that sets an attention weight for each pixel captured by the CASSI observation system; a mask generation unit that generates the coded aperture mask based on the attention degree weight for each pixel set by the attention weight setting unit; An imaging device comprising:

2. a modulation unit that modulates the output signal of the CASSI observation system using a multiplication mask; The imaging device according to claim 1 .

3. the mask generation unit generates the multiplication mask in addition to the coded aperture mask based on the weight of the attention degree, and sets the generated multiplication mask in the modulation unit. The imaging device according to claim 2 .

4. the attention weight setting unit is capable of setting an arbitrary value between 0 and 1 as the weight of the attention degree for each pixel. The imaging device according to claim 1 .

5. a reconstruction processing unit that receives an output signal from the modulation unit, performs reconstruction processing on the output signal using a reconstruction model that is pre-constructed by a neural network, and outputs an estimation result of a hyperspectral image; The imaging device according to claim 2 .

6. the mask generation unit constructs the coded aperture mask and the multiplication mask such that end-to-end input / output of the mask generation unit, the CASSI observation system, the modulation unit, and the reconstruction processing unit is optimized by the reconstruction model. The imaging device according to claim 5 .

7. An imaging method for measuring a hyperspectral image by compressed sensing, comprising: an imaging step of imaging an input signal using a CASSI observation system including: an encoding unit that applies a coded aperture mask to the input signal to encode the input signal and output the encoded signal; a dispersing unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measuring unit that images the input signal wavelength-dispersed by the dispersing unit; a modulating step of modulating the output signal of the CASSI observation system by a multiplication mask; An imaging method comprising:

8. A program for causing a computer to function as the imaging device according to any one of claims 1 to 6.

Images