Signal processing device, and signal processing method

Abstract

This signal processing method, which is executed by a computer, includes: acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength region (S101); acquiring reference spectral data including one or more items of spectral information associated with the subject (S102); and generating, from the compressed image data, a plurality of items of two-dimensional image data corresponding to a plurality of designated wavelength bands determined on the basis of the reference spectral data (S105).

Description

Signal processing device and signal processing method

[0001] The present disclosure relates to a signal processing device and a signal processing method.

[0002] By utilizing spectral information from multiple narrow wavelength bands, e.g., ten or more bands, it becomes possible to grasp detailed physical properties of an object that were previously impossible to grasp using RGB images, which only have information from three bands. Cameras that capture images in such multiple wavelength bands include "hyperspectral cameras" and "multispectral cameras." These cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Literature 1 discloses a hyperspectral camera using a compressed sensing method. Compressed sensing is a technique for recovering more data than observed data by assuming that the data distribution of an observed object is sparse in a certain space (e.g., frequency space). The estimation operation assuming the sparsity of the observed object is called "sparse reconstruction." The hyperspectral camera disclosed in Patent Literature 1 acquires a monochrome image through an array of filters whose spectral transmittance has local maxima at multiple wavelengths. The imaging device recovers a hyperspectral image from the monochrome image using an operation based on sparse reconstruction.

[0004] Non-Patent Document 1 discloses an example of a snapshot-type hyperspectral imaging device suitable for observing the spectrum of fluorescence emitted from a phosphor.

[0005] U.S. Pat. No. 9,599,511

[0006] Amicia D. Elliott et al., "Real-time hyperspectral fluorescence imaging of pancreatic b-cell dynamics with the image mapping spectrometer", Journal of Cell Science 125, 4833-4840 (2012).

[0007] The imaging device of Patent Document 1 is capable of capturing high-resolution, multi-wavelength moving images. However, since a high-load restoration calculation is performed using matrix data of a size equal to the product of the number of pixels of the image sensor and the number of wavelength bands, a processing circuit with high computing power is required.

[0008] The present disclosure provides techniques for reducing the load of reconstruction calculations by efficiently generating images of required wavelength bands.

[0009] A method according to one aspect of the present disclosure is a computer-implemented signal processing method that includes: acquiring compressed image data including two-dimensional image information of an object obtained by compressing hyperspectral information in a wavelength range of interest; acquiring reference spectral data including information on one or more spectra associated with the object; and generating, from the compressed image data, a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands determined based on the reference spectral data.

[0010] Another aspect of the present disclosure provides a method for generating mask data used to restore spectral image data for each wavelength band from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range, the method including: acquiring first mask data for restoring first spectral image data corresponding to a first set of wavelength bands in the target wavelength range; acquiring reference spectral data including information about at least one spectrum; determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectral data; and generating second mask data for restoring second spectral image data corresponding to a second set of wavelength bands in the one or more designated wavelength ranges based on the first mask data.

[0011] According to yet another aspect of the present disclosure, there is provided a computer-implemented signal processing method, the method including: acquiring compressed image data including two-dimensional image information of an object obtained by compressing hyperspectral information in a target wavelength range; acquiring reference spectral data including information on one or more spectra associated with the object; and displaying, on a display, a graphical user interface for allowing a user to specify restoration conditions for generating, from the compressed image data, a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands, and an image based on the reference spectral data.

[0012] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices.

[0013] According to one aspect of the present disclosure, the load of reconstruction calculations can be reduced by efficiently generating images of required wavelength bands.

[0014] FIG. 1A is a diagram schematically illustrating an example configuration of an imaging system. FIG. 1B is a diagram schematically illustrating another example configuration of an imaging system. FIG. 1C is a diagram schematically illustrating yet another example configuration of an imaging system. FIG. 1D is a diagram schematically illustrating yet another example configuration of an imaging system. FIG. 2A is a diagram schematically illustrating an example of a filter array. FIG. 2B is a diagram illustrating an example of the spatial distribution of light transmittance of each of multiple wavelength bands included in a target wavelength range. FIG. 2C is a diagram illustrating an example of the spectral transmittance of a region A1 included in the filter array shown in FIG. 2A. FIG. 2D is a diagram illustrating an example of the spectral transmittance of a region A2 included in the filter array shown in FIG. 2A. FIG. 3A is a diagram for explaining an example relationship between a target wavelength range and multiple wavelength bands included therein. FIG. 3B is a diagram for explaining another example relationship between a target wavelength range and multiple wavelength bands included therein. FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of a filter array. FIG. 4B is a diagram illustrating the result of averaging the spectral transmittance shown in FIG. 4A for each wavelength band. FIG. 5 is a block diagram showing an example of the configuration of a system for reducing the load of calculation processing. FIG. 6 is a diagram showing a modified example of the system of FIG. 5. FIG. 7 is a diagram showing an example of mask data before conversion stored in a memory. FIG. 8A is a diagram showing an example of spectra of four types of samples that may be included in an object. FIG. 8B is a diagram showing an example of bands to be synthesized. FIG. 9 is a flowchart showing an example of a mask data conversion process. FIG. 10 is a diagram for explaining an example of a method for synthesizing mask information of multiple bands and converting it into new mask information. FIG. 11A is a diagram showing an example of reference spectra of four types of samples that may be included in an object. FIG. 11B is a diagram showing an example of band synthesis. FIG. 12 is a block diagram showing an example of the configuration of a system that performs labeling. FIG. 13A is a diagram showing an example of reference spectra of four types of samples that may be included in an object. FIG. 13B is a diagram showing an example of band synthesis. FIG. 14 is a diagram showing an example of a graphical user interface (GUI). FIG. 15A is a first diagram for explaining a method for excluding a specific band from a restoration target. FIG. 15B is a second diagram for explaining a method for excluding a specific band from a restoration target.FIG. 16 is a flowchart showing an example of the operation of the signal processing circuit when a specific band is excluded from restoration. FIG. 17 is a flowchart showing an example of the operation when reference spectral data is translated into mask data conversion conditions and sent to the signal processing circuit. FIG. 18 is a diagram schematically showing the configuration of a system according to a fourth embodiment. FIG. 19 is a diagram showing an example of the relationship between excitation light and fluorescence spectra and excluded bands. FIG. 20A is a diagram showing absorption spectra of five types of fluorescent dyes. FIG. 20B is a diagram showing fluorescence spectra of five types of fluorescent dyes. FIG. 21A is a diagram showing the relationship between the excitation wavelength and the absorption spectra of the fluorescent dyes in the first imaging. FIG. 21B is a diagram showing an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and the restoration band. FIG. 22A is a diagram showing the relationship between the excitation wavelength and the absorption spectra of the fluorescent dyes in the second imaging. FIG. 22B is a diagram showing an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and the restoration band. FIG. 23A is a diagram showing the relationship between the excitation wavelength and the absorption spectra of the fluorescent dyes in the third imaging. FIG. 23B is a diagram showing the relationship between each fluorescence spectrum and the cutoff wavelength range and the restoration band.

[0015] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangements, positions and connection forms, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0016] In the present disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the connection relationships within the LSI or set up circuit sections within the LSI, can also be used for the same purpose.

[0017] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0018] (Findings Forming the Basis of the Present Disclosure) Before describing embodiments of the present disclosure, an overview of image restoration processing based on sparsity and synthesis and editing processing of mask data used during restoration will be described.

[0019] Sparsity is the property that elements that characterize an observation target exist sparsely in a certain space (for example, frequency space). Sparsity is widely observed in nature. Utilizing sparsity makes it possible to efficiently observe necessary information. Sensing technology that utilizes sparsity is called compressed sensing. Utilizing compressed sensing makes it possible to build highly efficient devices and systems. An example of the application of compressed sensing to a hyperspectral camera is disclosed in Patent Document 1. The hyperspectral camera disclosed in Patent Document 1 is capable of capturing high wavelength resolution, high resolution, and multi-wavelength video in a single shot.

[0020] An imaging device using compressed sensing includes, for example, an array of optical filters having random light transmission characteristics with respect to space and / or wavelength. Such an array of optical filters may be referred to as a "coding mask" or "coding element." The coding mask is placed on the optical path of light incident on the image sensor and transmits light from a subject with different light transmission characteristics depending on the region. This process using the coding mask is referred to as "encoding." An image of the light encoded by the coding mask is captured by the image sensor. An image generated by imaging using the coding mask is referred to as a "compressed image" in this specification. Mask data indicating the light transmission characteristics of the coding mask is recorded in advance in a storage device. A processing circuit in the imaging device performs a restoration process based on the compressed image and the mask data. The restoration process generates a restored image that contains information about more wavelengths than the compressed image. The mask data is, for example, information indicating the spatial distribution of the spectral transmittance of the coding mask. The restoration process based on such mask data allows images of multiple wavelength bands to be reconstructed from a single compressed image.

[0021] The restoration process includes an estimation operation that assumes the sparsity of the captured image. The operation performed in the sparse reconstruction may be, for example, a data estimation operation by minimizing an evaluation function incorporating a regularization term, such as a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, or a total variation (TV), as disclosed in Patent Document 1. Such an estimation operation is a high-load operation using mask data whose size corresponds to the product of the number of pixels and the number of wavelength bands of the image sensor. Therefore, a processing circuit with high computing power is required. If the time required for such a high-load operation is longer than the exposure time during image capture, the calculation time will limit the operating speed (e.g., frame rate) of the camera.

[0022] In many applications of hyperspectral cameras, such as fluorescence observation and absorption spectrum observation, the expected spectrum of a subject is known (see, for example, Non-Patent Document 1). When the expected spectrum of a subject is known, the amount of calculation can be reduced by appropriately editing or reducing the mask data used in the restoration process. For example, the mask data can be edited or reduced based on information about the wavelength range required for processing such as analysis or classification performed after image restoration.

[0023] Based on the above findings, the present disclosure discloses a method for reducing the load of computational processing by referring to information on a known spectrum when the spectrum that a subject may have is known. In the following description, a known spectrum assumed for each substance contained in the subject, i.e., the observation target, is referred to as a "reference spectrum." Data indicating the reference spectra of one or more substances that the observation target may have is collectively referred to as "reference spectral data."

[0024] An outline of an embodiment of the present disclosure will be described below.

[0025] A signal processing method according to an exemplary embodiment of the present disclosure includes acquiring compressed image data including two-dimensional image information of an object obtained by compressing hyperspectral information in a target wavelength range, acquiring reference spectral data including information of one or more spectra associated with the object, and generating, from the compressed image data, a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands determined based on the reference spectral data.

[0026] "Hyperspectral information in a target wavelength range" refers to information about the spatial distribution of luminance for each of multiple wavelength bands included in the target wavelength range. "Compressing hyperspectral information" refers to compressing information about the spatial distribution of luminance for multiple wavelength bands into a single monochrome two-dimensional image using an encoding element such as a filter array, which will be described later.

[0027] According to the above method, multiple pieces of two-dimensional image data corresponding to multiple designated wavelength bands determined based on the reference spectral data are restored from the compressed image data, which reduces the computational load compared to restoring two-dimensional image data corresponding to all wavelength bands included in the target wavelength range.

[0028] The one or more spectra may be associated with one or more substances assumed to be contained in the subject. For example, data defining the correspondence between substances and spectra may be pre-recorded in a storage medium such as a memory. By referencing such data, it is possible to easily obtain information about a spectrum corresponding to, for example, a substance specified by a user.

[0029] Each of the plurality of designated wavelength bands may include a peak wavelength in the spectrum associated with a corresponding one of the one or more materials, such that each designated wavelength band corresponds to one material, facilitating classification based on the recovered image.

[0030] The reference spectral data may include information on a plurality of spectra associated with a plurality of types of materials assumed to be contained in the subject. The plurality of designated wavelength bands may include a first designated wavelength band having no overlapping spectra and a second designated wavelength band having overlapping spectra. This facilitates classification based on the restored image even when the spectra of two or more materials overlap.

[0031] As used herein, a specified wavelength band "has no overlap among multiple spectra" means that one of the multiple spectra has significant intensity in the specified wavelength band, and the other spectra do not. Conversely, a specified wavelength band "has overlap among multiple spectra" means that two or more spectra have significant intensity in the specified wavelength band. Whether a spectrum has "significant intensity" can be determined, for example, based on the integral value obtained by integrating the intensity of each spectrum from the lower limit wavelength to the upper limit wavelength of the specified wavelength band. The maximum integral value among the integral values ​​of the multiple spectra is defined as signal S, and the sum of the remaining integral values ​​is defined as noise N. If the S / N ratio, which is the value obtained by dividing signal S by noise N, is equal to or greater than a threshold value (e.g., 1, 2, or 3), the specified wavelength band can be determined to have no overlap among the multiple spectra. Conversely, if the S / N ratio is less than the threshold value, the specified wavelength band can be determined to have overlap among the multiple spectra.

[0032] The compressed image data may be generated using an image sensor and a filter array including a plurality of optical filters having different spectral transmittances. The method may further include acquiring mask data reflecting a spatial distribution of the spectral transmittances. The plurality of two-dimensional image data may be generated based on the compressed image data and the mask data.

[0033] The mask data may include mask matrix information having elements corresponding to the spatial distribution of the transmittance of the filter array for each of a plurality of component bands included in the target wavelength range. The method may further include generating composite mask information by combining the mask matrix information corresponding to non-designated wavelength bands in the target wavelength range that are different from the designated wavelength band, and generating composite image data for the non-designated wavelength bands based on the compressed image data and the composite mask information. This reduces the amount of data for the non-designated wavelength bands by combining the mask matrix information, thereby further reducing the load on the calculation process.

[0034] Generating the plurality of two-dimensional image data may include generating and outputting the plurality of two-dimensional image data corresponding to the plurality of designated wavelength bands without generating, from the compressed image data, image data corresponding to non-designated wavelength bands in the target wavelength range that are different from the designated wavelength band. In other words, the method does not need to include generating, from the compressed image data, image data corresponding to non-designated wavelength bands in the target wavelength range that are different from the designated wavelength band. This prevents restoration processing from being performed on non-designated wavelength bands that are less important, thereby further reducing the computational load.

[0035] The plurality of designated wavelength bands may be determined based on the intensities or derivatives of the intensities of the one or more spectra indicated by the reference spectral data. For example, each designated wavelength band may be determined to include a peak wavelength at which the intensity of the corresponding spectrum peaks. Alternatively, each designated wavelength band may be determined to avoid wavelength ranges where the absolute value of the derivative of the intensity of the corresponding spectrum is close to zero. This method allows for the determination of designated wavelength bands that include characteristic portions of the spectrum, facilitating processing such as classification after restoration.

[0036] The reference spectral data may include information on the fluorescence spectra of one or more substances assumed to be contained in the subject, thereby enabling restoration processing with an appropriate band configuration based on the fluorescence spectra of the fluorescent substances.

[0037] The reference spectral data may include information on the absorption spectra of one or more substances assumed to be contained in the subject, thereby enabling restoration processing with an appropriate band configuration based on the absorption spectra of the substances.

[0038] The method may further include displaying a graphical user interface (GUI) on a display that allows a user to specify the one or more spectra or one or more substances associated with the one or more spectra. The display may be connected to or installed in the computer. The reference spectral data may be acquired according to the specified one or more spectra or the specified one or more substances. This allows the restoration process to be performed with a band configuration according to the spectra or substances specified by the user on the GUI.

[0039] Another embodiment of the present disclosure provides a method for generating mask data used to restore spectral image data for each wavelength band from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range, the method including: acquiring first mask data for restoring first spectral image data corresponding to a first set of wavelength bands in the target wavelength range; acquiring reference spectral data including information about at least one spectrum; determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectral data; and generating second mask data for restoring second spectral image data corresponding to a second set of wavelength bands in the one or more designated wavelength ranges based on the first mask data.

[0040] The first wavelength band group may be a collection of all or some of the wavelength bands included in the target wavelength range. For example, the first wavelength band group may be a collection of multiple unit wavelength bands having a very small bandwidth included in the target wavelength range. The second wavelength band group may be a collection of all or some of the wavelength bands included in a specified wavelength range. For example, the second wavelength band group may be a collection of multiple unit wavelength bands included in the specified wavelength range. Each of the first wavelength band group and the second wavelength band group may be a collection of synthesized bands formed by synthesizing two or more unit wavelength bands. When such band synthesis is performed, mask data conversion processing is performed according to the band synthesis mode.

[0041] According to the above configuration, it is possible to generate second mask data for restoring second spectral image data corresponding to the second wavelength band group in the specified wavelength range determined based on the reference spectral data. Because the second mask data has a smaller data size than the first mask data, efficient restoration is possible using the second mask data.

[0042] The above-mentioned method for generating second mask data may be performed by an apparatus that restores spectral image data for each wavelength band from compressed image data based on the second mask data, or may be performed by another apparatus connected to the apparatus.

[0043] The compressed image data may be generated using a filter array including a plurality of types of optical filters having different spectral transmittances and an image sensor. The first mask data and the second mask data may be data reflecting the spatial distribution of the spectral transmittance of the filter array. The first mask data may include first mask information indicating the spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second mask data may include second mask information indicating the spatial distribution of the spectral transmittance corresponding to the second wavelength band group.

[0044] The second mask data may further include third mask information obtained by combining a plurality of pieces of information, each of which indicates the spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength range other than the designated wavelength range within the target wavelength range.

[0045] The second mask data may not include information regarding the spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength range other than the designated wavelength range.

[0046] A signal processing method according to yet another embodiment of the present disclosure includes acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range, acquiring reference spectral data including information on one or more spectra associated with the subject, and displaying on a display both a graphical user interface for specifying restoration conditions for generating multiple two-dimensional image data corresponding to multiple specified wavelength bands from the compressed image data and an image based on the reference spectral data.

[0047] According to the above configuration, the user can specify restoration conditions for generating multiple sets of two-dimensional image data corresponding to multiple designated wavelength bands, thereby efficiently generating multiple sets of two-dimensional image data corresponding to desired designated wavelength bands.

[0048] The present disclosure also includes a computer program for causing a computer to execute the above-described methods. The present disclosure also includes a signal processing device including a processor that executes the above-described methods and a memory that stores the computer program executed by the processor.

[0049] The embodiments of the present disclosure will be described in more detail below. The embodiments described below are merely examples, and various modifications and alterations are possible in each embodiment.

[0050] First Embodiment 1. Imaging System First, a configuration example of an imaging system used in an exemplary embodiment of the present disclosure will be described.

[0051] FIG. 1A is a diagram schematically illustrating an example configuration of an imaging system. This system includes an imaging device 100 and a processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject. In the example of FIG. 1A, the filter array 110 is disposed between the optical system 140 and the image sensor 160.

[0052] FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple and may be any object. The image sensor 160 generates data for a compressed image 10 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The processing device 200 can generate image data for each of a plurality of wavelength bands included in a predetermined target wavelength range based on the data for the compressed image 10 generated by the image sensor 160. The plurality of image data that correspond one-to-one to the plurality of wavelength bands generated may be referred to as a "hyperspectral (HS) data cube" or "hyperspectral image data." Here, the number of wavelength bands included in the target wavelength range is set to N (N is an integer equal to or greater than 4). In the following description, the plurality of image data that correspond one-to-one to the plurality of wavelength bands generated will be referred to as a restored image 20W. １ , restored image 20W ２ , ..., restored image 20W Ｎ These are sometimes collectively referred to as a "hyperspectral image 20." The restored image data for each wavelength band may also be referred to as "spectral image data" or simply as a "spectral image." In this specification, the image data or signals, i.e., a collection of data or signals representing multiple pixel values ​​of multiple pixels included in an image, may also be simply referred to as an "image."

[0053] In this disclosure, the term "target wavelength range" refers to a wavelength range determined by the upper and lower wavelength limits of wavelength components contained in a spectral image output by a system. The target wavelength range may correspond to the wavelength range of light detectable by a photodetector, such as an image sensor, in the system. For example, in a system that captures images through a bandpass filter that suppresses transmission of light other than 400-700 nm, the target wavelength range may be 400-700 nm. In a system that captures images through an additional filter that absorbs light in the 500-600 nm range, the target wavelength range may be the combined wavelength range of 400-500 nm and 600-700 nm that the photodetector can detect.

[0054] In this embodiment, the filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependencies of light transmittance. The filter array 110 functions as the coding mask described above, modulating the intensity of incident light for each wavelength and outputting the modulated light.

[0055] 1A, the filter array 110 is disposed near or directly above the image sensor 160. Here, "near" means close enough that a light image from the optical system 140 is formed on the surface of the filter array 110 with a certain degree of clarity. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0056] 1A, optical system 140 includes at least one lens. Although optical system 140 is shown as a single lens in FIG. 1A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 110.

[0057] The filter array 110 may be disposed away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is disposed away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position away from the image sensor 160. In the example of FIG. 1C, the filter array 110 is disposed between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, with the filter array 110 disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.

[0058] The image sensor 160 is a monochrome photodetection device having a plurality of photodetection elements (also referred to herein as "pixels") arranged two-dimensionally. The image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor. The photodetection elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome sensor. For example, a color sensor may be used. Color sensors include sensors having filters that transmit red light, green light, and blue light; sensors having filters that transmit red light, green light, blue light, and infrared light; or sensors having filters that transmit red light, green light, blue light, and white light. The use of a color type sensor can increase the amount of information about wavelengths, thereby improving the accuracy of reconstructing the hyperspectral image 20. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, but may also be the ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength range.

[0059] The processing device 200 may be a computer including one or more processors and one or more storage media such as a memory. The processing device 200 generates a decompressed image 20W based on the compressed image 10 acquired by the image sensor 160. １ , restored image 20W ２ , ...Restored image 20W Ｎ Generate data.

[0060] 2A is a diagram schematically illustrating an example of a filter array 110. The filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, each of the plurality of regions may be referred to as a "cell." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0061] 2A , the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0062] FIG. 2B shows the wavelength band W included in the target wavelength range. １ , wavelength band W ２ , ..., wavelength band W Ｎ 2B is a diagram showing an example of the spatial distribution of light transmittance of each of the wavelength bands. In the example shown in FIG. 2B, the difference in shading of each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0063] 2C and 2D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 110 shown in FIG. 2A . The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary for all regions to have different spectral transmittances. In the filter array 110, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In some examples, the number of spectral transmittance patterns of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0064] 3A and 3B show the target wavelength range W and the wavelength bands W included therein. １ , wavelength band W ２ , ..., wavelength band W Ｎ 1 is a diagram for explaining the relationship between the wavelengths of visible light and ultraviolet light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. In this manner, the wavelength range used is not limited to the visible light range. In this specification, the term "light" refers to radiation in general, including not only visible light but also infrared and ultraviolet light.

[0065] In the example shown in FIG. 3A , N is an arbitrary integer equal to or greater than 4, and each wavelength range obtained by dividing the target wavelength range W into N equal parts is called a wavelength band W １ , wavelength band W ２ , ..., wavelength band W ＮHowever, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the method of determining the multiple wavelength bands is arbitrary.

[0066] 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has multiple maximum values ​​(i.e., maximum values ​​P1 to P5) and multiple minimum values ​​for wavelengths within the target wavelength band W. In the example shown in FIG. 4A, the optical transmittance within the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 4A, the wavelength band W ２ , and the wavelength band W Ｎ－１ In this way, the spectral transmittance of each region is expressed as a wavelength band W １ From wavelength band W Ｎ In the example of Fig. 4A, the maximum value P1, the maximum value P3, the maximum value P4, and the maximum value P5 are each 0.5 or more.

[0067] As such, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 110 transmits a large amount of components in a certain wavelength range among the incident light, while not transmitting components in other wavelength ranges as much. For example, the transmittance of light in k wavelength bands among the N wavelength bands may be greater than 0.5, while the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light is white light that contains all visible light wavelength components equally, the filter array 110 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light in a superimposed form.

[0068] FIG. 4B shows an example of the spectral transmittance shown in FIG. 4A in a wavelength band W １ , wavelength band W２ , ..., wavelength band W Ｎ This figure shows the results of averaging the spectral transmittance for each wavelength band. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the transmittance value averaged for each wavelength band in this way is referred to as the transmittance for that wavelength band. In this example, the transmittance is remarkably high in the three wavelength ranges that have maximum values ​​P1, P3, and P5. In particular, the transmittance exceeds 0.8 in the two wavelength ranges that have maximum values ​​P3 and P5.

[0069] In the examples shown in Figures 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary scale transmittance distribution may be employed in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two wavelength ranges out of multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.

[0070] A portion of all cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength bands W included in the wavelength range W of interest. １ From W Ｎ The filter array 110 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0071] Such data indicating the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the processing device 200. This data is used in the calculation process described below.

[0072] The filter array 110 can be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing a metal. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is formed differently for each cell. This allows different spectral characteristics to be achieved for each cell. By using a multilayer film, sharp rises and falls in the spectral transmittance can be achieved. A configuration using an organic material can be achieved by containing different pigments or dyes in each cell or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each cell. When a microstructure containing a metal is used, it can be fabricated using spectral separation due to the plasmon effect.

[0073] Next, an example of signal processing by the processing device 200 will be described. The processing device 200 reconstructs a hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. The generated hyperspectral image 20 includes multiple images. The multiple images correspond to multiple wavelength bands, and the number of the multiple wavelength bands is greater than, for example, three, which is the number of wavelength bands acquired by a typical color camera (e.g., red light wavelength band, green light wavelength band, and blue light wavelength band). The number of wavelength bands may be, for example, between four and 100. The number of wavelength bands is referred to as the "number of bands." Depending on the application, the number of bands may exceed 100.

[0074] The data to be obtained is the data of the hyperspectral image 20, and the data is denoted as f. If the number of bands is N, f is the wavelength band W １ Image data f corresponding to １ , wavelength band W ２ Image data f corresponding to ２ , ..., wavelength band W Ｎ Image data f corresponding to ＮHere, as shown in FIGS. 1A to 1D, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is m and the number of pixels in the y direction is n, then the image data f １ , image data f ２ , ..., image data f Ｎ Each of the elements f is two-dimensional data with n×m pixels. Therefore, data f is three-dimensional data with n×m×N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube." Meanwhile, data g of compressed image 10 obtained by encoding and multiplexing using filter array 110 has n×m elements. Data g can be expressed by the following equation (1):

[0075]

[0076] where f １ , f ２ , ..., f Ｎ Each of the elements in the matrix H is data having n×m elements. Therefore, the vector on the right side is a one-dimensional vector with n×m×N rows and one column. The compressed image 10 is converted into a one-dimensional vector g with n×m rows and one column, and is then calculated. The matrix H is a matrix of each component f of the vector f. １ , f ２ , ..., f Ｎ represents a transformation in which different encoding information (also referred to as "mask information") is used for each wavelength band to encode and intensity-modulate the signal, and then these information is added together. Therefore, H is a matrix with n×m rows and n×m×N columns. The mask information may be interpreted as the matrix H in equation (1).

[0077] Given the vector g and the matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the processing device 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

[0078]

[0079] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means that f that minimizes the sum of the first and second terms is found. The function in the parentheses in Equation (2) is called the evaluation function. The processing device 200 can converge the solution through recursive iterative calculations and calculate the f that minimizes the evaluation function as the final solution f'.

[0080] The first term in the parentheses in Equation (2) represents an operation to calculate the sum of squares of the difference between the acquired data g and Hf, which is obtained by transforming f in the estimation process using the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0081] Included in formula (1) and formula (2)

[0082]

[0083] is sometimes denoted as g in descriptions relating to formulas (1) and (2).

[0084] In the configurations of FIGS. 1B and 1C , the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned matrix H, the hyperspectral image 20 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to one pixel on an image spreads due to blurring into a k×k pixel region around that pixel, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the influence on the pixel values ​​of each pixel within that region. The hyperspectral image 20 can be reconstructed by reflecting the influence of blurring of the encoding pattern by the PSF in the matrix H. The filter array 110 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 110 to be lost due to excessive diffusion can be selected.

[0085] <2. Band Synthesis Processing Based on Reference Spectrum> Through the above processing, it is possible to restore images of each of a plurality of wavelength bands from the compressed image 10 acquired by the image sensor 160. However, in order to restore images of all wavelength bands included in the target wavelength range, it is necessary to perform an operation using a matrix including a number of elements corresponding to the product of the number of pixels of the image sensor 160 and the number of wavelength bands. This operation imposes a heavy load, and the processing device 200 is required to have high computing power.

[0086] On the other hand, in some cases, such as in fluorescence observation and absorption spectrum observation, the expected emission spectrum or absorption spectrum of the substance being observed is known. In such cases, the amount of calculation can be reduced by editing or reducing the mask data based on the expected known spectrum.

[0087] An example of a method for reducing the load of computational processing by using reference spectral data showing a known spectrum will be described below. The example of the reference spectrum shown below is merely a representative example, and various modifications or variations are possible. In the following description, as an example, a system will be described for capturing an image of a subject containing one or more specific substances (e.g., fluorescent substances) and analyzing or classifying the substances based on the captured image.

[0088] 5 is a block diagram showing an example configuration of a system for reducing the load of calculation processing by using reference spectral data. The system includes an image capture device 100, a processing device 200, a display device 300, and an input user interface (UI) 400. The processing device 200 is an example of a signal processing device defined in the present disclosure.

[0089] The imaging device 100 includes an image sensor 160 and a control circuit 150 that controls the image sensor 160. As shown in FIGS. 1A to 1D , the imaging device 100 also includes a filter array 110 and at least one optical system 140. The image sensor 160 acquires a compressed image, which is a monochrome image based on light whose intensity has been modulated for each region by the filter array 110. Data for each pixel of the compressed image has information for multiple wavelength bands included in the target wavelength range superimposed thereon. Therefore, this compressed image can be said to be hyperspectral information within the target wavelength range compressed as two-dimensional image information. In this specification, data representing the compressed image is referred to as "compressed image data."

[0090] The processing device 200 includes a signal processing circuit 250 and a memory 210 such as a RAM and a ROM. The signal processing circuit 250 may be an integrated circuit including a processor such as a CPU or a GPU. The signal processing circuit 250 performs restoration processing based on compressed image data output from the image sensor 160. The memory 210 stores computer programs executed by a processor included in the signal processing circuit 250, various data referenced by the signal processing circuit 250, and various data generated by the signal processing circuit 250. The memory 210 stores mask data reflecting the spatial distribution of the spectral transmittance of the filter array 110 in the imaging device 100. The mask data is data including information representing the matrices in the above equations (1) and (2) or information for deriving the matrices (hereinafter, sometimes referred to as "mask matrix information"). The mask matrix information may be information in a matrix format or a format similar to a matrix having elements corresponding to the spatial distribution of the transmittance of the filter array 110 for each of multiple component bands included in the target wavelength range. The mask data is created in advance and stored in the memory 210 .

[0091] The display device 300 includes an image processing circuit 320 and a display 330. The image processing circuit 320 performs necessary processing on the image restored by the signal processing circuit 250 and then displays the image on the display 330. The display 330 may be any display, such as a liquid crystal display or an organic light-emitting diode (OLED).

[0092] The input UI 400 includes hardware and software for setting various conditions such as imaging conditions. The input UI 400 may include input devices such as a keyboard and a mouse. The input UI 400 may be realized by a device capable of both input and output, such as a touch screen. In this case, the touch screen may also function as the display 330. The imaging conditions may include, for example, resolution, gain, and exposure time. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 causes the image sensor 160 to capture an image in accordance with the imaging conditions.

[0093] The memory 410 stores spectral data. The spectral data includes information on the expected spectra of one or more types of substances that may be contained in the subject. Spectral data is prepared in advance for each substance and recorded in the memory 410. The memory 410 may be an external memory or may be built into the imaging device 100. The spectral data may be acquired by downloading it via a network such as the Internet.

[0094] The user can select specific spectral data as reference spectral data by operating the input UI 400. For example, when the user selects a specific material or substance on the input UI 400, the spectral data corresponding to the material or substance can be determined as reference spectral data. When the reference spectral data is determined by the user's operation, the reference spectral data is sent to the signal processing circuit 250.

[0095] The signal processing circuit 250 determines the synthesis conditions for the mask data based on the reference spectral data. The synthesis conditions for the mask data are conditions for determining multiple designated wavelength bands for which restoration processing is performed. In other words, the signal processing circuit 250 determines multiple designated wavelength bands for which restoration processing is performed based on the reference spectral data. The wavelength range formed by the designated wavelength bands is referred to as the "designated wavelength range." The signal processing circuit 250 may automatically determine the synthesis conditions based on the reference spectral data, or may determine the synthesis conditions according to conditions specified by the user using the input UI 400. The synthesis conditions specify which of multiple component bands included in the target wavelength range are to be synthesized and treated as a single band. Each of the multiple component bands is a narrow wavelength band included in the target wavelength range. For example, in the observation of a sample containing one or more substances, multiple component bands included in wavelength ranges of relatively low importance may be synthesized into a single band. Alternatively, multiple component bands included in wavelength ranges expected to best represent the characteristics of individual substances may be synthesized into a single band. The synthesized, relatively wide band may be referred to as the "synthesized band" in the following description. The image data corresponding to the composite band may be referred to as composite image data. The composite conditions may include information on wavelength ranges for which restoration processing is not performed. For example, the computational load can be reduced by not performing restoration processing on component bands included in wavelength ranges that are less important for observation.

[0096] The signal processing circuit 250 converts the mask data into smaller-sized mask data based on the determined synthesis conditions and the mask data stored in the memory 210. Here, the mask data before conversion is referred to as "first mask data," and the mask data after conversion is referred to as "second mask data." The first mask data can be used to reconstruct first spectral image data corresponding to a first group of wavelength bands in a target wavelength range. The first group of wavelength bands can be, for example, a set of multiple component bands included in the target wavelength range. The first spectral image data can be data including image information for each component band included in the first group of wavelength bands. The second mask data can be used to reconstruct second spectral image data corresponding to a second group of wavelength bands in one or more designated wavelength ranges. The second group of wavelength bands can be a set of multiple component bands included in the designated wavelength range. The second spectral image data can be data including image information for each component band included in the designated wavelength range. The first mask data can include first mask information indicating the spatial distribution of spectral transmittance corresponding to the first group of wavelength bands in the filter array 110. The second mask data may include second mask information indicating the spatial distribution of spectral transmittance corresponding to a second wavelength band group in the filter array 110. The second mask data may further include third mask information obtained by combining multiple pieces of information corresponding to one or more non-designated wavelength bands other than the designated wavelength band in the first mask data. Each of the multiple pieces of information in the third mask information may indicate the spatial distribution of spectral transmittance in a corresponding unit wavelength band included in the non-designated wavelength band. The third mask information can be considered as composite mask information obtained by combining mask matrix information corresponding to the non-designated wavelength band (i.e., the non-designated wavelength band) in the first mask data. In this embodiment, the compressed second mask data is generated by combining the information of the multiple unit bands included in the non-designated wavelength band in the first mask data. The signal processing circuit 250 generates multiple pieces of two-dimensional image data corresponding to the multiple designated wavelength bands based on the compressed image data and the second mask information in the second mask data.The signal processing circuit 250 further generates one or more composite image data corresponding to one or more non-designated wavelength bands based on the compressed image data and third mask information (i.e., composite mask information) in the second mask data.

[0097] Specifically, the signal processing circuit 250 compresses the size of the first mask data by processing such as averaging matrix elements corresponding to the multiple component bands to be combined. The signal processing circuit 250 performs a restoration calculation corresponding to the above equation (2) using the converted second mask data and the compressed image data output from the imaging device 100. As a result, the signal processing circuit 250 generates a restored image (i.e., a spectral image) for each of the combined bands. The signal processing circuit 250 sends the generated restored image data to the image processing circuit 320. The image processing circuit 320 renders the restored image of each combined band on the display 330. The image processing circuit 320 may perform processing such as determining the layout within the screen, linking each restored image to band information, or coloring according to wavelength, before displaying the restored image on the display 330.

[0098] In this embodiment, the signal processing circuit 250 determines the band synthesis conditions based on the reference spectral data, but the present disclosure is not limited to this configuration. FIG. 6 illustrates a modification of the system of FIG. 5 . In this modification, a processor 420 is provided that determines mask data conversion conditions based on the reference spectral data output from the input UI 400. The processor 420 determines the band synthesis conditions based on the reference spectral data and sends the information to the signal processing circuit 250 as mask data conversion conditions. The signal processing circuit 250 reads information on the necessary component bands from the memory 210 in accordance with the transmitted mask data conversion conditions. The signal processing circuit 250 constructs compressed mask data from the read information and generates a restored image using the mask data. This configuration further reduces the computational load on the processing device 200. The modification illustrated in FIG. 6 can also be used in the various embodiments described below.

[0099] Next, a detailed example of mask data will be described with reference to FIG. 7 . FIG. 7 shows an example of first mask data before conversion stored in memory 210. In this example, the first mask data includes information indicating the spatial distribution of the transmittance of the filter array 110 for each of multiple component bands included in the target wavelength range. In this example, the first mask data includes mask information for each of a large number of component bands divided every 1 nm, and information regarding the conditions for acquiring the mask information. Each component band is identified by a lower limit wavelength and an upper limit wavelength. The mask information includes information on a mask image and a background image. Each mask image shown in FIG. 7 is acquired by capturing an image of a certain background through the filter array 110 with the image sensor 120. The background image shown in FIG. 7 is acquired by capturing an image of the background with the image sensor 120 without passing the background through the filter array 110. Such mask image and background image information is recorded for each component band. By dividing the value of each pixel in each mask image by the value of the corresponding pixel in the corresponding background image, a value indicating the transmittance of the corresponding filter can be calculated for each component band.

[0100] For example, the multiple component bands are the first component band, . . . , the kth component band, . . . , and the Nth component band. The image data f1, . . . corresponding to the first component band, the image data fk, . . . corresponding to the kth component band, and the image data fN corresponding to the Nth component band are each data in n×m rows and 1 column. In this case, equation (1) is

[0101]

[0102] Each of D1, ..., Dk, ..., DN may be a submatrix of matrix H and may be a diagonal matrix with n x m rows and n x m columns. An example of a case where each of these submatrices may be a diagonal matrix may include a case where it is determined that the crosstalk between pixels (p, q) and pixels (r, s) of image sensor 160 during actual measurement calibration to acquire information about matrix H is the same as the crosstalk between pixels (p, q) and pixels (r, s) of image sensor 160 when an end user captures an image of subject 70 (1 ≤ p, r ≤ n, 1 ≤ q, s ≤ m, pixel (p, q) ≠ pixel (r, s)). Whether the above-described crosstalk-related conditions are met may be determined taking into account the imaging environment, including the optical lens used during imaging, and may also be determined taking into account whether the image quality of each restored image can achieve the end user's objectives.

[0103] When the kth light, which includes light of the kth unit band and does not include light of bands other than the kth unit band, is irradiated onto the filter array and the light output from the filter array is incident on the image sensor, the data output by the image sensor (i.e., data of the mask image) is

[0104]

[0105] When the image sensor is irradiated with the k-th light, which includes light of the k-th unit band and does not include light of bands other than the k-th unit band, without passing through a filter array, and the data output by the image sensor (i.e., background image data) is fk', then equation (4) becomes

[0106]

[0107] That is, when formula (6) is written using matrix components, it becomes formula (7).

[0108]

[0109] Therefore, the diagonal elements hk(1,1), . . . , hk(n×m, n×m) of Dk can be determined using the mask image data and the background image data, that is, all elements of the matrix H become known.

[0110] fk'(i,j) is the pixel value of pixel (i,j) in the background image,

[0111]

[0112] may be the pixel value of pixel (i,j) in the mask pixels, where 1≦i≦n and 1≦j≦m.

[0113] The information about the acquisition conditions includes information about the exposure time and gain. The information about the acquisition conditions does not have to be included in the mask data. In the example of FIG. 7 , information about a mask image and a background image is recorded for each of a plurality of unit bands each having a width of 1 nm. The width of each unit band is not limited to 1 nm and can be set to any value. Furthermore, if the background image has high uniformity, the mask data does not have to include information about the background image. For example, in a configuration in which the image sensor 120 and the filter array 110 are integrated closely facing each other, the mask information nearly matches the mask image, so the mask data does not have to include information about the background image.

[0114] The mask data is, for example, data that defines the matrix H in the above-mentioned equation (2). The format of the mask data may vary depending on the system configuration. The mask data shown in FIG. 7 includes information for calculating the spatial distribution of the spectral transmittance of the filter array 110. The mask data is not limited to this format, and may also be data that directly indicates the spatial distribution of the spectral transmittance of the filter array 110. For example, the mask data may include a sequence of values ​​obtained by dividing each pixel value of the mask image shown in FIG. 7 by the corresponding pixel value of the background image.

[0115] In this embodiment, first mask data including information such as that shown in Fig. 7 is converted into second mask data having a smaller size. An example of a method for converting mask data will be described in more detail below.

[0116] The amount of calculation can be reduced by determining wavelength ranges that are considered to have a small contribution to analysis or classification from the acquired reference spectral data, synthesizing bands with a small contribution, and performing a reconstruction calculation. The bands to be synthesized may be determined by manual input by the user, or may be determined automatically based on the reference spectral data.

[0117] When the bands to be combined are determined manually, the input UI 400 has a function that allows the user to select the bands to be combined. The input UI 400 may also have a function that allows the user to select not only the bands to be combined but also the wavelength range to be subjected to the restoration calculation or restoration conditions, such as wavelength resolution. For example, a list of spectra corresponding to individual substances (e.g., phosphors) that may be contained in the subject or the types of individual substances may be displayed on the display 330. The user can select a specific combination of spectra from the displayed spectra or a specific combination of substances from a list. The user can further select the bands to be combined on the UI from the spectra of the selected substances. In this way, both the reference spectral data and a UI that allows the user to select the restoration conditions for the restoration calculation may be displayed on the display. This allows the user to more efficiently generate images of the wavelength bands required by the user.

[0118] When the bands to be combined are automatically selected, the contribution of each band to the analysis or classification can be automatically estimated based on the type or combination of selected substances. For example, the signal processing circuit 250 may combine bands whose contribution to the analysis or classification is less than a threshold and treat them as a single band. The magnitude of the contribution can be determined, for example, by the signal intensity in the spectrum, the wavelength derivative of the signal intensity, or pre-training. For example, if it is known that the signal intensity of the spectrum of a substance contained in a subject in a certain band is less than a threshold, it can be determined that the band contributes little to the analysis or classification results. Furthermore, if the absolute value of the wavelength derivative of the signal intensity across multiple consecutive bands is extremely small, i.e., if the signal intensities in multiple consecutive bands are almost the same, combining the corresponding multiple bands will not result in any loss of information. Therefore, such multiple bands can be combined without affecting the analysis or classification results. Pre-training is based on statistical methods such as principal component analysis or regression analysis. An example of pre-learning would be to refer to a database that records the "wavelength ranges used for classification" for each individual substance, and determine from the reference spectrum wavelength ranges that do not fall within the "wavelength ranges used for classification" as "wavelength ranges with little contribution."

[0119] 8A and 8B are diagrams illustrating a method for estimating bands that contribute little to calculations based on reference spectra and determining band synthesis conditions. FIG. 8A shows example spectra of four types of samples 31, 32, 33, and 34 that may be included in a subject. FIG. 8B shows an example of a band to be synthesized. Given reference spectrum data such as that shown in FIG. 8A, bands with low signal intensities in all reference spectra can be synthesized as shown in FIG. 8B. This processing allows a restored image to be obtained with sufficient wavelength resolution for bands with relatively high signal intensities in the reference spectra, thereby reducing the amount of calculations without affecting the accuracy of analysis or classification. In this example, wavelength bands that contribute greatly to analysis or classification are selected as "designated wavelength bands." Wavelength bands that contribute little to analysis or classification are selected as "non-designated wavelength bands."

[0120] FIG. 9 is a flowchart illustrating an example of a mask data conversion process executed by the signal processing circuit 250 shown in FIG. 5 . In step S101, the signal processing circuit 250 acquires compressed image data generated by the imaging device 100. In step S102, the signal processing circuit 250 acquires reference spectral data from the input UI 400. In step S103, the signal processing circuit 250 determines whether or not to perform mask data conversion process based on the reference spectral data. If it is determined that conversion process is necessary, the signal processing circuit 250 converts the mask data based on the reference spectral data. As described above, the determination of whether or not to perform mask data conversion process may be made based on whether or not there is a band estimated to have a small contribution to analysis or classification. Alternatively, if the user determines bands to be combined, the signal processing circuit 250 determines whether or not to convert the mask data based on the user's input. If there are bands to be combined, the process proceeds to step S104, where the signal processing circuit 250 performs conversion process. If there are no bands to be combined, the conversion process is omitted. In the following step S105, the signal processing circuit 250 performs the restoration calculation shown in the above-mentioned equation (2) based on the compressed image and mask data, and generates a restored image for each band. If the mask data was converted in step S104, the signal processing circuit 250 performs restoration processing using the converted mask data. The restored image is sent to the display device 300, undergoes necessary processing by the image processing circuit 320, and is displayed on the display 330. Note that the acquisition of the compressed image in step S101 may be performed at any timing before step S105.

[0121] FIG. 10 is a diagram illustrating an example of a method for synthesizing mask information of multiple bands and converting it into new mask information. In this example, mask information for component bands #1 to #20 is pre-stored in memory 210 as pre-conversion mask information. In the example of FIG. 10 , no synthesis process is performed on component bands #1 to #5, but synthesis process is performed on component bands #6 to #20. For component bands #1 to #5, the transmittance distribution of filter array 110 is calculated by dividing the value of each region in the mask image by the value of the corresponding region in the background image. Here, the data for each mask image stored in memory 210 is referred to as "unit mask image data," and the data for each saved background image is referred to as "unit background image data." For bands #6 to #20, the synthesized transmittance distribution is obtained by dividing the data obtained by adding up the unit mask image data for bands #6 to #20 for each pixel by the data obtained by adding up the unit background image data for bands #6 to #20 for each pixel. In this example, the matrix H' corresponding to the matrix H shown in equation (1) is H' = (F1F2 ... F20), where F1, F2, ..., F20 are each submatrices of the matrix H' and may be diagonal matrices with n x m rows and n x m columns. By performing such an operation, mask information can be synthesized for any number of bands. Furthermore, if the background image has a very high degree of uniformity, the mask information will nearly match the mask image. In this case, data obtained by adding or averaging the mask image data of bands #6 to #20 may be used as the synthesized mask data for bands #6 to #20. In this case, matrix H' is changed to matrix H'', where matrix H'' is H'' = (F1F2 ... F5F'),

[0122]

[0123] where i = 1, ..., 20.

[0124] h'(1,1) = (h6(1,1) + ... + h20(1,1) / 15, ..., h'(nxm,nxm) = (h6(nxm,nxm) + ... + h20(nxm,nxm) / 15 may also be used.

[0125] By performing the band synthesis process described above, calculations for 20 bands are required to restore all of the component bands, whereas when bands #6 to #20 are synthesized, calculations for only 6 bands are sufficient. Even when such synthesis is performed, it is possible to restore the images of each band while maintaining the wavelength resolution in the wavelength ranges of bands #1 to #5. This makes it possible to reduce the amount of calculations.

[0126] The mask data conversion process by synthesis may be performed in the environment where the end user uses the mask data, or may be performed at a manufacturing site, such as a factory where the system or device is manufactured. If the mask data conversion process is performed at the manufacturing site, the converted second mask data is stored in memory 210 during the manufacturing process, instead of or in addition to the unconverted first mask data. In this case, when the end user uses the mask data, signal processing circuit 250 can perform restoration processing using the pre-stored converted mask data in response to user input. This further reduces the processing load.

[0127] (Embodiment 2) Next, a second embodiment will be described. In embodiment 1, the amount of calculation required for restoration processing is reduced by combining multiple component bands that contribute little to analysis or classification into a single band. In contrast, in this embodiment, when it is considered that there is little overlap between multiple reference spectra, the signal processing circuit 250 performs band synthesis such that the restored image becomes a classification image as is. This further reduces the burden of signal processing.

[0128] 11A and 11B are diagrams illustrating the band synthesis process of this embodiment. FIG. 11A shows an example of reference spectra for four types of samples 31, 32, 33, and 34 that may be included in the subject. FIG. 11B shows an example of band synthesis. As shown in FIG. 11A, when the overlap between the reference spectra is small, it is effective to perform band synthesis based on the peaks of each spectrum. This makes it possible to create a situation in which each restored image shows a substance corresponding to approximately one type of spectrum. When band synthesis is performed as shown in FIG. 11B, sample 31 has substantial signal intensity in the image corresponding to band #1. Therefore, the restored image corresponding to band #1 can be treated as a classification image of sample #1. Similarly, the restored image of band #2 can be treated as a classification image of sample 32, the restored image of band #3 can be treated as a classification image of sample 33, and the restored image of band #4 can be treated as a classification image of sample #4. While classification would be performed after restoration if band synthesis were not performed, classification can be performed collectively during restoration. This significantly reduces the burden of signal processing.

[0129] The degree of overlap between reference spectra can be determined, for example, by the following method. Consider the wavelength range from λ1 to λ2. When the values ​​of each reference spectrum are integrated from λ1 to λ2, the integral value of the reference spectrum with the largest integral value is defined as signal S, and the sum of the integral values ​​of the other reference spectra is defined as noise N. The degree of overlap between reference spectra can be determined by the signal-to-noise ratio (S / N ratio) obtained by dividing signal S by noise N. For example, when the S / N ratio is lower than 1, it can be determined that the overlap is large, and that overlap occurs between the reference spectra. Conversely, when the S / N ratio is 1 or greater, it can be determined that the overlap is small, and that no overlap occurs between the reference spectra. Alternatively, when the S / N ratio is 2 or greater, it can be determined that no overlap occurs between the reference spectra, and when the S / N ratio is less than 2, it can be determined that overlap occurs between the reference spectra.

[0130] In this example, the signal processing circuit 250 determines the plurality of designated wavelength bands so that the designated wavelength bands do not overlap with the reference spectra, and synthesizes the plurality of component bands included in each designated wavelength band into one band. In this example, each of the plurality of designated wavelength bands includes a peak wavelength of the spectrum associated with a corresponding one of the plurality of substances.

[0131] Alternatively, each reference spectrum may be displayed on the input UI 400, and the user may determine the range of the band to be synthesized by moving the band edge, etc. The S / N ratio may be displayed on the screen to assist the user in making a decision.

[0132] In this example, the signal processing circuit 250 generates compressed second mask data by processing such as averaging multiple elements corresponding to each designated wavelength band in the first mask data. The signal processing circuit 250 generates image data corresponding to each designated wavelength band by performing an operation equivalent to the above-mentioned equation (2) based on the compressed image data and the second mask data. By using the compressed second mask data, the load of the restoration operation can be significantly reduced.

[0133] An example of a case where the overlap between reference spectra is small is the relationship between the excitation light and fluorescence spectra during fluorescence observation. According to this embodiment, observation can be performed under conditions such that the restored image can be separated into an excitation light image and a fluorescence image.

[0134] If it is considered that there is little overlap between the reference spectra, the content input through the input UI 400 can be used to label the restored image. Here, "labeling" refers to associating a known substance name or a classification code with a certain region in the restored image, or with a restored image corresponding to a band having a signal intensity biased toward a certain region.

[0135] Fig. 12 is a block diagram showing an example of the configuration of a system for performing labeling. In the example of Fig. 12, the input UI 400 determines labeling conditions corresponding to a substance or spectrum selected by the user and sends the information to the memory 310 of the display device 300. The image processing circuit 320 of the display device 300 performs labeling processing on each restored image as needed in accordance with the sent labeling conditions, and displays the labeled restored images on the display 330.

[0136] Labeling can take various forms and methods. An example of the labeling process will be described here using the relationship between excitation light and fluorescence spectra during fluorescence observation as an example. Consider a case in which band synthesis is performed so that a restored image of a certain band X becomes an excitation light image and a restored image of another band Y becomes a fluorescence image. In this case, the input UI 400 can determine that band X corresponds to the excitation light and band Y corresponds to a specific fluorophore based on band synthesis conditions determined from reference spectral data. The input UI 400 can, for example, determine labeling conditions that assign a name or a classification code for a fluorophore to the restored image of band Y or a specific region of the restored image. A band determined to be in the wavelength range of the excitation light may be assigned a name or classification code such as "excitation light band." Labeling information, such as a name entered by the user via the input UI 400, may be used. Labeling may also be performed automatically based on known physical property information.

[0137] 13A and 13B are diagrams showing examples of reference spectra for four samples 31 to 34 with large overlap between their spectra. When there is a large overlap between the reference spectra, as in this example, band synthesis cannot be performed so that the restored image and the classification image match. However, by performing band synthesis as shown in FIG. 13B, it is possible to reduce the overlap for some spectra. For bands and samples with no or small spectral overlap, classification is possible during restoration. For bands and samples with large overlap, it is possible to narrow down the substances that are expected to be contained in the band. If necessary, displaying substances that may be contained in the band can help the observer understand.

[0138] In this manner, the multiple designated wavelength bands may include one or more first designated wavelength bands that do not overlap among the multiple spectra and one or more second designated wavelength bands that overlap among the multiple spectra. In the example of Figure 13B, Band #1, Band #3, and Band #4 correspond to the first designated wavelength bands, and Band #X and Band #Y correspond to the second designated wavelength bands. For each first designated wavelength band, classification can be performed simultaneously with restoration. For each second designated wavelength band, a small number of substances that are expected to be included in that band can be narrowed down simultaneously with restoration.

[0139] FIG. 14 is a diagram illustrating an example of a graphical user interface (GUI) that enables operations performed in this embodiment. This GUI displays a compressed image, a restored image, reference spectra, and band synthesis information, as well as a table showing the correspondence between bands and samples (i.e., substances). If necessary, displays related to the restoration of the hyperspectral image, displays related to the analysis of the hyperspectral image, and / or displays of the restored hyperspectral image may be added or deleted. As described above, when there is little overlap between the reference spectra, band synthesis can be performed to achieve a one-to-one correspondence between band numbers and classifications. Regarding the display portion of the "reference spectra and band synthesis information," the results of band synthesis as shown in FIG. 10 or the results of band synthesis in which some bands are excluded from the restoration target, as shown in FIG. 15B (described later), may also be displayed.

[0140] Third Embodiment Next, a third embodiment of the present disclosure will be described. In this embodiment, bands that are considered to be less important among bands included in the target wavelength range are excluded from the restoration target, thereby further reducing the load of the restoration calculation.

[0141] Typically, the reconstruction calculation is performed using information on all bands included in the target wavelength range. If even one band included in the target wavelength range is not used in the reconstruction calculation, the relationship g = Hf in the above formula (1) will not be satisfied. In this case, the optical signals of wavelengths belonging to the excluded band are assigned to other bands as noise, resulting in reconstruction errors and reducing the accuracy of subsequent analysis or classification. However, when the spectrum of the observation target is known, such as in fluorescence observation, bands in the target wavelength range that are predicted to have zero or very small signal intensities may occur depending on the combination of observation targets. If the signal intensity of a band included in the target wavelength range is zero or very small, excluding that band from the reconstruction calculation will result in no or very small reconstruction errors. Excluding such a band will not substantially affect subsequent analysis or classification. Therefore, if a band included in the target wavelength range is predicted to have zero or very small signal intensity emitted from the observation target, that band can be excluded from the reconstruction calculation. This can also be explained as follows: The formula that is satisfied when reconstructing all bands included in the target wavelength range is g = Hf, as described above. The equation that is satisfied when some bands are excluded is g' = H'f', where H' = H - ΔH and f' = f - Δf. ΔH and Δf represent elements corresponding to the excluded bands. Since Δf is 0 or very small, g' = H'f' = H'f approximately holds. Therefore, the restoration calculation expressed by g' = H'f, that is, the restoration calculation when some bands are excluded, holds.

[0142] 15A and 15B are diagrams illustrating a method for excluding specific bands from the restoration target based on reference spectral data. If four samples 31, 32, 33, and 34 having spectra as shown in FIG. 15A are considered to be present in the imaging area, there is a band at wavelengths between samples 32 and 33 where all samples are predicted to have no signal intensity, i.e., a completely dark image is predicted to be output. Such bands predicted to have no signal intensity can be excluded from the restoration target, as shown in FIG. 15B. Reducing the number of bands to be restored makes it possible to reduce the load on signal processing.

[0143] FIG. 16 is a flowchart showing an example of the operation of the signal processing circuit 250 when excluding a specific band from reconstruction. The flowchart shown in FIG. 16 replaces the synthesis process for a specific band (steps S103 and S104) in the flowchart shown in FIG. 9 with a process for deleting information about the specific band (steps S203 and S204). In step S203, the signal processing circuit 250 determines whether to delete information about the specific band from the mask data. If, based on the reference spectrum data, there is a band from which an image with no signal intensity is expected to be output for all expected samples, the signal processing circuit 250 proceeds to step S204 and deletes information about that band from the mask data. If there is no band from which an image with no signal intensity is expected to be output, step S204 is omitted. This operation reduces the amount of calculation in the reconstruction calculation process in the subsequent step S105, thereby shortening the calculation time.

[0144] FIG. 17 is a flowchart showing an example of an operation in which reference spectral data is translated into mask data conversion conditions and sent to the signal processing circuit 250, as in the example shown in FIG. 6 . In this flowchart, steps S102, S103, and S104 in the flowchart of FIG. 9 are replaced with steps S302 and S303. In the example of FIG. 17 , the signal processing circuit 250 acquires mask data conversion conditions, i.e., information on bands to be used for restoration, from the processor 420 shown in FIG. 6 in step S302. Based on this information, specific bands not to be used for restoration can be determined. In the subsequent step S303, the signal processing circuit 250 acquires mask data other than the specific bands from the memory 210. In other words, the mask data of the specific bands to be deleted is not read. This reduces the number of processing steps compared to the example of FIG. 16 , in which mask data of all bands is read once and then the mask data of the specific bands is deleted. In this case, the reference spectral data does not need to be stored in the memory 410. For example, the name of a material or substance input or selected via the input UI 400 may be associated with the conversion conditions. Such linking allows the processor 420 or the signal processing circuit 250 to determine bands to be excluded without referring to the reference spectral data. Alternatively, labeling information such as the name of a substance corresponding to each spectrum may be extracted from the reference spectral data stored in the memory 410, and the labeling information may be used to label each restored image.

[0145] Fourth Embodiment Next, a fourth embodiment of the present disclosure will be described. This embodiment relates to a system for performing fluorescence observation.

[0146] 18 is a diagram schematically illustrating the configuration of a system according to this embodiment. The system includes a light source 610, an optical system 620, and a detector 630. The optical system 620 includes an interference filter 621, a dichroic mirror 622, an objective lens 623, and a long-pass filter 624. The light source 610 may include a laser light-emitting element that emits excitation light. The detector 630 includes the imaging device 100 described above. A sample 80 containing a fluorescent material is irradiated with the excitation light, and the generated fluorescence is detected by the detector 630.

[0147] Excitation light emitted from the light source 610 passes through an interference filter 621 that selectively transmits light of a specific wavelength that excites a fluorescent material, and then enters a dichroic mirror 622. The dichroic mirror 622 reflects light in a certain wavelength range that includes the wavelength of the excitation light, and transmits light in other wavelength ranges. The excitation light reflected by the dichroic mirror 622 enters the sample 80 through an objective lens 623. The sample 80 exposed to the excitation light emits fluorescence. The fluorescence passes through the dichroic mirror 622 and a long-pass filter 624 and is detected by the detector 630. A portion of the excitation light incident on the sample 80 is reflected. Most of the excitation light reflected by the sample 80 is reflected by the dichroic mirror 622, but a portion passes through the dichroic mirror and heads toward the detector 630. Although not much excitation light passes through the dichroic mirror 622, excitation light typically has an intensity several orders of magnitude higher than that of fluorescence. Therefore, if excitation light enters the detector 630 together with fluorescence, it may saturate the charge in the image sensor, hindering the observation of fluorescence. To prevent this phenomenon, a long-pass filter 624 is placed in front of the detector 630, and the optical system 620 is constructed so that excitation light does not enter the detector 630. The long-pass filter 624 is used because, in fluorescence observation, excitation light has higher energy, i.e., a shorter wavelength, than fluorescence.

[0148] The detector 630 may be, for example, a hyperspectral camera including the imaging device 100 and the processing device 200 in embodiment 2. As described above, the detector 630 performs restoration processing based on mask data from which information on unnecessary bands corresponding to excitation light has been removed.

[0149] FIG. 19 illustrates an example of the relationship between the excitation light and fluorescence spectra and the excluded bands. In this example, the sample 80 contains multiple types of fluorescent materials. The fluorescence spectra emitted by these fluorescent materials are different from one another. Because the energy of the excitation light is higher than the energy of the fluorescence, the wavelength of the excitation light is shorter than the wavelength of the fluorescence. In the example of FIG. 19, due to the characteristics of the dichroic mirror 622 and the long-pass filter 624, the wavelength range of the light actually incident on the detector 630 is narrower than the target wavelength range detectable by the detector 630. In this embodiment, in order to cut the excitation light, the optical system 620 is configured so that the dichroic mirror 622 and the long-pass filter 624 do not transmit short-wavelength light. In this case, even if the target wavelength range of the detector 630 overlaps with the cut wavelength range, it is known in advance that the cut wavelength range has no signal intensity. That is, it is known in advance that the aforementioned equation g' = H'f' = H'f is satisfied. Therefore, the band in this wavelength range can be excluded from the reconstruction calculation.

[0150] The characteristics of the long-pass filter 624 and the dichroic mirror 622 are selected depending on the fluorescence to be observed and the wavelength of the excitation light to be used. Therefore, the configuration of this embodiment, which allows any band to be selected and excluded from the restoration calculation, makes it possible to perform observation with the minimum amount of calculation processing required depending on the object to be observed.

[0151] Example An example will be described in which a method according to an embodiment of the present disclosure is applied to the m-FISH method, which is a method of fluorescence observation.

[0152] FISH (fluorescent in-situ hybridization) is a method in which a probe containing a gene sequence complementary to a specific gene sequence is labeled with a fluorescent dye, and the hybridized site or chromosome is identified by fluorescence. In m-FISH (multicolor FISH), multiple probes, each labeled with a different fluorescent dye, are used simultaneously.

[0153] The m-FISH method is used to test for certain cancers, such as leukemia, and for congenital genetic abnormalities. For example, Cambio's m-FISH probes are designed with five fluorescent dyes, with different attachment ratios for each chromosome number in humans and mice. Therefore, by staining a set of chromosomes with this probe, the number of each chromosome can be identified.

[0154] When there are no translocations, which are the cause of cancer and congenital genetic abnormalities, each chromosome exhibits a single fluorescence spectrum. However, when a translocation occurs, the fluorescence spectrum changes depending on the part of the chromosome. This property can be used to detect translocations.

[0155] Figure 20A shows the absorption spectra of the five fluorescent dyes (Cy3, Cy3.5, Cy5, FITC, and DEAC) used in Cambio's m-FISH fluorescent labeling. Figure 20B shows the fluorescence spectra of each of these fluorescent dyes. As is clear from Figure 20A, there is no single wavelength that can simultaneously induce the fluorescence of all fluorescent dyes. Therefore, the distribution of the five fluorescent dyes can be determined by the following procedure.

[0156] <STEP 1: Excitation with First Wavelength and Hyperspectral Imaging> The first imaging will be described with reference to Figures 21A and 21B. Figure 21A shows the relationship between excitation wavelength and absorption spectrum of fluorescent dyes. The solid curves show the absorption spectra of two types of dyes (DEAC and FITC) that induce fluorescence. The dotted curves show the absorption spectra of three types of dyes (Cy3, Cy3.5, and Cy5) that do not induce fluorescence because sufficient absorption does not occur at the excitation wavelength. Figure 21B shows an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and recovery band.

[0157] In this example, the system configuration shown in Figure 18 is used. The wavelength of the excitation light is set to 405 nanometers (nm). The cutoff wavelength of the dichroic mirror 622 and the long-pass filter 624 is set to 450 nm. In other words, light with wavelengths of 450 nm or longer is incident on the detector 630. In this example, the wavelength range shorter than 450 nm is the cutoff wavelength range. The wavelength of 405 nm of the excitation light is defined as the first wavelength.

[0158] The absorption spectra of the fluorescent dyes show that when irradiated with 405 nm excitation light, fluorescence is induced from the FITC and DEAC dyes. Light in the wavelength range of 450 nm or less is blocked by the dichroic mirror 622 and long-pass filter 624, so light in this wavelength range does not enter the detector 630. Furthermore, because the absorption spectra of the fluorescent dyes Cy3, Cy3.5, and Cy5 are longer than the excitation wavelength, no fluorescence is induced from the fluorescent dyes Cy3, Cy3.5, and Cy5. Therefore, for example, no fluorescence is present at wavelengths of 650 nm or more, and the output image is completely dark.

[0159] Therefore, it is effective to perform restoration by setting wavelengths from 450 nm to 500 nm as the first band, wavelengths from 500 nm to 550 nm as the second band, and wavelengths from 550 nm to 650 nm as the third band, which allows the fluorescence spectra of FITC and DEAC emitted under these conditions to be distinguished and their respective distributions to be determined.

[0160] <STEP 2: Excitation with a Second Wavelength and Hyperspectral Imaging> The second imaging will be described with reference to Figures 22A and 22B. Figure 22A shows the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the second imaging. The solid curve shows the absorption spectrum of the dye (Cy5) in which fluorescence is induced. The dotted curves show the absorption spectra of the remaining four dyes in which fluorescence is not induced because sufficient absorption does not occur at the excitation wavelength. Figure 22B shows an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and recovery band.

[0161] In the second imaging, the wavelength of the excitation light is set to the second wavelength of 633 nm, and the cutoff wavelengths of the dichroic mirror 622 and the long-pass filter 624 are set to 650 nm. That is, light with wavelengths of 650 nm or longer is incident on the detector 630.

[0162] 22A, it can be seen that when irradiated with 633 nm excitation light, fluorescence from the Cy5 dye is efficiently induced. In this case, the distribution of the Cy5 dye can be identified from the image output from the detector 630 without spectral decomposition. In other words, the restoration calculation process can be omitted.

[0163] <STEP 3: Excitation with a Third Wavelength and Hyperspectral Imaging> The third imaging will be described with reference to Figures 23A and 23B. Figure 23A shows the relationship between the excitation wavelength and the absorption spectrum of fluorescent dyes in the third imaging. The solid curves show the absorption spectra of two types of dyes (Cy3 and Cy3.5) in which fluorescence is induced. The dashed and dotted curves show the absorption spectra of the remaining three types of dyes in which fluorescence is not induced due to insufficient absorption at the excitation wavelength. Figure 23B shows the relationship between each fluorescence spectrum and the cutoff wavelength range and recovery band.

[0164] In the third imaging, the wavelength of the excitation light is set to a third wavelength of 532 nm, and the cutoff wavelengths of the dichroic mirror 622 and the long-pass filter 624 are set to 550 nm. That is, light with wavelengths of 550 nm or longer is incident on the detector 630.

[0165] The absorption spectra of the fluorescent dyes shown in Figure 23A show that the fluorescence of the Cy3 and Cy3.5 dyes is efficiently induced when irradiated with 532 nm excitation light. Note that the FITC and Cy5 dyes also have slight absorption, so these dyes may also emit weak fluorescence.

[0166] In this example, the restoration bands are set as follows: First band: wavelengths from 550 nm to 575 nm.

[0167] This band contains Cy3 and Cy3.5 fluorescence as the main components, and may contain a small amount of FITC fluorescence. - Second band: Wavelength 575 nm to 625 nm.

[0168] This band contains Cy3 and Cy3.5 fluorescence as the main components, and may contain a small amount of FITC fluorescence. - Third band: Wavelength 625 nm to 650 nm.

[0169] This band contains mainly Cy3 and Cy3.5 fluorescence, and may be slightly contaminated with FITC and Cy5 fluorescence. ・Fourth band: Wavelengths from 650 nm to 800 nm. This band contains mainly Cy3 and Cy3.5 fluorescence, and may be slightly contaminated with Cy5 fluorescence.

[0170] Of these components, the distribution of FITC is identified in STEP 1, and the distribution of Cy5 is identified in STEP 2. The fluorescence of Cy3 and Cy3.5 is contained in all of bands 1 to 4. However, the fluorescence intensity ratio of these fluorescent dyes in each band is known from the emission spectra of the fluorescent dyes. Therefore, the distributions of Cy3 and Cy3.5 can be determined by solving simultaneous equations related to intensity or by simulating the distribution of dyes that reproduces the imaging results.

[0171] In this way, even when a sample is labeled with multiple fluorochromes, it is possible to limit the excitation light spectrum so that only a portion of the fluorochromes are emitted in each measurement. By selecting the regeneration band accordingly, it is possible to identify the distribution of each fluorochrome with less computational effort.

[0172] (Other 1) Modifications of the embodiment of the present disclosure may be as follows.

[0173] A signal processing method executed by a computer, comprising: performing a first process based on a first instruction; performing a second process based on a second instruction; and performing a third process based on a third instruction, wherein the first process comprises: (a-1) receiving a plurality of first pixel values; an image sensor outputting the plurality of first pixel values ​​corresponding to first light from a filter array, the first light corresponding to second light from a first object incident on the filter array; and (a-2) generating a plurality of pixel values ​​I(11) of an image corresponding to a first wavelength range of the first object, through I(1p), corresponding to a p-th wavelength range of the first object, based on a first matrix and the plurality of first pixel values. the plurality of first pixel values ​​correspond to a plurality of first pixels arranged in m rows and n columns, the first matrix is ​​(A1A2...Ap), the first matrix includes a plurality of first submatrices, the plurality of first submatrices are A1, A2,...,Ap, the plurality of first submatrices include a q-th submatrix Aq, the plurality of first submatrices include a plurality of second submatrices, the plurality of second submatrices are the r-th submatrix Ar,..., the (r+s)-th submatrix A(r+s), p, q, r, and s are each natural numbers, q<r or (r+s)<q, 1≦q≦p, 1≦r≦p, 1≦r+s≦p; the second processing includes: (b-1) receiving a plurality of second pixel values; the image sensor outputs the plurality of second pixel values ​​in response to third light from the filter array, the third light corresponding to fourth light from a second object incident on the filter array; (b-2) generating a plurality of pixel values ​​I(2q) of an image corresponding to the q wavelength range of the second object based on the q submatrix Aq and a plurality of second pixel values, wherein the plurality of second pixel values ​​correspond to a plurality of second pixels arranged in m rows and n columns, and generating a plurality of pixel values ​​I(2r), to a plurality of pixel values ​​I(2(r+s)) of an image corresponding to the r wavelength range of the second object based on the plurality of second submatrices and the plurality of second pixel values, wherein the third processing includes: (c-1) receiving a plurality of third pixel values, wherein the image sensor outputs the plurality of third pixel values ​​in response to a fifth light from the filter array, the fifth light corresponding to a sixth light from a third object incident on the filter array,(c-2) generating, based on the q submatrix Aq and a plurality of third pixel values, a plurality of pixel values ​​I(3q) of an image corresponding to the q wavelength range of the third object; and (c-3) generating, based on the plurality of third pixel values ​​and a second matrix generated based on the plurality of second submatrices, a plurality of pixel values ​​of an image I3c corresponding to the r wavelength range to the (r+s) wavelength range of the third object, wherein the plurality of third pixel values ​​correspond to a plurality of third pixels arranged in m rows and n columns; and not generating, based on the plurality of second submatrices and the plurality of third pixel values, a plurality of pixel values ​​I(3r) to I(3(r+s)) of an image corresponding to the (r+s) wavelength range of the third object.

[0174] <Details of the Signal Processing Method> p, q, r, and s are natural numbers, q<r or (r+s)<q, 1≦q≦p, 1≦r≦p, and 1≦r+s≦p.

[0175] Although equations (1) and (2) are written in n×m, n×m in equations (1) and (2) may be rewritten as m×n.

[0176] The first instruction, the second instruction, and the third instruction may each be given by the user using the input UI 400 .

[0177] The first instruction is an instruction to generate an image corresponding to the first wavelength range of the first subject, . . . , an image corresponding to the p-th wavelength range of the first subject.

[0178] The second instructions include an instruction to generate an image corresponding to the q wavelength range of the second subject, an instruction not to generate an image corresponding to the r wavelength range of the second subject, ..., an instruction not to generate an image corresponding to the (r+s) wavelength range of the second subject.

[0179] The third instructions include an instruction to generate an image corresponding to the qth wavelength range of the third subject, an instruction to generate an image corresponding to the rth wavelength range to the (r+s)th wavelength range of the third subject, an instruction not to generate an image corresponding to the rth wavelength range of the third subject, to, an instruction not to generate an image corresponding to the (r+s)th wavelength range of the third subject.

[0180] The first process includes processes (a-1) and (a-2).

[0181] <Process (a-1)> Signal processing device 250 receives a plurality of first pixel values ​​from image sensor 160. Second light from a first object is incident on filter array 110. In response to this incidence, filter array 110 outputs first light. Image sensor 160 outputs a plurality of first pixel values ​​in response to the first light from filter array 110.

[0182] The plurality of first pixel values ​​can be written in a matrix format of m rows and n columns as follows:

[0183]

[0184] (i,j) may be thought of as corresponding to the position of a pixel in the image, where 1≦i≦m and 1≦j≦n.

[0185] The plurality of first pixel values ​​can be written in a matrix format of m×n rows and 1 column as follows:

[0186]

[0187] <Process (a-2)> The signal processing circuit 250 generates a plurality of pixel values ​​I(11), ..., a plurality of pixel values ​​I(1p) of an image corresponding to the p-th wavelength range of the first object, based on the first matrix and a plurality of first pixel values ​​recorded in the memory 210. This generation method has already been explained using equations (1) and (2).

[0188] Although equations (1) and (2) are written in n×m, n×m in equations (1) and (2) is rewritten as m×n for application.

[0189] The first example is the matrix H with m×n rows and m×n×N columns shown in equation (1). Here, if matrix H is H1 and submatrices A1, ..., Ap are used, then

[0190]

[0191] Each of the submatrices A1, ..., Ap may be a diagonal matrix.

[0192]

[0193] If the multiple pixel values ​​I(11), ..., multiple pixel values ​​I(1p) of the image corresponding to the first wavelength range of the first subject are written in a matrix format with m rows and n columns, they can be written as follows:

[0194]

[0195] (i,j) may be thought of as corresponding to the position of a pixel in the image, where 1≦i≦m and 1≦j≦n.

[0196] The multiple pixel values ​​I(11), ..., multiple pixel values ​​I(1p) of the image corresponding to the first wavelength range of the first subject can be written in a matrix format of m x n rows and 1 column as follows:

[0197]

[0198] If written in the form of equation (1),

[0199]

[0200] is.

[0201] The plurality of first submatrices A1, A2, ..., Ap includes the qth submatrix Aq.

[0202]

[0203] The plurality of first submatrices A1, A2, ..., Ap includes a plurality of second submatrices, r-th submatrices Ar, ..., and (r+s)-th submatrix A(r+s).

[0204]

[0205] The second process includes processes (b-1) and (b-2).

[0206] <Process (b-1)> The signal processing device 250 receives a plurality of second pixel values ​​from the image sensor 160. A fourth light from a second object is incident on the filter array 110. In response to this incidence, the filter array 110 outputs a third light. The image sensor 160 outputs a plurality of second pixel values ​​corresponding to the third light from the filter array 110.

[0207] The plurality of second pixel values ​​can be written as follows if written in a matrix format of m rows and n columns.

[0208]

[0209] (i,j) may be thought of as corresponding to the position of a pixel in the image, where 1≦i≦m and 1≦j≦n.

[0210] The plurality of second pixel values ​​can be written in a matrix format of m×n rows and 1 column as follows:

[0211]

[0212] <Process (b-2)> The signal processing circuit 250 generates a plurality of pixel values ​​I(2q) of an image corresponding to the q wavelength range of the second object, based on the q submatrix Aq and a plurality of second pixel values ​​recorded in the memory 210. The signal processing circuit 250 does not generate a plurality of pixel values ​​I(2r), to, of an image corresponding to the r wavelength range of the second object, based on the plurality of second submatrices and a plurality of second pixel values, or a plurality of pixel values ​​I(2(r+s)) of an image corresponding to the (r+s) wavelength range of the second object.

[0213] For example, the signal processing circuit 250 may perform the following processing.

[0214] The signal processing circuit 250 deletes the r-th submatrix Ar, ~, the (r+s)-th submatrix A(r+s) from the first matrix shown in equation (12) recorded in the memory 210, and generates a matrix H2 with n × m rows and n × m × (p-(s+1)) columns.

[0215]

[0216] Then, based on the following formula (22), and in accordance with the method disclosed in the explanation of formulas (1) and (2), (i) generate a plurality of pixel values ​​I(21), ..., of an image corresponding to the first wavelength range of the second subject, a plurality of pixel values ​​I(2q), ..., of an image corresponding to the q wavelength range of the second subject, and a plurality of pixel values ​​I(2(r-1)) of an image corresponding to the (r-1) wavelength range of the second subject; (ii) do not generate a plurality of pixel values ​​I(2r), ..., of an image corresponding to the s wavelength range of the second subject, and a plurality of pixel values ​​I(2(r+s)) of an image corresponding to the (r+s) wavelength range of the second subject; and (iii) generate a plurality of pixel values ​​I(2(r+s+1)), ..., of an image corresponding to the p wavelength range of the second subject.

[0217] Although equations (1) and (2) are written in n×m, n×m in equations (1) and (2) is rewritten as m×n for application.

[0218]

[0219] The third process includes processes (c-1), (c-2), and (c-3).

[0220] <Process (c-1)> The signal processing device 250 receives a plurality of third pixel values ​​from the image sensor 160. A sixth light from a third object is incident on the filter array 110. In response to this incidence, the filter array 110 outputs a fifth light. The image sensor 160 outputs a plurality of third pixel values ​​in response to the fifth light from the filter array 110.

[0221] The plurality of third pixel values ​​can be written as follows if written in a matrix format of m rows and n columns.

[0222]

[0223] (i,j) may be thought of as corresponding to the position of a pixel in the image, where 1≦i≦m and 1≦j≦n.

[0224] The plurality of third pixel values ​​can be written in a matrix format of m×n rows and 1 column as follows:

[0225]

[0226] <Process (c-2)> The signal processing circuit 250 generates multiple pixel values ​​I(3q) of an image corresponding to the q wavelength range of the third subject based on the qth submatrix Aq and multiple third pixel values ​​recorded in the memory 210.

[0227] <Process (c-3)> The signal processing circuit 250 generates a second matrix based on the multiple second submatrices recorded in the memory 210. The signal processing circuit 250 generates multiple pixel values ​​I3c of an image corresponding to the r wavelength range of the third subject to the (r+s) wavelength range of the third subject based on the generated second matrix and multiple third pixel values. The signal processing circuit 250 does not generate multiple pixel values ​​I(3r) to I(3(r+s)) of an image corresponding to the r wavelength range of the third subject, based on the multiple second submatrices and multiple third pixel values.

[0228] For example, the signal processing circuit 250 may perform the following processing.

[0229] The signal processing circuit 250 generates a second matrix H3 based on the r-th submatrix Ar, ~, and the (r+s)-th submatrix A(r+s) included in the first matrix (see equation (12)) recorded in the memory 210.

[0230]

[0231]

[0232] W(1,1) = (hr(1,1) + ... + h(r+s)(1,1)) / (s+1), ... W(mxn,mxn) = (hr(mxn,mxn) + ... + h(r+s)(mxn,mxn)) / (s+1).

[0233] The signal processing circuit 250 generates a third matrix H4 based on the first matrix H1 and the second matrix H3.

[0234]

[0235] The third matrix H4 is a matrix with n×m rows and n×m×(ps) columns.

[0236] Then, based on equation (27), and in accordance with the method disclosed in the explanation of equations (1) and (2), (i) generate a plurality of pixel values ​​I(31), ..., of an image corresponding to the first wavelength range of the third subject, a plurality of pixel values ​​I(3q), ..., of an image corresponding to the q wavelength range of the third subject, and a plurality of pixel values ​​I(3(r-1)) of an image corresponding to the (r-1) wavelength range of the third subject; (ii) do not generate a plurality of pixel values ​​I(3r), ..., of an image corresponding to the s wavelength range of the third subject, and a plurality of pixel values ​​I(3(r+s)) of an image corresponding to the (r+s) wavelength range of the third subject; (iii) generate a plurality of pixel values ​​I3c of an image corresponding to the r wavelength range to the (r+s) wavelength range of the third subject; and (iv) generate a plurality of pixel values ​​I(3(r+s+1)), ..., of an image corresponding to the p wavelength range of the third subject.

[0237] Although equations (1) and (2) are written in n×m, n×m in equations (1) and (2) may be rewritten as m×n.

[0238]

[0239] The pixel values ​​I3c of the image corresponding to the r-th wavelength region of the third subject to the (r+s)-th wavelength region of the third subject can be written in a matrix format of m rows and n columns as follows:

[0240]

[0241] (i,j) may be thought of as corresponding to the position of a pixel in the image, where 1≦i≦m and 1≦j≦n.

[0242] If the multiple pixel values ​​I3c of the image corresponding to the r-th wavelength region of the third subject to the (r+s)-th wavelength region of the third subject are written in a matrix format of m×n rows and 1 column, they can be written as follows:

[0243]

[0244] (Other 2) In the present disclosure, a compressed image may be generated by imaging using a method other than imaging using a filter array including a plurality of optical filters.

[0245] For example, the imaging device 100 may be configured such that the light-receiving characteristics of the image sensor are changed for each pixel by processing the image sensor 160, and a compressed image may be generated by capturing an image using the processed image sensor 160. That is, instead of encoding the light incident on the image sensor using the filter array 110, the compressed image may be generated by providing the image sensor with a function for encoding the incident light. In this case, the mask data corresponds to the light-receiving characteristics of the image sensor.

[0246] Furthermore, an optical element such as a metalens may be introduced into at least a portion of the optical system 140, thereby changing the optical characteristics of the optical system 140 spatially and wavelength-wise, and encoding the incident light. A compressed image may be generated by an imaging device including such a configuration. In this case, the mask data is information corresponding to the optical characteristics of the optical element such as a metalens. In this way, by using an imaging device 100 with a configuration different from that using the filter array 110, the intensity of the incident light may be modulated for each wavelength, and a compressed image and a restored image may be generated.

[0247] (Other 3) The present disclosure is not limited to the first to fourth embodiments, examples, and variations. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art may make to the above-described embodiments, examples, and variations, and forms constructed by combining components of different embodiments, examples, and / or different variations may also be included within the scope of the present disclosure.

[0248] The technology disclosed herein can be applied not only to fluorescence observation but also to other applications where the spectrum of the object to be observed is known, such as observation of absorption spectra, observation of blackbody radiation (e.g., temperature estimation), and estimation of light sources (e.g., LEDs, halogen lamps, etc.).

[0249] The technology disclosed herein is useful, for example, in cameras and measuring instruments that capture multi-wavelength images. The technology disclosed herein can also be applied to, for example, fluorescence observation, absorption spectrum observation, sensing for biological, medical, and cosmetic applications, systems for inspecting food for foreign matter and pesticide residues, remote sensing systems, and in-vehicle sensing systems.

[0250] 10 Compressed image 20 Decompressed image 31, 32, 33, 34 Sample 70 Object 80 Specimen 100 Imaging device 110 Filter array 140 Optical system 150 Control circuit 160 Image sensor 200 Processing device 210 Memory 250 Signal processing circuit 300 Display device 320 Image processing circuit 330 Display 400 Input UI 410 Memory 420 Processor 610 Light source 620 Optical system 621 Interference filter 622 Dichroic mirror 623 Objective lens 624 Long-pass filter 630 Detector

Claims

1. A signal processing method executed by a computer, comprising: acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range; acquiring reference spectral data including information on one or more spectra associated with the subject; generating a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands determined based on the reference spectral data from the compressed image data; A method comprising the steps of:

2. The method according to claim 1, wherein the one or more spectra are associated with one or more substances assumed to be included in the subject.

3. The method according to claim 1 or 2, wherein each of the plurality of designated wavelength bands includes a peak wavelength of the spectrum associated with a corresponding one of the one or more substances.

4. The reference spectral data includes information on a plurality of spectra associated with a plurality of substances assumed to be included in the subject, The plurality of designated wavelength bands includes a first designated wavelength band having no overlap among the plurality of spectra and a second designated wavelength band having overlap among the plurality of spectra, The method according to claim 1 or 2.

5. The compressed image data is generated using a filter array including a plurality of types of optical filters having different spectral transmittances and an image sensor, The method further includes acquiring mask data reflecting a spatial distribution of the spectral transmittance, The plurality of two-dimensional image data is generated based on the compressed image data and the mask data, The method according to claim 1 or 2.

6. The mask data includes mask matrix information having elements corresponding to a spatial distribution of transmittance of the filter array for each of a plurality of unit bands included in the target wavelength range, The method includes: generating composite mask information by synthesizing the mask matrix information corresponding to non-designated wavelength bands different from the designated wavelength bands in the target wavelength range; generating composite image data for the non-designated wavelength bands based on the compressed image data and the composite mask information; The method according to claim 5, further comprising:

7. Generating the plurality of two-dimensional image data includes generating and outputting the plurality of two-dimensional image data corresponding to the plurality of designated wavelength bands without generating image data corresponding to non-designated wavelength bands different from the designated wavelength band in the target wavelength range from the compressed image data, according to the method of claim 5.

8. The plurality of designated wavelength bands are determined based on the intensity of the one or more spectra indicated by the reference spectrum data or the differential value of the intensity, according to the method of claim 1 or 2.

9. The reference spectrum data includes information on fluorescence spectra of one or more substances assumed to be included in the subject, according to the method of claim 1 or 2.

10. The reference spectrum data includes information on absorption spectra of one or more substances assumed to be included in the subject, according to the method of claim 1 or 2.

11. Further including displaying on a display a graphical user interface for allowing a user to specify the one or more spectra, or one or more substances associated with the one or more spectra, wherein the reference spectrum data is acquired according to the specified one or more spectra, or the specified one or more substances, according to the method of claim 1 or 2.

12. A method for generating mask data used to restore spectral image data for each wavelength band from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range, acquiring first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength range; acquiring reference spectrum data including information on at least one spectrum; determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectrum data; generating second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more designated wavelength ranges based on the first mask data; including the method.

13. The compressed image data is generated using a filter array including a plurality of types of optical filters having different spectral transmittances and an image sensor. The first mask data and the second mask data are data reflecting the spatial distribution of the spectral transmittance of the filter array, the first mask data includes first mask information indicating the spatial distribution of the spectral transmittance corresponding to the first wavelength band group, the second mask data includes second mask information indicating the spatial distribution of the spectral transmittance corresponding to the second wavelength band group, The method according to claim 12.

14. the second mask data further includes third mask information obtained by synthesizing a plurality of pieces of information, each of the plurality of pieces of information indicates the spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength region other than the designated wavelength region in the target wavelength region, The method according to claim 13.

15. The method according to claim 13, wherein the second mask data does not include information regarding the spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength region other than the designated wavelength region.

16. A signal processing method executed by a computer, acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength region, acquiring reference spectral data including information on one or more spectra associated with the subject, displaying, on a display, both a graphical user interface for specifying restoration conditions for generating a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands from the compressed image data, and an image based on the reference spectral data, The method including the above.

17. A signal processing apparatus including a processor and a memory storing a computer program executed by the processor, wherein the computer program causes the processor to acquire compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength region, acquire reference spectral data including information on one or more spectra associated with the subject, generate a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands determined based on the reference spectral data from the compressed image data, The signal processing apparatus.

18. A processor and ​ A memory storing a computer program executed by the processor, A signal processing apparatus comprising: The computer program causes the processor to acquire first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength range from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in the target wavelength range; acquire reference spectral data including information on at least one spectrum; determine one or more specified wavelength ranges included in the target wavelength range based on the reference spectral data; generate second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more specified wavelength ranges based on the first mask data; A signal processing apparatus that executes the above.

19. A processor, A memory storing a computer program executed by the processor, A signal processing apparatus comprising: The computer program causes the processor to acquire compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in the target wavelength range; acquire reference spectral data including information on one or more spectra associated with the subject; display, on a display, both a graphical user interface for specifying restoration conditions for generating a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands from the compressed image data and an image based on the reference spectral data; A signal processing apparatus that executes the above.

20. A computer is caused to acquire compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in the target wavelength range; acquire reference spectral data including information on one or more spectra associated with the subject; generate a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands included in the target wavelength range and determined based on the reference spectral data from the compressed image data; A computer program that executes the above.

21. A computer is caused to Obtaining first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength range from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in the target wavelength range; Obtaining reference spectral data including information regarding at least one spectrum; Determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectral data; Generating second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more designated wavelength ranges based on the first mask data; A computer program for causing the above to be executed.

22. For a computer, Obtaining compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in the target wavelength range; Obtaining reference spectral data including information regarding one or more spectra associated with the subject; Displaying, on a display, both a graphical user interface for specifying restoration conditions for generating a plurality of two-dimensional image data corresponding to a plurality of designated wavelength bands from the compressed image data and an image based on the reference spectral data; A computer program for causing the above to be executed.