Signal processing device and signal processing method

Abstract

This signal processing method, which is executed by a computer, includes: acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength region (S101); acquiring reference spectral data including one or more items of spectral information associated with the subject (S102); and generating, from the compressed image data, a plurality of items of two-dimensional image data corresponding to a plurality of designated wavelength bands determined on the basis of the reference spectral data (S105).

Description

[Technical Field]

[0001] This disclosure relates to a signal processing device and a signal processing method. [Background technology]

[0002] By utilizing spectral information from numerous wavelength bands, each with a narrow bandwidth, such as ten or more bands, it becomes possible to understand the detailed physical properties of an object, which was impossible with conventional RGB images that only contain information from three bands. Cameras that acquire images from many wavelength bands include "hyperspectral cameras" and "multispectral cameras." These cameras are used in various fields such as food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Document 1 discloses a hyperspectral camera using a compressed sensing method. Compressed sensing is a technique that reconstructs more data than observed by assuming that the data distribution of the observed object is sparse in a certain space (e.g., frequency space). The estimation calculation that assumes the sparsity of the observed object is called "sparse reconstruction." The hyperspectral camera disclosed in Patent Document 1 acquires a monochrome image through an array of filters whose spectral transmittance has maximum values ​​at multiple wavelengths. The imaging device reconstructs a hyperspectral image from the monochrome image by calculations based on sparse reconstruction.

[0004] Non-patent document 1 discloses an example of a snapshot-type hyperspectral imaging device suitable for observing the fluorescence spectrum emitted from a phosphor. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Patent No. 9599511 [Non-patent literature]

[0006] [Non-Patent Document 1] Amicia D. Elliott et al., "Real-time hyperspectral fluorescence imaging of pancreatic b-cell dynamics with the image mapping spectrometer", Journal of Cell Science 125, 4833-4840 (2012). [Overview of the Initiative]

[0007] According to the imaging device described in Patent Document 1, high-resolution and multi-wavelength video recording is possible. On the other hand, since a highly computationally intensive reconstruction calculation is performed using matrix data of a size equal to the product of the number of pixels and the number of wavelength bands of the image sensor, a processing circuit with high computational power is required.

[0008] This disclosure provides a technique for reducing the load of reconstruction calculations by efficiently generating images of the required wavelength bands.

[0009] A method according to one aspect of the present disclosure is a signal processing method performed by a computer. The method includes obtaining compressed image data which includes two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range; obtaining reference spectral data which includes information of one or more spectra associated with the subject; and generating a plurality of two-dimensional image data from the compressed image data which correspond to a plurality of specified wavelength bands determined based on the reference spectral data.

[0010] Another method according to the present disclosure is a method for generating mask data used to reconstruct spectral image data for each wavelength band from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range. The method includes: obtaining first mask data for reconstructing first spectral image data corresponding to a first group of wavelength bands in the target wavelength range; obtaining reference spectral data including information on at least one spectrum; determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectral data; and generating second mask data for reconstructing second spectral image data corresponding to a second group of wavelength bands in the one or more designated wavelength ranges based on the first mask data.

[0011] A method according to yet another aspect of the present disclosure is a signal processing method performed by a computer. The method includes: obtaining compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range; obtaining reference spectral data including information of one or more spectra associated with the subject; and displaying on a display a graphical user interface for specifying restoration conditions for generating a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands from the compressed image data, and an image based on the reference spectral data.

[0012] The comprehensive or specific embodiments of this disclosure may be implemented as systems, apparatus, methods, integrated circuits, computer programs, or computer-readable recording media, or as any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. Computer-readable recording media include, for example, non-volatile recording media such as CD-ROMs (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. If an apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. In this specification and in the claims, “apparatus” may mean not only a single device but also a system consisting of multiple devices.

[0013] According to one aspect of this disclosure, the load on the reconstruction calculation can be reduced by efficiently generating images of the required wavelength bands. [Brief explanation of the drawing]

[0014] [Figure 1A] Figure 1A is a schematic diagram showing an example of the configuration of an imaging system. [Figure 1B] Figure 1B schematically shows another example configuration of the imaging system. [Figure 1C] Figure 1C schematically shows yet another example of the imaging system configuration. [Figure 1D] Figure 1D schematically shows yet another example configuration of the imaging system. [Figure 2A] Figure 2A is a schematic diagram showing an example of a filter array. [Figure 2B] Figure 2B shows an example of the spatial distribution of light transmittance for each of the multiple wavelength bands included in the target wavelength range. [Figure 2C] Figure 2C shows an example of the spectral transmittance of region A1 included in the filter array shown in Figure 2A. [Figure 2D]FIG. 2D is a diagram showing an example of the spectral transmittance of region A2 included in the filter array shown in FIG. 2A. [Figure 3A] FIG. 3A is a diagram for explaining an example of the relationship between a target wavelength range and a plurality of wavelength bands included therein. [Figure 3B] FIG. 3B is a diagram for explaining another example of the relationship between a target wavelength range and a plurality of wavelength bands included therein. [Figure 4A] FIG. 4A is a diagram for explaining the characteristics of spectral transmittance in a region of a filter array. [Figure 4B] FIG. 4B is a diagram showing a result obtained by averaging the spectral transmittance shown in FIG. 4A for each wavelength band. [Figure 5] FIG. 5 is a block diagram showing a configuration example of a system for reducing the load of arithmetic processing. [Figure 6] FIG. 6 is a diagram showing a modification of the system in FIG. 5. [Figure 7] FIG. 7 is a diagram showing an example of mask data before conversion stored in a memory. [Figure 8A] FIG. 8A is a diagram showing an example of spectra of four types of samples that can be included in a subject. [Figure 8B] FIG. 8B is a diagram showing an example of bands to be combined. [Figure 9] FIG. 9 is a flowchart showing an example of mask data conversion processing. [Figure 10] FIG. 10 is a diagram for explaining an example of a method of combining mask information of a plurality of bands and converting the combined mask information into new mask information. [Figure 11A] FIG. 11A is a diagram showing an example of reference spectra of four types of samples that can be included in a subject. [Figure 11B] FIG. 11B is a diagram showing an example of band combining. [Figure 12] FIG. 12 is a block diagram showing a configuration example of a system that performs labeling. [Figure 13A] FIG. 13A is a diagram showing an example of reference spectra of four types of samples that can be included in a subject. [Figure 13B] Figure 13B shows an example of band synthesis. [Figure 14] Figure 14 shows an example of a graphical user interface (GUI). [Figure 15A] Figure 15A is the first diagram illustrating how to exclude specific bands from the reconstruction. [Figure 15B] Figure 15B is a second diagram illustrating how to exclude specific bands from the reconstruction. [Figure 16] Figure 16 is a flowchart illustrating an example of how a signal processing circuit operates when a specific band is excluded from the reconstruction. [Figure 17] Figure 17 is a flowchart illustrating an example of the operation when reference spectral data is translated into conversion conditions for mask data and sent to a signal processing circuit. [Figure 18] Figure 18 is a schematic diagram showing the configuration of the system of Embodiment 4. [Figure 19] Figure 19 shows an example of the relationship between the excitation light and fluorescence spectra and the excluded bands. [Figure 20A] Figure 20A shows the absorption spectra of five different fluorescent dyes. [Figure 20B] Figure 20B shows the fluorescence spectra of five different fluorescent dyes. [Figure 21A] Figure 21A shows the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the first imaging. [Figure 21B] Figure 21B shows an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and recovery band. [Figure 22A] Figure 22A shows the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the second imaging. [Figure 22B] Figure 22B shows an example of the relationship between each fluorescence spectrum and the cutoff wavelength range and recovery band. [Figure 23A] Figure 23A shows the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the third imaging step. [Figure 23B] Figure 23B shows the relationship between each fluorescence spectrum, the cutoff wavelength range, and the recovery band. [Modes for carrying out the invention]

[0015] The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement, position and connection configurations of components, steps, and order of steps shown in the embodiments below are examples and are not intended to limit the art of this disclosure. Components in the embodiments below that are not described in the independent claim representing the highest-level concept are described as optional components. The figures are schematic and not necessarily strictly illustrative. Furthermore, in each figure, substantially identical or similar components are denoted by the same reference numerals. Duplication of explanation may be omitted or simplified.

[0016] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large-scale integration (LSI). The LSI or IC may be integrated on a single chip or may be composed of multiple chips combined. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSIs or ICs, but the name may change depending on the degree of integration, and they may also be called system LSIs, VLSIs (very large-scale integrations), or ULSIs (ultra-large-scale integrations). Field-programmable gate arrays (FPGAs) that are programmed after the manufacture of the LSI, or reconfigurable logic devices that allow for the reconfiguration of junction relationships within the LSI or the setup of circuit compartments within the LSI, can also be used for the same purpose.

[0017] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROMs, optical disks, or hard disk drives, and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0018] (Knowledge that forms the basis of this disclosure) Before describing embodiments of this disclosure, we will outline the sparsity-based image restoration process and the synthesis and editing processes of the mask data used during restoration.

[0019] Sparsity is the property that the elements characterizing an observed object exist sparsely (sparsely) in a given space (e.g., frequency space). Sparsity is widely observed in nature. By utilizing sparsity, it becomes possible to efficiently observe the necessary information. Sensing technology that utilizes sparsity is called compressed sensing. By using compressed sensing, it is possible to construct highly efficient devices and systems. An example of the application of compressed sensing to a hyperspectral camera is disclosed in Patent Document 1. According to the hyperspectral camera disclosed in Patent Document 1, it is possible to capture high wavelength resolution, high resolution, and multi-wavelength video in a single shot.

[0020] An imaging device utilizing compressed sensing includes, for example, an array of optical filters having random light transmission characteristics with respect to space and / or wavelength. Such an array of optical filters is sometimes called an "encoding mask" or "encoding element." The encoding mask is placed on the optical path of light incident on the image sensor, and transmits light incident from the subject with different light transmission characteristics depending on the region. This process by the encoding mask is called "encoding." The image of light encoded by the encoding mask is captured by the image sensor. The image generated by imaging using the encoding mask is referred to as a "compressed image" in this specification. Mask data indicating the light transmission characteristics of the encoding mask is recorded in advance in a storage device. The processing circuit in the imaging device performs a restoration process based on the compressed image and the mask data. The restoration process generates a restored image that contains more wavelength information than the compressed image. The mask data is, for example, information indicating the spatial distribution of the spectral transmittance of the encoding mask. By performing a restoration process based on such mask data, it is possible to reconstruct images for multiple wavelength bands from a single compressed image.

[0021] The reconstruction process includes estimation calculations that assume the sparsity of the image subject. The calculations performed in sparse reconstruction may be, for example, data estimation calculations by minimizing an evaluation function that incorporates a regularization term such as discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV), as disclosed in Patent Document 1. Such estimation calculations are computationally intensive and use mask data with a size equivalent to the product of the number of pixels and wavelength bands of the image sensor. Therefore, a processing circuit with high computational power is required. If the time required for such computationally intensive calculations is longer than the exposure time during shooting, the calculation time will become the limiting factor in the camera's operating speed (e.g., frame rate).

[0022] In many applications of hyperspectral cameras, such as fluorescence and absorption spectroscopy, the expected spectrum of the subject is known (see, for example, Non-Patent Document 1). When the expected spectrum of the subject is known, the computational load can be reduced by appropriately editing or reducing the mask data used in the reconstruction process. The mask data can be edited or reduced based on information about the wavelength range required for processing such as analysis or classification performed after image reconstruction.

[0023] Based on the above findings, this disclosure discloses a method for reducing the computational load by referencing information about known spectra when the spectra that an object may possess are known. In the following description, the known spectra expected for each substance contained in the object, i.e., the object being observed, are referred to as "reference spectra." Data showing the reference spectra of one or more substances that the object being observed may possess are collectively referred to as "reference spectrum data."

[0024] The embodiments of this disclosure are outlined below.

[0025] A signal processing method according to an exemplary embodiment of the present disclosure includes: acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range; acquiring reference spectral data including information of one or more spectra associated with the subject; and generating a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands determined based on the reference spectral data from the compressed image data.

[0026] "Hyperspectral information in the target wavelength range" refers to information about the spatial distribution of luminance for each of the multiple wavelength bands included in the target wavelength range. "Compressing hyperspectral information" means compressing the information about the spatial distribution of luminance of multiple wavelength bands into a single monochrome two-dimensional image using encoding elements such as a filter array, which will be described later.

[0027] According to the method described above, multiple 2D image data corresponding to multiple specified wavelength bands determined based on reference spectral data are reconstructed from the compressed image data. Therefore, the computational load can be reduced compared to reconstructing 2D image data corresponding to all wavelength bands included in the target wavelength range.

[0028] The one or more spectra may be associated with one or more substances that are assumed to be contained in the subject. For example, data defining the correspondence between substances and spectra may be pre-recorded in a storage medium such as memory. By referring to such data, information on the spectrum corresponding to a substance specified by the user can be easily obtained.

[0029] Each of the plurality of specified wavelength bands may include the peak wavelength of the spectrum associated with one of the one or more substances. This ensures that each specified wavelength band corresponds to one substance, facilitating classification based on the reconstructed image.

[0030] The reference spectral data may include information on multiple spectra associated with multiple types of substances assumed to be contained in the subject. The multiple designated wavelength bands may include a first designated wavelength band that does not overlap among the multiple spectra and a second designated wavelength band that does overlap among the multiple spectra. This facilitates classification based on the reconstructed image even when there is overlap in the spectra of two or more substances.

[0031] In this specification, a specified wavelength band "has no overlap between multiple spectra" means that in that specified wavelength band, one of the multiple spectra has a significant intensity, while the other spectra do not. Conversely, a specified wavelength band "has overlap between multiple spectra" means that two or more spectra in that specified wavelength band have significant intensity. Whether or not there is a "significant intensity" can be determined, for example, based on the integral value obtained by integrating the intensity of each spectrum from the lower limit wavelength to the upper limit wavelength of the specified wavelength band. The largest integral value among the integral values ​​of multiple spectra is the signal, and the sum of the other integral values ​​is the noise N. If the signal-to-noise ratio (S / N ratio), which is the value obtained by dividing the signal S by the noise N, is greater than or equal to a threshold (e.g., 1, 2, or 3), then the specified wavelength band can be determined to have no overlap between multiple spectra. Conversely, if the S / N ratio is less than the threshold, then the specified wavelength band can be determined to have overlap between multiple spectra.

[0032] The compressed image data may be generated using a filter array and an image sensor that include multiple types of optical filters with different spectral transmittances. The method may further include obtaining mask data that reflects the spatial distribution of the spectral transmittances. The multiple two-dimensional image data may be generated based on the compressed image data and the mask data.

[0033] The mask data may include mask matrix information having elements corresponding to the spatial distribution of the transmittance of the filter array for each of a plurality of unit bands included in the target wavelength range. The method may further include generating composite mask information by combining the mask matrix information corresponding to unspecified wavelength bands different from the specified wavelength bands in the target wavelength range, and generating composite image data for the unspecified wavelength bands based on the compressed image data and the composite mask information. As a result, the amount of data is reduced by combining the mask matrix information for the unspecified wavelength bands, and the computational load can be further reduced.

[0034] The generation of the plurality of two-dimensional image data may include generating and outputting the plurality of two-dimensional image data corresponding to the plurality of specified wavelength bands without generating image data corresponding to unspecified wavelength bands different from the specified wavelength bands in the target wavelength range from the compressed image data. In other words, the method does not have to include generating image data corresponding to unspecified wavelength bands different from the specified wavelength bands in the target wavelength range from the compressed image data. As a result, the restoration process is not performed for unspecified wavelength bands of low importance, and the computational load can be further reduced.

[0035] The aforementioned multiple designated wavelength bands may be determined based on the intensity of one or more spectra represented by the reference spectral data, or the derivative of such intensity. For example, each designated wavelength band may be determined to include a peak wavelength where the intensity of the corresponding spectrum peaks. Alternatively, each designated wavelength band may be determined to avoid wavelength ranges where the absolute value of the derivative of the intensity of the corresponding spectrum is close to zero. Such methods allow for the determination of designated wavelength bands that contain characteristic portions of the spectrum, facilitating processing such as classification after reconstruction.

[0036] The reference spectral data may include information on the fluorescence spectra of one or more substances assumed to be contained in the subject. This allows for reconstruction with an appropriate band configuration based on the fluorescence spectra of the fluorescent substances.

[0037] The reference spectral data may include information on the absorption spectra of one or more substances assumed to be contained in the subject. This allows for reconstruction with an appropriate band configuration based on the absorption spectra of the substances.

[0038] The method may further include displaying a graphical user interface (GUI) on a display for the user to specify one or more spectra, or one or more substances associated with one or more spectra. The display may be connected to or mounted on the computer. The reference spectral data may be acquired according to the specified one or more spectra, or the specified one or more substances. This allows for restoration processing with a band configuration corresponding to the spectra or substances specified by the user on the GUI.

[0039] A method according to another embodiment of the present disclosure is a method for generating mask data used to reconstruct spectral image data for each wavelength band from compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range. The method includes: obtaining first mask data for reconstructing first spectral image data corresponding to a first group of wavelength bands in the target wavelength range; obtaining reference spectral data including information on at least one spectrum; determining one or more designated wavelength ranges included in the target wavelength range based on the reference spectral data; and generating second mask data for reconstructing second spectral image data corresponding to a second group of wavelength bands in the one or more designated wavelength ranges based on the first mask data.

[0040] The first wavelength band group may be a collection of all or some of the wavelength bands included in the target wavelength range. For example, the first wavelength band group may be a collection of multiple unit wavelength bands with small bandwidths included in the target wavelength range. The second wavelength band group may be a collection of all or some of the wavelength bands included in the specified wavelength range. For example, the second wavelength band group may be a collection of multiple unit wavelength bands included in the specified wavelength range. Each of the first and second wavelength band groups may be a collection of composite bands formed by combining two or more unit wavelength bands. When such band synthesis is performed, a mask data transformation process is performed according to the manner in which the bands are synthesized.

[0041] According to the above configuration, a second mask data can be generated to reconstruct the second spectral image data corresponding to the second wavelength band group in a specified wavelength range determined based on the reference spectral data. Since the second mask data has a smaller data size than the first mask data, it becomes possible to efficiently reconstruct the image using the second mask data.

[0042] The method for generating the second mask data described above may be performed by an apparatus that reconstructs spectral image data for each wavelength band from compressed image data based on the second mask data, or by other apparatus connected to the said apparatus.

[0043] The compressed image data may be generated using a filter array and an image sensor that include multiple types of optical filters with different spectral transmittances. The first mask data and the second mask data may be data that reflects the spatial distribution of the spectral transmittance of the filter array. The first mask data may include first mask information showing the spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second mask data may include second mask information showing the spatial distribution of the spectral transmittance corresponding to the second wavelength band group.

[0044] The second mask data may further include a third mask information obtained by synthesizing multiple pieces of information. Each of the multiple pieces of information represents the spatial distribution of spectral transmittance in the corresponding wavelength band included in the unspecified wavelength range other than the specified wavelength range within the target wavelength range.

[0045] The second mask data does not necessarily have to include information regarding the spatial distribution of the spectral transmittance in the corresponding wavelength bands included in the non-specified wavelength range other than the specified wavelength range.

[0046] A signal processing method according to yet another embodiment of the present disclosure includes: acquiring compressed image data including two-dimensional image information of a subject obtained by compressing hyperspectral information in a target wavelength range; acquiring reference spectral data including information of one or more spectra associated with the subject; and displaying on a display a graphical user interface for specifying restoration conditions for generating a plurality of two-dimensional image data corresponding to a plurality of specified wavelength bands from the compressed image data, and an image based on the reference spectral data.

[0047] According to the above configuration, the user can specify restoration conditions for generating multiple 2D image data corresponding to multiple specified wavelength bands. This makes it possible to efficiently generate multiple 2D image data corresponding to desired specified wavelength bands.

[0048] This disclosure also includes computer programs for causing a computer to perform each of the above methods. This disclosure also includes a signal processing device comprising a processor for performing each of the above methods and a memory storing a computer program to be performed by the processor.

[0049] The embodiments of this disclosure will be described in more detail below. The embodiments described below are illustrative only, and various modifications or changes are possible in each embodiment.

[0050] (Embodiment 1) <1. Imaging System> First, an example configuration of an imaging system used in an exemplary embodiment of this disclosure will be described.

[0051] Figure 1A is a schematic diagram showing an example of the configuration of an imaging system. This system comprises an imaging device 100 and a processing device 200. The imaging device 100 has a configuration similar to the imaging device disclosed in Patent Document 1. The imaging device 100 comprises an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are arranged on the optical path of light incident from the object 70, which is the subject. In the example in Figure 1A, the filter array 110 is arranged between the optical system 140 and the image sensor 160.

[0052] Figure 1A shows an apple as an example of the object 70. The object 70 is not limited to an apple; it can be any object. The image sensor 160 generates compressed image 10 data, in which information from multiple wavelength bands is compressed as a two-dimensional monochrome image. The processing unit 200 can generate image data for each of the multiple wavelength bands included in a predetermined target wavelength range based on the compressed image 10 data generated by the image sensor 160. These multiple image data, which correspond one-to-one to the multiple wavelength bands generated, are sometimes referred to as "hyperspectral (HS) data cubes" or "hyperspectral image data." Here, the number of wavelength bands included in the target wavelength range is N (where N is an integer of 4 or more). In the following description, the multiple image data, which correspond one-to-one to the multiple wavelength bands generated, are referred to as reconstructed image 20W1, reconstructed image 20W2, ..., reconstructed image 20W N These are sometimes referred to as "hyperspectral images 20". The data of the reconstructed images for each wavelength band may also be referred to as "spectral image data" or simply "spectral images". In this specification, the data or signals of an image, that is, a set of data or signals representing multiple pixel values ​​of multiple pixels contained in an image, may be simply referred to as "image".

[0053] In this disclosure, "target wavelength range" refers to the wavelength range determined by the upper and lower wavelength limits of the wavelength components included in the spectral image output by the system. The target wavelength range may correspond to the wavelength range of light that can be detected by a photodetector such as an image sensor in the system. For example, in the case of a system that images through a bandpass filter that suppresses the transmission of light other than 400-700 nm, the target wavelength range may be 400-700 nm. In the case of a system that further images through a filter that absorbs light in the 500-600 nm range, the target wavelength range may be the combined wavelength range of 400-500 nm and 600-700 nm that can be detected by the photodetector.

[0054] The filter array 110 in this embodiment is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters whose spectral transmittance, i.e., wavelength dependence of light transmittance, differs from one another. The filter array 110 functions as the aforementioned coding mask, modulating the intensity of the incident light for each wavelength and outputting it.

[0055] In the example shown in Figure 1A, the filter array 110 is positioned near or directly above the image sensor 160. Here, "nearby" means that the image of light from the optical system 140 is formed on the surface of the filter array 110 in a reasonably clear state. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0056] The optical system 140 includes at least one lens. In Figure 1A, the optical system 140 is shown as a single lens, but the optical system 140 may be a combination of multiple lenses. The optical system 140 forms an image on the imaging plane of the image sensor 160 via the filter array 110.

[0057] The filter array 110 may be positioned away from the image sensor 160. Figures 1B to 1D show examples of the configuration of an imaging device 100 in which the filter array 110 is positioned away from the image sensor 160. In the example in Figure 1B, the filter array 110 is positioned between the optical system 140 and the image sensor 160, but away from the image sensor 160. In the example in Figure 1C, the filter array 110 is positioned between the object 70 and the optical system 140. In the example in Figure 1D, the imaging device 100 comprises two optical systems 140A and 140B, with the filter array 110 positioned between them. As in these examples, an optical system including one or more lenses may be positioned between the filter array 110 and the image sensor 160.

[0058] The image sensor 160 is a monochrome type photodetector having a plurality of photodetectors (also referred to as "pixels" in this specification) arranged in two dimensions. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The photodetectors include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor may be used. A color type sensor includes a sensor having a filter that transmits red light, a filter that transmits green light, a filter that transmits blue light, a sensor having a filter that transmits red light, a filter that transmits green light, a filter that transmits blue light, and an infrared light filter, or a sensor having a filter that transmits red light, a filter that transmits green light, a filter that transmits blue light, and a filter that transmits white light. By using a color type sensor, the amount of information regarding wavelength can be increased, and the accuracy of reconstructing the hyperspectral image 20 can be improved. The wavelength range to be acquired can be determined arbitrarily and is not limited to the visible wavelength range; it may also include ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength ranges.

[0059] The processing unit 200 may be a computer comprising one or more processors and one or more storage media such as memory. Based on the compressed image 10 acquired by the image sensor 160, the processing unit 200 recovers image 20W1, recovered image 20W2, ... recovered image 20W N Generate data.

[0060] Figure 2A is a schematic diagram showing an example of a filter array 110. The filter array 110 has multiple regions arranged in two dimensions. In this specification, each of these multiple regions may be referred to as a "cell". Each region is equipped with an optical filter having an individually set spectral transmittance. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take values ​​between 0 and 1, inclusive.

[0061] In the example shown in Figure 2A, the filter array 110 has 48 rectangular regions arranged in a 6x8 grid. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be, for example, roughly equivalent to the number of pixels in the image sensor 160. The number of filters included in the filter array 110 is determined according to the application, for example, ranging from tens of thousands to tens of millions.

[0062] Figure 2B shows wavelength bands W1, W2, ..., and W included in the target wavelength range. N This figure shows an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in Figure 2B, the difference in intensity in each region represents the difference in transmittance. Lighter regions have higher transmittance, and darker regions have lower transmittance. As shown in Figure 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0063] Figures 2C and 2D show examples of spectral transmittances for regions A1 and A2, respectively, included in the filter array 110 shown in Figure 2A. The spectral transmittances of region A1 and region A2 are different from each other. Thus, the spectral transmittance of the filter array 110 differs by region. However, it is not necessary for the spectral transmittances of all regions to be different. In the filter array 110, the spectral transmittances of at least some of the regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns for multiple regions included in the filter array 110 may be equal to or greater than the number of wavelength bands N included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0064] Figures 3A and 3B show the target wavelength range W and the wavelength bands W1, W2, ...,2, W2, ..., W1, W2, ..., W2, W2, ..., W2, W2, ..., W2, W2, ..., W2, W2, ..., W2, N This diagram illustrates the relationship. The target wavelength range W can be set to various ranges depending on the application. For example, the target wavelength range W may be the visible light wavelength range from about 400 nm to about 700 nm, the near-infrared wavelength range from about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range from about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. Thus, the wavelength range used is not limited to the visible light range. In this specification, "light" refers to all radiation, including infrared and ultraviolet rays, not just visible light.

[0065] In the example shown in Figure 3A, N is any integer greater than or equal to 4, and the target wavelength range W is divided into N equal parts, with each wavelength range being called wavelength band W1, wavelength band W2, ..., wavelength band W Nas described above. However, the present invention is not limited to such examples. The plurality of wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be made non-uniform depending on the wavelength band. There may be a gap or an overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. As described above, the method of determining the plurality of wavelength bands is arbitrary.

[0066] FIG. 4A is a diagram for explaining spectral transmittance characteristics in a region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has a plurality of local maximum values (that is, from the local maximum value P1 to the local maximum value P5) and a plurality of local minimum values for wavelengths within the target wavelength range W. In the example shown in FIG. 4A, normalization is performed such that the maximum value of light transmittance within the target wavelength range W is 1 and the minimum value is 0. In the example shown in FIG. 4A, the wavelength band W2 and the wavelength band W N-1 have a local maximum value of spectral transmittance in wavelength bands such as these. As described above, the spectral transmittance of each region ranges from the wavelength band W1 to the wavelength band W N can be designed to have local maximum values in at least two of the plurality of wavelength ranges. In the example of FIG. 4A, the local maximum value P1, the local maximum value P3, the local maximum value P4, and the local maximum value P5 are 0.5 or more.

[0067] As described above, the light transmittance of each region differs depending on the wavelength. Therefore, the filter array 110 transmits a large amount of components in a certain wavelength range among incident light, and does not transmit components in other wavelength ranges so much. For example, for light in k wavelength bands among N wavelength bands, the transmittance may be greater than 0.5, and for light in the remaining N−k wavelength ranges, the transmittance may be less than 0.5. k is an integer satisfying 2≦k<N. If the incident light is white light that uniformly contains wavelength components of all visible light, the filter array 110 modulates the incident light for each region into light having a plurality of discrete intensity peaks with respect to wavelength, and superimposes and outputs these multi-wavelength lights.

[0068] FIG. 4B is a diagram illustrating, as an example, a result obtained by averaging the spectral transmittance shown in FIG. 4A for each of wavelength band W1, wavelength band W2, ..., wavelength band W N . The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In the present specification, the transmittance value averaged for each wavelength band in this manner is defined as the transmittance in that wavelength band. In this example, the transmittance is prominently high in three wavelength regions having maximum values P1, P3, and P5. In particular, in the two wavelength regions having maximum values P3 and P5, the transmittance exceeds 0.8.

[0069] In the examples shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed where the transmittance of each region can take any value from 0 to 1. However, it is not absolutely necessary to employ a grayscale transmittance distribution. For example, a binary-scale transmittance distribution may be adopted, in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary-scale transmittance distribution, each region transmits most of the light in at least two wavelength regions among the plurality of wavelength regions included in the target wavelength range, and does not transmit most of the light in the remaining wavelength regions. Here, the term "most" refers to approximately 80% or more.

[0070] Some of all cells, for example half of the cells, may be replaced with transparent regions. Such a transparent region covers all wavelength bands W1 to W N included in the target wavelength range W and transmits the light of each wavelength band at a similarly high transmittance, for example, a transmittance of 80% or higher. In such a configuration, the plurality of transparent regions may be arranged, for example, in a checkerboard pattern. That is, in the two arrangement directions of the plurality of regions in the filter array 110, regions whose light transmittance varies depending on wavelength and transparent regions can be arranged alternately.

[0071] Data indicating the spatial distribution of the spectral transmittance of such a filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium included in the processing device 200. This data is used for arithmetic processing described later.

[0072] The filter array 110 may be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing a metal. When using a multilayer film, for example, a dielectric multilayer film or a multilayer film containing a metal layer may be used. In this case, each cell is formed so that at least one of the thickness, material, and stacking order of each multilayer film is different. This allows for different spectral characteristics depending on the cell. Using a multilayer film enables sharp rise and fall in spectral transmittance. A configuration using an organic material can be achieved by having different pigments or dyes contained in each cell, or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each cell. When using a microstructure containing a metal, it can be fabricated using spectroscopy by the plasmon effect.

[0073] Next, an example of signal processing by the processing unit 200 will be described. The processing unit 200 reconstructs a hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. The generated hyperspectral image 20 contains multiple images. These multiple images correspond to multiple wavelength ranges, and the number of these multiple wavelength ranges is greater than the number of wavelength ranges acquired by a typical color camera (for example, the wavelength range of red light, the wavelength range of green light, and the wavelength range of blue light), which is three. This number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is called the "number of bands". Depending on the application, the number of bands may exceed 100.

[0074] The data we want is the data from the hyperspectral image 20, and we will call this data f. If the number of bands is N, then f is the image data f1 corresponding to wavelength band W1, image data f2 corresponding to wavelength band W2, ..., wavelength band W N Image data f corresponding to NThis is data that includes the following. Here, as shown in Figures 1A to 1D, the horizontal direction of the image is the x-direction and the vertical direction of the image is the y-direction. If the number of pixels in the x-direction of the image data to be obtained is m and the number of pixels in the y-direction is n, then image data f1, image data f2, ..., image data f N Each of these is two-dimensional data with n × m pixels. Therefore, data f is three-dimensional data with n × m × N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube". On the other hand, the number of elements in data g of the compressed image 10, which is obtained by encoding and multiplexing by the filter array 110, is n × m. Data g can be expressed by the following equation (1).

[0075]

number

[0076] Here, f1, f2, ..., f N Each of these is data with n × m elements. Therefore, the vector on the right side is a one-dimensional vector of n × m × N rows and 1 column. The compressed image 10 is transformed into an n × m row and 1 column one-dimensional vector g and represented and computed. The matrix H is the components of the vector f f1, f2, ..., f N This represents a transformation that encodes and intensity modulates the signal using different encoding information (also called "mask information") for each wavelength band, and then adds them together. Therefore, H is an n×m row n×m×N column matrix. The mask information can also be interpreted as the matrix H in equation (1).

[0077] Given a vector g and a matrix H, it appears that f can be calculated by solving the inverse problem of equation (1). However, since the number of elements n × m × N of the data f to be obtained is greater than the number of elements n × m of the acquired data g, this problem is poorly set up and cannot be solved as is. Therefore, the processing unit 200 uses the redundancy of the image contained in the data f and employs a compressed sensing technique to find a solution. Specifically, the data f to be obtained is estimated by solving the following equation (2).

[0078]

number

[0079] Here, f' represents the estimated data for f. The first term in parentheses in the above equation represents the difference between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares may also be used as the residual term. The second term in parentheses is the regularization term or stabilization term. Equation (2) means finding the f that minimizes the sum of the first and second terms. The function in parentheses in equation (2) is called the evaluation function. The processing unit 200 can converge the solution through recursive iterative operations and calculate the f that minimizes the evaluation function as the final solution f'.

[0080] The first term in parentheses in equation (2) represents the operation of calculating the sum of squares of the differences between the acquired data g and Hf, which is obtained by transforming the estimation process f by matrix H. The second term, Φ(f), is a constraint in the regularization of f and is a function that reflects the sparse information of the estimated data. This function has the effect of making the estimated data smoother or more stable. The regularization term can be represented by, for example, the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation allows for obtaining stable estimated data that suppresses the effects of noise in the observed data g. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term may be selected that makes the texture of the object 70 more sparse in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the reduction of redundant data and the higher the compression rate. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to a moderate value that allows f to converge to a certain extent without overcompression.

[0081] The contents of equations (1) and (2)

[0082]

number

[0083] In descriptions related to formulas (1) and (2), this is sometimes denoted as g.

[0084] In the configurations shown in Figures 1B and 1C, the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, by storing this blur information in advance and reflecting it in the matrix H mentioned above, the hyperspectral image 20 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree to which a point image spreads to surrounding pixels. For example, if a point image corresponding to one pixel on the image spreads to a k×k pixel region around that pixel due to blurring, the PSF can be defined as a set of coefficients, i.e., a matrix, that shows the influence on the pixel value of each pixel in that region. By reflecting the effect of blurring of the encoding pattern due to the PSF in the matrix H, the hyperspectral image 20 can be reconstructed. The position where the filter array 110 is placed is arbitrary, but a position can be selected where the encoding pattern of the filter array 110 does not spread too much and disappear.

[0085] <2. Band synthesis process based on reference spectrum> Through the above processing, images of each of the multiple wavelength bands can be reconstructed from the compressed image 10 acquired by the image sensor 160. However, in order to reconstruct images of all wavelength bands included in the target wavelength range, it is necessary to perform calculations using a matrix containing a number of elements equivalent to the product of the number of pixels and the number of wavelength bands of the image sensor 160. This calculation is computationally intensive, and the processing unit 200 is required to have high processing power.

[0086] On the other hand, in some cases, such as fluorescence and absorption spectroscopy, the expected emission or absorption spectrum of the substance being observed is known. In such cases, the computational load can be reduced by editing or reducing the mask data based on the expected known spectrum.

[0087] The following describes an example of how to reduce the computational load by using reference spectral data that shows known spectra. The reference spectra shown below are representative examples and can be modified or altered in various ways. In the following description, as an example, a system is described for imaging a subject containing one or more specific substances (e.g., fluorescent substances) and for analyzing or classifying the substances based on the acquired images.

[0088] Figure 5 is a block diagram illustrating an example of a system configuration for reducing the computational load using reference spectral data. This system comprises an imaging device 100, a processing device 200, a display device 300, and an input user interface (UI) 400. The processing device 200 is an example of a signal processing device in this disclosure.

[0089] The imaging device 100 includes an image sensor 160 and a control circuit 150 that controls the image sensor 160. As shown in Figures 1A to 1D, the imaging device 100 also includes a filter array 110 and at least one optical system 140. The image sensor 160 acquires a compressed image, which is a monochrome image based on light whose intensity is modulated region by the filter array 110. The data of each pixel in the compressed image has information of multiple wavelength bands included in the target wavelength range superimposed on it. Therefore, this compressed image can be said to be a compressed image of hyperspectral information within the target wavelength range. In this specification, data representing the compressed image is referred to as "compressed image data".

[0090] The processing unit 200 comprises a signal processing circuit 250 and a memory 210 such as RAM and ROM. The signal processing circuit 250 may be an integrated circuit equipped with a processor such as a CPU or GPU. The signal processing circuit 250 performs decompression processing based on compressed image data output from the image sensor 160. The memory 210 stores computer programs executed by the processor included in the signal processing circuit 250, various data referenced by the signal processing circuit 250, and various data generated by the signal processing circuit 250. The memory 210 stores mask data that reflects the spatial distribution of spectral transmittance of the filter array 110 in the imaging device 100. The mask data is data that includes information representing the matrix in equations (1) and (2) above, or information for deriving said matrix (hereinafter sometimes referred to as "mask matrix information"). The mask matrix information may be information in matrix form or a matrix-like form having elements corresponding to the spatial distribution of transmittance of the filter array 110 for each of a plurality of unit bands included in the target wavelength range. The mask data is created in advance and stored in the memory 210.

[0091] The display device 300 comprises an image processing circuit 320 and a display 330. The image processing circuit 320 performs the necessary processing on the image restored by the signal processing circuit 250 and then displays it on the display 330. The display 330 may be any display, such as a liquid crystal or organic LED (OLED).

[0092] The input UI 400 includes hardware and software for setting various conditions, such as imaging conditions. The input UI 400 may include input devices such as a keyboard and a mouse. The input UI 400 may also be implemented by a device capable of both input and output, such as a touchscreen. In that case, the touchscreen may also function as the display 330. Imaging conditions may include conditions such as resolution, gain, and exposure time. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 causes the image sensor 160 to perform imaging according to the imaging conditions.

[0093] Memory 410 stores spectral data. The spectral data includes information about the expected spectra of one or more substances that may be contained in the subject. Spectral data is prepared in advance for each substance and recorded in memory 410. Memory 410 may be an external memory or may be built into the imaging device 100. Spectral data may be obtained by downloading it, for example, via a network such as the Internet.

[0094] The user can select specific spectral data as reference spectral data by operating on the input UI 400. For example, by selecting a specific material or substance on the input UI 400, the spectral data corresponding to that material or substance may be determined as reference spectral data. Once the reference spectral data is determined by the user's operation, that reference spectral data is sent to the signal processing circuit 250.

[0095] The signal processing circuit 250 determines the mask data synthesis conditions based on the reference spectral data. The mask data synthesis conditions are the conditions that determine the multiple specified wavelength bands on which the reconstruction process will be performed. In other words, the signal processing circuit 250 determines the multiple specified wavelength bands on which the reconstruction process will be performed based on the reference spectral data. The wavelength range composed of the specified wavelength bands is called the "specified wavelength range". The signal processing circuit 250 may automatically determine the synthesis conditions based on the reference spectral data, or it may determine the synthesis conditions according to the conditions specified by the user using the input UI 400. The synthesis conditions specify which of the multiple unit bands included in the target wavelength range will be synthesized and treated as a single band. Each of the multiple unit bands is a narrowband wavelength band included in the target wavelength range. For example, in the observation of a sample containing one or more substances, multiple unit bands included in a relatively less important wavelength range may be synthesized as a single band. Alternatively, multiple unit bands included in a wavelength range that is assumed to best represent the characteristics of each substance may be synthesized as a single band. The relatively broad band that is synthesized may be referred to as the "synthesized band" in the following description. Furthermore, image data corresponding to the composite band is sometimes called composite image data. The synthesis conditions may include information in wavelength ranges where reconstruction processing is not performed. For example, the computational load can be reduced by not performing reconstruction processing on unit bands included in wavelength ranges of low importance in observation.

[0096] The signal processing circuit 250 converts the mask data to smaller size mask data based on the determined synthesis conditions and the mask data stored in the memory 210. Here, the mask data before conversion is referred to as "first mask data," and the mask data after conversion is referred to as "second mask data." The first mask data may be used to reconstruct first spectral image data corresponding to a first wavelength band group in the target wavelength range. The first wavelength band group may be, for example, a set of multiple unit bands included in the target wavelength range. The first spectral image data may be data containing image information for each unit band included in the first wavelength band group. The second mask data may be used to reconstruct second spectral image data corresponding to a second wavelength band group in one or more specified wavelength ranges. The second wavelength band group may be a set of multiple unit bands included in the specified wavelength range. The second spectral image data may be data containing image information for each unit band included in the specified wavelength range. The first mask data may include first mask information showing the spatial distribution of spectral transmittance corresponding to the first wavelength band group in the filter array 110. The second mask data may include second mask information showing the spatial distribution of spectral transmittance corresponding to the second wavelength band group in the filter array 110. The second mask data may further include third mask information obtained by synthesizing multiple pieces of information corresponding to one or more unspecified wavelength ranges other than the specified wavelength range in the first mask data. Each of the multiple pieces of information in the third mask information may show the spatial distribution of spectral transmittance in the corresponding unit wavelength band included in the unspecified wavelength range. The third mask information can be said to be composite mask information obtained by synthesizing mask matrix information corresponding to the unspecified wavelength range (i.e., unspecified wavelength bands) in the first mask data. In this embodiment, compressed second mask data is generated by synthesizing information of multiple unit bands included in the unspecified wavelength range in the first mask data. The signal processing circuit 250 generates multiple two-dimensional image data corresponding to multiple specified wavelength bands based on the compressed image data and the second mask information in the second mask data.The signal processing circuit 250 further generates one or more composite image data corresponding to one or more unspecified wavelength bands based on the compressed image data and the third mask information (i.e., composite mask information) in the second mask data.

[0097] Specifically, the signal processing circuit 250 compresses the size of the first mask data by processing such as averaging the matrix elements corresponding to the multiple unit bands to be combined. The signal processing circuit 250 uses the converted second mask data and the compressed image data output from the imaging device 100 to perform a reconstruction operation corresponding to equation (2) above. As a result, the signal processing circuit 250 generates a reconstructed image (i.e., a spectral image) for each of the combined bands. The signal processing circuit 250 sends the generated reconstructed image data to the image processing circuit 320. The image processing circuit 320 draws the reconstructed images of each combined band on the display 330. The image processing circuit 320 may, for example, perform processing such as determining the arrangement on the screen, associating each reconstructed image with band information, or coloring according to wavelength, before displaying the reconstructed images on the display 330.

[0098] In this embodiment, the signal processing circuit 250 determines the band synthesis conditions based on reference spectral data, but the disclosure is not limited to this form. Figure 6 shows a modified version of the system in Figure 5. In this modified version, a processor 420 is provided that determines the mask data conversion conditions based on reference spectral data output from the input UI 400. The processor 420 determines the band synthesis conditions based on the reference spectral data and sends this information to the signal processing circuit 250 as mask data conversion conditions. The signal processing circuit 250 reads the necessary unit band information from the memory 210 according to the mask data conversion conditions sent. The signal processing circuit 250 constructs a size-compressed mask data from the read information and generates a reconstructed image using the mask data. This configuration can further reduce the computational load on the processing unit 200. Modifications such as the one shown in Figure 6 can be similarly used in various subsequent embodiments.

[0099] Next, a detailed example of mask data will be described with reference to Figure 7. Figure 7 shows an example of the first mask data before conversion stored in memory 210. In this example, the first mask data includes information indicating the spatial distribution of the transmittance of the filter array 110 for each of a plurality of unit bands included in the target wavelength range. In this example, the first mask data includes mask information for each of a number of unit bands divided into 1 nm intervals, and information regarding the conditions for acquiring the mask information. Each unit band is identified by a lower limit wavelength and an upper limit wavelength. The mask information includes information on the mask image and the background image. The individual mask images shown in Figure 7 are acquired by imaging a background through the filter array 110 with the image sensor 120. The background image shown in Figure 7 is acquired by imaging the same background with the image sensor 120 without passing through the filter array 110. Such mask image and background image information is recorded for each unit band. By dividing the value of each pixel in each mask image by the value of the corresponding pixel in the corresponding background image, a value indicating the transmittance of the corresponding filter can be calculated for each unit band.

[0100] For example, let the multiple unit bands be the first unit band, ~, the kth unit band, ~, and the nth unit band. The image data f1, ~ corresponding to the first unit band, the image data fk, ~ corresponding to the kth unit band, and the image data fN corresponding to the nth unit band are each n × m rows and 1 column data. In this case, equation (1) is

[0101]

number

[0102] The result is as follows. Each of D1, ..., Dk, ..., DN is a submatrix of matrix H and may be an n × m row n × m column diagonal matrix. An example of when each of these submatrixes may be a diagonal matrix may include the case where the crosstalk between pixels (p,q) and pixels (r,s) of the image sensor 160 during the actual calibration when acquiring information about matrix H is determined to be the same as the crosstalk between pixels (p,q) and pixels (r,s) of the image sensor 160 when the end user images the subject 70 (1 ≤ p, r ≤ n, 1 ≤ q, s ≤ m, pixel (p,q) ≠ pixel (r,s)). Whether or not the above-mentioned conditions regarding crosstalk are met may be determined by considering the imaging environment, including the optical lens used during imaging, and may also be determined by considering whether the image quality of each restored image can achieve the end user's objective.

[0103] When light of the kth unit band, which includes light of the kth unit band and does not include light of any other bands, is shone onto a filter array, and the light output from the filter array is incident on an image sensor, the data output by the image sensor (i.e., the data of the mask image) is

[0104]

number

[0105] If we assume that the k-th unit light, which includes light from the k-th unit band and does not include light from any other band, is shone onto the image sensor without passing through a filter array, and the data output by the image sensor (i.e., the background image data) is fk', then equation (4) is

[0106]

number

[0107] This is the result. In other words, when equation (6) is written using the components of the matrix, it becomes equation (7).

[0108]

number

[0109] Therefore, using the data of the mask image and the data of the background image, we can find the diagonal elements hk(1,1), ..., hk(n×m,n×m) of Dk, that is, all the elements of matrix H become known.

[0110] fk'(i,j) is the pixel value of pixel (i,j) in the background image.

[0111]

number

[0112] This can also be the pixel value of pixel (i,j) in the mask pixel. 1 ≤ i ≤ n and 1 ≤ j ≤ m.

[0113] Information regarding acquisition conditions includes exposure time and gain. Information regarding acquisition conditions does not necessarily have to be included in the mask data. In the example in Figure 7, mask image and background image information is recorded for each of the multiple unit bands with a width of 1 nm. The width of each unit band is not limited to 1 nm and can be determined to any value. Also, if the uniformity of the background image is high, the mask data does not need to include background image information. For example, in a configuration in which the image sensor 120 and the filter array 110 are integrated so as to be close to each other, the mask information will almost match the mask image, so the mask data does not need to include background image information.

[0114] The mask data is, for example, the data that defines the matrix H in equation (2) above. The format of the mask data may vary depending on the system configuration. The mask data shown in Figure 7 contains information for calculating the spatial distribution of the spectral transmittance of the filter array 110. The mask data is not limited to this format and may also be data that directly shows the spatial distribution of the spectral transmittance of the filter array 110. For example, the mask data may include a column of values ​​obtained by dividing each pixel value of the mask image shown in Figure 7 by the corresponding pixel value of the background image.

[0115] In this embodiment, the first mask data containing the information shown in Figure 7 is converted into a smaller second mask data. An example of how to convert the mask data is described in more detail below.

[0116] By determining wavelength ranges that are considered to have a small contribution to analysis or classification from the acquired reference spectral data, and then synthesizing the bands with low contributions to perform the reconstruction calculation, the amount of computation can be reduced. The determination of the bands to be synthesized may be done manually by the user or automatically based on the reference spectral data.

[0117] If the bands to be combined are determined manually, the input UI 400 includes a function to allow the user to select the bands to be combined. The input UI 400 may also include a function to allow the user to select restoration conditions such as the wavelength range to be targeted for the restoration calculation, or the wavelength resolution, not limited to the bands to be combined. For example, the display 330 may display spectra corresponding to individual substances (such as phosphors) that may be included in the subject, or a list of the types of individual substances. The user can select a specific combination of spectra from the displayed spectra, or select a specific combination of substances from the list. The user can further select the bands to be combined from the spectra of the selected substances on the UI. In this way, both the reference spectral data and the UI that allows the user to select the restoration conditions for the restoration calculation may be displayed on the display. This allows the user to generate images of the required wavelength bands more efficiently.

[0118] When the bands to be synthesized are automatically selected, the contribution of each band to the analysis or classification can be automatically estimated from the type or combination of substances selected. For example, the signal processing circuit 250 may synthesize bands whose contribution to the analysis or classification is less than a threshold and treat them as a single band. The magnitude of the contribution can be determined, for example, by the signal intensity in the spectrum, the wavelength derivative of the signal intensity, or by pre-training. For example, if it is found that the signal intensity of the spectrum of a substance contained in the subject in a certain band is less than a threshold, then it can be determined that the band has a small contribution to the result of the analysis or classification. Also, if the absolute value of the wavelength derivative of the signal intensity is extremely small across multiple consecutive bands, that is, if the signal intensity is almost the same in multiple consecutive bands, then no loss of information occurs by synthesizing the relevant bands. For this reason, such multiple bands can be synthesized without affecting the result of the analysis or classification. Pre-training is learning based on statistical methods such as principal component analysis or regression analysis. One possible method for pre-training is to refer to a database that records the "wavelength range used for classification" for each substance, and then determine from the reference spectrum that wavelength ranges that do not fall under the "wavelength range used for classification" as "wavelength ranges with low contribution."

[0119] Figures 8A and 8B illustrate a method for estimating bands with low contribution to the calculation based on a reference spectrum and determining the band synthesis conditions. Figure 8A shows example spectra of four types of samples 31, 32, 33, and 34 that may be included in the subject. Figure 8B shows an example of bands to be synthesized. Given reference spectral data as shown in Figure 8A, bands with low signal intensity from all reference spectra can be synthesized as shown in Figure 8B. This process allows for the acquisition of a reconstructed image with sufficient wavelength resolution for bands with relatively high signal intensity in the reference spectrum, reducing the computational load without affecting the accuracy of analysis or classification. In this example, wavelength bands with a high contribution to the analysis or classification are selected as "specified wavelength bands." Wavelength bands with a low contribution to the analysis or classification are selected as "unspecified wavelength bands."

[0120] Figure 9 is a flowchart illustrating an example of the mask data conversion process performed by the signal processing circuit 250 shown in Figure 5. In step S101, the signal processing circuit 250 acquires compressed image data generated by the imaging device 100. In step S102, the signal processing circuit 250 acquires reference spectral data from the input UI 400. In step S103, the signal processing circuit 250 determines whether or not to perform the mask data conversion process based on the reference spectral data. If it is determined to be necessary, the signal processing circuit 250 converts the mask data based on the reference spectral data. The decision of whether or not to perform the mask data conversion process may be based on whether or not there are bands that are estimated to have a small contribution in the analysis or classification, as described above. Alternatively, if the user determines the bands to be combined, the signal processing circuit 250 determines whether or not to convert the mask data based on the user's input. If there are bands to be combined, the process proceeds to step S104, and the signal processing circuit 250 performs the conversion process. If there are no bands to be combined, the conversion process is omitted. In the subsequent step S105, the signal processing circuit 250 performs the restoration calculation shown in equation (2) above based on the compressed image and mask data, and generates a restored image for each band. If the mask data was converted in step S104, the signal processing circuit 250 performs the restoration process using the converted mask data. The restored image is sent to the display device 300, and after necessary processing by the image processing circuit 320, it is displayed on the display 330. Note that the acquisition of the compressed image in step S101 may be performed at any time before step S105.

[0121] Figure 10 illustrates an example of a method for combining mask information from multiple bands to convert it into new mask information. In this example, mask information for unit bands #1 to #20 is pre-stored in memory 210 as mask information before conversion. In the example in Figure 10, no combining process is performed for unit bands #1 to #5, while combining is performed for unit bands #6 to #20. For unit bands #1 to #5, the transmittance distribution of the filter array 110 is calculated by dividing the value of each region in the mask image by the value of the corresponding region in the background image. Here, the data of each mask image stored in memory 210 is referred to as "unit mask image data," and the data of each background image stored is referred to as "unit background image data." For bands #6 to #20, the combined transmittance distribution is obtained by dividing the data obtained by summing the unit mask image data of bands #6 to #20 for each pixel by the data obtained by summing the unit background image data of bands #6 to #20 for each pixel. In this example, the matrix H' corresponding to the matrix H shown in equation (1) is H'=(F1F2···F20), where F1, F2, ···, and F20 are submatrices of matrix H' and may be n×m rows and n×m columns diagonal matrices. By performing this operation, mask information can be synthesized for any multiple bands. Also, if the uniformity of the background image is very high, the mask information will almost match the mask image. In this case, data obtained by summing or averaging the mask image data of bands #6 to 20 may be used as the synthesized mask data for bands #6 to 20. In this case, matrix H' is changed to matrix H''. Matrix H'' is H''=(F1F2···F5F'),

[0122]

number

[0123] i=1,..., 20.

[0124] h'(1,1)=(h6(1,1)+···+h20(1,1) / 15, ... h'(n×m,n×m)=(h6(n×m,n×m)+···+h20(n×m,n×m) / 15 That's fine.

[0125] By performing the band synthesis process described above, while restoring all unit bands requires calculations for 20 bands, synthesizing bands #6 through #20 requires calculations for only 6 bands. Even with this synthesis, it is possible to restore the image of each band while maintaining the wavelength resolution in the wavelength range of bands #1 through #5. This reduces the amount of computation required.

[0126] The mask data conversion process by synthesis may be performed in the environment used by the end user, or it may be performed at the manufacturing site, such as a factory that manufactures the system or device. When the mask data conversion process is performed at the manufacturing site, the converted second mask data is stored in the memory 210 in place of, or in addition to, the first mask data before conversion during the manufacturing process. In this case, when the end user uses the system, the signal processing circuit 250 can perform a restoration process using the pre-stored converted mask data in response to the user's input. This further reduces the processing load.

[0127] (Embodiment 2) Next, a second embodiment will be described. In Embodiment 1, the computational load of the reconstruction process is reduced by combining multiple unit bands with low contribution to analysis or classification into a single band. In contrast, in this embodiment, the signal processing circuit 250 performs band synthesis such that the reconstructed image becomes the classification image when the overlap between multiple reference spectra is considered to be small. This further reduces the burden on signal processing.

[0128] Figures 11A and 11B illustrate the band synthesis process in this embodiment. Figure 11A shows examples of reference spectra for four types of samples 31, 32, 33, and 34 that may be included in the subject. Figure 11B shows an example of band synthesis. As shown in Figure 11A, when the overlap between reference spectra is small, it is effective to perform band synthesis according to the peaks of each spectrum. This makes it possible to create a situation where each reconstructed image shows a substance corresponding to approximately one type of spectrum. As shown in Figure 11B, when band synthesis is performed, sample 31 substantially has signal intensity in the image corresponding to band #1. Therefore, the reconstructed image corresponding to band #1 can be treated as the classification image of sample #1. Similarly, the reconstructed image of band #2 can be treated as the classification image of sample 32, the reconstructed image of band #3 as the classification image of sample 33, and the reconstructed image of band #4 as the classification image of sample #4. If such band synthesis is not performed, classification processing will be performed after the reconstruction process, but with this method, classification processing can be performed all at once during the reconstruction process. This greatly reduces the burden of signal processing.

[0129] The degree of overlap between reference spectra can be determined, for example, by the following method. For example, consider the wavelength range from wavelength λ1 to λ2. When the values ​​of each reference spectrum are integrated from λ1 to λ2, the integral value of the reference spectrum with the largest integral value is taken as signal S, and the sum of the integral values ​​of the other reference spectra is taken as noise N. The degree of overlap between reference spectra can be determined by the signal-to-noise ratio (S / N ratio) obtained by dividing signal S by noise N. For example, if the S / N ratio is lower than 1, the overlap is judged to be large, and it can be judged that there is overlap between the reference spectra. Conversely, if the S / N ratio is 1 or greater, the overlap is judged to be small, and it can be judged that there is no overlap between the reference spectra. Alternatively, if the S / N ratio is 2 or greater, it can be judged that there is no overlap between the reference spectra, and if the S / N ratio is less than 2, it can be judged that there is overlap between the reference spectra.

[0130] In this example, the signal processing circuit 250 determines multiple specified wavelength bands such that each specified wavelength band does not overlap with the reference spectrum, and combines the multiple unit bands contained in each specified wavelength band into a single band. Each of the multiple specified wavelength bands in this example contains the peak wavelength of the spectrum associated with one of the multiple substances.

[0131] Alternatively, each reference spectrum may be displayed on the input UI400, allowing the user to determine the range of bands to be combined by performing operations such as moving the band edges. The signal-to-noise ratio may also be displayed on the screen to assist the user's decision.

[0132] In this example, the signal processing circuit 250 generates compressed second mask data by processing such as averaging multiple elements corresponding to each specified wavelength band in the first mask data. Based on the compressed image data and the second mask data, the signal processing circuit 250 generates image data corresponding to each specified wavelength band by performing an operation equivalent to equation (2) described above. By using compressed second mask data, the load of the restoration operation can be greatly reduced.

[0133] An example of a case where the overlap between reference spectra is small is the relationship between the excitation light spectrum and the fluorescence spectrum during fluorescence observation. According to this embodiment, observation can be performed under conditions in which the reconstructed image is directly separated into the excitation light image and the fluorescence image.

[0134] If the overlap between reference spectra is considered small, the content entered in input UI400 can also be used for labeling the reconstructed image. Here, "labeling" refers to associating a known substance name or a classification code with a reconstructed image corresponding to a certain region in the reconstructed image, or a band in the reconstructed image that has a biased signal intensity in a certain region.

[0135] Figure 12 is a block diagram showing an example of a system configuration for labeling. In the example in Figure 12, the input UI 400 determines the labeling conditions corresponding to the substance or spectrum selected by the user and sends this information to the memory 310 of the display device 300. The image processing circuit 320 of the display device 300 performs labeling processing on each restored image as necessary according to the sent labeling conditions and displays the labeled restored image on the display 330.

[0136] Labeling can take various forms and methods. Here, we will explain an example of labeling processing using the relationship between the excitation light and fluorescence spectra during fluorescence observation as an example. Consider a case where band synthesis is performed such that the reconstructed image of a certain band X becomes the excitation light image, and the reconstructed image of another band Y becomes the fluorescence image. In this case, the input UI400 can identify that band X corresponds to the excitation light and band Y corresponds to a specific phosphor, based on the band synthesis conditions determined from the reference spectral data. The input UI400 can determine labeling conditions, for example, to assign the name or classification code of the phosphor to the reconstructed image of band Y or a specific region of said reconstructed image. Bands determined to be in the wavelength range of the excitation light may be assigned a name or classification code such as "excitation light band". Regarding the content of the labeling, information such as names entered by the user in the input UI400 may be used for labeling. Labeling may also be performed automatically based on known physical property information.

[0137] Figures 13A and 13B show examples of reference spectra for four samples 31 to 34, where the spectra overlap significantly. In cases of significant overlap between reference spectra, as in these examples, band synthesis that results in a matching reconstructed image and a classification image is not possible. However, as shown in Figure 13B, band synthesis can reduce the overlap for some spectra. For bands and samples with little or no spectral overlap, classification is possible during reconstruction. For bands and samples with significant overlap, it is possible to narrow down the substances that are expected to be contained within that band. Displaying the substances that may be contained within the band can aid the observer's understanding.

[0138] Thus, multiple designated wavelength bands may include one or more first designated wavelength bands that do not overlap among the multiple spectra, and one or more second designated wavelength bands that do overlap among the multiple spectra. In the example in Figure 13B, bands #1, #3, and #4 correspond to the first designated wavelength bands, and bands #X and #Y correspond to the second designated wavelength bands. For each first designated wavelength band, classification is possible at the same time as reconstruction. For each second designated wavelength band, it is possible to narrow down the number of substances that are expected to be included in that band at the same time as reconstruction.

[0139] Figure 14 shows an example of a graphical user interface (GUI) that enables the operations performed in this embodiment. This GUI displays information on the compressed image, the reconstructed image, the reference spectrum and band synthesis, and a table showing the correspondence between bands and samples (i.e., substances). If necessary, displays related to the reconstruction of the hyperspectral image and / or displays related to the analysis of the hyperspectral image and / or displays of the reconstructed hyperspectral image may be added or removed. As described above, when the overlap between reference spectra is small, band synthesis is possible such that there is a one-to-one correspondence between band numbers and classifications. The display section for "Reference Spectrum and Band Synthesis Information" can also display the results of band synthesis as shown in Figure 10, or the results of band synthesis in which some bands are excluded from the reconstruction, as shown in Figure 15B, which will be described later.

[0140] (Embodiment 3) Next, a third embodiment of the present disclosure will be described. In this embodiment, the load of the reconstruction calculation is further reduced by excluding bands that are considered to be of low importance from the bands included in the target wavelength range from the reconstruction target.

[0141] Normally, the reconstruction calculation is performed using information from all bands included in the target wavelength range. If even one band included in the target wavelength range is not used in the reconstruction calculation, the relationship g=Hf in equation (1) above will not be satisfied. In that case, the optical signal of wavelengths belonging to the excluded band will be assigned to other bands as noise, resulting in reconstruction errors that reduce the accuracy of subsequent analysis or classification. However, in cases where the spectrum of the observed object is known, such as in fluorescence observation, depending on the combination of observed objects, there may be bands within the target wavelength range that are predicted to have zero or very low signal intensity. If the signal intensity of a band included in the target wavelength range is zero or very low, the reconstruction error caused by excluding that band from the reconstruction calculation is zero or very small. Even if such a band is excluded, it will not substantially affect subsequent analysis or classification. Therefore, if the signal intensity emitted from an observed object in a band included in the target wavelength range is predicted to be zero or very low, that band can be excluded from the reconstruction calculation. This can also be explained as follows: The equation that is satisfied when reconstructing all bands included in the target wavelength range is g=Hf, as mentioned above. Let g'=H'f' be the equation that is satisfied when some bands are excluded, where H'=H-ΔH and f'=f-Δf. ΔH and Δf represent the elements corresponding to the excluded bands. Since Δf is 0 or very small, g'=H'f'=H'f approximately holds. Therefore, the reconstruction operation expressed as g'=H'f, i.e., the reconstruction operation with some bands excluded, is valid.

[0142] Figures 15A and 15B illustrate how to exclude specific bands from reconstruction based on reference spectral data. If four samples 31, 32, 33, and 34 with spectra as shown in Figure 15A are assumed to be present within the imaging area, then there is a band between sample 32 and sample 33 at a wavelength where all samples are expected to have no signal intensity, i.e., a completely dark image. Such bands, which are expected to have no signal intensity, can be excluded from reconstruction, as shown in Figure 15B. Reducing the number of bands to be reconstructed reduces the signal processing load.

[0143] Figure 16 is a flowchart illustrating an example of the operation of the signal processing circuit 250 when a specific band is excluded from reconstruction. The flowchart in Figure 16 replaces the synthesis process of a specific band (steps S103 and S104) in the flowchart in Figure 9 with a process to delete information about a specific band (steps S203 and S204). In step S203, the signal processing circuit 250 determines whether or not to delete information about a specific band from the mask data. Based on the reference spectral data, if there is a band for all assumed samples that is expected to output an image with no signal intensity, the signal processing circuit 250 proceeds to step S204 and deletes the information about that band from the mask data. If there is no band for which an image with no signal intensity is expected, step S204 is omitted. This operation reduces the computational load of the reconstruction calculation process in step S105 and shortens the calculation time.

[0144] Figure 17 is a flowchart illustrating an example of the operation when reference spectral data is translated into conversion conditions for mask data and sent to the signal processing circuit 250, as shown in the example in Figure 6. This flowchart replaces steps S102, S103, and S104 in the flowchart of Figure 9 with steps S302 and S303. In the example in Figure 17, in step S302, the signal processing circuit 250 obtains information on the conversion conditions for mask data, i.e., the bands to be used for reconstruction, from the processor 420 shown in Figure 6. Based on this information, it can determine specific bands that will not be used for reconstruction. In the subsequent step S303, the signal processing circuit 250 obtains mask data other than the specific band from the memory 210. That is, the mask data of the specific band to be deleted is not read. Therefore, the processing steps can be reduced compared to the example in Figure 16, where the mask data of all bands is read first and then the mask data of the specific band is deleted. In this case, the reference spectral data does not need to be stored in the memory 410. For example, the name of the material or substance input or selected in the input UI 400 may be associated with the conversion conditions. Such associations allow the processor 420 or signal processing circuit 250 to determine which bands to exclude without referring to the reference spectral data. Alternatively, labeling information, such as the substance name corresponding to each spectrum, may be retrieved from the reference spectral data stored in the memory 410, and this labeling information may be used to label each reconstructed image.

[0145] (Embodiment 4) Next, a fourth embodiment of the present disclosure will be described. This embodiment relates to a system for performing fluorescence observation.

[0146] Figure 18 is a schematic diagram showing the configuration of the system of this embodiment. This system comprises a light source 610, an optical system 620, and a detector 630. The optical system 620 includes an interference filter 621, a dichroic mirror 622, an objective lens 623, and a long-pass filter 624. The light source 610 may include a laser light-emitting element that emits excitation light. The detector 630 comprises the imaging device 100 described above. A sample 80 containing a fluorescent material is irradiated with excitation light, and the generated fluorescence is detected by the detector 630.

[0147] Excitation light emitted from the light source 610 passes through an interference filter 621 that selectively transmits light of a specific wavelength that excites the fluorescent material, and then enters the dichroic mirror 622. The dichroic mirror 622 reflects light in a certain wavelength range, including the wavelength of the excitation light, and transmits light in other wavelength ranges. The excitation light reflected by the dichroic mirror 622 enters the sample 80 through the objective lens 623. The sample 80, upon receiving the excitation light, emits fluorescence. The fluorescence is detected by the detector 630 after passing through the dichroic mirror 622 and the long-pass filter 624. A portion of the excitation light incident on the sample 80 is reflected. Most of the excitation light reflected by the sample 80 is reflected by the dichroic mirror 622, but some passes through the dichroic mirror and goes to the detector 630. Although not much excitation light passes through the dichroic mirror 622, the excitation light typically has an intensity several orders of magnitude higher than fluorescence. Therefore, if the excitation light enters the detector 630 along with the fluorescence, the charge in the image sensor may saturate, potentially hindering fluorescence observation. To suppress this phenomenon, a long-pass filter 624 is placed in front of the detector 630, and the optical system 620 is constructed so that the excitation light does not enter the detector 630. The long-pass filter 624 is used because, in fluorescence observation, the excitation light has higher energy, i.e., a shorter wavelength, than the fluorescence.

[0148] The detector 630 may be a hyperspectral camera including, for example, the imaging device 100 and the processing device 200 in Embodiment 2. As described above, the detector 630 performs a restoration process based on mask data from which information of unwanted bands corresponding to the excitation light has been excluded.

[0149] Figure 19 shows an example of the relationship between the excitation light and fluorescence spectra and the excluded bands. In this example, sample 80 contains multiple types of fluorescent materials. The fluorescence spectra emitted from these fluorescent materials are different from each other. Since the energy of the excitation light is higher than the energy of the fluorescence, the wavelength of the excitation light is shorter than the wavelength of the fluorescence. In the example in Figure 19, due to the characteristics of the dichroic mirror 622 and the long-pass filter 624, the wavelength range of the incident light is actually narrower than the target wavelength range that the detector 630 can detect. In this embodiment, the optical system 620 is constructed so that the dichroic mirror 622 and the long-pass filter 624 do not transmit short-wavelength light in order to cut off the excitation light. In this case, even if the target wavelength range of the detector 630 and the cut-off wavelength range overlap, it is known in advance that the cut-off wavelength range has no signal intensity. That is, it is known in advance that the aforementioned equation g'=H'f'=H'f is satisfied. Therefore, the band in that wavelength range can be excluded from the reconstruction calculation.

[0150] The characteristics of the long-pass filter 624 and the dichroic mirror 622 are selected based on the fluorescence to be observed and the wavelength of the excitation light used. Therefore, the configuration of this embodiment, which allows for the selection and exclusion of any band from the reconstruction calculation, enables observation with the minimum necessary computational processing depending on the object being observed.

[0151] (Examples) An example of applying the method according to the embodiments of this disclosure to the m-FISH method, a technique for fluorescence observation, will be described.

[0152] FISH (fluorescent in-situ hybridization) is a method that identifies the hybridized location or chromosome by labeling probes with complementary gene sequences to specific gene sequences with fluorescent dyes. In m-FISH (Multicolor FISH), multiple probes labeled with different fluorescent dyes are used simultaneously.

[0153] The m-FISH method is used for testing some cancers such as leukemia and congenital genetic abnormalities. For example, the m-FISH probe from Cambio is designed using five types of fluorescent dyes such that the attachment ratio of the five fluorescent dyes differs for each chromosome number in humans and mice. Therefore, by staining a set of chromosomes with this probe, the number of each chromosome can be identified.

[0154] When there is no translocation that causes cancer or congenital genetic abnormalities, the entire of each chromosome exhibits a single fluorescence spectrum. However, when a translocation has occurred, the fluorescence spectrum varies depending on the portion of the chromosome. This property can be utilized to detect translocations.

[0155] Figure 20A shows the absorption spectra of five types of fluorescent dyes (Cy3, Cy3.5, Cy5, FITC, DEAC) used in fluorescent labeling for the m-FISH method manufactured by Cambio. Figure 20B shows the fluorescence spectrum of each of these fluorescent dyes. As is clear from Figure 20A, there is no single wavelength that can simultaneously induce fluorescence from all fluorescent dyes. Therefore, the distribution of the five types of fluorescent dyes can be identified by the following procedure.

[0156] <STEP1: Excitation by a first wavelength and hyperspectral imaging> First imaging will be described with reference to Figures 21A and 21B. Figure 21A is a diagram showing the relationship between excitation wavelength and the absorption spectra of fluorescent dyes. The solid curves show the absorption spectra of two dyes (DEAC and FITC) in which fluorescence is induced. The dotted curves show the absorption spectra of three dyes (Cy3, Cy3.5, Cy5) in which fluorescence is not induced because sufficient absorption does not occur at the excitation wavelength. Figure 21B is a diagram showing an example of the relationship between each fluorescence spectrum, the cutoff wavelength range, and the restoration band.

[0157] In this example, the system configuration shown in FIG. 18 is used. The wavelength of the excitation light is set to 405 nanometers (nm). The cutoff wavelengths of the dichroic mirror 622 and the long-pass filter 624 are set to 450 nm. That is, light with a wavelength of 450 nm or more enters the detector 630. In this example, the wavelength range shorter than 450 nm is the cutoff wavelength range. The wavelength of 405 nm of the excitation light is defined as the first wavelength.

[0158] From the absorption spectra of the fluorescent dyes, it can be seen that when irradiated with excitation light of 405 nm, fluorescence of the dyes FITC and DEAC is induced. Since light in the wavelength range of 450 nm or less is blocked by the dichroic mirror 622 and the long-pass filter 624, light in this wavelength range does not enter the detector 630. Further, since the absorption spectra of the fluorescent dyes Cy3, Cy3.5, and Cy5 are longer than the excitation wavelength, no fluorescence is induced from the fluorescent dyes Cy3, Cy3.5, and Cy5. Therefore, for example, there is no fluorescence at a wavelength of 650 nm or more, and the output image is completely dark.

[0159] Therefore, it is effective to perform restoration by, for example, setting a wavelength range from 450 nm to 500 nm as the first band, a wavelength range from 500 nm to 550 nm as the second band, and a wavelength range from 550 nm to 650 nm as the third band. This makes it possible to distinguish between the fluorescence spectra of FITC and DEAC that emit light under this condition, and to obtain the respective distributions.

[0160] <STEP2:Excitation with the second wavelength and hyperspectral imaging> The second imaging will be described with reference to FIGS. 22A and 22B. FIG. 22A is a diagram showing the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the second imaging. The solid line curve shows the absorption spectrum of the dye (Cy5) in which fluorescence is induced. The dotted line curves show the absorption spectra of the remaining four dyes in which fluorescence is not induced because sufficient absorption does not occur at the excitation wavelength. FIG. 22B is a diagram showing an example of the relationship between each fluorescence spectrum, the cutoff wavelength range, and the restoration bands.

[0161] In the second imaging, the wavelength of excitation light is set to the second wavelength of 633 nm, and the cutoff wavelengths of the dichroic mirror 622 and the long-pass filter 624 are set to 650 nm. That is, light with a wavelength of 650 nm or more enters the detector 630.

[0162] From the absorption spectrum of the fluorescent dye shown in FIG. 22A, it can be seen that when 633 nm excitation light is irradiated, fluorescence of the dye Cy5 is efficiently induced. In this case, the distribution of the dye Cy5 can be identified from the image output from the detector 630 without performing spectral decomposition. That is, restoration arithmetic processing can be omitted.

[0163] <STEP3:Excitation by the third wavelength and hyperspectral imaging> The third imaging will be described with reference to FIGS. 23A and 23B. FIG. 23A is a diagram showing the relationship between the excitation wavelength and the absorption spectrum of the fluorescent dye in the third imaging. The solid curve shows the absorption spectra of two types of dyes (Cy3, Cy3.5) in which fluorescence is induced. The dashed and dotted curves show the absorption spectra of the remaining three types of dyes in which fluorescence is not induced because sufficient absorption does not occur at the excitation wavelength. FIG. 23B is a diagram showing the relationship between each fluorescence spectrum, the cutoff wavelength range, and the restoration band.

[0164] In the third imaging, the wavelength of excitation light is set to the third wavelength of 532 nm, and the cutoff wavelengths of the dichroic mirror 622 and the long-pass filter 624 are set to 550 nm. That is, light with a wavelength of 550 nm or more enters the detector 630.

[0165] From the absorption spectrum of the fluorescent dye shown in FIG. 23A, it can be seen that when 532 nm excitation light is irradiated, fluorescence of the dyes Cy3 and Cy3.5 is efficiently induced. Note that since the dye FITC and the dye Cy5 also have slight absorption, these dyes may also emit weak fluorescence.

[0166] In this example, the restoration band is set as follows. • First band: Wavelengths from 550nm to 575nm.

[0167] This band primarily contains fluorescence from Cy3 and Cy3.5, with a small amount of FITC fluorescence potentially present. • Second band: Wavelengths from 575nm to 625nm.

[0168] This band primarily contains fluorescence from Cy3 and Cy3.5, with a small amount of FITC fluorescence potentially present. • Third band: Wavelengths from 625nm to 650nm.

[0169] This band primarily contains fluorescence from Cy3 and Cy3.5, with a small amount of fluorescence from FITC and Cy5 potentially present. • Fourth band: Wavelengths from 650 nm to 800 nm. This band mainly contains fluorescence from Cy3 and Cy3.5, with a small amount of fluorescence from Cy5 potentially present.

[0170] Of these components, the distribution of FITC was identified in STEP 1, and the distribution of Cy5 was identified in STEP 2. The fluorescence of Cy3 and Cy3.5 is present in all four bands from the first to the fourth. However, the fluorescence intensity ratio of these fluorescent dyes in each band is known from the emission spectra of the fluorescent dyes. Therefore, the distributions of Cy3 and Cy3.5 can be determined by solving simultaneous equations related to intensity or by simulating the distribution of dyes that reproduce the imaging results.

[0171] Thus, even when labeled with multiple fluorescent dyes, it is possible to restrict the excitation light spectrum to cause some of the fluorescent dyes to emit light in each measurement. By selecting a reconstruction band accordingly, the distribution of each fluorescent dye can be identified with less computation.

[0172] (Other 1) Modifications of the embodiments of this disclosure may include those shown below.

[0173] A signal processing method performed by a computer, Based on the first instruction, perform the first process. Based on the second instruction, perform the second process. Based on the third instruction, the third process is carried out. The first process is, (a-1) Receiving multiple first pixel values, The image sensor outputs the plurality of first pixel values ​​corresponding to the first light from the filter array, and the first light corresponds to the second light from the first subject incident on the filter array. (a-2) Based on the first matrix and the plurality of first pixel values, the process includes generating a plurality of pixel values ​​I(11), ~ of an image corresponding to the first wavelength range of the first subject, and a plurality of pixel values ​​I(1p) of an image corresponding to the p wavelength range of the first subject, The aforementioned plurality of first pixel values ​​correspond to a plurality of first pixels arranged in an m x n grid, the first matrix is ​​(A1A2···Ap), the first matrix contains a plurality of first submatrices, the plurality of first submatrices are A1, A2, ~, Ap, the plurality of first submatrices contains the q-th submatrice Aq, the plurality of first submatrices contains a plurality of second submatrices, the plurality of second submatrices are the r-th submatrice Ar, ~, the (r+s)-th submatrice A(r+s), where p, q, r, and s are natural numbers, q <rまたは(r+s)<q、1≦q≦p、1≦r≦p、1≦r+s≦pであり、 The second process is, (b-1) Receive multiple second pixel values, The image sensor outputs the plurality of second pixel values ​​corresponding to the third light from the filter array, and the third light corresponds to the fourth light from the second subject incident on the filter array. (b-2) The process includes generating a plurality of pixel values ​​I(2q) of an image corresponding to the q wavelength range of the second subject based on the q submatrix Aq and a plurality of second pixel values, The aforementioned plurality of second pixel values ​​correspond to a plurality of second pixels arranged in an m x n column pattern, Based on the plurality of second submatrices and the plurality of second pixel values, a plurality of pixel values ​​I(2r), ~ of the image corresponding to the r wavelength range of the second subject, and a plurality of pixel values ​​I(2(r+s)) of the image corresponding to the (r+s) wavelength range of the second subject are not generated. The third process described above is: (c-1) Receive multiple third pixel values, The image sensor outputs the plurality of third pixel values ​​corresponding to the fifth light from the filter array, and the fifth light corresponds to the sixth light from the third subject incident on the filter array. (c-2) Based on the q submatrix Aq and the plurality of third pixel values, a plurality of pixel values ​​I(3q) of the image corresponding to the q wavelength range of the third subject are generated, (c-3) The process includes generating a plurality of pixel values ​​of an image I3c corresponding to the r wavelength range to the (r+s) wavelength range of the third subject, based on a plurality of third pixel values ​​and a second matrix generated based on the plurality of second submatrices, The aforementioned plurality of third pixel values ​​correspond to a plurality of third pixels arranged in an m x n column pattern, Based on the plurality of second submatrices and the plurality of third pixel values, a plurality of pixel values ​​I(3r), ~ of the image corresponding to the r wavelength range of the third subject, and a plurality of pixel values ​​I(3(r+s)) of the image corresponding to the (r+s) wavelength range of the third subject are not generated. Signal processing method.

[0174] <Details of the signal processing method described above> p, q, r, and s are natural numbers, and q <rまたは(r+s)<q、1≦q≦p、1≦r≦p、1≦r+s≦pである。

[0175] Although equations (1) and (2) are written in terms of n × m, we may substitute n × m with m × n in equations (1) and (2) here.

[0176] The first, second, and third instructions may each be given by the user using the input UI 400.

[0177] The first instruction is to generate an image corresponding to the first wavelength range of the first subject, ~, and an image corresponding to the p-th wavelength range of the first subject.

[0178] The second instruction includes instructions to generate an image corresponding to the q wavelength range of the second subject, instructions not to generate an image corresponding to the r wavelength range of the second subject, ~, and instructions not to generate an image corresponding to the (r+s) wavelength range of the second subject.

[0179] The third instruction includes an instruction to generate an image corresponding to the q wavelength range of the third subject, an instruction to generate an image corresponding to the r wavelength range to the (r+s) wavelength range of the third subject, an instruction not to generate an image corresponding to the r wavelength range of the third subject, and an instruction not to generate an image corresponding to the (r+s) wavelength range of the third subject.

[0180] The first process includes processes (a-1) and (a-2).

[0181] <Processing (a-1)> The signal processing device 250 receives multiple first pixel values ​​from the image sensor 160. Second light from the first subject enters the filter array 110. In response to this entry, the filter array 110 outputs first light. The image sensor 160 outputs multiple first pixel values ​​in response to the first light from the filter array 110.

[0182] Multiple first pixel values ​​can be written in m x n matrix format as follows:

[0183]

number

[0184] (i,j) can be thought of as corresponding to the pixel position in the image. 1≦i≦m and 1≦j≦n.

[0185] Multiple first pixel values ​​can be written in m x n x 1 matrix format as follows:

[0186]

number

[0187] <Processing (a-2)> The signal processing circuit 250 generates multiple pixel values ​​I(11), ~ of the image corresponding to the first wavelength range of the first subject, and multiple pixel values ​​I(1p) of the image corresponding to the p-th wavelength range of the first subject, based on the first matrix and multiple first pixel values ​​recorded in the memory 210. This generation method has already been explained using equations (1) and (2).

[0188] Equations (1) and (2) are written in terms of n × m, but here we will replace n × m in equations (1) and (2) with m × n and apply them.

[0189] The first example is the m × n row m × n × N column matrix H shown in equation (1). Here, let H be H1, and using submatrices A1, ~, and Ap,

[0190]

number

[0191] This can be written as follows. Each of the submatrices A1, ~, and Ap may be a diagonal matrix.

[0192]

number

[0193] Multiple pixel values ​​I(11), ~ of the image corresponding to the first wavelength range of the first subject, and multiple pixel values ​​I(1p) of the image corresponding to the p wavelength range of the first subject, can be written in m x n matrix form as follows.

[0194]

number

[0195] (i,j) can be thought of as corresponding to the pixel position in the image. 1≦i≦m and 1≦j≦n.

[0196] Multiple pixel values ​​I(11), ~ of the image corresponding to the first wavelength range of the first subject, and multiple pixel values ​​I(1p) of the image corresponding to the p wavelength range of the first subject, can be written in m x n x 1 matrix format as follows.

[0197]

number

[0198] If expressed in the form of equation (1):

[0199]

number

[0200] That is the case.

[0201] Multiple first submatrices A1, A2, ~, Ap contain the q-th submatrice Aq.

[0202]

number

[0203] Multiple first submatrices A1, A2, ~, Ap contain multiple second submatrices, the r-th submatrice Ar, ~, and the (r+s)-th submatrice A(r+s).

[0204]

number

[0205] The second process includes processes (b-1) and (b-2).

[0206] <Processing (b-1)> The signal processing device 250 receives multiple second pixel values ​​from the image sensor 160. The fourth light from the second subject enters the filter array 110. In response to this entry, the filter array 110 outputs a third light. The image sensor 160 outputs multiple second pixel values ​​in response to the third light from the filter array 110.

[0207] Multiple second pixel values ​​can be written in m x n matrix format as follows:

[0208]

number

[0209] (i,j) can be thought of as corresponding to the pixel position in the image. 1≦i≦m and 1≦j≦n.

[0210] Multiple second pixel values ​​can be written in m x n x 1 matrix format as follows:

[0211]

number

[0212] <Processing (b-2)> The signal processing circuit 250 generates multiple pixel values ​​I(2q) of the image corresponding to the q wavelength range of the second subject based on the q-th submatrix Aq and multiple second pixel values ​​recorded in the memory 210. The signal processing circuit 250 does not generate multiple pixel values ​​I(2r) of the image corresponding to the r-th wavelength range of the second subject, ~, or multiple pixel values ​​I(2(r+s)) of the image corresponding to the (r+s) wavelength range of the second subject, based on the multiple second submatrixes and multiple second pixel values.

[0213] For example, the signal processing circuit 250 may perform the following processing.

[0214] The signal processing circuit 250 deletes the r-th submatrix Ar, . . . , the (r+s)-th submatrix A(r+s) from the first matrix shown in formula (12) recorded in the memory 210, and generates a matrix H2 of n×m rows and n×m×(p−(s+1)) columns.

[0215] [Math.]

[0216] Then, based on the following formula (22), according to the method disclosed in the description of formulas (1) and (2), (i) A plurality of pixel values I(21) of the image corresponding to the first wavelength band of the second object, . . . , a plurality of pixel values I(2q) of the image corresponding to the q-th wavelength band of the second object, . . . , a plurality of pixel values I(2(r-1)) of the image corresponding to the (r-1)-th wavelength band of the second object are generated, (ii) a plurality of pixel values I(2r) of the image corresponding to the s-th wavelength band of the second object, . . . , a plurality of pixel values I(2(r+s)) of the image corresponding to the (r+s)-th wavelength band of the second object are not generated, (iii) a plurality of pixel values I(2(r+s+1)) of the image corresponding to the (r+s+1)-th wavelength band of the second object, . . . , a plurality of pixel values I(2p) of the image corresponding to the p-th wavelength band of the second object are generated.

[0217] Although formulas (1) and (2) are described as n×m, here n×m in formulas (1) and (2) is rewritten as m×n for application.

[0218] [Math.]

[0219] The third process includes processes (c-1), (c-2), and (c-3).

[0220] <Process (c-1)> The signal processing device 250 receives multiple third pixel values ​​from the image sensor 160. The sixth light from the third subject enters the filter array 110. In response to this entry, the filter array 110 outputs the fifth light. The image sensor 160 outputs multiple third pixel values ​​in response to the fifth light from the filter array 110.

[0221] Multiple third pixel values ​​can be written in m x n matrix format as follows:

[0222]

number

[0223] (i,j) can be thought of as corresponding to the pixel position in the image. 1≦i≦m and 1≦j≦n.

[0224] Multiple third pixel values ​​can be written in m x n x 1 matrix format as follows:

[0225]

number

[0226] <Processing (c-2)> The signal processing circuit 250 generates multiple pixel values ​​I(3q) of an image corresponding to the q wavelength range of the third subject, based on the q-th submatrix Aq and multiple third pixel values ​​recorded in the memory 210.

[0227] <Processing (c-3)> The signal processing circuit 250 generates a second matrix based on a plurality of second submatrices recorded in the memory 210. Based on the generated second matrix and a plurality of third pixel values, the signal processing circuit 250 generates a plurality of pixel values ​​I3c of an image corresponding to the r wavelength range to the (r+s) wavelength range of the third subject. The signal processing circuit 250 does not generate, based on the plurality of second submatrices and the plurality of third pixel values, the plurality of pixel values I(3r) of an image corresponding to an r-th wavelength band of a third subject, ..., the plurality of pixel values I(3(r+s)) of an image corresponding to an (r+s)-th wavelength band of the third subject.

[0228] For example, the signal processing circuit 250 may execute the following processing.

[0229] The signal processing circuit 250 generates a second matrix H3 based on an r-th submatrix Ar, ..., an (r+s)-th submatrix A(r+s) included in a first matrix recorded in the memory 210 (see formula (12)).

[0230]

Math

[0231]

Math

[0232] W(1,1)=(hr(1,1)+···+h(r+s)(1,1)) / (s+1), ..., W(m×n,m×n)=(hr(m×n,m×n)+···+h(r+s)(m×n,m×n)) / (s+1).

[0233] The signal processing circuit 250 generates a third matrix H4 based on the first matrix H1 and the second matrix H3.

[0234]

Math

[0235] The third matrix H4 is a matrix with n×m rows and n×m×(p-s) columns.

[0236] Then, based on formula (27), in accordance with the method disclosed in the description of formula (1) and formula (2), (i) Generate multiple pixel values ​​I(31), ~ of the image corresponding to the first wavelength range of the third subject, multiple pixel values ​​I(3q), ~ of the image corresponding to the q wavelength range of the third subject, and multiple pixel values ​​I(3(r-1)) of the image corresponding to the (r-1) wavelength range of the third subject. (ii) Do not generate multiple pixel values ​​I(3r), ~ of the image corresponding to the s wavelength range of the third subject, and multiple pixel values ​​I(3(r+s)) of the image corresponding to the (r+s) wavelength range of the third subject. (iii) Generate multiple pixel values ​​I3c of the image corresponding to the r wavelength range to the (r+s) wavelength range of the third subject, (iv) Generate multiple pixel values ​​I(3(r+s+1)) of the image corresponding to the (r+s+1) wavelength range of the third subject, and multiple pixel values ​​I(3p) of the image corresponding to the p wavelength range of the third subject.

[0237] Although equations (1) and (2) are written in terms of n × m, we may substitute n × m with m × n in equations (1) and (2) here.

[0238]

number

[0239] The multiple pixel values ​​I3c of the image corresponding to the r-wavelength range to the (r+s)-wavelength range of the third subject can be written in m x n matrix format as follows:

[0240]

number

[0241] (i,j) can be thought of as corresponding to the pixel position in the image. 1≦i≦m and 1≦j≦n.

[0242] The multiple pixel values ​​I3c of the image corresponding to the r-wavelength range to the (r+s)-wavelength range of the third subject can be written in m x n x 1 matrix format as follows.

[0243]

number

[0244] (Other 2) In this disclosure, the compressed image may be generated by imaging in a manner different from imaging using a filter array that includes multiple optical filters.

[0245] For example, the imaging device 100 may be configured such that the image sensor 160 is modified to change the light-receiving characteristics of the image sensor for each pixel, and a compressed image may be generated by imaging using the modified image sensor 160. In other words, instead of encoding the light incident on the image sensor with the filter array 110, a compressed image may be generated by giving the image sensor the function of encoding the incident light. In this case, the mask data corresponds to the light-receiving characteristics of the image sensor.

[0246] Furthermore, by introducing an optical element such as a metalens into at least a part of the optical system 140, the optical properties of the optical system 140 may be changed spatially and wavelengthly, and the incident light may be encoded. Compressed images may be generated by an imaging device including such a configuration. In this case, the mask data will be information corresponding to the optical properties of the optical element such as the metalens. Thus, by using an imaging device 100 with a configuration different from the configuration using the filter array 110, the intensity of the incident light may be modulated for each wavelength, and compressed and restored images may be generated.

[0247] (Other 3) This disclosure is not limited to Embodiments 1-4, examples, and modifications. Within the scope of this disclosure, various modifications conceivable by those skilled in the art, applied to the above embodiments, examples, and modifications, as well as forms constructed by combining components from different embodiments, examples, and / or modifications, may also be included, without departing from the spirit of this disclosure.

[0248] Furthermore, the technology disclosed herein can be applied not only to fluorescence observation but also to other applications where the spectrum of the object to be observed is known. For example, the technology disclosed herein can be applied to various applications such as observation of absorption spectra, observation of blackbody radiation (e.g., temperature estimation), and estimation of light sources (e.g., LEDs, halogen lamps, etc.). [Industrial applicability]

[0249] The technology disclosed herein is useful, for example, in cameras and measuring instruments that acquire multi-wavelength images. The technology disclosed herein can also be applied, for example, to fluorescence observation, absorption spectrum observation, sensing for biological, medical, and cosmetic applications, foreign object and pesticide residue testing systems for food, remote sensing systems, and in-vehicle sensing systems. [Explanation of symbols]

[0250] 10 Compressed Images 20 Restored Images Samples 31, 32, 33, and 34. 70 Objects 80 samples 100 Imaging device 110 filter array 140 Optical system 150 Control circuits 160 Image Sensors 200 Processing Units 210 memory 250 Signal Processing Circuits 300 display device 320 Image Processing Circuits 330 displays 400 Input UI 410 memory 420 processors 610 Light source 620 Optical system 621 Interference Filter 622 Dichroic Mirror 623 Objective lens 624 Long-Pass Filter 630 detectors

Claims

1. A signal processing method performed by a computer, This involves obtaining compressed image data, which includes two-dimensional image information of a subject, generated using a filter array and an image sensor containing multiple types of optical filters with different spectral transmittances, and obtained by compressing hyperspectral information in the target wavelength range. To obtain mask data that reflects the spatial distribution of the spectral transmittance, To obtain reference spectral data containing information on one or more spectra associated with the subject, Based on the aforementioned reference spectral data, a plurality of specified wavelength bands included in the target wavelength range are determined, Based on the compressed image data and the mask data, a plurality of two-dimensional image data corresponding to the plurality of specified wavelength bands are generated, A method that includes this.

2. The method according to claim 1, wherein the one or more spectra are associated with one or more substances that are assumed to be contained in the subject.

3. The method according to claim 1 or 2, wherein each of the plurality of specified wavelength bands includes the peak wavelength of the spectrum associated with a corresponding one of the one or more substances.

4. The aforementioned reference spectral data includes information on multiple spectra associated with multiple types of substances assumed to be contained in the subject, The plurality of designated wavelength bands include a first designated wavelength band that does not overlap among the plurality of spectra and a second designated wavelength band that overlaps among the plurality of spectra. The method according to claim 1 or 2.

5. The mask data includes mask matrix information having elements corresponding to the spatial distribution of the transmittance of the filter array for each of a plurality of unit bands included in the target wavelength range. The aforementioned method, To generate composite mask information by combining the mask matrix information corresponding to unspecified wavelength bands different from the specified wavelength band in the target wavelength range, Based on the compressed image data and the composite mask information, composite image data for the unspecified wavelength band is generated. The method according to claim 1 or 2, further comprising:

6. The method according to claim 1 or 2, wherein generating the plurality of two-dimensional image data includes generating and outputting the plurality of two-dimensional image data corresponding to the plurality of specified wavelength bands without generating image data corresponding to unspecified wavelength bands different from the specified wavelength bands in the target wavelength range from the compressed image data.

7. The method according to claim 1 or 2, wherein the plurality of specified wavelength bands are determined based on the intensity of one or more spectra shown in the reference spectral data or the differential value of the intensity.

8. The method according to claim 1 or 2, wherein the reference spectral data includes information on the fluorescence spectra of one or more substances assumed to be contained in the subject.

9. The method according to claim 1 or 2, wherein the reference spectral data includes information on the absorbance spectra of one or more substances assumed to be contained in the subject.

10. The method further includes displaying a graphical user interface on a display that allows the user to specify one or more spectra, or one or more substances associated with one or more spectra, The aforementioned reference spectral data is obtained in accordance with one or more specified spectra, or one or more specified substances. The method according to claim 1 or 2.

11. Processor and A memory containing a computer program executed by the aforementioned processor, A signal processing device comprising, The computer program is used by the processor This involves obtaining compressed image data, which includes two-dimensional image information of a subject, generated using a filter array and an image sensor containing multiple types of optical filters with different spectral transmittances, and obtained by compressing hyperspectral information in the target wavelength range. To obtain mask data that reflects the spatial distribution of the spectral transmittance, To obtain reference spectral data containing information on one or more spectra associated with the subject, Based on the aforementioned reference spectral data, a plurality of specified wavelength bands included in the target wavelength range are determined, Based on the compressed image data and the mask data, a plurality of two-dimensional image data corresponding to the plurality of specified wavelength bands are generated, A signal processing device that executes a signal.

12. On the computer, This involves obtaining compressed image data, which includes two-dimensional image information of a subject, generated using a filter array and an image sensor containing multiple types of optical filters with different spectral transmittances, and obtained by compressing hyperspectral information in the target wavelength range. To obtain mask data that reflects the spatial distribution of the spectral transmittance, To obtain reference spectral data containing information on one or more spectra associated with the subject, Based on the aforementioned reference spectral data, a plurality of specified wavelength bands included in the target wavelength range are determined, Based on the compressed image data and the mask data, a plurality of two-dimensional image data corresponding to the plurality of specified wavelength bands are generated, A computer program that executes something.

Images