Imaging system, method used in imaging system, and computer program used in imaging system

Abstract

This imaging system comprises a filter array (20) including a plurality of filters having different transmission spectra, an image sensor (30) that images light transmitted through the filter array and generates image data, and a processing circuit (70), wherein the processing circuit: acquires luminance pattern data, which is generated by predicting a luminance pattern detected, on the basis of subject data including spectral information about at least one substance, when the substance is imaged by the image sensor; acquires first image data obtained by imaging a target scene with the image sensor; and generates output data regarding the presence or absence of the substance in the target scene by comparing the luminance pattern data and the first image data.

Description

Imaging system, method used in imaging system, and computer program used in imaging system

[0001] The present disclosure relates to an imaging system, a method for use in an imaging system, and a computer program for use in an imaging system.

[0002] Classifying one or more objects in an image into different types is essential in the fields of factory automation and medicine. This classification process uses features such as spectral and geometric information about the object. Hyperspectral cameras can capture hyperspectral images containing a wealth of spectral information for each pixel. Therefore, hyperspectral cameras are expected to be used for such classification processes.

[0003] Patent Documents 1 and 2 disclose imaging devices that obtain hyperspectral images using compressed sensing technology.

[0004] U.S. Patent No. 9,599,511 International Publication No. 2020 / 080045

[0005] The present disclosure provides an imaging system that can reduce the processing load for classifying subjects captured in an image by type.

[0006] An imaging system according to one aspect of the present disclosure comprises a filter array including a plurality of filters having different transmission spectra from one another, an image sensor that captures light transmitted through the filter array and generates image data, and a processing circuit, wherein the processing circuit acquires brightness pattern data generated by predicting a brightness pattern that will be detected when the substance is imaged by the image sensor based on subject data including spectral information of the at least one substance, acquires first image data obtained by imaging a target scene with the image sensor, and generates output data regarding the presence or absence of the substance in the target scene by comparing the brightness pattern data with the first image data.

[0007] A comprehensive or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. When an apparatus consists of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices. The multiple devices included in a "system" may include devices installed in remote locations away from the other devices and connected via a communication network.

[0008] According to the technology of the present disclosure, it is possible to realize an imaging device that can reduce the processing load of classifying subjects captured in an image by type.

[0009] FIG. 1A is a diagram for explaining the relationship between a target wavelength range and multiple bands included therein. FIG. 1B is a diagram schematically showing an example of a hyperspectral image. FIG. 2A is a diagram schematically showing an example of a filter array. FIG. 2B is a diagram showing an example of the transmission spectrum of a first filter included in the filter array shown in FIG. 2A. FIG. 2C is a diagram showing an example of the transmission spectrum of a second filter included in the filter array shown in FIG. 2A. FIG. 2D is a diagram showing an example of the transmission spectrum of multiple bands W included in the target wavelength range. １ , W ２ , ..., W ｉ3A and 3B are diagrams showing examples of spatial distributions of light transmittance for each of the four fluorescent dyes A to D. FIG. 3 is a diagram schematically showing examples of fluorescence spectra of four types of fluorescent dyes A to D. FIG. 4 is a block diagram schematically showing the configuration of an imaging device according to a first exemplary embodiment of the present disclosure. FIG. 5A is a diagram schematically showing the spatial distribution of luminance values ​​in a portion of a compressed image. FIG. 5B is a diagram schematically showing the spatial distribution of luminance in the reference region shown in FIG. 5A among nine luminance patterns. FIG. 6A is a diagram schematically showing the fluorescence intensities in bands 1 to 8 of two types of fluorescent dyes. FIG. 6B is a diagram schematically showing the fluorescence intensities in bands 1, 3, and 5 of two types of fluorescent dyes. FIG. 7A is a diagram schematically showing an example of a GUI displayed on the output device before classifying the fluorescent dyes. FIG. 7B is a diagram schematically showing an example of a GUI displayed on the output device after classifying the fluorescent dyes. FIG. 8A is a flowchart showing an example of an operation performed by a processing circuit in the first exemplary embodiment. FIG. 8B is a flowchart showing another example of an operation performed by a processing circuit in the first exemplary embodiment. FIG. 8C is a flowchart showing yet another example of the operation performed by the processing circuit in embodiment 1. FIG. 9 is a block diagram schematically showing the configuration of an imaging system according to exemplary embodiment 2 of the present disclosure. FIG. 10 is a block diagram schematically showing the configuration of an imaging system according to exemplary embodiment 3 of the present disclosure. FIG. 11A is a flowchart showing an example of the operation performed by the processing circuit in embodiment 3. FIG. 11B is a flowchart showing an example of the operation performed by an external processing circuit between step S201 and step S202 shown in FIG. 11A. FIG. 12 is a diagram schematically showing an example of an imaging device capturing an image of a color chart as a target scene. FIG. 13A is a diagram schematically showing an example of an imaging device individually capturing images of multiple medicine bags carried by a belt conveyor. FIG. 13B is a diagram schematically showing an example of a GUI displayed on an output device after the medicines have been sorted.

[0010] In the present disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the connection relationships within the LSI or set up circuit sections within the LSI, can also be used for the same purpose.

[0011] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and required hardware devices, such as interfaces.

[0012] Exemplary embodiments of the present disclosure will be described below. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in the independent claims that represent the highest concepts will be described as optional components. Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0013] Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

[0014] First, an example of a hyperspectral image will be briefly described with reference to FIGS. 1A and 1B. A hyperspectral image is image data that contains more wavelength information than a typical RGB image. An RGB image has a pixel value for each pixel in three bands: red (R), green (G), and blue (B). In contrast, a hyperspectral image has a pixel value for each pixel in the three or more bands described above. In this specification, a "hyperspectral image" refers to multiple images corresponding to four or more bands included in a predetermined target wavelength range. The value of each pixel is referred to as a "pixel value." In this specification, the multiple pixel values ​​of multiple pixels included in an image may be referred to as an "image." The number of bands in a hyperspectral image is typically 10 or more, and may exceed 100 in some cases. A "hyperspectral image" may also be referred to as a "hyperspectral data cube" or a "hyperspectral cube."

[0015] FIG. 1A shows a target wavelength range W and a plurality of bands W included therein. １ , W ２ , ..., W ｉ1 is a diagram for explaining the relationship between the wavelengths of visible light and near-infrared light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a mid-infrared or far-infrared wavelength range. In this manner, the wavelength range used is not limited to the visible light range. In this specification, for convenience, the term "light" is not limited to visible light, and electromagnetic waves with wavelengths outside the visible light wavelength range, such as ultraviolet and near-infrared rays, are also referred to as "light."

[0016] In the example shown in FIG. 1A, the target wavelength range W is divided into i equal parts, where i is an arbitrary integer equal to or greater than 4, and each of the divided wavelength ranges is called a band W １ , Band W ２ , ..., Band W ｉ However, the present invention is not limited to this example. The number of bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidths corresponding to the bands may be different from each other. There may be gaps between adjacent bands. If the number of bands is four or more, more information can be obtained from a hyperspectral image than from an RGB image.

[0017] 1B is a diagram schematically illustrating an example of a hyperspectral image 22. In the example illustrated in FIG. 1B, the imaged object is an apple. The hyperspectral image 22 is captured in the band W １ , Band W ２、 ..., Band W ｉ Image 22W corresponds one-to-one to １ , Image 22W ２ , ..., Image 22W ｉ Each of these images includes a plurality of pixels arranged two-dimensionally. FIG. 1B illustrates vertical and horizontal dashed lines indicating pixel divisions. The actual number of pixels per image can be as large as tens of thousands to tens of millions, but for ease of understanding, FIG. 1B illustrates pixel divisions as if the number of pixels were extremely small. In FIG. 1B, image 22W １ , Image 22W ２ , ..., Image 22W ｉIn the example shown, each of the images 22W includes 14×10=140 pixel values. When an object is illuminated with light, light based on the reflected light is detected by a plurality of photodetection elements in the image sensor. A signal indicating the amount of light detected by each photodetection element represents the pixel value of the pixel corresponding to that photodetection element. ｋ includes a plurality of pixels, each of which is a band W ｋ (k is a natural number equal to or less than i). Therefore, by acquiring the hyperspectral image 22, information on the two-dimensional distribution of the spectrum of the object can be obtained. Based on the spectrum of the object, the optical characteristics of the object can be accurately analyzed.

[0018] Next, an example of a method for generating a hyperspectral image will be briefly described. A hyperspectral image can be acquired by imaging using a spectroscopic element such as a prism or a grating. When a prism is used, when reflected or transmitted light from an object passes through the prism, the light is emitted from the exit surface of the prism at an emission angle according to the wavelength. When a grating is used, when reflected or transmitted light from an object enters the grating, the light is diffracted at a diffraction angle according to the wavelength.

[0019] A line-scan hyperspectral camera acquires a hyperspectral image as follows: The light generated by illuminating a subject with a line beam is split into bands using a prism or grating, and the separated light is detected for each band. This operation is repeated each time the line beam is shifted slightly. Line-scan hyperspectral cameras have high spatial and wavelength resolution, but require a long imaging time due to scanning using a line beam. Conventional snapshot hyperspectral cameras have a short imaging time because they do not require scanning, but their sensitivity and spatial resolution are not very high. Conventional snapshot hyperspectral cameras periodically arrange multiple types of narrow-band filters with different transmission bands on an image sensor. The average transmittance of each filter is approximately 5%. Increasing the number of narrow-band filters to improve wavelength resolution reduces spatial resolution.

[0020] A snapshot-type hyperspectral camera utilizing compressed sensing technology, as disclosed in Patent Document 1, can achieve high sensitivity and spatial resolution. In the compressed sensing technology disclosed in Patent Document 1, light reflected from an object is detected by an image sensor through a filter array called a coding element or coding mask. The filter array includes multiple filters arranged two-dimensionally. Each of these filters has its own unique transmission spectrum. By capturing an image using such a filter array, a compressed image is obtained in which image information from multiple bands is compressed into a single two-dimensional image. In the compressed image, spectral information of the object is compressed and recorded as a single pixel value for each pixel. In other words, each pixel in the compressed image contains information corresponding to multiple bands.

[0021] FIG. 2A is a schematic diagram illustrating an example of a filter array 20. The filter array 20 includes a plurality of filters arranged two-dimensionally. Each filter has an individually set transmission spectrum. The transmission spectrum is expressed by a function T(λ), where λ is the wavelength of incident light. The transmission spectrum T(λ) can take values ​​between 0 and 1. In the example shown in FIG. 2A, the filter array 20 includes 48 rectangular filters arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more filters may be provided. The number of filters included in the filter array 20 may be approximately the same as the number of pixels in the image sensor.

[0022] 2B and 2C are diagrams showing examples of the transmission spectra of the first filter A1 and the second filter A2, respectively, among the multiple filters included in the filter array 20 of FIG. 2A. The transmission spectrum of the first filter A1 and the transmission spectrum of the second filter A2 are different from each other. As such, the transmission spectra of the filter array 20 vary depending on the filter. However, it is not necessary for all filters to have different transmission spectra. In the filter array 20, the transmission spectra of at least two of the multiple filters are different from each other. That is, the filter array 20 includes multiple types of filters with different transmission spectra. Each type of filter may have two or more local maxima in the target wavelength range. The multiple types of filters may include four or more types of filters, and the transmission range of one type of filter may partially overlap with the transmission range of another type of filter. In some examples, the number of transmission spectrum patterns of the multiple types of filters included in the filter array 20 may be equal to or greater than the number i of bands included in the target wavelength range. The filter array 20 may be designed so that half or more of the filters have different transmission spectra.

[0023] FIG. 2D shows multiple bands W included in the wavelength range of interest. １ , W ２ , ..., W ｉ2D is a diagram showing an example of the spatial distribution of light transmittance of each filter. In the example shown in FIG. 2D, the difference in shade of each filter represents the difference in light transmittance. Lighter filters have higher light transmittance, while darker filters have lower light transmittance. As shown in FIG. 2D, the spatial distribution of light transmittance varies depending on the band. In this specification, data showing the spatial distribution of the transmittance spectrum of the filter array is referred to as a "reconstruction table." Using compressed sensing technology, a hyperspectral image can be reconstructed from a compressed image using the reconstruction table. Because compressed sensing technology does not require the use of a prism or grating, the hyperspectral camera can be made more compact. Furthermore, compressed sensing technology compresses information from multiple spectra into a single compressed image, thereby reducing the amount of data processed by the processing circuit. Furthermore, because each filter in the filter array does not need to be a narrow-band filter, higher sensitivity and spatial resolution can be achieved than conventional snapshot-type hyperspectral cameras.

[0024] Next, a method for restoring a hyperspectral image from a compressed image using the restoration table will be described. The compressed image data g acquired by the image sensor, the restoration table H, and the hyperspectral image data f satisfy the following equation (1).

[0025] Here, the compressed image data g and the hyperspectral image data f are vector data, and the restoration table H is matrix data. ｇ Then, the compressed image data g is N ｇ The number of pixels in each of the multiple images included in the hyperspectral image is expressed as N ｆ , and the number of wavelength bands is M, the hyperspectral image data f is ｆ 1B. The images are represented as a one-dimensional array or vector having 1×M elements. For example, the images 22W shown in FIG. １ , ..., Image 22W ｉ If so, N ｆ = 140, M = i. The restoration table H isｇ Row (N ｆ ×M) columns of elements. ｇ and N ｆ can be designed to be the same value.

[0026] If vector g and matrix H are given, it seems possible to calculate f by solving the inverse problem of equation (1). However, if the number of elements N of the data f to be calculated is ｆ ×M is the number of elements of the acquired data g ｇ Since the number of pixels is larger than the number of pixels in the image, this problem is an ill-posed problem and cannot be solved as is. Therefore, a solution is obtained using a compressed sensing technique by utilizing the image redundancy contained in the data f. Specifically, the desired data f is estimated by solving the following equation (2).

[0027] Included in formula (1) and formula (2)

[0028] is sometimes written as g in the descriptions relating to formulas (1) and (2).

[0029] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term, which will be described later. Equation (2) means that f is found to minimize the sum of the first and second terms. The arithmetic processing circuit can converge the solution through recursive iterative calculations and calculate the final solution f.

[0030] The first term in the parentheses in Equation (2) represents the sum of squares of the difference between the acquired data g and Hf, which is the system transformation of the estimation process f using the matrix H. The second term, Φ(f), is a constraint for the regularization of f and is a function that reflects the sparsity information of the estimation data. Its function is to smooth or stabilize the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object in the space of each regularization term varies depending on the texture of the object. A regularization term that makes the object's texture sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the calculation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0031] A more detailed method for obtaining hyperspectral images by compressed sensing techniques is disclosed in U.S. Patent No. 6,277,949, the disclosure of which is incorporated herein by reference in its entirety.

[0032] In a hyperspectral camera using compressed sensing technology, compressed image data is generated before generating hyperspectral image data. Patent Document 2 discloses a method for recognizing an object using a compressed image rather than a hyperspectral image. In this method, a compressed image of a known object is first acquired, and training data of the compressed image of the object is generated by machine learning. Then, the object shown in the newly acquired compressed image is recognized based on the training data. This method eliminates the need to generate hyperspectral image data, thereby reducing the processing load.

[0033] In the process of classifying objects in an image by type, the spectral information of the object may be known. For example, fluorescent dyes absorb excitation light and emit fluorescence of a specific wavelength. Pharmaceuticals and electronic components of the same type have unique spectral information with little individual variation. Until now, the process of classifying objects in an image by type has been performed by comparing hyperspectral image data with known spectral data. This method increases the burden of the classification process by generating hyperspectral image data. Until now, no one has considered taking advantage of the advantage of knowing the spectral information of the object to reduce the burden of the classification process.

[0034] Based on the above considerations, the inventors have conceived of an imaging device according to an embodiment of the present disclosure that can classify objects by type using the brightness pattern of image data obtained by photographing the object through a filter array, rather than using hyperspectral image data of the object. The imaging device according to this embodiment uses a coding element used for compressed sensing, as disclosed in Patent Document 1, as the filter array. Furthermore, the compressed image data obtained through the coding element is used to classify the object by type. The imaging device according to this embodiment can classify the object by type without acquiring hyperspectral image data of the object, thereby reducing the burden of the classification process. Furthermore, the imaging device according to this embodiment does not require each filter included in the filter array to be a narrowband filter, thereby achieving high sensitivity and spatial resolution. An imaging device and a computer program according to an embodiment of the present disclosure are described below.

[0035] The imaging system according to the first item comprises a filter array including a plurality of filters having different transmission spectra, an image sensor that captures light transmitted through the filter array and generates image data, and a processing circuit, wherein the processing circuit acquires brightness pattern data generated by predicting a brightness pattern that will be detected when the image sensor captures an image of at least one substance based on subject data including spectral information of the substance, acquires first image data obtained by capturing an image of a target scene with the image sensor, and generates output data regarding the presence or absence of the substance in the target scene by comparing the brightness pattern data with the first image data.

[0036] This imaging system can reduce the processing load of classifying the subjects captured in the image by type.

[0037] The imaging system according to a second aspect is the imaging system according to the first aspect, further comprising a storage device that stores the object data and a table indicating a spatial distribution of the transmission spectrum of the filter array, wherein the processing circuit acquires the object data and the table from the storage device and generates the luminance pattern data based on the object data and the table.

[0038] This imaging system can generate luminance pattern data without communicating with an external device.

[0039] The imaging system according to a third aspect is the imaging system according to the first aspect, further comprising a storage device that stores a table indicating the spatial distribution of the transmission spectrum, wherein the processing circuit acquires the table from the storage device, acquires the object data from an external device, and generates the luminance pattern data based on the object data and the table.

[0040] In this imaging system, it is not necessary to store subject data in a storage device in order to generate brightness pattern data, and therefore the amount of data stored in the storage device can be reduced.

[0041] An imaging system according to a fourth aspect is the imaging system according to the first aspect, wherein the processing circuitry acquires the luminance pattern data from an external source.

[0042] In this imaging system, there is no need to generate luminance pattern data, so the processing load can be reduced.

[0043] An imaging system according to a fifth item is an imaging system according to any one of the first to fourth items, wherein the spectral information of the at least one substance includes spectral information of a plurality of substances, and the output data relates to the presence or absence of each of the plurality of substances in the target scene.

[0044] This imaging system can detect the presence or absence of each of a plurality of types of subjects in a target scene.

[0045] The imaging system relating to the sixth item is an imaging system relating to any one of the first to fifth items, in which the processing circuit determines the presence or absence of the substance in the target scene by comparing the brightness pattern data with the first image data in a reference area including two or more pixels.

[0046] This imaging system can determine whether or not a subject is present in a reference area in a target scene.

[0047] An imaging system according to a seventh item is the imaging system according to the sixth item, wherein the number of the two or more pixels included in the reference region varies depending on the number of a plurality of substances.

[0048] In this imaging system, it is possible to select a reference region suitable for the number of types of subjects.

[0049] An imaging system according to an eighth item is the imaging system according to the sixth or seventh item, wherein the target wavelength range dispersed by the imaging system includes n bands, the two or more pixels included in the reference area include an evaluation pixel and n pixels that are pixels in the vicinity of the evaluation pixel, the reference area contains one material rather than multiple materials, the filter array includes n filters corresponding to the n pixels included in the reference area, the transmission spectra of the n filters are different from one another, and the transmittance of each of the n filters for the n bands is all non-zero.

[0050] This imaging system can efficiently determine whether or not a subject is present in a reference area in a target scene.

[0051] The imaging system according to the ninth item is an imaging system according to any one of the first to eighth items, wherein the output data includes information on the probability of the substance being present in each pixel of the first image data, and / or information on the probability of the substance being present in a plurality of pixels of the first image data corresponding to the object of observation.

[0052] In this imaging system, the presence or absence of a subject in a target scene can be determined based on the probability of the subject's presence.

[0053] An imaging system according to a tenth item is the imaging system according to any one of the first to ninth items, wherein the subject data further includes shape information of the at least one substance.

[0054] In this imaging system, the presence or absence of a subject in a target scene can be determined based on the shape of the subject.

[0055] An imaging system according to an eleventh item is the imaging system according to any one of the first to tenth items, further comprising an output device. The processing circuit causes the output device to output the classification result indicated by the output data.

[0056] In this imaging system, the user can know the classification results of the subjects in the target scene through the output device.

[0057] An imaging system according to a twelfth item is the imaging system according to the eleventh item, wherein the output device displays an image in which a label according to type is attached to a portion of the target scene where the substance exists.

[0058] In this imaging system, the user can know the type of subject present in the target scene by looking at the output device.

[0059] An imaging system according to a thirteenth item is the imaging system according to the eleventh or twelfth item, wherein the output device displays at least one of an image showing a graph of the spectrum of the substance and an image showing a description of the substance.

[0060] In this imaging system, the user can learn detailed information about the subject by looking at the output device.

[0061] The imaging system according to the fourteenth item is an imaging system according to any one of the eleventh to thirteenth items, in which the output device displays an image of an object of observation in the target scene where the probability of the substance being present is below a certain value, with a label attached indicating that the type of the object of observation cannot be classified.

[0062] In this imaging system, the user can know the object of observation that could not be determined by the output device.

[0063] An imaging system according to a fifteenth item is an imaging system according to any one of the first to fourteenth items, wherein each of the plurality of filters has two or more maxima in the target wavelength range that is spectrally separated by the imaging system.

[0064] In this imaging system, a filter array suitable for comparing brightness pattern data with image data can be realized.

[0065] An imaging system according to a sixteenth aspect is the imaging system according to any one of the first to fifteenth aspects, wherein the plurality of filters include four or more types of filters, and a transmission range of one type of filter among the four or more types of filters overlaps with a transmission range of another type of filter.

[0066] In this imaging system, a filter array suitable for comparing brightness pattern data with image data can be realized.

[0067] An imaging system according to a seventeenth item is the imaging system according to any one of the first to sixteenth items, wherein the first image data is compressed image data encoded by the filter array, and the processing circuit generates hyperspectral image data of the target scene based on the compressed image data of the target scene.

[0068] The imaging system is capable of generating hyperspectral image data of a scene of interest.

[0069] An imaging system according to an eighteenth item is the imaging system according to any one of the eleventh to fourteenth items, wherein the first image data is compressed image data encoded by the filter array, and the processing circuitry causes the output device to display a GUI for a user to instruct generation of hyperspectral image data of the target scene, and generates the hyperspectral image data of the target scene based on the compressed image data of the target scene in response to the user's instruction.

[0070] In this imaging system, a user can generate hyperspectral image data of a target scene by inputting data into a GUI displayed on an output device.

[0071] An imaging system according to a nineteenth item is the imaging system according to any one of the eleventh to fourteenth items, wherein the first image data is compressed image data encoded by the filter array, and the processing circuit causes the output device to display a GUI for a user to instruct switching between a first mode for generating the output data and a second mode for generating hyperspectral image data of the target scene, generates the output data in response to the user's instruction for the first mode, and generates the hyperspectral image data of the target scene based on the compressed image data of the target scene in response to the user's instruction for the second mode.

[0072] In this imaging system, the user can switch between the first mode and the second mode by inputting information into the GUI displayed on the output device.

[0073] A method according to a twentieth item is a computer-implemented method, the method including: acquiring first image data obtained by capturing an image of a target scene using an image sensor that generates image data by capturing light transmitted through a filter array including a plurality of filters having different transmission spectra from each other; acquiring luminance pattern data generated by predicting a luminance pattern that will be detected when the target is captured by the image sensor based on object data including spectral information of at least one type of object; and generating output data indicating the presence or absence of the object in the target scene by comparing the luminance pattern data with the first image data.

[0074] This method reduces the processing load for classifying the subjects in the image by type.

[0075] A computer program according to a twenty-first item is a computer program executed by a computer, the computer program causing the computer to acquire first image data obtained by capturing an image of a target scene using an image sensor that generates image data by capturing light transmitted through a filter array including a plurality of filters having different transmission spectra from each other, acquire luminance pattern data generated by predicting a luminance pattern that will be detected when the target scene is captured by the image sensor based on object data including spectral information of at least one type of object, and generate and output output data indicating the presence or absence of the object in the target scene by comparing the luminance pattern data with the first image data.

[0076] This computer program can reduce the processing load of classifying subjects in an image by type.

[0077] (Embodiment 1) Here, an example of fluorescence imaging using an imaging device according to embodiment 1 of the present disclosure will be described. Fluorescence imaging is widely performed, primarily in the fields of biology and medicine. In fluorescence imaging, a fluorescent dye is attached to an observation target having a specific molecule, tissue, or structure, and the observation target is irradiated with excitation light to obtain an image of the fluorescence emitted from the fluorescent dye. As a result, the observation target can be visualized. FIG. 3 is a diagram schematically illustrating examples of the fluorescence spectra of four types of fluorescent dyes A to D. The fluorescence spectra of fluorescent dyes A, B, and D each exhibit a single peak. The peak wavelength and peak width differ between fluorescent dyes A, B, and D. The fluorescence spectrum of fluorescent dye C exhibits two peaks with different peak wavelengths and peak widths. As shown in FIG. 3, in fluorescence imaging, the fluorescence spectrum information of the fluorescent dye attached to the observation target is known.

[0078] The configuration of an imaging device according to a first exemplary embodiment of the present disclosure will be described below with reference to FIG. 4 . The imaging device uses compressed image data, rather than hyperspectral image data, to classify multiple types of fluorescent dyes attached to multiple types of observation targets. FIG. 4 is a block diagram schematically illustrating the configuration of an imaging device 100 according to a first exemplary embodiment of the present disclosure. FIG. 4 illustrates a target scene 10 to be imaged. The target scene 10 includes multiple types of observation targets to which multiple types of fluorescent dyes 12 are attached. The multiple types of observation targets illustrated in FIG. 4 have elliptical, polygonal, and rectangular shapes. The number of types of fluorescent dyes attached to the imaging target may be multiple or may be one.

[0079] 4 includes a filter array 20, an image sensor 30, an optical system 40, a storage device 50, an output device 60, a processing circuit 70, and a memory 72. The imaging device 100 functions as a hyperspectral camera. The imaging device 100 may be part of the configuration of, for example, a mobile terminal or a personal computer.

[0080] The filter array 20 modulates the intensity of incident light for each filter and emits the modulated light. Details of the filter array 20 are as described above.

[0081] The image sensor 30 includes a plurality of photodetection elements arranged two-dimensionally along a photodetection surface. In this specification, the photodetection elements are also referred to as "pixels." The size of the photodetection surface of the image sensor 30 is approximately equal to the size of the light incident surface of the filter array 20. The image sensor 30 is positioned to receive light that has passed through the filter array 20. The photodetection elements included in the image sensor 30 may correspond to, for example, the plurality of filters included in the filter array 20. A single photodetection element may detect light that has passed through two or more filters. The image sensor 30 generates compressed image data based on the light that has passed through the filter array 20. The image sensor 30 may be, for example, a charge-coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor. The photodetection elements may include, for example, photodiodes. The image sensor 30 may be, for example, a monochrome sensor or a color sensor. The target wavelength range is a wavelength range that the image sensor 30 can detect.

[0082] The optical system 40 is located between the target scene 10 and the filter array 20. The target scene 10 and the filter array 20 are located on the optical axis of the optical system 40. The optical system 40 includes at least one lens. In the example shown in FIG. 4, the optical system 40 is composed of one lens, but it may also be composed of a combination of multiple lenses. The optical system 40 forms an image on the light detection surface of the image sensor 30 via the filter array 20.

[0083] The storage device 50 stores a restoration table corresponding to the transmission characteristics of the filter array 20 and dye data including fluorescence spectral information of multiple types of fluorescent dyes. In this specification, data including spectral information of at least one type of subject in the target scene 10 is referred to as "subject data." The fluorescent dye in this embodiment is an example of a subject in the target scene 10. Any subject may be used as long as its spectral information is known.

[0084] The term "at least one substance" in the claims may also mean the above-mentioned "at least one type of subject."

[0085] The output device 60 displays the classification results of the multiple types of fluorescent dyes contained in the target scene 10. This information may be displayed on a GUI (Graphical User Interface). The output device 60 may be, for example, a display of a mobile terminal or a personal computer. Alternatively, the output device 60 may be a speaker that conveys the classification results aloud. The output device 60 is not limited to a display or a speaker, as long as it is a device that can convey the classification results to the user.

[0086] The imaging device 100 may transmit an instruction to the output device 60 to output the classification result. The output device 100 may receive the instruction and output the classification result.

[0087] The processing circuitry 70 controls the operations of the image sensor 30, the storage device 50, and the output device 60. The processing circuitry 70 classifies the fluorescent dyes contained in the target scene 10 by type. This operation will be described in detail below. The computer program executed by the processing circuitry 70 is stored in a memory 72 such as a ROM or a RAM (Random Access Memory). As described above, the imaging device 100 includes a processing device including the processing circuitry 70 and the memory 72. The processing circuitry 70 and the memory 72 may be integrated on a single circuit board or provided on separate circuit boards. The functions of the processing circuitry 70 may be distributed across multiple circuits.

[0088] Next, we will explain a method for classifying multiple types of fluorescent dyes in the target scene 10. This classification method includes the following steps (1) to (3).

[0089] (1) Luminance pattern data is generated for each of multiple types of fluorescent dyes. The luminance pattern data is generated by predicting the luminance pattern detected when the fluorescent dyes are imaged by the image sensor 30. That is, luminance pattern data A1, ..., corresponding to fluorescent dye A1, luminance pattern data An corresponding to fluorescent dye An are generated (n is a natural number greater than or equal to 1). The luminance pattern data are multiple pixel values ​​that correspond one-to-one to multiple pixels included in the luminance pattern. More specifically, the luminance pattern data is data predicted to be generated when a virtual scene in which the corresponding fluorescent dyes are distributed throughout the scene is imaged by the image sensor 30 via the filter array 20. The luminance pattern represents the spatial distribution of luminance values ​​at multiple pixels. The luminance value is proportional to the value obtained by integrating, over the target wavelength range, a function obtained by multiplying the transmission spectrum of the corresponding filter by the fluorescence spectrum of the fluorescent dye. Note that if the region in which each type of fluorescent dye is distributed in the target scene 10 is determined, the luminance pattern data may be generated using a virtual scene in which each type of fluorescent dye is distributed over a portion of the scene rather than the entire scene.

[0090] (2) The target scene 10 is imaged by the image sensor 30 through the filter array 20 to generate compressed image data of the target scene 10 .

[0091] (3) By comparing the luminance pattern data with the compressed image data, the presence or absence of each type of fluorescent dye in the target scene can be determined.

[0092] Below, with reference to Figures 5A and 5B, an example will be described in which the fluorescence spectra of nine types of fluorescent dyes A to I are known and the light transmittance in the nine bands of each filter included in the filter array 20 is known.

[0093] FIG. 5A is a diagram schematically illustrating the spatial distribution of luminance values ​​in a portion of a compressed image. The luminance value of an evaluation pixel marked with a star in FIG. 5A originates from one of the nine fluorescent dyes A to I, which is determined by referring not only to the luminance value of the evaluation pixel but also to the luminance values ​​of the pixels surrounding it. In this specification, the region containing the pixel to be referenced is referred to as the "reference region." In the example shown in FIG. 5A, the reference region is a square region of three rows and three columns surrounded by a thick line, but the shape of the reference region is not limited to a square. The evaluation pixel marked with a star is the pixel located at the center of the square region.

[0094] 5B is a diagram showing a schematic representation of the spatial distribution of luminance in the same region as the reference region shown in FIG. 5A, among nine luminance patterns A to I predicted from nine types of fluorescent dyes A to I, respectively. The labels A to I represent the luminance patterns A to I, respectively. In this specification, the term "reference region" is used for the luminance patterns as well, just as it is for the compressed image.

[0095] The spatial distribution of luminance in the reference region of the compressed image shown in FIG. 5A matches the spatial distribution of luminance in the reference region of luminance pattern D shown in FIG. 5B. Therefore, it can be seen that fluorescent dye D is present in the portion of the target scene 10 corresponding to the evaluation pixel marked with a star in FIG. 5A. This pattern fitting may be performed, for example, by searching nine luminance patterns A to I for a luminance pattern that minimizes the mean squared error (MSE) or peak signal to noise ratio (PSNR) in the reference region between the luminance pattern and the compressed image. Alternatively, pattern fitting may be performed using machine learning. By performing the above pattern fitting on all pixels of the compressed image, the presence or absence of each type of fluorescent dye in the target scene can be determined.

[0096] The pattern match rate for each pixel between the luminance pattern and the compressed image can be quantified based on, for example, MSE or PSNR. The pattern match rate also represents the probability of the presence of a fluorescent dye in each pixel of the compressed image. The "probability of the presence of a fluorescent dye in each pixel of the compressed image" refers to the probability of the presence of a fluorescent dye in a portion of the target scene 10 that corresponds to each pixel of the compressed image.

[0097] Pattern fitting can be performed most efficiently when the number of pixels contained in the reference region is minimal. Below, we will explain a method for determining the reference region. This method is effective for any fluorescence spectrum.

[0098] Assume that a minimum of nine bands are used to classify nine types of fluorescent dyes. The brightness value g at the evaluation pixel x is ｘ is the light transmittance in the kth band of the filter, ｋ , the fluorescence intensity in the kth band of the fluorescent dye is I ｋ Then, it is expressed by the following equation (3).

[0099] Brightness value g ｘ is the sum of the light transmittance of the filter and the fluorescence intensity of the fluorescent dye for all bands. ｘ and the light transmittance t of the filter ｋ is known, equation (3) can be solved using nine variables I ｋ If there are at least nine simultaneous equations, then there are nine variables I ｋ As described above, the reference region includes pixels arranged in 3 rows and 3 columns with the evaluation pixel x at the center. In this embodiment, one type of fluorescent dye is present in the reference region, the nine filters included in the reference region have different transmission spectra, and the light transmittance t ｋ are all non-zero. In this case, the nine variables I ｋ can be derived.

[0100] To generalize the number of types of fluorescent dyes and the number of bands, let us say that n bands are used to classify m types of fluorescent dyes. This is the same as the degree of k in equation (3) becoming n. In this embodiment, pattern fitting can be performed most efficiently when the following requirements (A) to (D) are satisfied. (A) The reference area includes n pixels, which are the evaluation pixel and pixels located in its vicinity. (B) The reference area contains one type of fluorescent dye, not multiple types. (C) The transmission spectra of the n filters included in the reference area are different from each other. (D) The transmittance t for the n bands of each filter is ｋ are all non-zero.

[0101] In requirement (A), the "pixels located near the evaluation pixel" are pixels selected in order of shortest center-to-center distance from the evaluation pixel. In the example shown in FIG. 5A , the pixels with the shortest center-to-center distance from the evaluation pixel are the four pixels located above, below, left, and right of the evaluation pixel, and the pixels with the second shortest center-to-center distance from the evaluation pixel are the four pixels located to the upper left, upper right, lower left, and lower right of the evaluation pixel. For example, if the reference region includes seven pixels, the seven pixels include the evaluation pixel, the four pixels with the shortest center-to-center distance from the evaluation pixel, and any two pixels from the four pixels with the second shortest center-to-center distance from the evaluation pixel.

[0102] The requirement (D) is not satisfied by the filter arrays used in monochrome cameras, RGB cameras, and conventional snapshot-type hyperspectral cameras. ｋ is non-zero" means that the transmittance t ｋ This means that the pixel signal of the image sensor that detects the light transmitted through the filter has a value significantly larger than the noise. The filter array 20 suitable for generating hyperspectral image data is also suitable for pattern fitting.

[0103] In addition, two or more types of fluorescent dyes may be mixed in the reference region. For example, when fluorescent dye A and fluorescent dye E are mixed equally, the above variable I ｋis the average value of the fluorescence intensity of fluorescent dye A and the fluorescence intensity of fluorescent dye E in the kth band. ｋ is not limited to the average value of the multiple fluorescence intensities corresponding to the multiple mixed fluorescent dyes, but may be, for example, a weighted average or median value multiplied by weights according to the type of dye, etc.

[0104] Next, a method for determining the bands used to classify the fluorescent dyes will be described with reference to Figures 6A and 6B. Figure 6A is a diagram schematically illustrating the fluorescence intensities of two fluorescent dyes in bands 1 to 8. Each band has a width of approximately several nanometers. In the example shown in Figure 6A, the fluorescence intensities of the two fluorescent dyes are equal in bands 6 to 8. These bands are not necessary for classifying the two fluorescent dyes. Therefore, the bands used to classify the fluorescent dyes can be reduced from bands 1 to 8 to bands 1 to 5. Further reduction from bands 1 to 5 can be achieved by using a dimensionality reduction method. Figure 6B is a diagram schematically illustrating the fluorescence intensities of two fluorescent dyes in bands 1, 3, and 5. In the example shown in Figure 6B, the bands used to classify the fluorescent dyes are reduced from bands 1 to 5 to bands 1, 3, and 5 using dimensionality reduction. Examples of dimensionality reduction methods that can be used include Auto Encoder, Principle Component Analysis, and Singular Value Decomposition.

[0105] Once the bands used to classify the fluorescent dyes are known, a reference region that satisfies the above requirements (A) to (D) can be selected. The number of bands used to classify the fluorescent dyes increases with the number of types of fluorescent dyes. In other words, the number of two or more pixels included in the reference region changes depending on the number of types of fluorescent dyes.

[0106] Next, an example of a GUI displayed on the output device 60 will be described with reference to Figures 7A and 7B. Figure 7A is a diagram schematically showing an example of a GUI displayed on the output device 60 before classifying fluorescent dyes. The output device 60 shown in Figure 7A is a display of a smartphone.

[0107] A compressed image of the target scene is displayed at the top of the GUI shown in Fig. 7A. The target scene includes nine observation targets, which are numbered 1 to 9. Each observation target is affixed with one of fluorescent dyes A to C, each with a known fluorescence spectrum.

[0108] The object of observation can be extracted from the compressed image by, for example, edge detection. If the position of the object of observation is known, pattern fitting can be performed only on the pixels in the compressed image where the object of observation is located. Therefore, it is not necessary to perform pattern fitting on all pixels.

[0109] A load button for loading spectral information of fluorescent dyes is displayed near the center of the GUI shown in Fig. 7A. When the user selects this button, the processing circuitry 70 receives a signal indicating the button selection, loads the dye data and restoration table from the storage device 50, and generates brightness pattern data for each type of fluorescent dye. The dye data includes fluorescence spectral information of fluorescent dyes A to C.

[0110] 7A, a pattern fitting button and a compressed sensing button are displayed at the bottom. These buttons are used by the user to instruct the execution of pattern fitting or compressed sensing. When the user selects the pattern fitting button, the processing circuitry 70 receives a button selection signal and compares the luminance pattern data with the compressed image data to classify which fluorescent dye has been added to the object of observation in the target scene 10.

[0111] 7B is a diagram schematically illustrating an example of a GUI displayed on the output device 60 after the fluorescent dyes have been classified. The upper part of the GUI shown in FIG. 7B displays observation targets 1 to 9 in the target scene, with labels A to C indicating the type of fluorescent dye next to the observation target numbers 1 to 9. In the example shown in FIG. 7B, fluorescent dye A is attached to elliptical observation targets 1, 3, 7, and 9, fluorescent dye B is attached to polygonal observation targets 2 and 5, and fluorescent dye C is attached to rectangular observation targets 4, 6, and 8. In the example shown in FIG. 7B, when the user selects the number of an observation target, the processing circuitry 70 may receive a selection signal and cause the GUI to display a graph of the spectrum of the fluorescent dye attached to the observation target and a description of the fluorescent dye.

[0112] At the bottom of the GUI shown in Fig. 7B, a table showing the pattern match rates between observation targets 1 to 9 and fluorescent dyes A to C is displayed. The pattern match rates shown in Fig. 7B represent the degree to which predicted brightness pattern data and compressed image data match for the observation target. The pattern match rates shown in Fig. 7B can be obtained, for example, by averaging the pattern match rates for multiple pixels included in the observation target. The fluorescent dye attached to each observation target is a fluorescent dye, out of fluorescent dyes A to C, that shows a pattern match rate exceeding 0.9. In the example shown in Fig. 7B, pattern match rates exceeding 0.9 for each observation target are indicated in bold.

[0113] In the example shown in FIG. 7B , the highest pattern match rate among the pattern match rates of fluorescent dyes A to C for each observation target exceeds 0.9, indicating high accuracy in fluorescent dye classification. If the highest pattern match rate is below 0.9, the classification accuracy of the fluorescent dyes is not necessarily high. In this case, instead of labels A to C, an "unknown" label indicating unclassifiable may be displayed next to the number of the observation target. The "unknown" label may be displayed, for example, when the pattern match rate for the observation target does not meet a predetermined standard. The standard for the pattern match rate may be set by the user. The "unknown" label is an example of an image displayed when the pattern match rate is insufficient to classify the type of observation target. Alternatively, the user may return to the GUI shown in FIG. 7A and select the compressed sensing button. In response to the button selection signal, the processing circuitry 70 generates hyperspectral image data of the target scene using compressed sensing technology and compares the hyperspectral image data of the target scene with the dye data of the fluorescent dyes to determine the presence or absence of each type of fluorescent dye in the target scene. Alternatively, if the highest pattern matching rate is below 0.9, the processing circuitry 70 may display a message on the GUI prompting the user to generate hyperspectral image data of the target scene using compressed sensing. Alternatively, if the highest pattern matching rate is below 0.9, the processing circuitry 70 may automatically use compressed sensing technology to determine the presence or absence of each type of fluorescent dye in the target scene. If there is no known dye data, the user should initially select the compressed sensing button rather than the pattern fitting button.

[0114] Note that there may be other applications that use compressed image data, and in that case, the GUI shown in Fig. 7A may further display a button for selecting the other application in addition to the pattern fitting button and the compressed sensing button.

[0115] When the type of fluorescent dye attached to an observation target is determined by the shape of the observation target, the type of fluorescent dye can be classified according to the shape of the observation target. In this case, the dye data includes information on the distribution shape of the fluorescent dye in addition to spectral information of the fluorescent dye. In the example shown in Figure 7B, fluorescent dyes A to C are attached to observation targets with an elliptical shape, a polygonal line shape, and a rectangular shape, respectively.

[0116] Next, an example of the operation performed by the processing circuitry 70 in classifying fluorescent dyes will be described with reference to Figures 8A to 8C. Figure 8A is a flowchart showing an example of the operation performed by the processing circuitry 70. The processing circuitry 70 performs the following steps S101 to S106.

[0117] <Step S101 > The processing circuitry 70 acquires the dye data and the restoration table from the storage device 50 .

[0118] <Step S102> The processing circuitry 70 generates a plurality of pieces of luminance pattern data for each of a plurality of types of fluorescent dyes.

[0119] <Step S103> The processing circuitry 70 causes the image sensor 30 to capture the target scene 10 via the filter array 20 and generate compressed image data.

[0120] <Step S104> The processing circuitry 70 compares the luminance pattern data with the compressed image data to generate and output output data indicating the presence or absence of each type of fluorescent dye in the target scene. The output data may include, for example, label information for each type of fluorescent dye attached to portions of the target scene where the observation target is present. The output data may include, for example, information on the probability of the fluorescent dye being present in each pixel of the compressed image, and / or information on the probability of the fluorescent dye being present in multiple pixels of the compressed image that correspond to the observation target. The processing circuitry 70 may store the output data in the storage device 50.

[0121] <Step S105> As shown in FIG. 7B, the processing circuitry 70 causes the output device 60 to output the classification result indicated by the output data.

[0122] <Step S106> The processing circuitry 70 determines whether the classification accuracy is equal to or greater than a reference value. This determination can be made, for example, by determining whether the highest pattern match ratio among the pattern match rates of fluorescent dyes A to C for each of observation targets 1 to 9 is equal to or greater than 0.9. If the determination is Yes, the processing circuitry 70 terminates operation. If the determination is No, the processing circuitry 70 executes steps S102 to S106 again. If the classification accuracy is still not equal to or greater than the reference value in the second determination, the processing circuitry 70 may execute steps S102 to S106 again, or may terminate operation.

[0123] In the imaging device 100 according to this embodiment, compressed image data is used in pattern fitting, eliminating the need to restore a hyperspectral image. As a result, the processing load for classifying objects in a target scene by type can be significantly reduced compared to a configuration that restores a hyperspectral image. In the imaging device 100 according to this embodiment, the processing circuit 70 does not require a GPU or FPGA used for high-speed processing; a low-spec CPU is sufficient. In the imaging device 100 according to this embodiment, the processing speed is approximately 100 times faster than a configuration that restores a hyperspectral image. In the imaging device 100 according to this embodiment, the filters included in the filter array 20 do not need to be narrow-band filters, thereby achieving high sensitivity and spatial resolution.

[0124] If the classification accuracy is below a reference value, the processing circuitry 70 may automatically use compressed sensing technology to determine the presence or absence of each type of fluorescent dye in the target scene. FIG. 8B is a flowchart showing another example of the operations performed by the processing circuitry 70. The processing circuitry 70 performs the operations of steps S101 to S104 and the following operations of S107 to S110. The operations of steps S101 to S104 shown in FIG. 8B are the same as the operations of steps S101 to S104 shown in FIG. 8A, respectively.

[0125] <Step S107> The processing circuitry 70 determines whether the classification accuracy is equal to or greater than a reference value. If the determination is Yes, the processing circuitry 70 executes the operation of step S108. If the determination is No, the processing circuitry 70 executes the operation of step S109.

[0126] <Step S108> The processing circuitry 70 causes the output device 60 to output the classification result indicated by the output data.

[0127] <Step S109> The processing circuitry 70 generates hyperspectral image data of the target scene based on the compressed image data and the restoration table.

[0128] <Step S110> The processing circuitry 70 generates output data by comparing the hyperspectral image data with the pigment data, and then performs the operation of step S108.

[0129] Furthermore, the user may switch between pattern fitting and compressed sensing. The processing circuitry 70 causes the output device 60 to display a GUI that allows the user to instruct switching between the first pattern fitting mode and the second compressed sensing mode. The processing circuitry 70 performs pattern fitting in response to the user's instruction for the first mode, and performs compressed sensing in response to the user's instruction for the second mode.

[0130] 8C is a flowchart showing yet another example of the operations performed by processing circuitry 70. Processing circuitry 70 performs the following operations of steps S111 and S112, as well as steps S101 to S104 and S107 to S110. The operations of steps S101 to S104 and S107 to S110 shown in FIG. 8C are the same as the operations of steps S101 to S104 and S107 to S110 shown in FIG. 8B, respectively.

[0131] <Step S111> The processing circuit 70 determines whether the first mode or the second mode has been selected by the user, i.e., whether a signal for selecting the first mode or the second mode button has been received. If the determination is Yes, the processing circuit 70 executes the operation of step S112. If the determination is No, the external processing circuit 90 executes the operation of step S111 again.

[0132] <Step S112> The processing circuit 70 further determines whether the first mode has been selected, i.e., whether the received signal is a signal in the first mode. If the determination is Yes, the processing circuit 70 executes the operation of step S101. If the determination is No, the second mode has been selected, i.e., the received signal is a signal in the second mode, so the processing circuit 70 executes the operation of step S109.

[0133] (Embodiment 2) The storage device 50 included in the imaging device 100 does not need to store pigment data. Furthermore, the storage device 50 included in the imaging device 100 does not need to store not only pigment data but also a restoration table. Next, with reference to FIG. 9 , an example configuration of an imaging system according to embodiment 2 of the present disclosure will be described. FIG. 9 is a block diagram schematically showing the configuration of an imaging system 200 according to exemplary embodiment 2 of the present disclosure. The imaging system 200 shown in FIG. 9 includes the imaging device 100 shown in FIG. 4 and an external storage device 80. Note that in this specification, the term "imaging system" is also used to refer to the standalone imaging device 100 shown in FIG. 4.

[0134] In an example of embodiment 2, the storage device 50 included in the imaging device 100 stores the restoration table, and the external storage device 80 stores the pigment data. In step S101 shown in FIG. 8A , the processing circuitry 70 acquires the restoration table from the storage device 50 and acquires the pigment data from the external storage device 80.

[0135] In another example of the second embodiment, the external storage device 80 stores the restoration table and the dye data. The processing circuitry 70 acquires the dye data and the restoration table from the external storage device 80 in step S101 shown in FIG.

[0136] (Embodiment 3) The processing circuit 70 included in the imaging device 100 does not need to generate luminance pattern data. Next, with reference to FIG. 10 , an example configuration of an imaging system according to embodiment 3 of the present disclosure will be described. FIG. 10 is a block diagram schematically showing the configuration of an imaging system 300 according to exemplary embodiment 3 of the present disclosure. The imaging system 300 shown in FIG. 10 includes the imaging device 100 shown in FIG. 4 , an external storage device 80 that stores a restoration table and pigment data, and an external processing circuit 90. In the imaging system 300 according to embodiment 3, the external processing circuit 90 generates luminance pattern data.

[0137] 11A and 11B, an example of the operation performed by the processing circuit 70 and the external processing circuit 90 will be described. Fig. 11A is a flowchart showing an example of the operation performed by the processing circuit 70. The processing circuit 70 performs the following operations in steps S201 to S206.

[0138] <Step S201> The processing circuit 70 transmits a request signal for luminance pattern data to the external processing circuit 90.

[0139] <Step S202> The processing circuit 70 acquires the luminance pattern data from the external processing circuit 90.

[0140] <Steps S203 to S206> The operations of steps S203 to S206 are the same as the operations of steps S103 to S106 shown in Fig. 8A, respectively. However, the operation of step S206 differs from the operation of step S106 in that if the determination is No, the processing circuitry 70 executes steps S201 to S205 again.

[0141] Fig. 11B is a flowchart showing an example of the operation performed by the external processing circuit 90 between step S201 and step S202 shown in Fig. 11A. The external processing circuit 90 performs the following operations of steps S301 to S304.

[0142] <Step S301> The external processing circuit 90 determines whether or not a request signal has been received. If the determination is Yes, the external processing circuit 90 executes the operation of step S302. If the determination is No, the external processing circuit 90 executes the operation of step S301 again.

[0143] <Step S302> The external processing circuit 90 acquires the dye data and the restoration table from the external storage device 80.

[0144] <Step S303> The external processing circuit 90 generates luminance pattern data based on the pigment data and the restoration table.

[0145] <Step S304> The external processing circuit 90 transmits the luminance pattern data to the processing circuit 70.

[0146] Next, a method for checking whether or not the pattern fitting technique is used in the imaging device will be described with reference to FIG.

[0147] 12 is a diagram schematically illustrating an example in which an imaging device 900 captures an image of a color chart as a target scene 10. In the color chart, multiple color regions having different colors are distributed in a mosaic pattern, and the color boundaries are clearly defined. Instead of a color chart, a sheet on which different types of phosphors are applied in a mosaic pattern may be used.

[0148] In this embodiment, when the reference region contains one type of spectral information, it is possible to classify the subject contained in the target scene 10. For example, when two adjacent color regions having different colors are present, if the reference region is located only within one of the color regions, high classification accuracy can be achieved. In contrast, when the reference region straddles the two color regions, i.e., when the reference region includes a portion of each color region, classification accuracy drops significantly. This is because the reference region contains two types of spectral information. If classification accuracy drops significantly when repeatedly capturing images with the color chart shifted slightly, it is clear that pattern fitting technology is used in the imaging device 900.

[0149] Furthermore, in this embodiment, the number of pixels included in the reference area changes depending on the spectral information and number of objects included in the object data. When the reference area is displayed on an output device (not shown) of the image capture device 900, if the number of pixels included in the reference area changes in accordance with changes in the object data, it can be seen that the image capture device 900 uses pattern fitting technology.

[0150] Furthermore, if the GUIs shown in FIGS. 7A and 7B are displayed on an output device (not shown) of the imaging device 900, it can be seen that the imaging device 900 uses a pattern fitting technique.

[0151] (Application to foreign substance inspection) The imaging device 100 according to embodiments 1 to 3 can be used for foreign substance inspection, for example, in addition to classifying fluorescent dyes. Next, application examples of the imaging device 100 according to embodiments 1 to 3 will be described with reference to Figures 13A and 13B. Here, foreign substance inspection of medicines will be taken as an example, but foreign substance inspection of electronic components, for example, is also possible.

[0152] FIG. 13A is a schematic diagram illustrating an example in which multiple drug bags 14 transported by a belt conveyer are individually imaged by an imaging device 100. The imaging device 100 illustrated in FIG. 13A includes a camera 110, a processing device 120, and an output device 60. The camera 110 illustrated in FIG. 13A includes the filter array 20, image sensor 30, and optical system 40 illustrated in FIG. 4. The processing device 120 illustrated in FIG. 13A includes the storage device 50, processing circuit 70, and memory 72 illustrated in FIG. 4. The drug bags 14 contain large and small tablet-shaped drugs A and B, and large and small capsule-shaped drugs C and D. The spectral information of drug A contained in the multiple drug bags 14 is approximately the same. The same is true for drugs B to D. The storage device 50 stores drug data including spectral information of drugs A to D.

[0153] FIG. 13B is a diagram schematically illustrating an example of a GUI displayed on the output device 60 after classifying medications. The left portion of the GUI shown in FIG. 13B displays observation targets 1 to 4 in the target scene, with labels A to D for each medication type next to the observation target numbers 1 to 4. The right portion of the GUI shown in FIG. 13B displays a table showing pattern matches between observation targets 1 to 4 and medications A to D. In the example shown in FIG. 13B, observation target 1 is medication C, observation target 2 is medication A, observation target 3 is medication B, and observation target 4 is medication D. In the table, a circle indicates medications A to D with a pattern match rate of 90% or higher for each observation target. In the example shown in FIG. 13B, medication bag 14 contains medications A to D, indicating that medication bag 14 does not contain medications of the same type, medications other than medications A to D, or foreign matter.

[0154] (Other 1) Modifications of the embodiment of the present disclosure may be as follows.

[0155] an image sensor that generates a plurality of pixel values ​​in response to light from the filter array; and a processing circuit, wherein the processing circuit obtains a plurality of first pixel values ​​calculated based on information indicative of a spectral characteristic of a substance and information indicative of the plurality of transmission spectra, the calculation being performed as light from the substance is incident on the filter array and the image sensor generates the plurality of first pixel values ​​in response to light responsive to the incidence; and the processing circuit obtains a plurality of second pixel values ​​generated by the image sensor by imaging a target scene through the filter array, and the processing circuit determines whether the target scene includes the substance based on the plurality of first pixel values ​​and the plurality of second pixel values.

[0156] The substance may be a fluorescent substance.

[0157] (Other 2) In the present disclosure, the luminance pattern data and the compressed image may be generated by capturing an image using a method other than capturing an image using a filter array including a plurality of optical filters.

[0158] For example, the imaging device 100 may be configured such that the light-receiving characteristics of the image sensor 30 are changed for each pixel by processing the image sensor 30, and image data may be generated by capturing an image using the processed image sensor 160. That is, instead of encoding the light incident on the image sensor using the filter array 20, the image sensor may be endowed with a function for encoding the incident light, thereby generating luminance pattern data and a compressed image. In this case, the restoration table corresponds to the light-receiving characteristics of the image sensor.

[0159] Furthermore, an optical element such as a metalens may be introduced into at least a portion of the optical system 40, thereby changing the optical characteristics of the optical system 40 spatially and wavelength-wise, thereby encoding the incident light, and the luminance pattern data and compressed image may be generated by an imaging device including such a configuration. In this case, the restoration table contains information corresponding to the optical characteristics of the optical element such as a metalens. In this way, by using an imaging device 100 with a configuration different from that using the filter array 20, the intensity of the incident light may be modulated for each wavelength, the compressed image and luminance pattern data may be generated, and output data regarding the presence or absence of a substance in the target scene may be generated.

[0160] That is, the present disclosure may include the following aspects.

[0161] an imaging system comprising: an imaging device including a plurality of light-receiving regions having mutually different photoresponse characteristics; and a processing circuit, wherein the processing circuit acquires brightness pattern data generated by predicting a brightness pattern that will be detected when a substance is imaged by the imaging device based on subject data including spectral information of the at least one substance; acquires first image data obtained by imaging a target scene with the imaging device; and generates output data regarding the presence or absence of the substance in the target scene by comparing the brightness pattern data with the first image data.

[0162] Each of the plurality of light receiving regions may correspond to a pixel included in an image sensor.

[0163] The imaging device may include an optical element, and the optical response characteristics of the plurality of light-receiving regions may correspond to the spatial distribution of the transmission spectrum of the optical element.

[0164] (Other 3) As long as they do not deviate from the spirit of this disclosure, various modifications that a person skilled in the art may make to each embodiment, and forms constructed by combining components of different embodiments, are also included within the scope of this disclosure.

[0165] The imaging device according to the present disclosure can be used to classify objects included in a target scene by type, and can also be used to inspect for foreign matter.

[0166] REFERENCE SIGNS LIST 10 Target scene 12 Fluorescent dye 14 Drug bag 20 Filter array 22 Hyperspectral image 30 Image sensor 40 Optical system 50 Storage device 60 Output device 70 Processing circuit 80 External storage device 90 External processing circuit 100 Imaging device 110 Camera 120 Processing device 900 Imaging device

Claims

1. A filter array including a plurality of filters having different transmission spectra from each other, An image sensor that images light transmitted through the filter array and generates image data, A processing circuit, comprising: The processing circuit acquires luminance pattern data generated by predicting a luminance pattern detected when the substance is imaged by the image sensor based on subject data including spectral information of at least one substance, acquires first image data obtained by imaging a target scene with the image sensor, and generates output data regarding the presence or absence of the substance in the target scene by comparing the luminance pattern data and the first image data. An imaging system.

2. further comprising a storage device that stores the subject data and a table indicating a spatial distribution of the transmission spectra of the filter array, The processing circuit acquires the subject data and the table from the storage device, and generates the luminance pattern data based on the subject data and the table. The imaging system according to claim 1.

3. further comprising a storage device that stores a table indicating a spatial distribution of the transmission spectra, The processing circuit acquires the table from the storage device, acquires the subject data from the outside, and generates the luminance pattern data based on the subject data and the table. The imaging system according to claim 1.

4. The processing circuit acquires the luminance pattern data from the outside. The imaging system according to claim 1.

5. The spectral information of the at least one substance includes the spectral information of a plurality of substances. The output data relates to the presence or absence of each of the plurality of substances in the target scene. The imaging system according to any one of claims 1 to 4.

6. The processing circuit determines the presence or absence of the substance in the target scene by comparing the luminance pattern data and the first image data in a reference region including two or more pixels. The imaging system according to any one of claims 1 to 4.

7. The number of the two or more pixels included in the reference region varies according to the number of the plurality of substances. The imaging system according to claim 6.

8. The target wavelength range spectrally split by the imaging system includes n bands. The two or more pixels included in the reference region include n pixels that are a certain evaluation pixel and pixels in the vicinity of the evaluation pixel. Only one substance, not a plurality of substances, exists in the reference region. The filter array includes n filters respectively corresponding to the n pixels included in the reference region, and the transmission spectra of the n filters are different from each other. The transmittance of each of the n filters for the n bands is all non-zero. The imaging system according to claim 6.

9. The output data includes information on the probability of existence of the substance in each pixel of the first image data and / or information on the probability of existence of the substance in a plurality of pixels corresponding to an observation target among the first image data. The imaging system according to any one of claims 1 to 4.

10. The subject data further includes shape information of the at least one substance. The imaging system according to any one of claims 1 to 4.

11. Further comprising an output device, The processing circuit causes the output device to output the classification result indicated by the output data. The imaging system according to any one of claims 1 to 4.

12. The output device displays an image in which a label of each type is attached to a portion where the substance of the target scene exists. The imaging system according to claim 11.

13. The output device displays at least one of a graph of the spectrum of the substance and an image showing a description of the substance. The imaging system according to claim 11.

14. The output device displays an image in which a label indicating that the type of the observation target cannot be classified is attached to an observation target in the target scene where the probability of existence of the substance is lower than a certain value. The imaging system according to claim 11.

15. Each of the plurality of filters has two or more maximum values in a target wavelength range to be spectroscopically analyzed by the imaging system. The imaging system according to any one of claims 1 to 4.

16. The plurality of filters includes four or more types of filters, and among the four or more types of filters, a transmission range of a certain type of filter overlaps with a part of a transmission range of another type of filter. The imaging system according to any one of claims 1 to 4.

17. The first image data is compressed image data encoded by the filter array, The processing circuit generates hyperspectral image data of the target scene based on the compressed image data of the target scene. The imaging system according to any one of claims 1 to 4.

18. The first image data is compressed image data encoded by the filter array, The processing circuit, causes the output device to display a GUI for a user to instruct generation of hyperspectral image data of the target scene, and generates the hyperspectral image data of the target scene based on the compressed image data of the target scene according to an instruction of the user. The imaging system according to claim 11.

19. The first image data is compressed image data encoded by the filter array, The processing circuit, causes the output device to display a GUI for a user to instruct switching between a first mode of generating the output data and a second mode of generating hyperspectral image data of the target scene, generates the output data according to an instruction of the user in the first mode, and generates the hyperspectral image data of the target scene based on the compressed image data of the target scene according to an instruction of the user in the second mode. The imaging system according to claim 11.

20. A method executed on a computer, the method comprising: acquiring first image data obtained by imaging a target scene with an image sensor that generates image data by imaging light that has passed through a filter array including a plurality of filters having different transmission spectra; acquiring luminance pattern data generated by predicting a luminance pattern detected when the subject is imaged by the image sensor based on subject data including spectral information of at least one type of subject; ​Generating output data indicating the presence or absence of the subject in the target scene by comparing the luminance pattern data and the first image data, Method.

21. A computer program executed by a computer, The computer program causes the computer to, Acquire first image data obtained by imaging a target scene with an image sensor that generates image data by imaging light transmitted through a filter array including a plurality of filters having different transmission spectra from each other, Acquire luminance pattern data generated by predicting a luminance pattern detected when the subject is imaged by the image sensor based on subject data including spectral information of at least one type of subject, Generate and output output data indicating the presence or absence of the subject in the target scene by comparing the luminance pattern data and the first image data, To execute, Computer program.

22. A first region, An image sensor, Including a processing circuit, The first region includes a plurality of filters having a plurality of transmission spectra, the plurality of filters and the plurality of transmission spectra are in one-to-one correspondence, and the plurality of transmission spectra are different from each other, The image sensor generates a plurality of pixel values in response to light from the first region, The processing circuit acquires a plurality of first pixel values calculated based on information indicating spectral characteristics of a substance and information indicating the plurality of transmission spectra, The calculation is performed assuming that light from the substance is incident on the first region and the image sensor generates the plurality of first pixel values in response to the light in response to the incidence, The processing circuit acquires a plurality of second pixel values generated by the image sensor capturing a target scene through the first region, The processing circuit determines whether the target scene includes the substance based on the plurality of first pixel values and the plurality of second pixel values, The first region is included in the image sensor, or the first region is not included in the image sensor An imaging system.