Signal processing method, signal processing device, and imaging system

Abstract

A signal processing method according to one aspect of the present disclosure comprises: acquiring compressed image data including two-dimensional image information obtained by compressing hyperspectral information in a target wavelength range; acquiring setting data that specifies one or more sub-wavelength regions that are a part of the target wavelength region; and generating, on the basis of the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength regions.

Description

[Technical Field]

[0001] The present disclosure relates to a signal processing method, a signal processing device, and an imaging system. [Background technology]

[0002] By utilizing spectral information from multiple narrow bands, for example, several dozen bands, it becomes possible to understand the detailed physical properties of an object, which was not possible with conventional RGB images. A camera that captures such multi-wavelength information is called a "hyperspectral camera." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Document 1 discloses an example of a hyperspectral imaging device that uses compressed sensing. The imaging device includes an encoding element, which is an array of multiple optical filters with different wavelength-dependent light transmittances, an image sensor that detects light transmitted through the encoding element, and a signal processing circuit. The encoding element is disposed on an optical path connecting the subject and the image sensor. The image sensor simultaneously detects light in which components of multiple wavelength bands are superimposed for each pixel to acquire a single wavelength-multiplexed image. The signal processing circuit reconstructs image data for each of the multiple wavelength bands by applying compressed sensing to the acquired wavelength-multiplexed image using information on the spatial distribution of the spectral transmittance of the encoding element. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,599,511 Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional hyperspectral imaging devices, image data is generated and displayed for each of all wavelength bands included in the wavelength range of the acquired wavelength-multiplexed image (hereinafter referred to as the "target wavelength range"). However, depending on the application, information on only a portion of the target wavelength range may be required. In such cases, it is not efficient to generate information on unnecessary wavelength bands with high wavelength resolution.

[0006] The present disclosure provides techniques for efficiently generating images of required wavelength bands. [Means for solving the problem]

[0007] A method according to one aspect of the present disclosure is a computer-implemented signal processing method that includes: acquiring compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges.

[0008] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices. [Effects of the Invention]

[0009] According to the present disclosure, images of required wavelength bands can be efficiently generated. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1A is a schematic diagram of an exemplary hyperspectral imaging system. [Figure 1B] FIG. 1B is a schematic diagram illustrating a first variation of an exemplary hyperspectral imaging system. [Figure 1C] FIG. 1C is a schematic diagram illustrating a second variation of an exemplary hyperspectral imaging system. [Figure 1D] FIG. 1D is a schematic diagram of a third variation of an exemplary hyperspectral imaging system. [Figure 2A] FIG. 2A is a diagram schematically illustrating an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of the spatial distribution of the light transmittance of each of a plurality of wavelength bands W1, W2, . . . , WN included in the target wavelength range. [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of the region A1 included in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of the region A2 included in the filter array shown in FIG. 2A. [Figure 3A] FIG. 3A is a diagram showing an example of the relationship between a target wavelength range W and a plurality of wavelength bands W1, W2, . . . , WN included therein. [Figure 3B] FIG. 3B is a diagram showing another example of the relationship between the target wavelength range W and the multiple wavelength bands W1, W2, . . . , WN included therein. [Figure 4A] FIG. 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array. [Figure 4B] FIG. 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W1, W2, . . . , WN. [Figure 5] FIG. 5 is a diagram schematically illustrating a usage scene of a hyperspectral camera. [Figure 6A] FIG. 6A is a diagram showing an example of a target wavelength range W and a specified sub-wavelength range Wa. [Figure 6B] FIG. 6B is a diagram showing an example in which a second sub-wavelength range is specified in addition to the first sub-wavelength range. [Figure 7] FIG. 7 is a diagram showing a configuration of an imaging system according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart showing the operation of the system. [Figure 9] FIG. 9 is a diagram showing an example of mask data before conversion stored in the memory. [Figure 10] FIG. 10 is a diagram showing an example of a GUI for inputting imaging conditions. [Figure 11] FIG. 11 is a diagram showing an example of a GUI for inputting restoration conditions. [Figure 12] FIG. 12 is a diagram showing an example of a GUI for inputting restoration conditions. [Figure 13] FIG. 13 is a diagram showing an example of a screen displaying a spectroscopic image generated as a result of the restoration calculation. [Figure 14] FIG. 14 is a diagram for explaining an example of a method for synthesizing mask information of a plurality of bands and converting it into new mask information. [Figure 15A] FIG. 15A is a diagram showing an example of converted mask data recorded in memory. [Figure 15B] FIG. 15B is a diagram showing another example of the converted mask data recorded in the memory. [Figure 16] FIG. 16 is a diagram showing an example of a method for generating an image for each of a plurality of wavelength bands included in the target wavelength range. [Figure 17] FIG. 17 is a diagram showing the configuration of a system in which the signal processing circuit does not convert mask information. [Figure 18] FIG. 18 is a diagram showing another example of a GUI for setting restoration conditions. [Figure 19] FIG. 19 is a diagram showing an example of displaying an image in an unspecified wavelength range. [Figure 20] FIG. 20 is a diagram showing an example of a method for restoring only a specific sub-wavelength range with high wavelength resolution by performing two-stage restoration. DETAILED DESCRIPTION OF THE INVENTION

[0011] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangements, positions and connection forms, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0012] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A field programmable gate array (FPGA), which is programmable after LSI fabrication, or a reconfigurable logic device, which can reconfigure connections within an LSI or set up circuit partitions within an LSI, may also be used for the same purpose.

[0013] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0014] First, a configuration example of a hyperspectral imaging system according to an embodiment of the present disclosure and findings discovered by the present inventors will be described.

[0015] FIG. 1A is a schematic diagram illustrating an exemplary hyperspectral imaging system. The system includes an imaging device 100 and a processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device 100 also includes an optical system 140, a filter array 110, and an image sensor 160. The filter array 110 has a structure and function similar to that of the "encoding element" disclosed in Patent Document 1. For this reason, in the following description, the filter array 110 may be referred to as the "encoding element." The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject of imaging. The filter array 110 is disposed between the optical system 140 and the image sensor 160.

[0016] FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple, but may be any object. The processing device 200 generates image data for each of a plurality of wavelength bands included in the target wavelength range based on the image data generated by the image sensor 160. This image data is referred to as "spectral image data" in this specification. Here, the number of wavelength bands included in the target wavelength range is N (N is an integer equal to or greater than 4). In the following description, the generated spectral image data for the plurality of wavelength bands will be referred to as spectral images 220W1, 220W2, ..., 220W N These are collectively referred to as a spectroscopic image 220. In this specification, a collection of data or signals representing an image, that is, data or signals representing the pixel values of each pixel, may be simply referred to as an "image."

[0017] The filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittance. The filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 110 is referred to as "encoding" in this specification.

[0018] 1A, the filter array 110 is disposed near or directly above the image sensor 160. Here, "near" means close enough that a reasonably clear image of light from the optical system 140 is formed on the surface of the filter array 110. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0019] 1A, optical system 140 includes at least one lens. Although optical system 140 is shown as a single lens in FIG. 1A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 10.

[0020] The filter array 110 may be located away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is located away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is located between the optical system 140 and the image sensor 160 and at a position away from the image sensor 160. In the example of FIG. 1C, the filter array 110 is located between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, with the filter array 110 located between them. As in these examples, an optical system including one or more lenses may be located between the filter array 110 and the image sensor 160.

[0021] The image sensor 160 is a monochrome photodetector having a plurality of photodetecting elements (also referred to herein as "pixels") arranged two-dimensionally. The image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), an infrared array sensor, a terahertz array sensor, or a millimeter-wave array sensor. The photodetecting elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome sensor. For example, a color sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may also be used. Using a color sensor can increase the amount of wavelength-related information and improve the accuracy of reconstructing the spectroscopic image 220. The wavelength range to be acquired may be determined arbitrarily and is not limited to the visible wavelength range, but may also be ultraviolet, near-infrared, mid-infrared, far-infrared, microwave, or radio wave wavelength ranges.

[0022] The processing device 200 is a computer including a processor and a storage medium such as a memory. The processing device 200 generates a plurality of spectral images 220W1, 220W2, ..., 220W3, each including information of a plurality of wavelength bands, based on the image 120 acquired by the image sensor 160. N Generate data.

[0023] FIG. 2A is a diagram schematically illustrating an example of a filter array 110. The filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, these regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0024] 2A, the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0025] FIG. 2B shows multiple wavelength bands W1, W2, . . . , W included in the target wavelength range. N 2B is a diagram showing an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in FIG. 2B, the difference in shading in each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0026] 2C and 2D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 110 shown in FIG. 2A, respectively. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary for all regions to have different spectral transmittances. In the filter array 110, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0027] 3A and 3B show a target wavelength range W and multiple wavelength bands W1, W2, . . . , W NThis figure explains the relationship between the wavelengths of the light and the wavelengths of the light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a radio wave range such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. Thus, the wavelength range used is not limited to the visible light range. In this specification, for convenience, "light" refers not only to visible light but also to non-visible light such as near-ultraviolet, near-infrared, and radio waves.

[0028] In the example shown in FIG. 3A, N is an arbitrary integer equal to or greater than 4, and the target wavelength range W is divided into N equal wavelength ranges, each of which is designated as wavelength bands W1, W2, . . . , W N However, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the multiple wavelength bands may be determined arbitrarily as long as they are different from each other.

[0029] 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has multiple maximum values P1 to P5 and multiple minimum values for wavelengths in the target wavelength band W. In the example shown in FIG. 4A, the optical transmittance in the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 4A, the optical transmittance in wavelength band W2 and wavelength band W3 is normalized so that the maximum value is 1 and the minimum value is 0. N-1 In this way, the spectral transmittance of each region is expressed as a multiple of wavelength bands W1 to W2. N In the example of Fig. 4A, the maximum values P1, P3, P4, and P5 are 0.5 or more.

[0030] As described above, the light transmittance of each region varies with wavelength. Therefore, the filter array 110 transmits a greater amount of components in a certain wavelength range of the incident light and less of the components in other wavelength ranges. For example, for the light in k of the N wavelength bands, the transmittance may be greater than 0.5, and for the light in the remaining N - k wavelength ranges, the transmittance may be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that equally contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of intensity peaks discrete with respect to wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0031] FIG. 4B shows, as an example, the spectral transmittance shown in FIG. 4A averaged for each of the wavelength bands W1, W2, ···, W N It is a diagram showing the result. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is taken as the transmittance in that wavelength band. In this example, in three wavelength ranges taking the maximum values P1, P3, and P5, the transmittance is prominently high. In particular, in two wavelength ranges taking the maximum values P3 and P5, the transmittance exceeds 0.8.

[0032] In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution in which the transmittance of each region can take any value from 0 to 1 is assumed. However, it is not necessarily a grayscale transmittance distribution. For example, a binary scale transmittance distribution in which the transmittance of each region can take either a value close to 0 or a value close to 1 may be adopted. In the binary scale transmittance distribution, each region transmits most of the light in at least two of the plurality of wavelength ranges included in the target wavelength range and does not transmit most of the light in the remaining wavelength ranges. Here, "most" generally refers to 80% or more.

[0033] A portion of the cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength bands W1 to W2 included in the wavelength range W of interest. N The filter array 110 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0034] Such data indicating the spatial distribution of the spectral transmittance of the filter array 110 is obtained in advance using design data or actual measurement calibration, and is stored in a storage medium provided in the processing device 200. This data is used in the calculation process described below.

[0035] The filter array 110 can be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing metal. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is different for each cell. This allows different spectral characteristics to be achieved for each cell. The use of a multilayer film allows for sharp rises and falls in the spectral transmittance. A configuration using organic materials can be achieved by using different pigments or dyes in each cell or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each cell. When a microstructure containing metal is used, it can be fabricated using plasmon effect-based spectral analysis.

[0036] Next, an example of signal processing by the processing device 200 will be described. The processing device 200 reconstructs a multi-wavelength spectral image 220 based on the image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, "multi-wavelength" means a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a typical color camera. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the number of bands. Depending on the application, the number of bands may exceed 100.

[0037] The data to be obtained is the data of the spectroscopic image 220, and this data is denoted as f. If the number of bands is N, f is the image data of each band, f1, f2, . . . , f N Here, as shown in Figure 1A, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is n, and the number of pixels in the y direction is m, then the image data f1, f2, ..., f N Each of the elements f is two-dimensional data with n×m pixels. Therefore, data f is three-dimensional data with n×m×N elements. This three-dimensional data is called a "hyperspectral data cube" or "hyperspectral cube." Meanwhile, data g of image 120, which is acquired by encoding and multiplexing using filter array 110, has n×m elements. Data g can be expressed by the following equation (1).

number

[0038] where f1, f2, . . . , f N Each of these is data with n × m elements. Therefore, the vector on the right side is strictly a one-dimensional vector with n × m × N rows and 1 column. The vector g is converted into a one-dimensional vector with n × m rows and 1 column, and then calculated. The matrix H is the sum of the components f1, f2, ..., f of the vector f. Nrepresents a transformation in which H is encoded and intensity-modulated with different encoding information (hereinafter also referred to as "mask information") for each wavelength band, and then added together. Therefore, H is a matrix with n x m rows and n x m x N columns. In this specification, the matrix H may be referred to as the "system matrix."

[0039] Given a vector g and a matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the processing device 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

number

[0040] Here, f' represents the estimated data of f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The processing device 200 can converge the solution by recursive iterative calculations and calculate the final solution f'.

[0041] The first term in the parentheses in equation (2) represents the sum of squares of the difference between the acquired data g and Hf, which is the result of transforming the estimation process f by the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT) of f, the wavelet transform, the Fourier transform, or the total variation (TV) of f. For example, using the total variation transform can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the regularization term space can be selected. Alternatively, multiple regularization terms can be included in the calculation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0042] In the configurations of FIGS. 1B and 1C , the image encoded by the filter array 110 is acquired in a blurred state on the imaging plane of the image sensor 160. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned system matrix H, the spectral image 220 can be reconstructed. Here, the blur information is expressed by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to a pixel on an image spreads due to blurring to a k×k pixel region around the pixel, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the influence on the luminance of each pixel within that region. The spectral image 220 can be reconstructed by reflecting the influence of blurring of the encoding pattern due to the PSF in the system matrix H. The filter array 110 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 110 to be lost due to excessive diffusion can be selected.

[0043] In the above configuration, as shown in FIGS. 3A and 3B, a plurality of wavelength bands W1 to W2 included in the target wavelength range W are N However, depending on the application, images for all of these wavelength bands may not be required. In such cases, it is inefficient to generate images with high wavelength resolution for all wavelength bands.

[0044] FIG. 5 is a diagram illustrating typical usage scenarios for a hyperspectral camera. In one scenario, a user may wish to acquire color information in the red wavelength range, for example, from 600 nm to 700 nm, to estimate the sugar content of an apple. In another scenario, a user may wish to acquire color information in the green wavelength range, for example, from 500 nm to 600 nm, to determine the exact spectrum of a leaf. In yet another scenario, a user may wish to acquire color information in the blue wavelength range, for example, from 400 nm to 500 nm, to determine the degree of fading of a blue product.

[0045] In such situations, conventional hyperspectral cameras use either of the following methods (1) or (2) to acquire color information in different wavelength ranges: (1) An image sensor capable of acquiring information in each wavelength range independently is used. (2) A camera capable of acquiring information over a wide wavelength range is used to acquire a wide range of color information, and only the information over the necessary wavelength range is displayed.

[0046] In the method (1), it is necessary to prepare a separate image sensor for an application that requires color information different from the color information required for a particular application.

[0047] In method (2), color information is acquired from a wavelength range wider than the required wavelength range, and images are generated for each of the many wavelength bands included in that wide wavelength range. As a result, many calculations are performed for unnecessary wavelength ranges, resulting in unnecessary high calculation costs.

[0048] Therefore, in an embodiment of the present disclosure, a user can specify one or more subwavelength bands that are part of the target wavelength range W based on image data acquired by a hyperspectral imaging device. The signal processing device generates a hyperspectral data cube that represents multiple two-dimensional images of multiple wavelength bands included in the specified one or more subwavelength bands. This allows detailed spectral information about the subwavelength band desired by the user to be obtained while reducing computational costs.

[0049] FIG. 6A shows the wavelength range of interest W and the designated sub-wavelength range W a In this example, one sub-wavelength range W a Only the sub-wavelength range W is specified. a is a part of the target wavelength range W and includes multiple wavelength bands W a1 , W a2 , , W ai where i is the sub-wavelength range W a The signal processing unit processes these wavelength bands W a1 , W a2 , , W ai Generate a hyperspectral data cube showing a two-dimensional image for each of the

[0050] FIG. 6B shows the first sub-wavelength range W a In addition, the second sub-wavelength range W b The first sub-wavelength range W a and the second sub-wavelength range W b are spaced apart and are all included in the target wavelength range W. The second sub-wavelength range W b is a set of multiple wavelength bands W b1 , W b2 , , Wbj where j is the second sub-wavelength range W b In this way, multiple sub-wavelength ranges may be specified.

[0051] In this way, the signal processing device generates images of multiple wavelength bands included in the specified subwavelength range based on the image data generated by the imaging device. The generated images of the multiple wavelength bands are displayed on the display. This operation reduces the computational cost required to generate unnecessary spectral images for the intended usage scenario.

[0052] An outline of an embodiment of the present disclosure will be described below.

[0053] A computer-implemented signal processing method according to an embodiment of the present disclosure includes: acquiring compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a wavelength range of interest; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the wavelength range of interest; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges.

[0054] "Hyperspectral information within a target wavelength range" refers to information on multiple images corresponding to multiple wavelength bands included in a predetermined target wavelength range. "Compressing hyperspectral information" includes compressing image information of multiple wavelength bands into a single monochrome two-dimensional image using an encoding element such as the filter array 120 described above, and compressing previously acquired image information of multiple wavelength bands into a single monochrome two-dimensional image using software processing.

[0055] The above method makes it possible to generate a hyperspectral data cube, which is a set of two-dimensional image data corresponding to multiple wavelength bands included in one or more subwavelength ranges specified by a user, thereby enabling the generation of only the hyperspectral data cubes required for a given application or purpose.

[0056] The hyperspectral information may be information on four or more wavelength bands included in the target wavelength range, and the two-dimensional image information may be data on a plurality of pixels included in the compressed image data. The data on each of the plurality of pixels may have information on the four or more wavelength bands superimposed thereon. In other words, the data on each pixel of the compressed image data may include a single value on which information on four or more wavelength bands included in the target wavelength range is superimposed. Depending on the application, the data on each pixel of the compressed image data may have information on 10 or more or 100 or more wavelength bands superimposed thereon.

[0057] The setting data may include information specifying a wavelength resolution in the one or more sub-wavelength bands. The two-dimensional images may be generated at the wavelength resolution. In this embodiment, the user can specify the sub-wavelength bands and also the wavelength resolution for each sub-wavelength band. This allows for flexible adjustment, such as increasing the wavelength resolution for a sub-wavelength band that requires detailed spectral information.

[0058] The one or more sub-wavelength ranges may include a first sub-wavelength range and a second sub-wavelength range, and the plurality of two-dimensional images may be generated for each of the first sub-wavelength range and the second sub-wavelength range.

[0059] The one or more sub-wavelength ranges may include a first sub-wavelength range and a second sub-wavelength range. The wavelength resolution may be independently specified for each of the first sub-wavelength range and the second sub-wavelength range. The plurality of two-dimensional images may be generated at the corresponding wavelength resolution for each of the first sub-wavelength range and the second sub-wavelength range.

[0060] The first sub-wavelength range and the second sub-wavelength range may be spaced apart, or the first sub-wavelength range and the second sub-wavelength range may be adjacent to each other or may partially overlap each other.

[0061] The method may further include displaying a graphical user interface (GUI) on a display connected to the computer for allowing a user to input the setting data. By displaying such a GUI, the user can easily perform operations such as specifying sub-wavelength bands and specifying wavelength resolution for each sub-wavelength band.

[0062] The method may further include displaying the plurality of two-dimensional images on a display connected to the computer, thereby allowing a user to easily check the generated spectral images for each wavelength band.

[0063] The compressed image data may be generated by capturing an image using an image sensor and a filter array including a plurality of optical filters with different spectral transmittances. The method may further include acquiring mask data that reflects the spatial distribution of the spectral transmittances of the filter array. The plurality of two-dimensional images may be generated based on the compressed image data and the mask data.

[0064] The method may further include acquiring mask data including information on a plurality of mask images acquired by capturing, with the image sensor, a plurality of backgrounds corresponding to a plurality of component bands included in the target wavelength range through the filter array. The plurality of two-dimensional images may be generated based on the compressed image data and the mask data.

[0065] The mask data may be, for example, data defining the matrix H in the above-described equation (2). The format of the mask data may vary depending on the configuration of the imaging system. The mask data may indicate the spatial distribution of the spectral transmittance of the filter array, or may include information for calculating the spatial distribution of the spectral transmittance of the filter array. For example, the mask data may include information on a background image for each component band in addition to the above-described mask image information. By dividing the mask image by the background image for each pixel, transmittance distribution information for each component band can be obtained. The mask data may include only information on the mask image. The mask image indicates a distribution of values obtained by multiplying the transmittance of the filter array by the sensitivity of the image sensor. Such mask data may be used in a configuration in which the filter array is arranged closely facing the image sensor.

[0066] The mask data may include mask information. The mask information may indicate a spatial distribution of transmittance of the filter array in each of a plurality of component bands included in the target wavelength range. The method may further include generating composite mask information by synthesizing a portion of the mask information, the portion corresponding to a plurality of component bands included in a non-designated wavelength range other than the one or more sub-wavelength ranges of the target wavelength range, and generating a composite image corresponding to the non-designated wavelength range based on the compressed image data and the composite mask information.

[0067] The method may further include generating a composite mask image by combining the mask images for multiple unit bands included in a non-designated wavelength range other than the one or more specified sub-wavelength ranges within the target wavelength range, and generating composite image data for the non-designated wavelength range based on the compressed image data and the composite mask image.

[0068] According to the above method, detailed spectral images are not generated for non-designated wavelength ranges, but are generated for designated sub-wavelength ranges only, thereby reducing the calculation time required to generate spectral images.

[0069] The mask data may further include information on a plurality of background images obtained by capturing the plurality of backgrounds with the image sensor without using the filter array. The method may further include generating a composite background image by combining the plurality of background images. The composite image may be generated based on the compressed image data, the composite mask image, and the composite background image.

[0070] The mask data may include a plurality of background images and a plurality of mask images. Each of the plurality of background images may be acquired, for example, by capturing an image of a corresponding background of the plurality of backgrounds with the image sensor without using the filter array. Each of the plurality of mask images may be acquired, for example, by capturing an image of the corresponding background of the plurality of backgrounds with the image sensor through the filter array. The composite mask information may be generated based on the plurality of mask images and the plurality of background images.

[0071] The method may further include displaying the composite image on a display connected to the computer, thereby allowing a user to easily check a rough image for an unspecified wavelength range.

[0072] The mask data may include mask information. The mask information may indicate a spatial distribution of transmittance of the filter array in each of a plurality of component bands included in the target wavelength range. The setting data may include information specifying a plurality of large sub-wavelength bands, each of which is part of the target wavelength range, and a plurality of small sub-wavelength bands included in at least one of the plurality of large sub-wavelength bands. The method may further include generating first composite mask information by synthesizing, for each of the plurality of large sub-wavelength bands, a portion of the mask information corresponding to a plurality of component bands included in each of the plurality of large sub-wavelength bands, and generating a first composite image for each of the plurality of large sub-wavelength bands based on the compressed image data and the first composite mask information. The plurality of two-dimensional images may be generated corresponding to the plurality of small sub-wavelength bands. The method may further include generating, for each of the plurality of small sub-wavelength bands, second composite mask information by combining the mask information for a plurality of unit bands included in the small sub-wavelength band, and generating, for each of the small sub-wavelength bands, a second composite image based on the first composite image for the at least one of the specified plurality of large sub-wavelength bands and the second composite mask information.

[0073] According to the above method, it is possible to generate a composite image for each of a plurality of large sub-wavelength bands included in a target wavelength band, and a plurality of small sub-wavelength bands included in at least one of the large sub-wavelength bands, as specified by a user, for example. This allows for flexible adjustment, such as setting small sub-wavelength bands only for wavelength bands for which detailed color information is required, and setting only large sub-wavelength bands for wavelength bands for which detailed color information is not required.

[0074] The target wavelength range may include a visible wavelength range. The method may further include generating an image corresponding to a red wavelength range, an image corresponding to a green wavelength range, and an image corresponding to a blue wavelength range based on the compressed image data and the synthesis mask information, and displaying an RGB image based on the image corresponding to the red wavelength range, the image corresponding to the green wavelength range, and the image corresponding to the blue wavelength range on a display connected to the computer. This allows a user to check the RGB image of the target object separately from detailed spectral images for the specified sub-wavelength ranges.

[0075] A method according to yet another embodiment of the present disclosure is a method for generating mask data. The mask data is used to restore spectral image data for each wavelength band from compressed image data acquired by an imaging device including a filter array including multiple types of optical filters with different spectral transmittances. That is, the method generates mask data used to restore spectral image data for each wavelength band from compressed image data acquired by an imaging device including a filter array including multiple types of optical filters with different spectral transmittances. The method includes acquiring first mask data for restoring first spectral images corresponding to a first set of wavelength bands in a target wavelength range, acquiring setting data specifying one or more sub-wavelength ranges that are part of the target wavelength range, and generating second mask data for restoring second spectral image data corresponding to a second set of wavelength bands in the one or more sub-wavelength ranges based on the first mask data and the setting data.

[0076] The first wavelength band group may be a collection of all or a portion of wavelength bands included in the target wavelength range. The second wavelength band group may be a collection of all or a portion of wavelength bands included in the sub-wavelength range. Each of the first wavelength band group and the second wavelength band group may be a collection of composite bands formed by combining two or more unit wavelength bands. When such band synthesis is performed, the mask data is converted according to the band synthesis mode. The setting data may include information regarding the band synthesis mode used in the mask data conversion process.

[0077] The first mask data and the second mask data may be data reflecting a spatial distribution of the spectral transmittance of the filter array. The first mask data may include first mask information indicating a spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second mask data may include second mask information indicating a spatial distribution of the spectral transmittance corresponding to the second wavelength band group.

[0078] The second mask data may further include third mask information obtained by combining a plurality of pieces of information, each of which indicates the spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength range other than the one or more sub-wavelength ranges within the target wavelength range.

[0079] A signal processing device according to another aspect of the present disclosure includes a processor and a memory storing a computer program executed by the processor, the computer program causing the processor to acquire compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range, acquire configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range, and generate, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges.

[0080] A signal processing device according to another aspect of the present disclosure includes a processor and a memory storing a computer program executed by the processor, the computer program causing the processor to acquire first mask data for reconstructing first spectral image data corresponding to a first set of wavelength bands in a target wavelength range, acquire setting data specifying one or more sub-wavelength ranges that are part of the target wavelength range, and reconstruct, based on the first mask data and the setting data, second mask data for generating second spectral image data corresponding to a second set of wavelength bands in the one or more sub-wavelength ranges.

[0081] An imaging system according to another aspect of the present disclosure includes the signal processing device and an imaging device that generates the compressed image data.

[0082] A computer program according to another aspect of the present disclosure causes a computer to acquire compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range, acquire setting data specifying one or more sub-wavelength ranges that are part of the target wavelength range, and generate, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges.

[0083] A computer program according to another aspect of the present disclosure causes a computer to acquire, from compressed image data acquired by an imaging device including a filter array including multiple types of optical filters with different spectral transmittances, first mask data for restoring first spectral image data corresponding to a first group of wavelength bands in a target wavelength range; acquire setting data for designating one or more sub-wavelength ranges that are part of the target wavelength range; and generate, based on the first mask data and the setting data, second mask data for restoring second spectral image data corresponding to a second group of wavelength bands in the one or more sub-wavelength ranges.

[0084] According to another aspect of the present disclosure, a computer-readable non-transitory storage medium stores a program for causing a computer to execute a process including: acquiring compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges.

[0085] According to another aspect of the present disclosure, a computer-readable non-transitory storage medium stores a program for causing a computer to execute a process including: acquiring first mask data for reconstructing first spectral image data corresponding to a first group of wavelength bands in a target wavelength range; acquiring setting data that designates one or more sub-wavelength ranges that are part of the target wavelength range; and generating, based on the first mask data and the setting data, second mask data for reconstructing second spectral image data corresponding to a second group of wavelength bands in the one or more sub-wavelength ranges.

[0086] More specific embodiments of the present disclosure will be described below. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims. In the following description, identical or similar components are designated by the same reference numerals. The x, y, and z coordinates shown in the drawings will be used in the following description.

[0087] (Embodiment) 7 is a diagram showing the configuration of an imaging system according to an exemplary embodiment of the present disclosure. The system includes an imaging device 100, a processing device 200, a display device 300, and an input user interface (UI) 400. The processing device 200 corresponds to the signal processing device in the present disclosure.

[0088] The imaging device 100 includes an image sensor 160 and a control circuit 150 that controls the image sensor 160. Although not shown in FIG. 7 , the imaging device 100 also includes a filter array 110 and at least one optical system 140, as shown in FIGS. 1A to 1D . The filter array 110 and the optical system 140 may be arranged in any of the arrangements shown in FIGS. 1A to 1D . The image sensor 160 acquires a monochrome image based on light whose intensity has been modulated by the filter array 110 for each region. Data for each pixel of this monochrome image contains superimposed information for multiple wavelength bands within the target wavelength range W. Therefore, this monochrome image can be considered to be a two-dimensional image in which hyperspectral information within the target wavelength range W has been compressed. Such a monochrome image is an example of a “compressed image” as used herein. Furthermore, data representing a compressed image is referred to as “compressed image data” in this specification.

[0089] The processing device 200 includes a signal processing circuit 250 and a memory 210 such as a RAM and a ROM. The signal processing circuit 250 may be an integrated circuit equipped with a processor such as a CPU or a GPU. The signal processing circuit 250 performs restoration processing based on compressed image data output from the image sensor 160. This restoration processing is essentially the same as the processing performed by the processing device 200 shown in FIGS. 1A to 1D. However, in this embodiment, the restoration processing is performed according to restoration conditions input from the input UI 400. The signal processing circuit 250 generates image data with high wavelength resolution only for a specified sub-wavelength range within the target wavelength range. This reduces calculation time. The memory 210 stores computer programs executed by the processor included in the signal processing circuit 250, various data referenced by the signal processing circuit 250, and various data generated by the signal processing circuit 250.

[0090] The display device 300 includes a memory 310, an image processing circuit 320, and a display 330. The memory 310 temporarily stores setting data indicating restoration conditions sent from the input UI 400. The image processing circuit 320 performs necessary processing on the image restored by the signal processing circuit 250 and then displays the image on the display 330. The display 330 may be any display, such as a liquid crystal display or an organic LED.

[0091] The input UI 400 includes hardware and software for setting various conditions, such as imaging conditions and restoration conditions. The imaging conditions may include, for example, resolution, gain, and exposure time. The restoration conditions may include, for example, the lower and upper wavelength limits of each subwavelength band, the number of wavelength bands included in each subwavelength band, and the number of calculations. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 then controls the image sensor 160 to capture images in accordance with the imaging conditions. The image sensor 160 then generates a compressed image in which information from multiple wavelength bands within the target wavelength band W is superimposed. The input restoration conditions are also sent to the signal processing circuit 250 and memory 310 for recording. The signal processing circuit 250 performs restoration processing in accordance with the set restoration conditions and generates a hyperspectral data cube for the specified subwavelength band. The image processing circuit 320 controls the display 330 to display images for each of the multiple wavelength bands in the specified subwavelength band in accordance with the set restoration conditions.

[0092] During restoration, the signal processing circuit 250 converts mask data pre-recorded in the memory 210 as necessary and uses it in accordance with restoration conditions input via the input UI 400. The mask data is data indicating the spatial distribution of the spectral transmittance of the filter array 110, and includes information equivalent to the matrix H in the above-described equation (2). The generated spectral image is processed as necessary by the image processing circuit 320. The image processing circuit 320 performs processing such as determining the layout within the screen, linking with band information, or coloring according to wavelength, and then displays the spectral image on the display 330.

[0093] In this embodiment, the signal processing circuit 250 generates an image for each of a plurality of wavelength bands only for at least one specified sub-wavelength band within the target wavelength range W. For wavelength ranges within the target wavelength range W other than the specified sub-wavelength band, consecutive wavelength bands are combined as a single wavelength band for calculation. This reduces calculation costs. The signal processing circuit 250 may also generate an image for each of a plurality of wavelength bands for the entire target wavelength range W. In this case, the image processing circuit 320 may extract and display data for the specified sub-wavelength band from the image data input from the signal processing circuit 250.

[0094] 8 is a flowchart showing the operation of the system of this embodiment. In this embodiment, first, in step S101, the user inputs imaging conditions and restoration conditions via the input UI 400 (step S101). Data indicating the input imaging conditions is sent to the control circuit 150. Data indicating the input restoration conditions is sent to the signal processing circuit 250 and the memory 310. The memory 310 temporarily stores the restoration conditions. When the image is displayed, these restoration conditions are referenced to associate the image with the set wavelength band conditions. Next, the imaging device 100 captures an image of the object according to the imaging conditions, thereby acquiring a compressed image (step S102).

[0095] When the compressed image is acquired, the signal processing circuit 250 determines whether or not the mask data needs to be converted based on the input restoration conditions (step S103). If conversion is necessary, the signal processing circuit 250 converts the mask data previously stored in the memory 210 (step S104). Here, conversion refers to combining mask information for multiple wavelength ranges and treating it as mask information for a single wavelength range. Details of combining mask information will be described later with reference to FIG. 14. If conversion is not necessary, step S104 is omitted. The signal processing circuit 250 performs a restoration calculation using the compressed image and the mask data converted as necessary in accordance with the input restoration conditions (step S105). This generates a spectral image from the compressed image. Next, the image processing circuit 320 of the display device 300 associates the generated spectral images with the restoration conditions stored in the memory 310 and labels them (step S106). For example, image data is generated by adding a label indicating the corresponding wavelength range to each of the generated spectral images. The image processing circuit 320 outputs the generated image data to the display 330, causing the image to be displayed (step S107).

[0096] FIG. 9 shows an example of pre-conversion mask data stored in memory 210. The mask data in this example includes mask information indicating the spatial distribution of transmittance for each of multiple component bands included in the target wavelength range. The mask data in this example includes mask information for each of multiple component bands divided into 1-nm intervals and information related to the conditions for acquiring the mask information. Each component band is identified by a lower-limit wavelength and an upper-limit wavelength. The mask information includes information on a mask image and a background image. The multiple mask images shown in FIG. 9 are acquired by capturing multiple backgrounds corresponding to the multiple component bands through filter array 110 using image sensor 120. The multiple background images are acquired by capturing the multiple backgrounds without passing them through filter array 110 using image sensor 120. Such mask image and background image data is recorded in advance for each component band. Information related to the acquisition conditions includes information on exposure time and gain. In the example of FIG. 9, mask image and background image data is recorded for each of multiple component bands with a width of 1 nm. The width of each component band is not limited to 1 nm and can be set to any value. Furthermore, if the background image is highly uniform, the mask data may not include information about the background image. For example, in a configuration in which the image sensor 120 and the filter array 110 are closely integrated and face each other, the mask information is nearly identical to the mask image, so the mask data does not need to include the background image.

[0097] Next, examples of graphical user interfaces (GUIs) displayed by the programs that perform the above information processing will be described with reference to Figures 10 to 13. Images for realizing these GUIs are generated by the signal processing circuit 250 and the image processing circuit 320 and displayed on the display 330.

[0098] Figure 10 shows an example of a GUI screen for inputting imaging conditions. In this example, the user sets the resolution, gain, exposure time, and frame rate before performing hyperspectral imaging. Resolution refers to the number of vertical and horizontal pixels of the displayed image. The resolution can be specified by, for example, selecting a name such as VGA, HD, or 4K from a pull-down menu or by directly entering the number of vertical and horizontal pixels. Gain is specified as a rational number greater than or equal to 0 and may be input by adding, subtracting, multiplying, or dividing rational numbers. For example, if 8 / 3 is input, the gain may be set to 2.6666... dB. The exposure time and frame rate do not need to be input together. The user may input at least one of the exposure time and frame rate, and if a conflict occurs (e.g., an exposure time of 100 ms and a frame rate of 30 fps), one of them may take priority. In addition to inputting the above four conditions, a function for automatically adjusting the gain, exposure time, and frame rate may also be provided. For example, the average brightness may be automatically adjusted to half the maximum brightness. As shown in the example of FIG. 10, the GUI for inputting imaging conditions may have a function for saving and loading the set imaging conditions. The GUI may also have a function for displaying a compressed image acquired under the set imaging conditions in real time. Here, the compressed image itself does not necessarily need to be displayed. Any image acquired under the imaging conditions set at that time may be displayed. For example, pixels that output only red (R), green (G), and blue (B) values may be arranged, and an RGB image acquired using only the values of these pixels may be displayed. Furthermore, for example, a three-band restoration may be performed using a component band synthesis process (described later) with a first band from 400 nm to 500 nm, a second band from 500 nm to 600 nm, and a third band from 600 nm to 700 nm, and the restoration result may be displayed as an RGB image.

[0099] 11 and 12 are diagrams showing examples of GUIs for inputting restoration conditions. In the example shown in FIG. 11, the user inputs the subwavelength range, the wavelength resolution or the number of band divisions, and the number of calculations. Here, the number of calculations represents the number of iterations of the restoration calculation shown in Equation (2). As shown in FIG. 11, the subwavelength range can be specified by setting the lower limit wavelength and the upper limit wavelength, for example, by dragging and dropping. In the example of FIG. 11, the subwavelength range from 420 nm to 480 nm and the subwavelength range from 600 nm to 690 nm are specified. Instead of specifying the ranges by dragging and dropping, as shown in FIG. 12, the ranges of the subwavelength range and each wavelength band within each subwavelength range may be input numerically. The area for inputting the ranges of the subwavelength range and each wavelength band within each subwavelength range may be displayed as an independent window or may be included in the screen for inputting other setting items.

[0100] In the example of Figure 11, the user inputs either the wavelength resolution or the number of band divisions. The number of calculations is specified as an integer equal to or greater than 1. Typically, a number between 10 and 10,000 can be specified. In the example of Figure 11, the predicted calculation time is also displayed. The predicted calculation time is not input by the user; it is automatically calculated and displayed based on the set resolution, number of band divisions, and number of calculations. Note that the functions for inputting the number of calculations and displaying the predicted calculation time may be omitted. Instead, a format may be used in which multiple modes, such as high-precision mode (low speed), balanced mode (medium speed), and high-speed mode (high speed), are selected, for example, from a pull-down menu. As shown in Figure 11, the system may also have a function for saving and loading the set restoration conditions.

[0101] FIG. 13 is a diagram showing an example of a screen that displays a spectroscopic image generated as a result of the reconstruction calculation. The generated spectroscopic image is linked to the set reconstruction conditions and is displayed in a format that allows distinction for each set band. For example, as shown in FIG. 13, the lower and upper limit wavelengths of each band may be displayed numerically along with the restored image of that band. Alternatively, each band may be displayed with a number counted from the short wavelength side or the long wavelength side. The image of each band may be displayed in the color contained in that band. In the above examples, all physical quantities expressed in wavelength (nm) are expressed in wave numbers (for example, cm -1 ) or frequency (e.g., Hz).

[0102] FIG. 14 is a diagram illustrating an example of a method for synthesizing mask information of multiple bands and converting it into new mask information. In this example, mask information for component bands #1 to #20 is pre-stored in memory 210 as pre-conversion mask information, as shown in FIG. 9. In the example of FIG. 14, synthesis processing is not performed for component bands #1 to #5, but synthesis processing is performed for component bands #6 to #20. For component bands #1 to #5, the transmittance distribution of filter array 110 is calculated by dividing the value of each region in the mask image by the value of the corresponding region in the background image. Here, the data for each mask image stored in memory 210 is referred to as "unit mask image data," and the data for each saved background image is referred to as "unit background image data." For bands #6 to #20, the synthesized transmittance distribution is obtained by dividing the data obtained by adding up the unit mask image data for bands #6 to #20 for each pixel by the data obtained by adding up the unit background image data for bands #6 to #20 for each pixel. By performing these operations, mask information can be synthesized for any number of bands. In addition, if the background image is highly uniform, the mask information will nearly match the mask image. In this case, the mask image data for bands #6 to #20 may be added together or averaged and used as the composite mask data for bands #6 to #20.

[0103] In the example shown in Fig. 9, mask information is recorded for each of a large number of component bands, each with a width of 1 nm. In contrast, in the example shown in Fig. 12, the width of each wavelength band specified by the user is a relatively wide 30 nm. In such a case, signal processing circuit 250 combines and restores the mask information of the multiple component bands for each specified wavelength band.

[0104] The conversion process of the mask data by synthesis may be performed in an environment where the end user uses the mask data, or may be performed at a manufacturing site such as a factory where the system is manufactured. When the conversion process of the mask data is performed at a manufacturing site, the converted mask data is stored in advance in memory 210 instead of or in addition to the mask data before conversion.

[0105] As described above, the mask data is used to restore spectral image data for each wavelength band from compressed image data acquired by an imaging device equipped with a filter array including multiple types of optical filters with different spectral transmittances. The method for converting mask data in this embodiment includes the following steps. First mask data is obtained to restore first spectral image data corresponding to a first wavelength band group in the target wavelength range. The first mask data is used to generate second spectral image data corresponding to a second wavelength band group in the sub-wavelength range, based on the first mask data and the setting data.

[0106] The first mask data may be, for example, data for restoring, from a compressed image, a spectral image for each of all component bands included in the target wavelength range. The second mask data may be, for example, data for restoring, from a compressed image, a spectral image for each of all component bands included in each specified sub-wavelength range. The second mask data may also be data for restoring, from a compressed image, a spectral image for each composite band obtained by combining multiple component bands. When such combination is performed, the setting data may include data regarding the combination mode of the bands. In the following description, a composite band obtained by combining multiple component bands is also referred to as an "edited band."

[0107] The first mask data and the second mask data are data reflecting the spatial distribution of the spectral transmittance of the filter array. The first mask data includes first mask information indicating the spatial distribution of the spectral transmittance corresponding to a first group of wavelength bands. The second mask data includes second mask information indicating the spatial distribution of the spectral transmittance corresponding to the second group of wavelength bands.

[0108] The second mask data may further include third mask information that combines information indicating the spatial distribution of spectral transmittance corresponding to a third wavelength band group included in a non-designated wavelength range other than the one or more designated sub-wavelength ranges. In this case, the signal processing circuit 250 can generate, based on the compressed image and the second mask data, a spectral image having a relatively high wavelength resolution for each of the designated sub-wavelength ranges and a spectral image having a relatively low wavelength resolution for the non-designated wavelength range that was not designated.

[0109] 15A and 15B are diagrams showing examples of converted second mask data stored in memory 210. In the example shown in FIG. 15A, for each edited band having a width of 10 nm, the mask image and background image are combined and saved as converted mask information. If the background image has very high uniformity, the mask data may not include background image information. In the example shown in FIG. 15B, for each edited band having a width of 10 nm, the converted mask data is saved as combined mask data obtained by dividing the mask image by the background image. The wavelength width of the edited band is not limited to 10 nm and can be set arbitrarily.

[0110] The wider the bandwidth of the composite mask image after synthesis, the more unit mask images are averaged. Similarly, the wider the bandwidth of the composite mask data after synthesis, the more unit mask images are averaged, resulting in mask data obtained by dividing the unit background images by the average. Therefore, the wider the bandwidth after synthesis, the lower the contrast of the composite mask image or composite mask data tends to be.

[0111] FIG. 16 illustrates an example of a method for generating images for each of multiple wavelength bands included in a target wavelength range. In this example, the target wavelength range includes four subwavelength ranges. The first subwavelength range includes bands #1 to #5. The second subwavelength range includes bands #6 to #10. The third subwavelength range includes bands #11 to #15. The fourth subwavelength range includes bands #16 to #20. In this example, the signal processing circuit 250 performs a reconstruction operation for each subwavelength range by combining the mask information of all component bands that do not belong to that subwavelength range. As shown in FIG. 16, even if the mask information of component bands that do not belong to a given subwavelength range is combined, a good spectral image can be generated for the bands within the subwavelength range. By performing this type of combination processing, the calculation time required to generate an image for each band can be reduced.

[0112] Next, a modification of this embodiment will be described.

[0113] 17 is a diagram showing the configuration of a system in which the signal processing circuit 250 does not convert mask information. In this example, the signal processing circuit 250 reads the restoration conditions provided from the input UI 400 and the mask information stored in the memory 210, and generates a spectral image from the compressed image acquired from the image sensor 160. In this case, the signal processing circuit 250 generates a spectral image across the entire target wavelength range and outputs it to the image processing circuit 320. The image processing circuit 320 displays only images for some wavelength bands from the acquired spectral image on the display 330 in accordance with the set restoration conditions.

[0114] FIG. 18 shows another example of a GUI for setting restoration conditions. In this example, a different wavelength resolution or band division number can be specified for each set sub-wavelength range. The user inputs either the wavelength resolution or the band division number for each sub-wavelength range. The signal processing circuit 250 performs restoration calculations according to the input wavelength resolution or band division number. This configuration makes it possible to generate spectral images with different resolutions for each sub-wavelength range.

[0115] FIG. 19 is a diagram illustrating an example of a UI displaying an image generated from mask information synthesized for a wavelength range that is included in the target wavelength range but not included in any of the sub-wavelength ranges (hereinafter referred to as a "non-designated wavelength range"). In the example of FIG. 19, one image generated for the non-designated wavelength range is displayed, but images for two or more non-designated wavelength ranges may be displayed. An RGB image may be displayed instead of or in addition to the image for the non-designated wavelength range. In this case, the target wavelength range includes the visible wavelength range, and the signal processing circuit 250 synthesizes mask information for each of the red (R), green (G), and blue (B) wavelength ranges. The signal processing circuit 250 uses this synthesized mask information to generate image data for each of the red, green, and blue wavelength ranges from the compressed image data. The image processing circuit 320 displays the generated RGB image on the display 330.

[0116] FIG. 20 illustrates an example of a method for restoring only a specific subwavelength band with high wavelength resolution by performing two-stage restoration. In this example, a compressed image of a target wavelength band including 20 component bands is acquired. Using the method described with reference to FIG. 16, the signal processing circuit 250 restores four large subwavelength bands (hereinafter referred to as "large subwavelength bands"), such as bands 1 to 5, bands 6 to 10, bands 11 to 15, and bands 16 to 20. The signal processing circuit 250 then performs restoration by dividing the specified specific large subwavelength band into multiple smaller bands (hereinafter referred to as "small subwavelength bands"). The number of large subwavelength bands to be divided into multiple small subwavelength bands can be determined arbitrarily. In the example of FIG. 20, only one large subwavelength band is divided into multiple small subwavelength bands, but two or more large subwavelength bands may also be divided into multiple small subwavelength bands. 20, the signal processing circuit 250 performs band division in two stages, but may generate a spectral image through division in three or more stages. The small sub-wavelength range may be a unit band.

[0117] Furthermore, each time band division is performed at each stage, a selection may be made of which wavelength ranges among the divided wavelength ranges to divide into finer sub-wavelength ranges, and this selection may be made by a user or automatically.

[0118] 20, the setting data includes information specifying a plurality of large sub-wavelength bands, each of which is a part of the target wavelength band, and a plurality of small sub-wavelength bands included in at least one of the plurality of large sub-wavelength bands. The signal processing circuit 250 performs the following processes. For each of the plurality of large sub-wavelength bands, first combined mask information is generated by combining mask information for the plurality of component bands included in that large sub-wavelength band. First composite image data is generated for each large sub-wavelength band based on the compressed image data and the first composite mask information. For each of a plurality of small sub-wavelength bands in the designated large sub-wavelength band, second combined mask information is generated by combining mask information for a plurality of component bands included in the small sub-wavelength band. The second composite image data is generated for each small sub-wavelength band based on the first composite image data for the designated large sub-wavelength band and the second composite mask information.

[0119] In this case, the generated hyperspectral data cube includes second composite image data for multiple small sub-wavelength ranges. By performing such processing, detailed spectral information can be obtained only for a specific large sub-wavelength range specified by the user.

[0120] The configuration of the imaging device, the compression algorithm for hyperspectral information, and the reconstruction algorithm for the hyperspectral data cube are not limited to those described in the above-described embodiments. For example, the arrangement of the filter array 110, the optical system 140, and the image sensor 160 is not limited to that shown in FIGS. 1A to 1D and may be modified as appropriate. Furthermore, the characteristics of the filter array 110 are not limited to those exemplified with reference to FIGS. 2A to 4B. A filter array 110 with optimal characteristics may be used depending on the application or purpose. Furthermore, spectral images for each wavelength band may be generated using methods other than the compressed sensing calculation shown in Equation (2) above. For example, other statistical methods such as maximum likelihood estimation or Bayesian estimation may be used.

[0121] In the above embodiment, the compressed image data is generated by the imaging device 100 equipped with the filter array 110. However, the compressed image data may be generated by other methods. For example, the compressed image data may be generated by applying an encoding matrix equivalent to the matrix H in the above equation (1) to a hyperspectral data cube generated by any hyperspectral camera. When it is necessary to reduce the amount of data for data storage or transmission, the compressed image data may be generated by such software processing. The processes in the above embodiments can also be applied to the compressed image data generated by such software processing to restore images for each wavelength band. [Industrial Applicability]

[0122] The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength images. The technology disclosed herein can also be applied to, for example, biomedical, medical, and cosmetic sensing, food foreign matter and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems. [Explanation of symbols]

[0123] 70 Objects 100 Imaging device 110 Filter Array 120 images 140 Optical system 150 control circuit 160 image sensors 200 Processing Equipment 210 memory 220 Spectral Images 250 Signal Processing Circuit 300 display device 310 memory 320 Image Processing Circuit 330 Display 400 Input UI

Claims

1. 1. A computer-implemented signal processing method comprising: Obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges based on the compressed image data; A method comprising:

2. the hyperspectral information is information on four or more wavelength bands included in the target wavelength range, the two-dimensional image information is data of a plurality of pixels included in the compressed image data, information of the four or more wavelength bands is superimposed on the data of each of the plurality of pixels; The method of claim 1.

3. the setting data includes information specifying wavelength resolution in the one or more sub-wavelength bands, the plurality of two-dimensional images are generated at the wavelength resolution; The method according to claim 1 or 2.

4. the one or more sub-wavelength ranges include a first sub-wavelength range and a second sub-wavelength range; the plurality of two-dimensional images are generated for each of the first sub-wavelength band and the second sub-wavelength band; The method according to any one of claims 1 to 3.

5. the one or more sub-wavelength ranges include a first sub-wavelength range and a second sub-wavelength range; the wavelength resolution is specified independently for each of the first sub-wavelength band and the second sub-wavelength band; the plurality of two-dimensional images are generated for each of the first sub-wavelength band and the second sub-wavelength band at the corresponding wavelength resolution; The method of claim 3.

6. The first sub-wavelength range and the second sub-wavelength range are spaced apart. The method according to claim 4 or 5.

7. and further comprising displaying a graphical user interface on a display connected to the computer to allow a user to input the setting data.

7. The method according to any one of claims 1 to 6.

8. further comprising displaying the plurality of two-dimensional images on a display connected to the computer.

8. The method according to any one of claims 1 to 7.

9. the compressed image data is generated by capturing an image using a filter array including a plurality of types of optical filters with different spectral transmittances and an image sensor; The method further includes acquiring mask data reflecting a spatial distribution of the spectral transmittance of the filter array; the plurality of two-dimensional images are generated based on the compressed image data and the mask data; 9. The method according to any one of claims 1 to 8.

10. the mask data includes mask information; the mask information indicates a spatial distribution of transmittance of the filter array in each of a plurality of component bands included in the target wavelength range, The method comprises: generating composite mask information by synthesizing a portion of the mask information corresponding to a plurality of component bands included in a non-designated wavelength range other than the one or more sub-wavelength ranges within the target wavelength range; generating a composite image corresponding to the non-designated wavelength range based on the compressed image data and the composite mask information; further comprising:

10. The method of claim 9.

11. the mask data includes a plurality of background images and a plurality of mask images; each of the plurality of background images is acquired by capturing an image of a corresponding one of a plurality of backgrounds with the image sensor without passing through the filter array; each of the plurality of mask images is obtained by imaging the corresponding one of the plurality of backgrounds with the image sensor through the filter array; the composite mask information is generated based on the plurality of mask images and the plurality of background images; The method of claim 10.

12. further comprising displaying the composite image on a display connected to the computer.

12. The method according to claim 10 or 11.

13. the mask data includes mask information; the mask information indicates a spatial distribution of transmittance of the filter array in each of a plurality of component bands included in the target wavelength range, the setting data includes information specifying a plurality of large sub-wavelength bands, each of which is a part of the target wavelength band, and a plurality of small sub-wavelength bands included in at least one of the plurality of large sub-wavelength bands; The method comprises: generating first composite mask information by combining, for each of the plurality of large sub-wavelength bands, a portion of the mask information corresponding to a plurality of component bands included in the plurality of large sub-wavelength bands; generating a first composite image for each of the plurality of large sub-wavelength bands based on the compressed image data and the first composite mask information; further comprising the plurality of two-dimensional images are generated corresponding to the plurality of small sub-wavelength ranges based on the first composite image; 10. The method of claim 9.

14. the target wavelength range includes the visible wavelength range, The method comprises: generating an image corresponding to a red wavelength range, an image corresponding to a green wavelength range, and an image corresponding to a blue wavelength range based on the compressed image data and the synthesis mask information; displaying an RGB image based on an image corresponding to the red wavelength region, an image corresponding to the green wavelength region, and an image corresponding to the blue wavelength region on a display connected to the computer; further comprising: The method of claim 10.

15. A method for generating mask data used to restore spectral image data for each wavelength band from compressed image data acquired by an imaging device including a filter array including multiple types of optical filters with different spectral transmittances, the method comprising: acquiring first mask data for restoring first spectral image data corresponding to a first set of wavelength bands in a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges based on the first mask data and the setting data; A method comprising:

16. the first mask data and the second mask data are data reflecting a spatial distribution of a spectral transmittance of the filter array, the first mask data includes first mask information indicating a spatial distribution of the spectral transmittance corresponding to the first wavelength band group; the second mask data includes second mask information indicating a spatial distribution of the spectral transmittance corresponding to the second wavelength band group; 16. The method of claim 15.

17. the second mask data further includes third mask information obtained by combining a plurality of pieces of information; each of the plurality of pieces of information indicates a spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength range other than the one or more sub-wavelength ranges within the target wavelength range; 17. The method of claim 16.

18. a processor; a memory storing a computer program to be executed by the processor; A signal processing device comprising: The computer program causes the processor to: Obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges based on the compressed image data; A signal processing device that executes the above.

19. a processor; a memory storing a computer program to be executed by the processor; A signal processing device comprising: The computer program causes the processor to: acquiring first mask data for restoring first spectral image data corresponding to a first set of wavelength bands in a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges based on the first mask data and the setting data; A signal processing device that executes the above.

20. A signal processing device according to claim 18; an imaging device that generates the compressed image data; An imaging system comprising:

21. On the computer, Obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating a plurality of two-dimensional images corresponding to a plurality of wavelength bands included in the one or more sub-wavelength ranges based on the compressed image data; A computer program that executes

22. On the computer, acquiring first mask data for restoring first spectral image data corresponding to a first set of wavelength bands in a target wavelength range; acquiring configuration data specifying one or more sub-wavelength ranges that are part of the target wavelength range; generating second mask data for restoring second spectral image data corresponding to a second wavelength band group in the one or more sub-wavelength ranges based on the first mask data and the setting data; A computer program that executes

Images