Signal processing method, signal processing device, and imaging system

Abstract

This signal processing method is implemented by a computer. The signal processing method includes: acquiring first compressed image data indicating a two-dimensional image, the first compressed image data including hyperspectral information as luminance information for each of at least four wavelength bands included in an object wavelength region, and being such that the hyperspectral information is compressed; extracting partial image data from the first compressed image data; and generating first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band from the partial image data.

Description

Signal processing method, signal processing device, and imaging system

[0001] The present disclosure relates to a signal processing method, a signal processing device, and an imaging system.

[0002] By utilizing spectral information from multiple narrow bands, such as several dozen bands, it becomes possible to understand the detailed physical properties of an object, which was not possible with conventional RGB images. A camera that captures such multi-wavelength information is called a "hyperspectral camera." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral composition analysis.

[0003] Patent Documents 1 and 2 disclose examples of imaging devices that acquire hyperspectral images using compressed sensing. The imaging devices disclosed in Patent Documents 1 and 2 include an array of optical filters with different transmission spectra on an optical path connecting a subject and an image sensor. In Patent Document 1, spatial compressed sensing is performed primarily using a single-mode interference filter array. On the other hand, in Patent Document 2, compressed sensing of wavelength information is performed using a multimode interference filter array. The methods disclosed in Patent Documents 1 and 2 are characterized by generating a hyperspectral image from a compressed image containing compressed hyperspectral information. Such methods are superior to other methods in terms of resolution and imaging time.

[0004] WO 2016 / 012980 U.S. Pat. No. 9,599,511

[0005] This disclosure provides techniques for further reducing the computational complexity and time required to generate hyperspectral images.

[0006] A signal processing method according to an aspect of the present disclosure, executed by a computer, includes: acquiring first compressed image data representing a two-dimensional image in which hyperspectral information, which is brightness information for each of at least four wavelength bands included in a target wavelength range, has been compressed using the hyperspectral information; extracting partial image data from the first compressed image data; and generating, from the partial image data, first two-dimensional image data corresponding to the first wavelength band and second two-dimensional image data corresponding to the second wavelength band.

[0007] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices.

[0008] According to one aspect of the present disclosure, the amount of calculation and time required to generate a hyperspectral image can be reduced.

[0009] 1A is a schematic diagram of an exemplary hyperspectral imaging system; FIG. 1B is a schematic diagram of a first modification of the exemplary hyperspectral imaging system; FIG. 1C is a schematic diagram of a second modification of the exemplary hyperspectral imaging system; and FIG. 1D is a schematic diagram of a third modification of the exemplary hyperspectral imaging system. FIG. 2A is a schematic diagram of an example filter array; and FIG. 2B is a schematic diagram of a filter array for capturing multiple wavelength bands W included in a wavelength range of interest. １ , W ２ , ..., WＮ 2C is a diagram showing an example of the spectral transmittance of an area A1 included in the filter array shown in FIG. 2A. FIG. 2D is a diagram showing an example of the spectral transmittance of an area A2 included in the filter array shown in FIG. 2A. FIG. 3A is a diagram showing an example of the spectral transmittance of a target wavelength range W and a plurality of wavelength bands W included therein. １ , W ２ , ..., W Ｎ 3B is a diagram showing an example of the relationship between the target wavelength range W and the multiple wavelength bands W included therein. １ , W ２ , ..., W Ｎ 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array. FIG. 4B is a diagram illustrating the spectral transmittance shown in FIG. 4A in a wavelength band W. １ , W ２ , ..., W ＮFIG. 5 is a diagram illustrating an example of a compressed image acquired by a hyperspectral camera. FIG. 6 is a diagram illustrating an example of the relationship between the error and the number of pixels of a hyperspectral image generated by an operation based on compressed sensing. FIG. 7 is a diagram illustrating a configuration of an imaging system according to an exemplary embodiment of the present disclosure. FIG. 8 is a flowchart illustrating the operation of the imaging system. FIG. 9 is a diagram illustrating an example of a GUI screen for inputting imaging conditions. FIG. 10 is a diagram illustrating an example of a GUI for setting a region to be extracted. FIG. 11A is a diagram illustrating a process for editing mask data. FIG. 11B is a diagram illustrating a process for editing mask data. FIG. 12 is a flowchart illustrating the operation of the imaging system according to a first modified example. FIG. 13A is a diagram illustrating an example of an error screen. FIG. 13B is a diagram illustrating a region enlargement process. FIG. 13C is a diagram illustrating a region enlargement process. FIG. 14 is a flowchart illustrating the operation of the imaging system according to a second modified example. FIG. 15 is a diagram illustrating an example of an error screen. FIG. 16 is a diagram illustrating an example of determining a region to be extracted from a compressed image by edge detection. 17A and 17B are diagrams illustrating an example in which an area to be extracted from a compressed image is determined based on the result of color discrimination;

[0010] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangements, positions and connection forms, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0011] In the present disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the connection relationships within the LSI or set up circuit sections within the LSI, can also be used for the same purpose.

[0012] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0013] First, a configuration example of a hyperspectral imaging system according to an embodiment of the present disclosure and findings discovered by the present inventors will be described.

[0014] FIG. 1A is a schematic diagram illustrating an exemplary hyperspectral imaging system. The system includes an imaging device 100 and a signal processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 2. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The filter array 110 has a structure and function similar to that of the "encoding element" disclosed in Patent Document 2. The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject of imaging. The filter array 110 is disposed between the optical system 140 and the image sensor 160.

[0015] FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple and may be any object. The image sensor 160 generates compressed image 120 data in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The signal processing device 200 generates image data for each of a plurality of wavelength bands included in the target wavelength range based on the compressed image 120 data generated by the image sensor 160. This generated image data of a plurality of wavelength bands is referred to as "hyperspectral image data" in this specification. Here, the number of wavelength bands included in the target wavelength range is set to N (N is an integer equal to or greater than 4). In the following description, the generated image data of a plurality of wavelength bands is referred to as a hyperspectral image 220W. １ , 220W ２ , ..., 220W Ｎ These are collectively referred to as a hyperspectral image 220. In this specification, a signal representing an image, that is, a set of signals representing the pixel values ​​of each pixel, may be simply referred to as an "image."

[0016] The filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittance. The filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 110 is referred to as "encoding" in this specification.

[0017] 1A, the filter array 110 is disposed near or directly above the image sensor 160. Here, "near" means close enough that a light image from the optical system 140 is formed on the surface of the filter array 110 with a certain degree of clarity. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0018] 1A, optical system 140 includes at least one lens. Although optical system 140 is shown as a single lens in FIG. 1A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 10.

[0019] The filter array 110 may be disposed away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is disposed away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position away from the image sensor 160. In the example of FIG. 1C, the filter array 110 is disposed between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, with the filter array 110 disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.

[0020] The image sensor 160 is a monochrome photodetector having a plurality of photodetection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, an infrared array sensor, a terahertz array sensor, or a millimeter-wave array sensor. The photodetection elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome sensor. For example, a color sensor having an R / G / B, R / G / B / IR, or R / G / B / W filter may also be used. Using a color sensor can increase the amount of information related to wavelengths, thereby improving the accuracy of reconstructing the hyperspectral image 220. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, but may also be ultraviolet, near infrared, mid-infrared, far infrared, microwave, or radio wave wavelength ranges.

[0021] The signal processing device 200 is a computer including a processor and a storage medium such as a memory. The signal processing device 200 generates a plurality of hyperspectral images 220W each including information of a plurality of wavelength bands based on the compressed image 120 acquired by the image sensor 160. １ , 220W ２ , ...220W Ｎ Generate data.

[0022] 2A is a diagram schematically illustrating an example of a filter array 110. The filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, these regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0023] 2A , the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0024] FIG. 2B shows multiple wavelength bands W included in the target wavelength range. １ , W ２ , ..., W Ｎ 2B is a diagram showing an example of the spatial distribution of light transmittance of each of the wavelength bands. In the example shown in FIG. 2B, the difference in shading of each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0025] 2C and 2D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 110 shown in FIG. 2A . The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary for all regions to have different spectral transmittances. In the filter array 110, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In some examples, the number of spectral transmittance patterns of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0026] 3A and 3B show a target wavelength range W and a plurality of wavelength bands W included therein. １ , W ２ , ..., W Ｎ1 is a diagram illustrating the relationship between the wavelengths of the visible light and the near-infrared light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a radio wave range such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. Thus, the wavelength range used is not limited to the visible light range. In this specification, for convenience, "light" refers not only to visible light but also to non-visible light such as near-ultraviolet light, near-infrared light, and radio waves.

[0027] In the example shown in FIG. 3A , N is an arbitrary integer equal to or greater than 4, and each wavelength range obtained by dividing the target wavelength range W into N equal parts is called a wavelength band W １ , W ２ , ..., W Ｎ However, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the multiple wavelength bands may be determined arbitrarily as long as they are different from each other.

[0028] 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has multiple maximum values ​​P1 to P5 and multiple minimum values ​​for wavelengths within the target wavelength band W. In the example shown in FIG. 4A, the optical transmittance within the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 4A, the wavelength band W ２ , and the wavelength band W Ｎ－１ In this way, the spectral transmittance of each region is expressed as a maximum value in a plurality of wavelength bands W １ From W Ｎ In the example of Fig. 4A, the maximum values ​​P1, P3, P4, and P5 are 0.5 or more.

[0029] As described above, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 110 transmits a large amount of components in a certain wavelength range among the incident light, and transmits only a small amount of components in other wavelength ranges. For example, the transmittance of light in k wavelength bands out of N wavelength bands may be greater than 0.5, and the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light is white light that contains all visible light wavelength components equally, the filter array 110 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light by superimposing it.

[0030] FIG. 4B shows an example of the spectral transmittance shown in FIG. 4A in a wavelength band W １ , W ２ , ..., W Ｎ This figure shows the results of averaging the spectral transmittance for each wavelength band. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the transmittance value averaged for each wavelength band in this way is referred to as the transmittance for that wavelength band. In this example, the transmittance is remarkably high in the three wavelength ranges that have maximum values ​​P1, P3, and P5. In particular, the transmittance exceeds 0.8 in the two wavelength ranges that have maximum values ​​P3 and P5.

[0031] In the examples shown in Figures 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary scale transmittance distribution may be employed in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two wavelength ranges out of multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.

[0032] A portion of all cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength bands W included in the wavelength range W of interest.１ From W Ｎ The filter array 110 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0033] Such data indicating the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the signal processing device 200. This data is used in the calculation processing described below.

[0034] The filter array 110 can be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing a metal. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is different for each filter. This allows different spectral characteristics to be achieved for each filter. The use of a multilayer film allows for sharp rises and falls in the spectral transmittance. A configuration using an organic material can be achieved by using different pigments or dyes for each filter or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each filter. When a microstructure containing a metal is used, it can be fabricated using spectroscopy due to the plasmon effect.

[0035] Next, an example of signal processing by the signal processing device 200 will be described. The signal processing device 200 reconstructs a multi-wavelength hyperspectral image 220 based on the compressed image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, "multi-wavelength" means a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a typical color camera, for example. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the number of bands. Depending on the application, the number of bands may exceed 100.

[0036] The data to be obtained is the data of the hyperspectral image 220, and this data is denoted as f. If the number of bands is N, f is the image data f of each band. １ , f ２ , ..., f Ｎ Here, as shown in FIG. 1A, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is n and the number of pixels in the y direction is m, then the image data f １ , f ２ , ..., f Ｎ Each of the elements f is two-dimensional data with n×m pixels. Therefore, data f is three-dimensional data with n×m×N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube." Meanwhile, data g of compressed image 120 obtained by encoding and multiplexing using filter array 110 has n×m elements. Data g can be expressed by the following equation (1):

[0037] where f １ , f ２ , ..., f Ｎ Each of the elements in the matrix H is data with n×m elements. Therefore, the vector on the right side is strictly a one-dimensional vector with n×m×N rows and one column. The vector g is converted into a one-dimensional vector with n×m rows and one column, and is then expressed and calculated. The matrix H is a matrix of each component f of the vector f. １ , f ２ , ..., f Ｎis encoded and intensity-modulated with different encoding information (hereinafter also referred to as "mask information") for each wavelength band, and then added together. Therefore, H is a matrix with n×m rows and n×m×N columns.

[0038] Given the vector g and the matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the signal processing device 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

[0039] Here, f' represents the estimated data of f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or a stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The signal processing device 200 can converge the solution by recursive iterative calculations and calculate the final solution f'.

[0040] The first term in the parentheses in Equation (2) represents an operation to calculate the sum of squares of the difference between the acquired data g and Hf, which is obtained by transforming f in the estimation process using the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0041] In the configurations of FIGS. 1B and 1C , the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned matrix H, the hyperspectral image 220 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to a pixel on an image spreads to a k×k pixel region around the pixel due to blurring, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the effect on the brightness of each pixel within that region. The hyperspectral image 220 can be reconstructed by reflecting the effect of blurring of the encoding pattern by the PSF in the matrix H. The filter array 110 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 110 to be lost due to excessive diffusion can be selected.

[0042] Through the above processing, a hyperspectral image 220 can be constructed from the compressed image 120 acquired by the image sensor 160. In the above example, the signal processing device 200 applies a compressed sensing algorithm to the entire compressed image 120 to generate the hyperspectral image 220. In this case, if the resolution of the compressed image 120 is high, the amount of calculation and time required to generate the hyperspectral image 220 increases.

[0043] Therefore, in an embodiment of the present disclosure, a method is used in which only a necessary area is extracted and restored from the acquired compressed image 120. An outline of an embodiment of the present disclosure will be described below.

[0044] A computer-implemented signal processing method according to an exemplary embodiment of the present disclosure includes: acquiring first compressed image data representing a two-dimensional image in which hyperspectral information, which is brightness information for each of at least four wavelength bands included in a target wavelength range, has been compressed, extracting partial image data from the first compressed image data, and generating, from the partial image data, first two-dimensional image data corresponding to the first wavelength band and second two-dimensional image data corresponding to the second wavelength band.

[0045] Here, "hyperspectral information" refers to brightness information for each of four or more wavelength bands included in a predetermined target wavelength range. For example, as described above, the compressed image data can be acquired by imaging using a filter array including multiple types of optical filters with different transmission spectra. Information on multiple wavelength bands, for example, four or more, can be superimposed on data for each pixel of the compressed image data. Depending on the application, information on 10 or more or 100 or more wavelength bands can be superimposed on data for each pixel. Furthermore, the compressed image data can include information on multiple wavelength bands, each of which is spatially encoded. Partial image data can be extracted, for example, based on image recognition or a user instruction. According to the above method, multiple two-dimensional image data for multiple wavelength bands are generated from the partial image data extracted from the compressed image data. This reduces the amount of calculation and the calculation time compared to generating multiple two-dimensional image data for multiple wavelength bands from the entire compressed image. It should be noted that the two-dimensional image data does not need to be generated for all wavelength bands included in the target wavelength range; it may be generated for only some of the wavelength bands. The two-dimensional image data can be generated, for example, by applying the compressed sensing algorithm described above to the partial image data.

[0046] The first compressed image data may include data of a plurality of pixels, and the data of each of the plurality of pixels may include first information and second information superimposed on each other, and each of the first information and the second information may correspond to one wavelength band among the at least four wavelength bands.

[0047] The first compressed image data may include spatially encoded first information and spatially encoded second information, each of the first information and the second information corresponding to one wavelength band among the at least four wavelength bands.

[0048] The method may further include determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data, in which case different actions are performed depending on the result of the determination.

[0049] That is, the method may further include: executing a first process when it is determined that the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data; and executing a second process different from the first process when it is determined that the partial image data is not suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data.

[0050] Determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data may be performed based on the number of pixels of the partial image data. For example, the plurality of two-dimensional image data may be generated only if the number of pixels of the partial image data exceeds a predetermined threshold.

[0051] The second processing may be to increase the number of pixels in the partial image data.

[0052] That is, the method may further include, when it is determined that the partial image data is not suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data, increasing the number of pixels of the partial image data extracted from the compressed image data. In this case, the plurality of two-dimensional image data may be generated from the partial image data with the increased number of pixels.

[0053] The first compressed image data may be generated, for example, by an imaging device having a filter array including multiple types of optical filters with different spectral transmittances, and an image sensor that acquires an image formed by light that has passed through the filter array.

[0054] The second processing may be to cause the imaging device to generate second compressed image data by enlarging and imaging an area corresponding to the partial image data, and the first two-dimensional image data and the second two-dimensional image data may be generated based on the second compressed image data.

[0055] That is, the method may further include, when it is determined that the partial image data is not suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data, causing the imaging device to generate new compressed image data by enlarging and imaging an area corresponding to the partial image data. The first two-dimensional image data and the second two-dimensional image data may be generated based on the new compressed image data.

[0056] The method may further include acquiring mask data reflecting a spatial distribution of the spectral transmittance of the filter array, and the first two-dimensional image data and the second two-dimensional image data may be generated based on data of a portion of the mask data corresponding to the partial image data and the partial image data.

[0057] The second process may be to display an error on a display device.

[0058] That is, the method may further include displaying an error on a display device when it is determined that the partial image data is not suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data.

[0059] The method may further include displaying on a display device a graphical user interface (GUI) that allows a user to specify an area of ​​the two-dimensional image to be extracted as the partial image data.

[0060] The method may further include detecting edges of the two-dimensional image, and the partial image data may be extracted based on the detected edges.

[0061] The method may further include extracting luminance information of a specific wavelength component from the first compressed image data. The partial image data may be extracted based on the luminance information.

[0062] The method may further include displaying on a display device a graphical user interface that allows a user to input threshold data for determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data.

[0063] In the method, the threshold data may include a parameter related to the number of pixels or a luminance error.

[0064] A signal processing device according to another embodiment of the present disclosure includes a processor and a memory storing a computer program executed by the processor, the computer program causing the processor to acquire compressed image data representing a two-dimensional image in which the hyperspectral information, the two-dimensional image including hyperspectral information being brightness information for each of at least four wavelength bands included in a target wavelength range, has been compressed, extracts partial image data from the compressed image data, and generates, from the partial image data, first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band.

[0065] An imaging system according to another embodiment of the present disclosure includes the signal processing device and an imaging device that generates the compressed image data.

[0066] A computer program according to another embodiment of the present disclosure causes a computer to perform the following steps: acquire compressed image data that includes hyperspectral information, which is brightness information for each of at least four wavelength bands included in a target wavelength range, and indicates a two-dimensional image in which the hyperspectral information has been compressed; extract partial image data from the compressed image data; and generate, from the partial image data, first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band.

[0067] A computer-readable non-transitory recording medium according to another embodiment of the present disclosure stores a computer program that, when executed by a computer, causes the computer to obtain compressed image data representing a two-dimensional image in which the hyperspectral information, which includes hyperspectral information being brightness information for each of at least four wavelength bands included in a target wavelength range, has been compressed; extract partial image data from the compressed image data; and generate, from the partial image data, first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band.

[0068] More specific embodiments of the present disclosure will be described below. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims. In the following description, identical or similar components are designated by the same reference numerals. In the following description, x, y, and z coordinates shown in the drawings will be used. In this specification, data representing an image may be simply referred to as an "image."

[0069] (Embodiment) First, a usage scenario of a hyperspectral camera, which is an example of an imaging system according to an embodiment of the present disclosure, will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a compressed image 120 acquired by the hyperspectral camera. In this example, a plurality of vegetables are photographed as subjects. A signal processing device 200 acquires the compressed image 120 from the image sensor 160 and performs the above-described arithmetic processing on the compressed image 120 to generate a hyperspectral image 220.

[0070] Consider a case where a hyperspectral image of only a specific object in a region 121 is required in a compressed image 120 containing multiple objects. This situation can occur in a wide range of situations, such as sorting fruits and vegetables, inspecting industrial products, or inspecting the exterior of buildings. In such a case, the hyperspectral camera of this embodiment extracts a portion of the region 121 from the compressed image 120 and performs arithmetic processing based on compressed sensing only on that region 121 to generate a hyperspectral image. This process can reduce the amount of calculation required compared to generating a hyperspectral image by performing arithmetic processing on the entire region of the compressed image 120. This enables improved calculation speed and reduced costs for the required arithmetic processing device (e.g., memory, CPU, etc.).

[0071] On the other hand, the inventors have found that the smaller the area to be calculated, the more likely errors are to occur in the colors of the generated hyperspectral image. This problem will be described with reference to FIG. 6.

[0072] FIG. 6 illustrates an example of the relationship between the error and the number of pixels in a hyperspectral image generated by the aforementioned compressed sensing-based calculation. The graph in FIG. 6 shows the dependency of the average luminance error between the generated hyperspectral image and the actual image on the number of pixels for each of the four images shown in parts (a) to (d) in FIG. 6 . Here, the average luminance error is the percentage obtained by dividing the absolute value of the difference between the luminance value of a pixel in the generated hyperspectral image and the luminance value of the corresponding pixel in the actual image, averaged over all pixels and all wavelength bands, by the maximum luminance value. As can be seen from FIG. 6 , it was confirmed that the luminance error in the generated hyperspectral image increases as the number of pixels in the compressed image decreases. In the example of FIG. 6 , the luminance error can increase, for example, to exceed 5% as the number of pixels decreases. Here, a luminance error of 5% means that when the luminance value of each pixel in the image is expressed, for example, in 10 bits (i.e., 1024 levels), the luminance value of a pixel in a certain wavelength band is calculated to deviate from its actual value by approximately 50. Thus, a compressed image with a small number of pixels is more likely to produce false colors. For example, a pixel that should normally have a very small signal for a certain wavelength band may have a non-negligible signal, or conversely, a pixel that should normally have a relatively large signal may not have a signal at all. Such behavior may lead to erroneous conclusions in applications such as foreign object inspection or visual inspection.

[0073] Therefore, in the process of extracting a specific region from a compressed image to generate a hyperspectral image, the imaging system of this embodiment determines whether the extracted region is large enough to obtain a hyperspectral image with small brightness errors, and if the extracted region is not large enough, performs an operation of enlarging the extracted region or outputting an error. By introducing such an operation, it is possible to suppress the occurrence of false colors caused by insufficient resolution or number of pixels in the partial region extracted from the compressed image generated by imaging, and to avoid generating a hyperspectral image that is significantly different from the actual image.

[0074] The configuration and operation of the imaging system of this embodiment will be described in more detail below.

[0075] 7 is a diagram showing the configuration of an image capturing system according to this embodiment. The system includes an image capturing apparatus 100, a signal processing apparatus 200, a display apparatus 300, and an input user interface (UI) 400.

[0076] The imaging device 100 includes an image sensor 160 and a control circuit 150 that controls the image sensor 160. Although not shown in FIG. 7 , the imaging device 100 also includes a filter array 110 and at least one optical system 140, as shown in FIGS. 1A to 1D . The filter array 110 and the optical system 140 may be arranged in any of the arrangements shown in FIGS. 1A to 1D . The filter array 110 includes multiple types of optical filters with different spectral transmittances. The image sensor 160 acquires a monochrome image based on light whose intensity has been modulated by the filter array 110 for each region. Data for each pixel of this monochrome image contains superimposed information for multiple wavelength bands within the target wavelength range W. Therefore, this monochrome image can be said to be a two-dimensional image in which hyperspectral information within the target wavelength range W has been compressed. Such a monochrome image is an example of a "compressed image." In this specification, data representing a compressed image is referred to as "compressed image data."

[0077] In the example of the compressed image described above, since information on multiple wavelength bands is superimposed on the data of each pixel and the brightness of each wavelength band is spatially encoded, it can be said that the hyperspectral information is compressed in the wavelength direction and the spatial direction. The compressed image is not limited to such an example, and may be a two-dimensional image in which information on each wavelength band is encoded only in the spatial direction, as disclosed in Patent Document 1, for example.

[0078] The signal processing device 200 includes a signal processing circuit 250 and a memory 210 such as a RAM and a ROM. The signal processing circuit 250 performs restoration processing based on compressed image data output from the image sensor 160. This restoration processing is basically the same as the processing performed by the signal processing device 200 shown in FIGS. 1A to 1D , but in this embodiment, the restoration processing is performed according to restoration conditions input from the input UI 400. The signal processing circuit 250 performs calculations based on a compressed sensing algorithm only on a specified partial region of the compressed image, thereby generating a hyperspectral image of the specified partial region. This reduces the amount of calculation and the calculation time.

[0079] Hyperspectral image data includes three-dimensional information: two spatial dimensions and one wavelength dimension. In this specification, data representing a hyperspectral image is also referred to as "three-dimensional hyperspectral data."

[0080] The display device 300 includes an image processing circuit 320 and a display 330. The image processing circuit 320 performs necessary processing on the image restored by the signal processing circuit 250 and then displays the image on the display 330. The display 330 may be any display, such as a liquid crystal display or an organic LED.

[0081] The input UI 400 includes hardware and software for setting various conditions, such as imaging conditions and restoration conditions. The imaging conditions may include, for example, resolution, gain, and exposure time. The restoration conditions may include, for example, information indicating whether to extract a specific region from the compressed image, information specifying the partial region to be extracted, and conditions such as the number of calculations. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 controls the image sensor 160 to capture an image in accordance with the imaging conditions. The image sensor 160 then generates a compressed image in which information from multiple wavelength bands within the target wavelength range W is superimposed. The input restoration conditions are sent to the signal processing circuit 250 and recorded. The signal processing circuit 250 performs restoration processing in accordance with the set restoration conditions and generates three-dimensional hyperspectral data for the specified partial region. The image processing circuit 320 controls the display 330 to display an image of multiple wavelength bands represented by the three-dimensional hyperspectral data.

[0082] During restoration, the signal processing circuit 250 edits and uses mask data pre-recorded in the memory 210 as necessary in accordance with restoration conditions input via the input UI 400. The mask data is data reflecting the spatial distribution of the spectral transmittance of the filter array 110 and corresponds to the matrix H in the above-described equation (2). The signal processing circuit 250 generates a hyperspectral image for a specified partial region using partial mask data obtained by extracting a portion corresponding to the specified partial region from the mask data. The generated hyperspectral image is processed as necessary by the image processing circuit 320. The image processing circuit 320 performs processing such as determining the layout within the screen, linking the image to band information, or coloring the image according to wavelength, and then displays the hyperspectral image on the display 330.

[0083] 8 is a flowchart showing the operation of the system of this embodiment. In this embodiment, first, the user inputs imaging conditions and restoration conditions via the input UI 400 (step S101). Data indicating the input imaging conditions is sent to the control circuit 150. Data indicating the input restoration conditions is sent to the signal processing circuit 250. Next, the imaging device 100 captures an image of the object in accordance with the imaging conditions to obtain a compressed image (step S102).

[0084] Once the compressed image is acquired, the signal processing circuit 250 determines whether or not it is necessary to extract a specific region from the compressed image (step S103). Whether or not it is necessary to extract a specific region from the compressed image may be determined, for example, based on restoration conditions entered by the user. The signal processing circuit 250 may automatically determine whether or not region extraction is necessary based on the content of the compressed image. For example, if a specific shape is detected in the compressed image, the signal processing circuit 250 may extract a region excluding that shape. Such processing corresponds to, for example, removing a shelf section with many straight lines from the compressed image when sorting fruit and vegetables from a display shelf. The signal processing circuit 250 may also limit the region based on rough color discrimination. For example, processing such as removing blue parts of the sky during the exterior inspection of a building is possible.

[0085] If it is determined that no specific region needs to be extracted, the signal processing circuit 250 generates a hyperspectral image based on the compressed image and the mask data using the method described above (step S104).The image processing circuit 320 displays the generated hyperspectral image on the display 330 (step S130).

[0086] If it is determined in step S103 that a specific region needs to be extracted from the compressed image, the signal processing circuit 250 determines candidate regions to be extracted (step S111). For example, the signal processing circuit 250 determines candidate regions designated by the user. Alternatively, the signal processing circuit 250 may determine candidate regions to be extracted from the compressed image or a similar image by edge detection, detection of luminance information of specific wavelength components, or other methods. The similar image to the compressed image may be, for example, a color image generated by restoring the three primary colors R (red), G (green), and B (blue) from the compressed image, or a monochrome image denoised by multiplying the compressed image by the inverse of the transmittance distribution indicated by mask data for all wavelength bands.

[0087] The signal processing circuit 250 determines whether the candidate region to be extracted has sufficient resolution for performing the restoration calculation, i.e., whether the amount of spatial information is sufficient (step S112). Here, whether the candidate region to be extracted has sufficient resolution for performing the restoration calculation can be determined, for example, based on whether the number of pixels in the candidate region exceeds a predetermined threshold. The threshold may vary depending on the target of the restoration calculation. The imaging system may be configured so that the user can set the threshold via the input UI 400. Here, the user may input the number of pixels as the threshold, but this is not limited to this. For example, the user may input a luminance error as the threshold, and the imaging system may convert the input luminance error into the number of pixels based on the relationship shown in FIG. 6. If the amount of spatial information is insufficient, the signal processing circuit 250 expands the candidate region to be extracted (step S113). For example, the candidate region to be extracted may be expanded by adding all pixels adjacent to the outermost periphery of the region determined in step S111 to the candidate region. This supplements the amount of spatial information and determines a new candidate region to be extracted. In step S112, the processes of steps S111 to S113 are repeated until it is determined that the size of the determined region candidate is sufficient.

[0088] Through the above operations, an extraction region having sufficient resolution for performing a restoration calculation is determined, and a partial image is extracted from the compressed image. The signal processing circuit 250 performs editing by cutting out data corresponding to the extracted region from the mask data stored in the memory 210 (step S121). Next, the signal processing circuit 250 generates a hyperspectral image using the partial image extracted from the compressed image and the cut-out mask data (step S122). The calculation for generating the hyperspectral image is the same as the calculation described with reference to equations (1) and (2). The image processing circuit 320 displays the generated hyperspectral image on the display 330 (step S130).

[0089] The extraction and calculation processes may be performed on a single region of the compressed image or on multiple regions, where the processes may be performed on each region individually.

[0090] Next, examples of graphical user interfaces (GUIs) displayed by the programs that perform the above-described information processing will be described with reference to Figures 9 and 10. Images for realizing these GUIs are generated by the signal processing circuit 250 and the image processing circuit 320 and displayed on the display 330.

[0091] FIG. 9 shows an example of a GUI screen for inputting imaging conditions. In this example, the user sets the resolution, gain, exposure time, and frame rate before performing hyperspectral imaging. Resolution refers to the number of vertical and horizontal pixels of the image to be displayed. The resolution can be specified by, for example, selecting a name such as VGA, HD, or 4K from a pull-down menu or by directly entering the number of vertical and horizontal pixels. Gain is specified as a rational number greater than or equal to 0 and may be input by adding, subtracting, multiplying, or dividing rational numbers. For example, if 8 / 3 is input, the gain may be set to 2.6666... ​​dB. The exposure time and frame rate do not need to be input together. The user may input at least one of the exposure time and frame rate, and if a conflict occurs (e.g., an exposure time of 100 ms and a frame rate of 30 fps), one of them may take priority. In addition to inputting the above four conditions, a function for automatically adjusting the gain, exposure time, and frame rate may be provided. For example, the average brightness may be automatically adjusted to half of the maximum brightness. As shown in the example of FIG. 9 , the GUI for inputting imaging conditions may have a function for saving and loading the set imaging conditions. The GUI may also have a function for displaying a compressed image 120 acquired under the set imaging conditions in real time. Here, the compressed image itself does not necessarily need to be displayed. Any image acquired under the currently set imaging conditions may be displayed. For example, pixels that output only red (R), green (G), and blue (B) values ​​may be arranged, and an RGB image acquired using only the values ​​of these pixels may be displayed. Furthermore, for example, a three-band restoration may be performed using a component band synthesis process (described later) with a first band from 400 nm to 500 nm, a second band from 500 nm to 600 nm, and a third band from 600 nm to 700 nm, and the restoration result may be displayed as an RGB image.

[0092] FIG. 10 illustrates an example of a GUI that allows a user to manually set an area to be extracted. This GUI allows a user to specify an area 121 to be extracted from a displayed compressed image 120 or an equivalent image, for example, by dragging and dropping or by inputting coordinates. The area to be calculated may be displayed on the GUI by, for example, enclosing it with a dotted line or by superimposing a graphic with a transparency of 1% or more. The area specified by the user may also be automatically corrected. For example, a hand-drawn circle may be automatically corrected to a perfect circle or ellipse. While FIG. 10 illustrates an example in which a single area is specified as the area to be calculated, multiple areas may also be specified as the area to be calculated. In this manner, the signal processing circuit 250 displays a GUI on the display device 300 that allows a user to specify an image area to be extracted as a partial image.

[0093] After determining the candidate region to be extracted using the method shown in FIG. 10 , the signal processing circuit 250 determines whether the determined region contains a sufficient amount of spatial information for the calculation process based on the number of pixels in the region. The minimum number of pixels required for the calculation process varies depending on the object of observation or the allowable brightness error. If the signal processing circuit 250 determines that the determined region does not contain the number of pixels required for the calculation process, it expands the periphery of the candidate region to obtain the number of pixels required for the calculation process. For example, the region can be expanded by repeating the process of incorporating all pixels adjacent to the periphery of the candidate region into the candidate region.

[0094] 11A and 11B are diagrams illustrating the editing process of the mask data in step S121 of FIG. 8. Consider the case where a region 121 included in a compressed image 120 is extracted to generate a hyperspectral image, as shown in FIG. 11A. Here, the compressed image 120 is assumed to be acquired by imaging using a filter array having transmittance information 170 as shown in FIG. 11B. The calculation to generate the hyperspectral image for region 121 is performed using transmittance information 171 of a region of the transmittance information 170 that corresponds to region 121. In this way, editing the mask data in step S121 may include extracting only a portion of the transmittance information included in the mask data to generate mask data to be used in the calculation.

[0095] As described above, according to this embodiment, the signal processing circuit 250 extracts a partial image from a compressed image, which is a two-dimensional monochrome image containing hyperspectral information, and generates a hyperspectral image from the partial image using a compressed sensing algorithm. The signal processing circuit 250 determines whether a hyperspectral image can be generated from the partial image based on the number of pixels in the partial image. If it is determined that a hyperspectral image can be generated from the partial image, the signal processing circuit 250 generates a hyperspectral image based on the partial image. On the other hand, if it is determined that a hyperspectral image cannot be generated from the partial image, the signal processing circuit 250 increases the number of pixels in the partial image extracted from the compressed image and generates a hyperspectral image based on the partial image with the increased number of pixels.

[0096] The above processing can reduce the amount of calculation required to generate a hyperspectral image, and furthermore, can suppress the occurrence of false colors in the hyperspectral image.

[0097] Next, a modification of this embodiment will be described.

[0098] 12 is a flowchart showing an example in which the operations of steps S114 and S115 are executed instead of step S113 in FIG. 8. In this example, if it is determined in step S112 that the size of the candidate area to be extracted is insufficient, the signal processing circuit 250 causes the display device 300 to display an error (step S114). The signal processing circuit 250 instructs the image capture device 100 to enlarge and capture the periphery of the extracted area (step S115). In response to this instruction, the image capture device 100 generates a new compressed image in which the area of ​​the object is enlarged compared to the previous image. In the subsequent step S111, a candidate area to be extracted is determined from this new compressed image.

[0099] 13A is a diagram showing an example of an error screen displayed in step S114. In this example, an error message is displayed indicating that the extracted area is too small and that the periphery of the target area should be enlarged and the image should be captured again. Here, "enlarging the periphery of the target area and capturing the image again" refers to generating a new compressed image 120B as shown in FIG. 13C by enlarging the periphery of the extracted area 121 from the original compressed image 120A as shown in FIG. 13B. The imaging device 100 generates a compressed image in which the periphery of the target area is enlarged, for example, using a zoom function.

[0100] 12, an enlarged compressed image is automatically generated, but the compressed image may also be generated manually by a user. In this case, after seeing the error message, the user operates the imaging device 100 to capture an image so that the object in the image is enlarged. This generates a compressed image in which the periphery of the target area is enlarged, and a hyperspectral image is generated based on the compressed image.

[0101] FIG. 14 is a flowchart showing a modified example in which, when it is determined in step S112 that the extracted region is not large enough for calculation, an error is displayed in step S114, and then the process is terminated and a prompt to reset the extracted region is made.

[0102] 15 shows an example of an error screen. In this example, an error message is displayed indicating that the extraction area is too small to generate hyperspectral data and that processing will be interrupted. Instead of this message, a message may be displayed urging the user to zoom in on the periphery of the target area and capture the image again.

[0103] FIG. 16 is a diagram illustrating an example of determining an area to be extracted from a compressed image or an image equivalent thereto by edge detection in step S111 in FIGS. 8, 12, and 14. In this example, in step S111, the signal processing circuit 250 detects edges in the compressed image and determines candidate areas to be extracted based on the detected edges. In the example of FIG. 16, buildings 201 and 202 are identified by detecting structures surrounded by straight lines. A user may be prompted to decide whether to perform a restoration calculation on the area extracted by edge detection. For example, a GUI such as that shown in FIG. 10 may display the detected area as a dotted line or a graphic with a transparency of 1% or more, allowing the user to manually select the area as an extraction target. Thus, in the example of FIG. 16, the signal processing circuit 250 detects edges in a two-dimensional image represented by compressed image data and extracts partial image data based on the detected edges.

[0104] 17A and 17B are diagrams illustrating an example of extracting an area based on the results of rough color discrimination in step S111 in FIGS. 8, 12, and 14. In this example, in step S111, the signal processing circuit 250 extracts luminance information of specific wavelength components from a compressed image or an image equivalent thereto, and extracts a partial image based on the luminance information of the specific wavelength components. For example, when generating a hyperspectral image of grapes growing in an orchard, as shown in FIG. 17A, a situation in which blue or purple grapes grow surrounded by green leaves and brown branches is conceivable. In this case, by extracting areas with a large amount of luminance information in the purple to blue wavelength range, a hyperspectral image can be generated by selecting an area containing grape information, as shown in FIG. 17B. In such a case, it is not necessary to generate images for each wavelength band across the entire target wavelength range; instead, it is possible to generate images for only a portion of the wavelength band (e.g., the band corresponding to purple to blue). Thus, in the example shown in Figures 17A and 17B, the signal processing circuit 250 extracts luminance information of a specific wavelength component from the compressed image data, and extracts partial image data based on the luminance information of the specific wavelength component.

[0105] The configuration of the imaging device, the method for compressing hyperspectral information, and the algorithm for reconstructing three-dimensional hyperspectral data are not limited to those described in the above-described embodiments. For example, the arrangement of the filter array 110, the optical system 140, and the image sensor 160 is not limited to that shown in Figures 1A to 1D and may be modified as appropriate. Furthermore, the characteristics of the filter array 110 are not limited to those exemplified with reference to Figures 2A to 4B, and a filter array 110 with optimal characteristics may be used depending on the application or purpose.

[0106] The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength images. The technology disclosed herein can also be applied to, for example, biomedical, cosmetic, and other sensing applications, food foreign matter and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

[0107] 70 Object 100 Imaging device 110 Filter array 120 Compressed image 140 Optical system 150 Control circuit 160 Image sensor 200 Signal processing device 210 Memory 220 Hyperspectral image 250 Signal processing circuit 300 Display device 320 Image processing circuit 330 Display 400 Input UI

Claims

1. A signal processing method executed by a computer, the method comprising: obtaining first compressed image data indicating a two-dimensional image including hyperspectral information which is luminance information for each of at least four wavelength bands included in a target wavelength range, the hyperspectral information being compressed; extracting partial image data from the first compressed image data; and generating first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band from the partial image data.

2. The method according to claim 1, wherein the first compressed image data includes data of a plurality of pixels, data of each of the plurality of pixels includes first information and second information that are superimposed on each other, and each of the first information and the second information corresponds to one wavelength band among the at least four wavelength bands.

3. The method according to claim 1 or 2, wherein the first compressed image data includes spatially encoded first information and spatially encoded second information, and each of the first information and the second information corresponds to one wavelength band among the at least four wavelength bands.

4. The method according to any one of claims 1 to 3, further comprising: determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data; executing a first process when it is determined that the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data; and executing a second process different from the first process when it is determined that the partial image data is not suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data.

5. The method according to claim 4, wherein determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data is executed based on the number of pixels of the partial image data.

6. The method according to claim 5, wherein the second process is to increase the number of pixels of the partial image data.

7. The first compressed image data is generated by an imaging device including a filter array including a plurality of types of optical filters having different spectral transmittances from each other, and an image sensor that acquires an image formed by light that has passed through the filter array. The method according to any one of claims 1 to 6.

8. The first compressed image data is generated by an imaging device including a filter array including a plurality of types of optical filters having different spectral transmittances from each other, and an image sensor that acquires an image formed by light that has passed through the filter array. The second process is to generate second compressed image data by causing the imaging device to image an area corresponding to the partial image data after expanding the area. The first two-dimensional image data and the second two-dimensional image data are generated based on the second compressed image data. The method according to claim 4.

9. The method further includes acquiring mask data reflecting a spatial distribution of the spectral transmittance of the filter array. The first two-dimensional image data and the second two-dimensional image data are data of a portion corresponding to the partial image data and data generated based on the partial image data among the mask data. The method according to claim 7 or 8.

10. The method according to claim 4, wherein the second process is to cause a display device to display an error.

11. The method according to any one of claims 1 to 10, further including causing a display device to display a graphical user interface for allowing a user to specify an area extracted as the partial image data in the two-dimensional image.

12. The method according to any one of claims 1 to 11, further including detecting an edge of the two-dimensional image, wherein the partial image data is extracted based on the edge.

13. The method according to any one of claims 1 to 12, further comprising extracting luminance information of a specific wavelength component from the first compressed image data, wherein the partial image data is extracted based on the luminance information.

14. The method according to claim 4, further comprising causing a display device to display a graphical user interface for allowing a user to input threshold data for determining whether the partial image data is suitable for generating the first two-dimensional image data and the second two-dimensional image data from the partial image data.

15. The method according to claim 14, wherein the threshold data includes a parameter related to the number of pixels or the luminance error.

16. A signal processing apparatus comprising: a processor; and a memory storing a computer program executed by the processor, wherein the computer program causes the processor to: acquire compressed image data indicating a compressed two-dimensional image including hyperspectral information which is luminance information for each of at least four wavelength bands included in a target wavelength range; extract partial image data from the compressed image data; and generate first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band from the partial image data.

17. An imaging system comprising: the signal processing apparatus according to claim 16; and an imaging device that generates the compressed image data.

18. A computer program that causes a computer to: acquire compressed image data indicating a compressed two-dimensional image including hyperspectral information which is luminance information for each of at least four wavelength bands included in a target wavelength range; extract partial image data from the compressed image data; and generate first two-dimensional image data corresponding to a first wavelength band and second two-dimensional image data corresponding to a second wavelength band from the partial image data.

Images