Signal processing method, signal processing device and camera system

Abstract

A signal processing method according to one embodiment of the present disclosure includes: obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information within an object band; obtaining setting data for specifying one or more sub-bands as part of the object band; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands.

Description

Technical Field

[0001] The present disclosure relates to a signal processing method, a signal processing device and a camera system. Background Art

[0002] By making full use of the spectrum information of multiple narrow bands, such as dozens of bands, it is possible to grasp the detailed physical properties of the object that could not be grasped in the previous RGB image. The camera that obtains such multi-wavelength information is called a "hyperspectral camera". Hyperspectral cameras are used in various fields such as food inspection, biological inspection, pharmaceutical development and mineral component analysis.

[0003] Patent document 1 discloses an example of a hyperspectral imaging device using compressed sensing. The imaging device comprises: a coding element which is an array of multiple filters having different wavelength dependencies of light transmittance, an image sensor which detects light transmitted through the coding element, and a signal processing circuit. A coding element is arranged on an optical path connecting the subject and the image sensor. The image sensor simultaneously detects light having components of multiple wavelength bands overlapped for each pixel, thereby obtaining a wavelength multiplexed image. The signal processing circuit applies compressed sensing to the obtained wavelength multiplexed image using information on the spatial distribution of the spectral transmittance of the coding element, thereby reconstructing image data about each of the multiple wavelength bands.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent No. 9599511 Summary of the invention

[0007] Problems to be Solved by the Invention

[0008] In the conventional hyperspectral imaging device, image data of each band among all bands included in the band of the acquired wavelength multiplexed image (hereinafter referred to as "target band") is generated and displayed. However, depending on the application, sometimes only information about a part of the target band is needed. In such a case, it cannot be said to be efficient to generate information of useless bands with high wavelength resolution.

[0009] The present disclosure provides techniques for efficiently generating images of desired wavelength bands.

[0010] Means for solving problems

[0011] A method according to one embodiment of the present disclosure is a signal processing method executed by a computer. The method comprises: obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information in a target band; obtaining setting data for specifying one or more sub-bands as a part of the target band; and generating a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands based on the compressed image data.

[0012] The general or specific aspects of the present disclosure may also be implemented through a system, device, method, integrated circuit, computer program, or a computer-readable recording disk or other recording medium, or through any combination of a system, device, method, integrated circuit, computer program, and recording medium. Computer-readable recording media may include, for example, non-volatile recording media such as CD-ROM (Compact Disc-Read Only Memory). A device may also be composed of more than one device. In the case where a device is composed of more than two devices, the two or more devices may be configured in one device or separately configured in two or more separate devices. In this specification and claims, "device" may refer not only to one device, but also to a system composed of multiple devices.

[0013] Effects of the Invention

[0014] According to the present disclosure, it is possible to efficiently generate an image in a desired wavelength band. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A FIG. 1 is a diagram schematically showing an exemplary hyperspectral imaging system.

[0016] Figure 1B FIG. 1 is a diagram schematically showing a first modification of an exemplary hyperspectral imaging system.

[0017] Figure 1C FIG. 1 is a diagram schematically showing a second modification of the exemplary hyperspectral imaging system.

[0018] Figure 1D FIG. 1 is a diagram schematically showing a third modification of the exemplary hyperspectral imaging system.

[0019] Figure 2A It is a diagram schematically showing an example of a filter array.

[0020] Figure 2B represents a plurality of bands W1, W2, ..., W N A diagram showing an example of the spatial distribution of the transmittance of each light.

[0021] Figure 2C Yes means Figure 2A Graph showing an example of the spectral transmittance of area A1 included in the filter array shown.

[0022] Figure 2D Yes means Figure 2A Graph showing an example of the spectral transmittance of area A2 included in the filter array shown.

[0023] Figure 3A represents the target band W and the multiple bands W1, W2, ..., W N A diagram showing an example of the relationship between .

[0024] Figure 3B represents the target band W and the multiple bands W1, W2, ..., W N A diagram of another example of the relationship between .

[0025] Figure 4A This is a diagram for explaining the characteristics of the spectral transmittance in a certain area of ​​the filter array.

[0026] Figure 4B It indicates that each band W1, W2, ..., W N right Figure 4A The graph shows the results of averaging the spectral transmittance shown.

[0027] Figure 5 The figure schematically shows the usage scenario of the hyperspectral camera.

[0028] Fig. 6A 1 is a diagram showing an example of a target band W and a designated sub-band Wa.

[0029] Figure 6B This is a diagram showing an example in which not only the first sub-band but also the second sub-band is specified.

[0030] Figure 7 It is a diagram showing the configuration of an imaging system in an exemplary embodiment of the present disclosure.

[0031] Figure 8 It is a flowchart showing the operation of the system.

[0032] Fig. 9 FIG. 1 is a diagram showing an example of mask data before conversion stored in a memory.

[0033] Fig.10 This is a diagram showing an example of a GUI for inputting imaging conditions.

[0034] Fig.11 It is a diagram showing an example of a GUI for inputting restoration conditions.

[0035] Fig.12 It is a diagram showing an example of a GUI for inputting restoration conditions.

[0036] Fig.13 This is a diagram showing an example of a screen displaying a spectral image generated as a result of restoration calculation.

[0037] Fig.14 This is a diagram for explaining an example of a method of synthesizing mask information of a plurality of bands to convert the information into new mask information.

[0038] Fig.15A FIG. 2 is a diagram showing an example of mask data after conversion recorded in the memory.

[0039] Fig. 15B It is a diagram showing another example of the converted mask data recorded in the memory.

[0040] Fig.16 This is a diagram showing an example of a method of generating an image for each of a plurality of bands included in a target band.

[0041] Fig.17 This is a diagram showing the configuration of a system when the signal processing circuit does not perform mask information conversion.

[0042] Fig.18 It is a diagram showing another example of the GUI for setting the restoration conditions.

[0043] Fig.19 The figure shows an example of displaying an image of an unspecified wavelength band.

[0044] Fig. 20 This is a diagram showing an example of a method of restoring only a specific sub-band with high wavelength resolution by performing two-stage restoration. DETAILED DESCRIPTION

[0045] The embodiments described below all represent general or specific examples. The numerical values, shapes, materials, constituent elements, configurations, positions and connection methods of constituent elements, steps and the order of steps shown in the following embodiments are examples, and their main purpose is not to limit the technology disclosed in the present invention. Among the constituent elements in the following embodiments, the constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. Each figure is a schematic diagram and is not necessarily a strict illustration. Furthermore, in each figure, the same mark is given to the constituent elements that are substantially the same or similar. Sometimes repeated descriptions are omitted or simplified.

[0046] In the present disclosure, all or part of a circuit, unit, device, component or part, or all or part of a functional module in a block diagram, for example, can be performed by a semiconductor device, a semiconductor integrated circuit (IC), or one or more electronic circuits including LSI (large-scale integration). LSI or IC can be integrated into one chip or composed of a combination of multiple chips. For example, functional modules other than storage elements can also be integrated into one chip. It is called LSI or IC here, but the name changes depending on the degree of integration, and it can also be called a system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). A field programmable gate array (FPGA, Field Programmable Gate Array) programmed after the manufacture of LSI or a reconfigurable logic device (reconfigurable logic device) that can reconstruct the bonding relationship inside LSI or set the circuit division inside LSI can also be used for the same purpose.

[0047] Furthermore, all or part of the functions or actions of a circuit, unit, device, component or part can also be executed by software processing. In this case, the software is recorded on one or more non-volatile recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processing device (processor), the functions determined by the software are executed by the processing device and peripheral devices. The system or device may also have one or more non-volatile recording media, a processing device, and required hardware devices such as interfaces that record the software.

[0048] First, a configuration example of a hyperspectral imaging system according to an embodiment of the present disclosure and findings of the present inventors will be described.

[0049] Figure 1A: is a diagram schematically showing an exemplary hyperspectral imaging system. The system includes an imaging device 100 and a processing device 200. The imaging device 100 has the same configuration as the imaging device disclosed in Patent Document 1. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The filter array 110 has the same structure and function as the "coding element" disclosed in Patent Document 1. Therefore, in the following description, the filter array 110 is sometimes referred to as a "coding element". The optical system 140 and the filter array 110 are arranged on the optical path of light incident from an object 70 as a subject. The filter array 110 is arranged between the optical system 140 and the image sensor 160.

[0050] exist Figure 1A , an apple is illustrated as an example of the object 70. The object 70 is not limited to an apple, but may be any object. The processing device 200 generates image data for each of the multiple bands included in the object band based on the image data generated by the image sensor 160. In this specification, the image data is referred to as "spectrographic image data". Here, the number of bands included in the object band is set to N (N is an integer greater than 4). In the following description, the generated spectral image data of the multiple bands are referred to as spectral images 220W1, 220W2, . . . , 220W N , which are collectively referred to as the spectral image 220. In this specification, the data or signal representing an image, that is, the set of data or signals representing the pixel value of each pixel is sometimes simply referred to as an "image".

[0051] The filter array 110 is an array of a plurality of light-transmitting filters arranged in rows and columns. The plurality of filters include a plurality of types of filters having different spectral transmittances, i.e., wavelength dependencies of light transmittances. The filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 110 is referred to as "encoding" in this specification.

[0052] exist Figure 1A In the example shown, the filter array 110 is arranged near or just above (directly above) the image sensor 160. Here, "near" means: close to the extent that the image of the light from the optical system 140 is formed on the surface of the filter array 110 in a somewhat clear state. "Just above (directly above)" means that the two are close to the extent that there is almost no gap. The filter array 110 and the image sensor 160 may also be integrated.

[0053] The optical system 140 includes at least one lens. Figure 1AAlthough the optical system 140 is shown as a single lens in FIG. 1 , the optical system 140 may be a combination of a plurality of lenses. The optical system 140 forms an image on an imaging surface of the image sensor 160 via the optical filter array 10 .

[0054] The filter array 110 may also be configured separately from the image sensor 160 . Figures 1B to 1D 1 is a diagram showing an example of the configuration of the imaging device 100 in which the filter array 110 and the image sensor 160 are arranged apart from each other. Figure 1B In the example of FIG. 1 , the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position away from the image sensor 160. Figure 1C In the example of FIG. 1 , the filter array 110 is arranged between the object 70 and the optical system 140. Figure 1D In the example of , the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.

[0055] The image sensor 160 is a black-and-white type photodetector having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged in two dimensions. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, an infrared array sensor, a terahertz array sensor, or a millimeter wave array sensor. The light detection element includes, for example, a photodiode. The image sensor 160 does not necessarily have to be a black-and-white type sensor. For example, a color type sensor having a filter of R / G / B, R / G / B / IR, or R / G / B / W may also be used. By using a color type sensor, the amount of information related to the wavelength can be increased, and the accuracy of the reconstruction of the spectral image 220 can be improved. The wavelength range of the acquisition object can be determined arbitrarily, and is not limited to the visible wavelength range, but may also be the wavelength range of ultraviolet, near infrared, mid-infrared, far infrared, microwave / radio waves.

[0056] The processing device 200 is a computer including a processor and a storage medium such as a memory. The processing device 200 generates a plurality of spectral images 220W1, 220W2, ..., 220W1, each including information of a plurality of wavelength bands, based on the image 120 acquired by the image sensor 160. N data.

[0057] Figure 2A1 is a diagram schematically showing an example of a filter array 110. The filter array 110 has a plurality of regions arranged in two dimensions. In this specification, the region is sometimes referred to as a "unit". In each region, a filter having an individually set spectral transmittance is arranged. When the wavelength of the incident light is set to λ, the spectral transmittance is represented by a function T(λ). The spectral transmittance T(λ) can take a value greater than 0 and less than 1.

[0058] exist Figure 2A In the example shown, the filter array 110 has 48 rectangular areas arranged in 6 rows and 8 columns. This is just an example, and in actual use, more areas can be provided. The number of such areas can be, for example, the same as the number of pixels of the image sensor 160. The number of filters included in the filter array 110 is determined, for example, in the range of tens to tens of millions depending on the use.

[0059] Figure 2B represents a plurality of bands W1, W2, ..., W N An example of the spatial distribution of the transmittance of each light. Figure 2B In the example shown, the difference in lightness and darkness of each area shows the difference in transmittance. The lighter the area, the higher the transmittance, and the darker the area, the lower the transmittance. Figure 2B As shown, the spatial distribution of light transmittance differs depending on the wavelength band.

[0060] Figure 2C and Figure 2D They are respectively Figure 2A FIG. 1 is a diagram showing an example of spectral transmittance of area A1 and area A2 included in the filter array 110 shown. The spectral transmittance of area A1 is different from the spectral transmittance of area A2. In this way, the spectral transmittance of the filter array 110 is different depending on the area. However, it is not necessarily necessary for the spectral transmittance of all areas to be different. In the filter array 110, the spectral transmittance of at least a part of the multiple areas is different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of modes of spectral transmittance of the multiple areas included in the filter array 110 can be the same as or more than the number N of wavelength bands included in the target wavelength band. The filter array 110 can also be designed so that the spectral transmittance of more than half of the areas is different.

[0061] Figure 3A and Figure 3B It is used to explain the object band W and the multiple bands W1, W2, ..., W NThe object band W can be set to various ranges depending on the purpose. The object band W can be, for example, a band of visible light from approximately 400nm to approximately 700nm, a band of near infrared from approximately 700nm to approximately 2500nm, or a band of near ultraviolet from approximately 10nm to approximately 400nm. Alternatively, the object band W can also be an electric wave band such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. In this way, the band used is not limited to the visible light domain. In this specification, for convenience, non-visible light such as near ultraviolet rays, near infrared rays, and electric waves is also referred to as "light" without being limited to visible light.

[0062] exist Figure 3A In the example shown, N is set to an arbitrary integer greater than 4, and the target band W is divided into N equally divided bands as bands W1, W2, . . . , W N However, this is not limited to such an example. The multiple bands included in the target band W can also be set arbitrarily. For example, the bandwidth can be made uneven according to the band. There can also be gaps or overlaps between adjacent bands. Figure 3B In the example shown, the bandwidth is different for each band, and there is a gap between two adjacent bands. As described above, the plurality of bands only need to be different from each other, and the method of determining the plurality of bands is arbitrary.

[0063] Figure 4A 1 is a diagram for explaining the characteristics of the spectral transmittance in a certain area of ​​the filter array 110. Figure 4A In the example shown, the spectral transmittance has a plurality of local maximum values ​​P1 to P5 and a plurality of local minimum values ​​with respect to the wavelength in the target wavelength band W. Figure 4A In the example shown, the light transmittance in the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. Figure 4A In the example shown, in band W2 and band W N-1 In this way, the spectral transmittance of each region has a maximum value in multiple wavelength bands W1 to W N There are maxima in at least two of the bands. Figure 4A In the example, the local maximum values ​​P1, P3, P4 and P5 are greater than 0.5.

[0064] As described above, the light transmittance of each region is different depending on the wavelength. Therefore, the filter array 110 transmits more components of a certain band among the incident light, and transmits less components of other bands. For example, the transmittance of light of k bands among N bands may be greater than 0.5, and the transmittance of light of the remaining Nk bands may be less than 0.5. k is an integer satisfying 2≤k<N. If the incident light is white light that evenly contains the wavelength components of all visible light, the filter array 110 modulates the incident light into light with multiple discrete intensity peaks with respect to the wavelength for each region, and overlaps and outputs these multi-wavelength lights.

[0065] Figure 4B As an example, each band W1, W2, ..., W N right Figure 4A . The averaged transmittance can be obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing it by the bandwidth of the wavelength band. In this specification, the value of the transmittance averaged for each wavelength band is set as the transmittance in the wavelength band. In this example, the transmittance is significantly higher in the three wavelength bands with the maximum values ​​P1, P3 and P5. In particular, the transmittance exceeds 0.8 in the two wavelength bands with the maximum values ​​P3 and P5.

[0066] exist FIG. 2A to FIG. 2D In the example shown, it is assumed that the transmittance distribution of each region can take any value of grayscale greater than 0 and less than 1. However, it is not necessarily necessary to set it as a grayscale transmittance distribution. For example, it is also possible to adopt a binary scale transmittance distribution in which the transmittance of each region can take a value of approximately 0 or approximately 1. In the binary scale transmittance distribution, each region transmits most of the light of at least two bands among the multiple bands included in the target band, and does not transmit most of the light of the remaining bands. Here, "most of" means approximately 80% or more.

[0067] It is also possible to replace a portion of all cells, for example, half of the cells, with transparent areas. Such transparent areas make all bands W1 to W2 included in the target band W N The light is transmitted with the same high transmittance, for example, 80% or more. In such a configuration, the plurality of transparent regions can be arranged, for example, in a checkerboard shape. That is, in the two arrangement directions of the plurality of regions in the filter array 110, the regions with different light transmittances depending on the wavelength and the transparent regions can be arranged alternately.

[0068] Data indicating the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance through design data or actual measurement calibration, and is stored in a storage medium provided in the processing device 200. This data is used in the calculation process described later.

[0069] The filter array 110 can be constructed, for example, by using a multilayer film, an organic material, a diffraction grating structure, or a fine structure containing metal. In the case of using a multilayer film, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is different for each unit. Thereby, different spectral characteristics can be achieved according to the unit. By using a multilayer film, a sharp increase and decrease in the spectral transmittance can be achieved. By making the contained pigments or dyes different according to the unit, or by stacking different types of materials, a structure using organic materials can be achieved. By setting a diffraction structure with different diffraction spacing or depth for each unit, a structure using a diffraction grating structure can be achieved. In the case of using a fine structure containing metal, it can be made using spectroscopy based on the plasmon effect.

[0070] Next, an example of signal processing by the processing device 200 is described. The processing device 200 reconstructs a multi-wavelength spectral image 220 based on the image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance of the filter array 110 at each wavelength. Here, multi-wavelength means, for example, more bands than the three-color bands of RGB acquired by a normal color camera. The number of the bands can be, for example, about 4 to 100. The number of the bands is called the number of bands. Depending on the application, the number of bands may exceed 100.

[0071] The data to be obtained is the data of the spectral image 220, and this data is denoted as f. If the number of bands is denoted as N, then f is the image data f1, f2, ..., f of each band. N Here, Figure 1A As shown in FIG. 1 , the horizontal direction of the image is set to the x direction, and the vertical direction of the image is set to the y direction. If the number of pixels in the x direction of the image data to be obtained is set to n, and the number of pixels in the y direction is set to m, then the image data f1, f2, . . . , f N are 2D data of n×m pixels respectively. Therefore, data f is 3D data of n×m×N elements. This 3D data is called "hyperspectral data cube" or "hyperspectral cube". On the other hand, the number of elements of data g of image 120 obtained by encoding and multiplexing the filter array 110 is n×m. Data g can be expressed by the following formula (1).

[0072] [Number 1]

[0073]

[0074] Here, f1, f2, ···, f N Each is data with n×m elements. Therefore, the vector on the right is strictly a 1-dimensional vector with n×m×N rows and 1 column. The vector g is transformed into a 1-dimensional vector with n×m rows and 1 column to express and calculate. The matrix H expresses the components f1, f2, ..., f2 of the vector f with different encoding information (hereinafter also referred to as "masking information") for each band. N The transformation is to perform coding and intensity modulation and add them. Therefore, H is a matrix of n×m rows and n×m×N columns. In this specification, the matrix H is sometimes referred to as a "system matrix".

[0075] If the vector g and the matrix H are given, f should be calculated by solving the inverse problem of equation (1). However, the number of elements n×m×N of the data f to be obtained is greater than the number of elements n×m of the acquired data g, so this problem is an ill-posed problem and cannot be solved directly. Therefore, the processing device 200 uses the redundancy of the image contained in the data f and solves it using the compressed sensing method. Specifically, the data f to be obtained is estimated by solving the following equation (2).

[0076] [Number 2]

[0077]

[0078] Here, f' represents the data of the estimated f. The first term in the brackets of the above formula represents the deviation between the estimated result Hf and the acquired data g, that is, the so-called residual term. Here, the sum of squares is set as the residual term, but the absolute value or the square root of the sum of squares can also be set as the residual term. The second term in the brackets is a regularization term or a stabilization term. Formula (2) means finding f that minimizes the sum of the first term and the second term. The processing device 200 converges the solution through iterative regression operations and can calculate the final solution f'.

[0079] The first term in the brackets of formula (2) means the operation of finding the sum of the squares of the differences between the acquired data g and Hf obtained by transforming f in the estimation process through the matrix H. The second term Φ(f) is a constraint condition in the regularization of f, and is a function that reflects the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be represented by, for example, discrete cosine transform (DCT), wavelet transform, Fourier transform or total variation (TV) of f. For example, when the total variation is used, stable estimated data that suppresses the influence of noise on the observed data g can be obtained. The sparsity of the object 70 in the space of each regularization term is different depending on the texture of the object 70. It is also possible to select a regularization term that makes the texture of the object 70 sparser in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weight coefficient. The larger the weight coefficient τ, the more redundant data is reduced and the higher the compression ratio. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to a moderate value that allows f to converge to a certain extent and does not compress excessively.

[0080] In addition, Figure 1B and Figure 1C In the configuration, the image encoded by the filter array 110 is obtained in a blurred state on the imaging plane of the image sensor 160. Therefore, by pre-holding the blur information and reflecting the blur information in the above-mentioned system matrix H, the spectral image 220 can be reconstructed. Here, the blur information is represented by a point spread function (Point Spread Function: PSF). PSF is a function that specifies the degree of diffusion of a point image to surrounding pixels. For example, in the case where a point image equivalent to 1 pixel on an image diffuses to an area of ​​k×k pixels around the pixel due to blur, the PSF can be specified as a coefficient group, i.e., a matrix, that represents the influence on the brightness of each pixel in the area. By reflecting the influence of the blur of the coding pattern based on the PSF in the system matrix H, the spectral image 220 can be reconstructed. The position at which the filter array 110 is configured is arbitrary, and a position can be selected so that the coding pattern of the filter array 110 does not diffuse excessively and disappear.

[0081] In the above composition, if Figure 3A and Figure 3B As shown, a plurality of bands W1 to W1 included in the target band W are generated. N However, depending on the application, images for all these bands may not be needed. In such cases, it is not efficient to generate images for all bands with high wavelength resolution.

[0082] Figure 5Schematically shows the use scenario of the hyperspectral camera. In some cases, the user sometimes wants to obtain color information in the red band, such as the band of 600nm to 700nm, in order to estimate the sugar content of apples. In other cases, the user sometimes wants to obtain color information in the green band, such as the band of 500nm to 600nm, in order to know the accurate spectrum of leaves. In still other cases, the user sometimes wants to obtain color information in the blue band, such as the band of 400nm to 500nm, in order to know the fading status of blue products.

[0083] In such a case, conventional hyperspectral cameras use one of the following methods (1) and (2) to obtain color information in different wavelength bands: (1) An image sensor capable of independently obtaining information in each wavelength band is used.

[0084] (2) Use a camera that can acquire information in a wide wavelength band to acquire color information in a wide range and display only the information in the required wavelength band.

[0085] In the method (1), it is necessary to prepare a different image sensor for an application that requires color information different from the color information required in a certain application.

[0086] In the method (2), color information of a band wider than the required band is obtained, and an image is generated for each of the multiple bands included in the wide band. Therefore, a large amount of calculation is performed even for unnecessary bands, which consumes more calculation cost than necessary.

[0087] Therefore, in the embodiment of the present disclosure, the user is allowed to specify one or more sub-bands as a part of the target band W based on the image data obtained by the hyperspectral imaging device. The signal processing device generates a hyperspectral data cube representing a plurality of 2D images of a plurality of bands included in the one or more specified sub-bands. Thus, the calculation cost can be suppressed and detailed spectral information on the sub-bands required by the user can be obtained.

[0088] Fig. 6A The target band W and the specified sub-band W a In this example, only one sub-band W is specified. a . Sub-band W a is part of the object band W and contains multiple bands W a1 , W a2 、···、W ai Here, i represents the sub-band W a The signal processing device generates a signal indicating the number of bands W included. a1 , W a2 、···、Wai The hyperspectral data cube of the 2D images of each waveband in .

[0089] Figure 6B Indicates that not only the first sub-band W is specified a And specify the second sub-band W b The first sub-band W a and the second sub-band W b The second sub-band W b Contains multiple bands W b1 , W b2 、···、W bj Here, j represents the second sub-band W b The number of bands to include. Like this, multiple sub-bands can also be specified.

[0090] In this way, the signal processing device generates images of multiple bands included in the designated sub-band based on the image data generated by the camera device. The generated images of multiple bands are displayed on the display. By such an operation, the computational cost of generating spectral images that are not required in the intended use scene can be reduced.

[0091] The following describes an outline of embodiments of the present disclosure.

[0092] A signal processing method according to an embodiment of the present disclosure is executed by a computer. The method includes: obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information in a target band; obtaining setting data for specifying one or more sub-bands as a part of the target band; and generating a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands based on the compressed image data.

[0093] “Hyperspectral information within the target band” means information of multiple images corresponding to multiple bands included in the predetermined target band. “Compressing hyperspectral information (compressing hyperspectral information)” includes compressing image information of multiple bands into one black-and-white two-dimensional image using a coding element such as the filter array 120 described above, and compressing image information of multiple bands acquired in advance into one black-and-white two-dimensional image by software processing.

[0094] According to the above method, for example, data of multiple two-dimensional images corresponding to multiple bands included in one or more sub-bands specified by the user, that is, a hyperspectral data cube can be generated. Therefore, only the required hyperspectral data cube can be generated according to the application or purpose.

[0095] The hyperspectral information may also be information of more than four bands included in the target band, and the two-dimensional image information may also be data of a plurality of pixels included in the compressed image data. The information of the more than four bands may also be superimposed on the data of each of the plurality of pixels. In other words, the data of each pixel of the compressed image data may include a value obtained by superimposing the information of more than four bands included in the target band. The data of each pixel of the compressed image data may also be superimposed with information of more than 10 or more than 100 bands depending on the purpose.

[0096] The setting data may include information for specifying the wavelength resolution in the one or more sub-bands. The plurality of 2D images may be generated at the wavelength resolution. In such a manner, the user can not only specify the sub-bands, but also specify the wavelength resolution for each sub-band. Therefore, the wavelength resolution of the sub-bands requiring detailed spectral information can be improved and adjusted flexibly.

[0097] The one or more sub-bands may include a first sub-band and a second sub-band. The plurality of two-dimensional images may be generated for each of the first sub-band and the second sub-band.

[0098] The one or more sub-bands may include a first sub-band and a second sub-band. The wavelength resolution may be independently specified for each of the first sub-band and the second sub-band. The plurality of two-dimensional images may be generated at the corresponding wavelength resolution for each of the first sub-band and the second sub-band.

[0099] The first sub-band and the second sub-band may be separated from each other. Alternatively, the first sub-band and the second sub-band may be adjacent to each other or partially overlap each other.

[0100] The method may further include causing a display connected to the computer to display a graphical user interface (GUI) for a user to input the setting data. By displaying such a GUI, the user can easily perform operations such as specifying a sub-band and specifying a wavelength resolution for each sub-band.

[0101] The method may further include causing a display connected to the computer to display the plurality of two-dimensional images. This allows a user to easily check the generated spectral images for each wavelength band.

[0102] The compressed image data may be generated by performing imaging using a filter array including a plurality of types of filters having mutually different spectral transmittances and an image sensor. The method may further include obtaining shielding data reflecting the spatial distribution of the spectral transmittance of the filter array. The plurality of two-dimensional images may be generated based on the compressed image data and the shielding data.

[0103] The method may also include: obtaining mask data including information of a plurality of mask images, the plurality of mask images being obtained by the image sensor through the filter array to capture a plurality of backgrounds corresponding to a plurality of unit bands included in the target band. The plurality of two-dimensional images may be generated based on the compressed image data and the mask data.

[0104] The shielding data may be, for example, data that specifies the matrix H in the above-mentioned formula (2). The form of the shielding data may vary depending on the configuration of the imaging system. The shielding data may represent the spatial distribution of the spectral transmittance of the filter array, or may include information for calculating the spatial distribution of the spectral transmittance of the filter array. For example, the shielding data may include not only the information of the above-mentioned shielding image, but also the information of the background image of each unit band. By dividing the shielding image by the background image for each pixel, information on the transmittance distribution can be obtained for each unit band. The shielding data may also include only the information of the shielding image. The shielding image represents the distribution of values ​​obtained by multiplying the transmittance of the filter array by the sensitivity of the image sensor. Such shielding data can be used in a configuration in which the filter array is arranged in a manner closely opposed to the image sensor.

[0105] The shielding data may include shielding information. The shielding information may represent the spatial distribution of the transmittance of the filter array in each of the plurality of unit bands included in the target band. The method may further include: generating synthetic shielding information by synthesizing a portion of the shielding information corresponding to the plurality of unit bands included in the non-designated bands other than the one or more sub-bands in the target band; and generating a synthetic image corresponding to the non-designated band based on the compressed image data and the synthetic shielding information.

[0106] The method may also include: generating a synthetic masked image obtained by synthesizing the masked images of multiple unit bands contained in a non-specified band other than the one or more designated sub-bands in the object band; and generating synthetic image data about the non-specified band based on the compressed image data and the synthetic masked image.

[0107] According to the above method, a detailed spectral image is not generated for non-specified wavelength bands, but a detailed spectral image is generated only for designated sub-wavelength bands. Therefore, the calculation time required to generate the spectral image can be shortened.

[0108] The masking data may also include information of a plurality of background images obtained by photographing the plurality of backgrounds by the image sensor without passing through the filter array. The method may also include generating a composite background image obtained by synthesizing the plurality of background images. The composite image may be generated based on the compressed image data, the composite masking image, and the composite background image.

[0109] The masking data may also include a plurality of background images and a plurality of masking images. Each of the plurality of background images is obtained, for example, by photographing a corresponding background among the plurality of backgrounds by the image sensor without passing through the filter array. Each of the plurality of masking images is obtained, for example, by photographing the corresponding background among the plurality of backgrounds by the image sensor through the filter array. The synthetic masking information may be generated based on the plurality of masking images and the plurality of background images.

[0110] The method may further include causing a display connected to the computer to display the synthesized image. Thus, the user can easily check a rough image for an unspecified wavelength band.

[0111] The masking data may include masking information. The masking information may represent the spatial distribution of the transmittance of the filter array in each of the plurality of unit bands included in the object band. The setting data may include information for specifying a plurality of large sub-bands each being a part of the object band and a plurality of small sub-bands included in at least one of the plurality of large sub-bands. The method may further include: synthesizing a portion of the masking information corresponding to the plurality of unit bands included in each of the plurality of large sub-bands, thereby generating a first synthesized masking information, with respect to each of the plurality of large sub-bands; and generating a first synthesized image with respect to each of the plurality of large sub-bands based on the compressed image data and the first synthesized masking information. The plurality of 2D images may be generated corresponding to the plurality of small sub-bands. The method may also include: generating, for each of the multiple small sub-bands, a second synthetic masking information obtained by synthesizing the masking information of the multiple unit bands contained in the small sub-band; and generating a second synthetic image for each of the small sub-bands based on the first synthetic image of at least one of the multiple designated large sub-bands and the second synthetic masking information.

[0112] According to the above method, for example, a synthesized image can be generated for each of the plurality of large sub-bands included in the target band and the plurality of small sub-bands included in at least one of the large sub-bands specified by the user. Therefore, it is possible to perform flexible adjustment such as setting only small sub-bands for bands that require detailed color information and setting only large sub-bands for bands that do not require detailed color information.

[0113] The object band may include a visible band. The method may also include: generating an image corresponding to a red band, an image corresponding to a green band, and an image corresponding to a blue band based on the compressed image data and the synthetic mask information; and causing a display connected to the computer to display an RGB image based on the image corresponding to the red band, the image corresponding to the green band, and the image corresponding to the blue band. Thus, the user can confirm the RGB image of the object different from the detailed spectral image of the designated sub-band.

[0114] Still another embodiment of the present disclosure relates to a method for generating masking data. The masking data is used to restore spectral image data of each wavelength band from compressed image data obtained by an imaging device including a filter array, the filter array including multiple types of filters with mutually different spectral transmittances. That is, the above method is a method for generating masking data for restoring spectral image data of each wavelength band from compressed image data obtained by an imaging device including a filter array, the filter array including multiple types of filters with mutually different spectral transmittances. The method includes: obtaining first masking data for restoring a first spectral image corresponding to a first wavelength band group in a target wavelength band; obtaining setting data for specifying one or more sub-bands as part of the target wavelength band; and generating second masking data for restoring second spectral image data corresponding to a second wavelength band group in the one or more sub-bands based on the first masking data and the setting data.

[0115] The first band group may be a collection of all or part of the bands included in the target band. The second band group may be a collection of all or part of the bands included in the sub-band. The first band group and the second band group may each be a collection of synthetic bands obtained by synthesizing more than two unit bands. When such band synthesis is performed, the mask data is transformed in accordance with the synthesis method of the band. The setting data may also include information related to the synthesis method of the band used in the transformation process of the mask data.

[0116] The first shielding data and the second shielding data may be data reflecting the spatial distribution of the spectral transmittance of the filter array. The first shielding data may include first shielding information indicating the spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second shielding data may include second shielding information indicating the spatial distribution of the spectral transmittance corresponding to the second wavelength band group.

[0117] The second mask data may further include third mask information obtained by synthesizing a plurality of information, each of which indicates a spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-designated wavelength band other than the one or more sub-bands in the target wavelength band.

[0118] A signal processing device according to another embodiment of the present disclosure includes: a processor and a memory storing a computer program executed by the processor. The computer program causes the processor to: obtain compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target band; obtain setting data for specifying one or more sub-bands as a part of the target band; and generate a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands based on the compressed image data.

[0119] A signal processing device according to another embodiment of the present disclosure includes: a processor and a memory storing a computer program executed by the processor. The computer program causes the processor to execute: obtaining first masking data for restoring first spectral image data corresponding to a first band group in a target band; obtaining setting data for specifying one or more sub-bands as a part of the target band; and restoring second masking data for generating second spectral image data corresponding to a second band group in the one or more sub-bands based on the first masking data and the setting data.

[0120] An imaging system according to another aspect of the present disclosure includes: the signal processing device described above; and an imaging device that generates the compressed image data.

[0121] A computer program according to another embodiment of the present disclosure causes a computer to execute: obtaining compressed image data including two-dimensional image information obtained by compressing hyperspectral information within an object band; obtaining setting data for specifying one or more sub-bands as part of the object band; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands.

[0122] A computer program involved in other embodiments of the present disclosure causes a computer to execute: obtaining first masking data for restoring first spectral image data corresponding to a first band group in an object band from compressed image data obtained by a camera device including a filter array, wherein the filter array includes a plurality of types of filters having mutually different spectral transmittances; obtaining setting data for specifying one or more sub-bands that are part of the object band; and generating second masking data for restoring second spectral image data corresponding to a second band group in the one or more sub-bands based on the first masking data and the setting data.

[0123] Another embodiment of the present disclosure relates to a non-volatile storage medium that can be read by a computer and stores a program for causing a computer to execute a process, the process comprising: obtaining compressed image data, the compressed image data including two-dimensional image information obtained by compressing hyperspectral information within an object band; obtaining setting data for specifying one or more sub-bands that are part of the object band; and generating, based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in the one or more sub-bands.

[0124] Other embodiments of the present disclosure involve a non-volatile storage medium that can be read by a computer and stores a program for causing a computer to execute a process, which process includes: obtaining first masking data for restoring first spectral image data corresponding to a first band group in an object band; obtaining setting data for specifying one or more sub-bands that are part of the object band; and generating second masking data for restoring second spectral image data corresponding to a second band group in the one or more sub-bands based on the first masking data and the setting data.

[0125] The following describes a more specific embodiment of the present disclosure. In which, sometimes an overly detailed description is omitted. For example, sometimes a detailed description of a well-known matter and a repeated description of substantially the same structure are omitted. This is to avoid the following description being too lengthy and to make it easy for those skilled in the art to understand. In addition, the inventors provide the drawings and the following description in order to enable those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims by these. In the following description, the same reference numerals are attached to the same or similar components. In the following description, the xyz coordinates shown in the figure are used.

[0126] (Implementation Method)

[0127] Figure 71 is a diagram showing the configuration of an imaging system in an exemplary embodiment of the present disclosure. The system includes an imaging device 100, a processing device 200, a display device 300, and an input user interface (UI) 400. The processing device 200 corresponds to a signal processing device in the present disclosure.

[0128] The imaging device 100 includes an image sensor 160 and a control circuit 150 for controlling the image sensor 160. Figure 7 Although not shown in the figure, the camera device 100 is as shown in FIG. Figures 1A to 1D As shown in FIG. 1 , the optical filter array 110 and at least one optical system 140 are also provided. The configuration of the optical filter array 110 and the optical system 140 can be Figures 1A to 1D The image sensor 160 obtains a black and white image based on the light whose intensity is modulated by the filter array 110 for each area. In the data of each pixel of the black and white image, information of multiple bands within the target band W is superimposed. Therefore, the black and white image can be said to be an image obtained by compressing the hyperspectral information within the target band W into a two-dimensional image. Such a black and white image is an example of a "compressed image" in this specification. In addition, in this specification, data representing a compressed image is referred to as "compressed image data".

[0129] The processing device 200 includes a signal processing circuit 250 and a memory 210 such as a RAM and a ROM. The signal processing circuit 250 may be an integrated circuit having a processor such as a CPU or a GPU. The signal processing circuit 250 performs a restoration process based on the compressed image data output from the image sensor 160. The restoration process is similar to Figures 1A to 1D The processing performed by the processing device 200 shown is basically the same, but in this embodiment, the restoration processing is performed according to the restoration condition input from the input UI 400. The signal processing circuit 250 generates image data with high wavelength resolution only for the designated sub-band in the target band. As a result, the calculation time can be shortened. The memory 210 stores a computer program executed by a processor included in the signal processing circuit 250, various data referenced by the signal processing circuit 250, and various data generated by the signal processing circuit 250.

[0130] The display device 300 includes a memory 310, an image processing circuit 320, and a display 330. The memory 310 temporarily stores setting data indicating restoration conditions sent from the input UI 400. The image processing circuit 320 performs required processing on the image restored by the signal processing circuit 250 and then displays it on the display 330. The display 330 may be any display such as a liquid crystal or an organic LED.

[0131] The input UI 400 includes hardware and software for setting various conditions such as imaging conditions and restoration conditions. The imaging conditions may include, for example, conditions such as resolution, gain, and exposure time. The restoration conditions may include, for example, conditions such as the lower limit wavelength and the upper limit wavelength of each sub-band, the number of bands contained in each sub-band, and the number of calculations. The input imaging conditions are sent to the control circuit 150 of the imaging device 100. The control circuit 150 causes the image sensor 160 to perform imaging according to the imaging conditions. Thus, the image sensor 160 generates a compressed image in which information of multiple bands in the target band W is overlapped. In addition, the input restoration conditions are sent to the signal processing circuit 250 and the memory 310 and recorded. The signal processing circuit 250 performs restoration processing according to the set restoration conditions and generates a hyperspectral data cube for the designated sub-band. The image processing circuit 320 causes the display 330 to display images of each of the multiple bands in the designated sub-band according to the set restoration conditions.

[0132] The signal processing circuit 250 converts and uses the mask data pre-recorded in the memory 210 according to the restoration conditions input by the input UI 400 when necessary. The mask data is data indicating the spatial distribution of the spectral transmittance of the filter array 110, and includes information equivalent to the matrix H in the above-mentioned equation (2). The generated spectral image is processed as necessary by the image processing circuit 320. The image processing circuit 320 displays the spectral image on the display 330 after performing processing such as determining the arrangement in the screen, associating with the band information, or coloring according to the wavelength.

[0133] In the present embodiment, the signal processing circuit 250 generates an image for each of the plurality of bands only for at least one designated sub-band in the object band W. For bands other than the designated sub-band in the object band W, continuous bands are summed up as one band for calculation. Thus, the calculation cost can be suppressed. In addition, the signal processing circuit 250 may also generate an image for each of the plurality of bands for the entire object band W. In this case, the image processing circuit 320 may also extract data on the designated sub-band from the image data input by the signal processing circuit 250 and display it.

[0134] Figure 8is a flowchart showing the operation of the system of the present embodiment. In the present embodiment, first, in step S101, the user inputs the imaging condition and the restoration condition via the input UI 400 (step S101). The data representing the input imaging condition is sent to the control circuit 150. The data representing the input restoration condition is sent to the signal processing circuit 250 and the memory 310. The memory 310 temporarily stores the restoration condition. When displaying an image, the restoration condition is referred to in order to associate the image with the condition of the set band. Next, the imaging device 100 captures the object according to the imaging condition, thereby obtaining a compressed image (step S102).

[0135] If the compressed image is obtained, the signal processing circuit 250 determines whether the mask data needs to be transformed based on the input restoration condition (step S103). If transformation is necessary, the signal processing circuit 250 transforms the mask data pre-stored in the memory 210 (step S104). Here, transformation means synthesizing the mask information about multiple bands and treating it as the mask information of one band. For details on the synthesis of the mask information, refer to Fig.14 To be described later. If no transformation is required, step S104 is omitted. The signal processing circuit 250 uses the compressed image and the masking data transformed as needed to perform a restoration operation according to the input restoration conditions (step S105). Thus, a spectral image is generated based on the compressed image. Next, the image processing circuit 320 of the display device 300 associates the generated spectral image with the restoration conditions stored in the memory 310 and attaches labels (step S106). For example, image data is generated after labels representing corresponding bands are attached to the generated spectral images. The image processing circuit 320 outputs the generated image data to the display 330 and causes it to display the image (step S107).

[0136] Fig. 9 An example of mask data before conversion stored in the memory 210 is shown. The mask data in this example includes mask information indicating the spatial distribution of transmittance of each of the multiple unit bands included in the target band. The mask data in this example includes: mask information of each of the multiple unit bands divided into 1 nm, and information related to the conditions for obtaining the mask information. Each unit band is determined based on the lower limit wavelength and the upper limit wavelength. The mask information includes information of the mask image and the background image. Fig. 9The plurality of shielding images shown are obtained by photographing a plurality of backgrounds corresponding to the plurality of unit bands respectively by the image sensor 120 through the filter array 110. The plurality of background images are obtained by photographing the plurality of backgrounds by the image sensor 120 without the filter array 110. Such shielding image and background image data are recorded in advance for each unit band. The information related to the acquisition conditions includes the information of the exposure time and the gain. In addition, Fig. 9 In the example, data of the shielding image and the background image are recorded for each of the plurality of unit bands having a width of 1 nm. The width of each unit band is not limited to 1 nm and can be determined to be any value. In addition, when the uniformity of the background image is very high, the shielding data may not include information of the background image. For example, in a configuration in which the image sensor 120 and the filter array 110 are integrated in a manner that they are closely opposed to each other, the shielding information is substantially consistent with the shielding image, and therefore the shielding data may not include the background image.

[0137] Next, refer to Figures 10 to 13 , an example of a graphical user interface (GUI) displayed by the program for performing the above-mentioned information processing is described. Images for realizing these GUIs are generated by the signal processing circuit 250 and the image processing circuit 320 and displayed on the display 330.

[0138] Fig.10 An example of a GUI screen for inputting camera conditions is shown. In this example, the user sets the resolution, gain, exposure time, and frame rate before performing hyperspectral imaging. The resolution represents the number of vertical and horizontal pixels of the displayed image. The resolution can be specified, for example, by the user selecting a name such as VGA, HD, 4K, etc. from a drop-down menu, or by directly inputting the number of vertical and horizontal pixels. The gain can also be specified using a rational number greater than 0, and is input by adding, subtracting, multiplying, and dividing rational numbers. For example, when 8 / 3 is input, the gain can be set to 2.6666...dB. Both the exposure time and the frame rate do not need to be input. The user can also input at least one of the exposure time and the frame rate, and in the event of a conflict (for example, the exposure time is 100ms and the frame rate is 30fps, etc.), one of them will take priority. In addition to the input of the above four conditions, a function for automatically adjusting the gain, exposure time, and frame rate can also be set. For example, it can also be automatically adjusted so that the average brightness is 1 / 2 of the maximum brightness. It can also be like Fig.10As shown in the example, the GUI for inputting the imaging conditions has the function of saving and loading the set imaging conditions. In addition, the GUI may also have the function of displaying the compressed image obtained with the set imaging conditions in real time. Here, the compressed image itself does not necessarily have to be displayed. It is also possible to display any image obtained with the set imaging conditions at that moment. For example, it is also possible to configure pixels that only output red (R), green (G), and blue (B) values, and display an RGB image obtained using only the values ​​of these pixels. In addition, for example, through the synthesis processing of the unit bands described later, restoration based on three bands can be performed with 400nm to 500nm as the first band, 500nm to 600nm as the second band, and 600nm to 700nm as the third band, and the restoration result can be displayed as an RGB image.

[0139] Fig.11 and Fig.12 FIG. 4 is a diagram showing an example of a GUI for inputting restoration conditions. Fig.11 In the example shown, the user inputs the sub-band, wavelength resolution or number of band divisions, and the number of calculations. Here, the number of calculations represents the number of times the restoration operation shown in equation (2) is repeated. Fig.11 As shown, for example, the lower limit wavelength and the upper limit wavelength can be specified by setting them by dragging and dropping. Fig.11 In the example of , the sub-bands of 420nm to 480nm and 600nm to 690nm are specified. Fig.12 As shown, it is set to be possible to input the sub-bands and the ranges of each band in each sub-band as numerical values. The area for inputting the ranges of each sub-band and each band in each sub-band can be displayed as an independent window or in the screen for inputting other setting items.

[0140] exist Fig.11 In the example, the user inputs either the wavelength resolution or the number of band divisions. The number of calculations is specified by an arbitrary integer greater than 1. Typically, a number of 10 to 10,000 can be specified. Fig.11 In the example, the predicted calculation time is also displayed. The predicted calculation time is not input by the user, but is automatically calculated and displayed based on the set resolution, number of band divisions, and number of calculations. In addition, the functions of inputting the number of calculations and displaying the predicted calculation time can also be omitted. Instead, a form of selecting from multiple modes such as high-precision mode (slow speed), balanced mode (medium speed), and high-speed mode (high speed) can be adopted, for example, using a drop-down menu. Fig.11 As shown, a function of saving and loading the set restoration conditions may also be provided.

[0141] Fig.13FIG. 1 is a diagram showing an example of a screen displaying a spectral image generated as a result of restoration operation. The generated spectral image is associated with the set restoration conditions and displayed in a form that can be distinguished for each set band. For example, Fig.13 As shown, the lower and upper wavelength limits of each band can be displayed numerically together with the restored image of each band. Alternatively, each band can be represented by a number starting from the short wavelength side or the long wavelength side. The image of each band can also be displayed in the color contained in the band. In the above example, all physical quantities represented by wavelength (nm) can also be represented by wave numbers (e.g. cm -1 ) or frequency (e.g. Hz).

[0142] Fig.14 1 is a diagram for explaining an example of a method of synthesizing mask information of a plurality of bands to transform into new mask information. In this example, as mask information before transformation, mask information of unit bands #1 to #20 is as follows: Fig. 9 The data shown is stored in the memory 210 in advance. Fig.14 In the example, the synthesis process is not performed on the unit bands #1 to 5, but the synthesis process is performed on the unit bands #6 to 20. For the unit bands #1 to 5, the transmittance distribution of the filter array 110 is calculated by dividing the value of each area in the mask image by the value of the corresponding area in the background image. Here, the data of each mask image stored in the memory 210 is referred to as "unit mask image data", and the data of each background image stored is referred to as "unit background image data". For bands #6 to 20, the synthesized transmittance distribution is obtained by dividing the data obtained by summing the unit mask image data of bands #6 to 20 for each pixel by the data obtained by summing the unit background image data of bands #6 to 20 for each pixel. By performing such an operation, the mask information can be synthesized for any plurality of bands. In addition, when the uniformity of the background image is very high, the mask information is roughly consistent with the mask image. In this case, the data obtained by summing or averaging the mask image data of bands #6 to 20 can also be used as the synthesized mask data of bands #6 to 20.

[0143] exist Fig. 9 In the example shown, mask information is recorded for each of a plurality of unit bands each having a width of 1 nm. Fig.12 In the example shown, the width of each wavelength band designated by the user is 30 nm, which is relatively wide. In such a case, the signal processing circuit 250 synthesizes and restores the mask information of a plurality of unit bands for each designated wavelength band.

[0144] The mask data conversion process performed by synthesis can be performed in an environment used by an end user or in a manufacturing site such as a factory of a manufacturing system. When the mask data conversion process is performed at the manufacturing site, the mask data after conversion is stored in the memory 210 in place of the mask data before conversion or in addition to the mask data before conversion.

[0145] As described above, the mask data is used to restore the spectral image data for each wavelength band from the compressed image data acquired by the imaging device having a filter array including a plurality of types of filters having mutually different spectral transmittances. The method of converting the mask data in this embodiment includes the following steps.

[0146] Acquire first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength band.

[0147] Acquiring setting data for specifying one or more sub-bands as a part of the target band. Generating second mask data for restoring second spectral image data corresponding to a second band group in the sub-bands based on the first mask data and the setting data.

[0148] The first masking data may be, for example, data for restoring the spectral image of each unit band in all the unit bands included in the target band from a compressed image. The second masking data may be, for example, data for restoring the spectral image of each unit band in all the unit bands included in each designated sub-band from a compressed image. The second masking data may also be data for restoring the spectral image of each synthesized band obtained by synthesizing a plurality of unit bands from a compressed image. In the case of performing such a synthesis, the setting data may include data related to the synthesis method of the band. In the following description, a synthesized band obtained by synthesizing a plurality of unit bands is also referred to as an "edited band".

[0149] The first mask data and the second mask data are data reflecting the spatial distribution of the spectral transmittance of the filter array. The first mask data includes first mask information indicating the spatial distribution of the spectral transmittance corresponding to the first band group. The second mask data includes second mask information indicating the spatial distribution of the spectral transmittance corresponding to the second band group.

[0150] The second masking data may also include third masking information, which is information synthesized by synthesizing information indicating the spatial distribution of spectral transmittance corresponding to the third band group included in the non-designated band other than the designated one or more sub-bands. In this case, the signal processing circuit 250 can generate a spectral image with a relatively high wavelength resolution for each designated sub-band and a spectral image with a relatively low wavelength resolution for an undesignated non-designated band based on the compressed image and the second masking data.

[0151] Fig.15A and Fig. 15B 2 is a diagram showing an example of the second mask data after the conversion recorded in the memory 210. Fig.15A In the example shown, for each edited band having a width of 10 nm, the mask image and the background image are combined to form the transformed mask information and stored. In the case where the uniformity of the background image is very high, the mask data may not include the background image information. Fig. 15B In the example shown, for each edited band having a width of 10 nm, the converted mask data is stored as a composite mask data obtained by dividing the mask image by the background image. The wavelength width of the edited band is not limited to 10 nm, and can be set arbitrarily.

[0152] The synthesized mask image becomes an image obtained by averaging more unit mask images as the synthesized bandwidth becomes wider. Similarly, the synthesized mask data becomes data obtained by averaging mask data obtained by dividing more unit mask images by the unit background image as the synthesized bandwidth becomes wider. Therefore, there is a tendency that the wider the synthesized bandwidth becomes, the smaller the contrast of the synthesized mask image or the synthesized mask data becomes.

[0153] Fig.16 1 is a diagram showing an example of a method for generating an image for each of a plurality of bands included in a target band. In this example, the target band includes four sub-bands. The first sub-band includes bands #1 to 5. The second sub-band includes bands #6 to 10. The third sub-band includes bands #11 to 15. The fourth sub-band includes bands #16 to 20. In this example, the signal processing circuit 250 performs a restoration operation by synthesizing the masking information of all unit bands that do not belong to the sub-band for each sub-band. Fig.16 As shown, in any sub-band, even if the mask information of the unit band not included in the sub-band is synthesized, a good spectral image can be generated for the band within the sub-band. By performing such synthesis processing, the calculation time required to generate the image of each band can be shortened.

[0154] Next, a modification of the present embodiment will be described.

[0155] Fig.173 is a diagram showing the configuration of the system when the signal processing circuit 250 does not perform the conversion of the mask information. In this example, the signal processing circuit 250 reads the restoration condition given from the input UI 400 and the mask information stored in the memory 210, and generates a spectral image based on the compressed image obtained from the image sensor 160. In this case, the signal processing circuit 250 generates a spectral image in the entire target wavelength band and outputs it to the image processing circuit 320. The image processing circuit 320 displays only an image of a part of the wavelength band from the acquired spectral image on the display 330 according to the set restoration condition.

[0156] Fig.18 2 is a diagram showing another example of a GUI for setting restoration conditions. In this example, different wavelength resolutions or band division numbers can be specified for each sub-band to be set. The user inputs either the wavelength resolution or the band division number for each sub-band. The signal processing circuit 250 performs restoration calculations according to the input wavelength resolution or the band division number. With such a configuration, a spectral image can be generated with different resolutions for each sub-band.

[0157] Fig.19 FIG. 1 is a diagram showing an example of a UI for displaying an image generated based on mask information synthesized about a band included in the target band but not included in any sub-band (hereinafter referred to as a “non-specified band”). Fig.19 In the example, one image generated for the non-specified band is displayed, but more than two images for the non-specified band may be displayed. Alternatively, an RGB image may be displayed instead of or in addition to the image for the non-specified band. In this case, the object band includes a visible band, and the signal processing circuit 250 synthesizes the mask information for each of the red (R), green (G), and blue (B) bands. The signal processing circuit 250 uses the synthesized mask information to generate image data for each of the red, green, and blue bands based on the compressed image data. The image processing circuit 320 displays the generated RGB image on the display 330.

[0158] Fig. 20 2 is a diagram showing an example of a method for restoring only a specific sub-band with high wavelength resolution by performing two-stage restoration. In this example, a compressed image of a target band including 20 unit bands is obtained. Fig.16According to the method described above, the signal processing circuit 250 performs restoration on four large sub-bands (hereinafter referred to as "large sub-bands"), such as the 1st to 5th bands, the 6th to 10th bands, the 11th to 15th bands, and the 16th to 20th bands. Thereafter, the signal processing circuit 250 performs restoration in a manner that further divides the designated specific large sub-band into a plurality of small bands (hereinafter referred to as "small sub-bands"). The number of large sub-bands divided into a plurality of small sub-bands can be arbitrarily determined. Fig. 20 In the example, only one large sub-band is divided into a plurality of small sub-bands, but two or more large sub-bands may be divided into a plurality of small sub-bands. Fig. 20 In the example of , the signal processing circuit 250 performs two-stage band division, but the spectral image may be generated through three or more stages of division. In addition, the small sub-band may be a unit band.

[0159] Furthermore, each time each stage of band division is performed, it is possible to select which band among the divided bands is to be further divided into fine sub-bands. This selection may be performed by the user or automatically.

[0160] exist Fig. 20 In the example of , the setting data includes information for specifying a plurality of large subbands each being a part of the target band and a plurality of small subbands included in at least one of the plurality of large subbands. The signal processing circuit 250 performs the following processing.

[0161] For each of the plurality of large subbands, first synthesized mask information is generated by synthesizing mask information about a plurality of unit bands included in the large subband.

[0162] Based on the compressed image data and the first synthesis mask information, first synthesized image data is generated for each large sub-band.

[0163] For each of the plurality of small subbands in the designated large subband, second synthesized mask information is generated by synthesizing mask information about a plurality of unit bands included in the small subband.

[0164] Based on the first synthetic image data on the designated large sub-band and the second synthetic mask information, the second synthetic image data is generated for each small sub-band.

[0165] In this case, the generated hyperspectral data cube includes the second synthetic image data for a plurality of small sub-bands. Through such processing, detailed spectral information can be obtained only for a specific large sub-band designated by the user.

[0166] In addition, the configuration of the imaging device, the compression algorithm of the hyperspectral information, and the reconstruction algorithm of the hyperspectral data cube are not limited to the above-mentioned embodiments. For example, the configuration of the filter array 110, the optical system 140, and the image sensor 160 is not limited to Figures 1A to 1D The configuration shown in FIG. 1 may also be modified as appropriate. Figures 2A to 4B The characteristics shown in the examples can be used to use the filter array 110 with the best characteristics according to the application or purpose. Furthermore, a method other than the operation using compressed sensing shown in the above formula (2) can also be used to generate a spectral image for each wavelength band. For example, other statistical methods such as maximum likelihood estimation or Bayesian estimation can also be used.

[0167] In the above-mentioned embodiment, the compressed image data is generated by the camera device 100 having the filter array 110, but the compressed image data may be generated by other methods. For example, the encoding matrix equivalent to the matrix H in the above-mentioned formula (1) may be applied to the hyperspectral data cube generated by any hyperspectral camera to generate compressed image data. In the case where the amount of data needs to be reduced in order to save or transmit the data, the compressed image data may be generated by such software processing. Even if the processing in each of the above-mentioned embodiments is applied to the compressed image data generated by such software processing, the image of each band can be restored.

[0168] Industrial Applicability

[0169] The technology disclosed in the present invention is useful in cameras and measuring devices that acquire multi-wavelength images, and can also be applied to sensing for organisms, medical treatment, and beauty, foreign matter and pesticide residue inspection systems for food, remote sensing systems, and vehicle-mounted sensing systems.

[0170] Description of reference numerals:

[0171] 70 Objects

[0172] 100 Camera Device

[0173] 110 Filter Array

[0174] 120 images

[0175] 140 Optical System

[0176] 150 Control circuit

[0177] 160 Image Sensor

[0178] 200 Processing device

[0179] 210 Memory

[0180] 220 Spectral images

[0181] 250 Signal Processing Circuit

[0182] 300 Display device

[0183] 310 Memory

[0184] 320 Image Processing Circuit

[0185] 330 Display

[0186] 400 Input UI

Claims

1. A signal processing method, executed by a computer, comprising: Acquire compressed image data, the compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in each of the one or more sub-bands are generated.

2. The method according to claim 1, The hyperspectral information is information of four or more bands included in the target band. The two-dimensional image information is data of a plurality of pixels included in the compressed image data. Information on the four or more wavelength bands is superimposed on the data of each of the plurality of pixels.

3. The method according to claim 1 or 2, The setting data includes information for specifying the wavelength resolution in the one or more sub-bands. The plurality of 2-dimensional images are generated at the wavelength resolution.

4. The method according to claim 1 or 2, The one or more sub-bands include a first sub-band and a second sub-band, The plurality of two-dimensional images are generated for each of the first sub-band and the second sub-band.

5. The method according to claim 3, The one or more sub-bands include a first sub-band and a second sub-band, The wavelength resolution is independently specified for each of the first sub-band and the second sub-band, The plurality of two-dimensional images are generated at the corresponding wavelength resolution for each of the first sub-band and the second sub-band.

6. The method according to claim 4, The first sub-band is separated from the second sub-band.

7. The method according to claim 1 or 2, further comprising: A display connected to the computer is caused to display a graphical user interface for a user to input the setting data.

8. The method according to claim 1 or 2, further comprising: The plurality of two-dimensional images are displayed on a display connected to the computer.

9. The method according to claim 1 or 2, The compressed image data is generated by capturing an image using a filter array and an image sensor, wherein the filter array includes a plurality of types of filters having mutually different spectral transmittances. The method further comprises: obtaining shielding data reflecting the spatial distribution of the spectral transmittance of the filter array, The plurality of 2D images are generated based on the compressed image data and the mask data.

10. The method according to claim 9, The shielding data includes shielding information, The mask information represents a spatial distribution of transmittance of the filter array in each of a plurality of unit bands included in the target wavelength band. The method further comprises: generating synthesized mask information by synthesizing a part of the mask information corresponding to a plurality of unit bands included in a non-specified band other than the one or more sub-bands in the target band; as well as A synthetic image corresponding to the non-specified wavelength band is generated based on the compressed image data and the synthetic mask information.

11. The method according to claim 10, The shielding data includes a plurality of background images and a plurality of shielding images. Each of the plurality of background images is obtained by photographing a corresponding background among the plurality of backgrounds by the image sensor without passing through the filter array. Each of the plurality of masking images is obtained by photographing the corresponding background among the plurality of backgrounds by the image sensor through the filter array, The composite mask information is generated based on the plurality of mask images and the plurality of background images.

12. The method of claim 10, further comprising: The composite image is displayed on a display connected to the computer.

13. The method of claim 9, The shielding data includes shielding information, The mask information represents a spatial distribution of transmittance of the filter array in each of a plurality of unit bands included in the target wavelength band. The setting data includes information for specifying a plurality of large sub-bands each of which is a part of the target band and a plurality of small sub-bands included in at least one of the plurality of large sub-bands. The method further comprises: For each of the plurality of large sub-bands, synthesizing a portion of the mask information corresponding to a plurality of unit bands included in each of the plurality of large sub-bands, thereby generating first synthesized mask information; as well as generating a first synthetic image for each of the plurality of large sub-bands based on the compressed image data and the first synthetic mask information; The plurality of two-dimensional images are generated based on the first synthetic image and corresponding to the plurality of small sub-bands.

14. The method according to claim 10, The object band includes a visible band, The method further comprises: generating an image corresponding to a red band, an image corresponding to a green band, and an image corresponding to a blue band based on the compressed image data and the synthetic mask information; and An RGB image based on the image corresponding to the red wavelength band, the image corresponding to the green wavelength band, and the image corresponding to the blue wavelength band is displayed on a display connected to the computer.

15. A method for generating mask data for restoring spectral image data for each wavelength band from compressed image data acquired by an imaging device including a filter array, the filter array including a plurality of types of filters having mutually different spectral transmittances, the method comprising: Acquiring first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the first mask data and the setting data, second mask data for restoring the second spectral image data corresponding to the second band group in each of the one or more sub-bands is generated.

16. The method of claim 15, The first mask data and the second mask data are data reflecting the spatial distribution of the spectral transmittance of the filter array. The first mask data includes first mask information indicating a spatial distribution of the spectral transmittance corresponding to the first wavelength band group. The second mask data includes second mask information indicating a spatial distribution of the spectral transmittance corresponding to the second band group.

17. The method of claim 16, The second masking data further includes third masking information obtained by synthesizing a plurality of information. The plurality of information items each represent a spatial distribution of the spectral transmittance in a corresponding wavelength band included in a non-specified wavelength band other than the one or more sub-bands in the target wavelength band.

18. A signal processing device comprising: Processor; and a memory storing a computer program executed by the processor, The computer program causes the processor to execute: Acquire compressed image data, the compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in each of the one or more sub-bands are generated.

19. A signal processing device comprising: Processor; and a memory storing a computer program executed by the processor; The computer program causes the processor to execute: Acquiring first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the first mask data and the setting data, second mask data for restoring the second spectral image data corresponding to the second band group in each of the one or more sub-bands is generated.

20. A camera system comprising: The signal processing device according to claim 18 or 19; and The camera device generates the compressed image data.

21. A computer program product comprising a computer program / instructions, characterized in that The computer program / instructions, when executed by a processor, achieve: Acquire compressed image data, the compressed image data including two-dimensional image information obtained by compressing hyperspectral information within a target band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the compressed image data, a plurality of two-dimensional images corresponding to a plurality of bands included in each of the one or more sub-bands are generated.

22. A computer program product comprising a computer program / instructions, characterized in that The computer program / instructions, when executed by a processor, achieve: Acquiring first mask data for restoring first spectral image data corresponding to a first wavelength band group in the target wavelength band; acquiring setting data for specifying one or more sub-bands as a part of the target band; and Based on the first mask data and the setting data, second mask data for restoring the second spectral image data corresponding to the second band group in each of the one or more sub-bands is generated.

Images