Image processing device, imaging system, and method for estimating errors in reconstructed images

Abstract

An image processing device according to the present invention is provided with: a storage device that stores encoding information representing light transmission characteristics of an encoding mask including a plurality of optical filters arrayed two-dimensionally and having different light transmission characteristics; and a signal processing circuit that generates a restored image on the basis of a compressed image and the encoding information, the compressed image being generated by means of image capturing using the encoding mask, that estimates an error in the restored image, and that outputs a signal representing the error.

Description

[Technical Field]

[0001] This disclosure relates to an image processing device, an imaging system, and a method for estimating errors in a reconstructed image. [Background technology]

[0002] Compressed sensing is a technique that reconstructs more data than observed by assuming that the data distribution of the observed object is sparse in a certain space (e.g., frequency space). Compressed sensing can be applied, for example, to imaging devices that reconstruct images containing more information from a small amount of observed data. When compressed sensing is applied to an imaging device, an optical filter that has the function of encoding the image of light spatially or wavelengthly is used. The imaging device captures the subject through the optical filter and generates a reconstructed image through calculation. This makes it possible to obtain various effects such as higher resolution, wider wavelength range, shorter imaging time, or higher sensitivity of the image.

[0003] Patent Document 1 discloses an example of applying compressed sensing technology to a hyperspectral camera that acquires images of multiple wavelength bands, each of which is narrowband. According to the technology disclosed in Patent Document 1, it is possible to realize a hyperspectral camera that generates high-resolution, multi-wavelength images.

[0004] Patent Document 2 discloses a super-resolution method that generates high-resolution monochrome images from limited observational information using compressed sensing technology.

[0005] Patent Document 3 discloses a method for generating an image with a higher resolution than the acquired image by applying a convolutional neural network (CNN) to the acquired image. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent No. 9599511 [Patent Document 2] Patent No. 6672070 [Patent Document 3] U.S. Patent Application Publication No. 2019 / 0340497 Specification [Overview of the project] [Problems that the invention aims to solve]

[0007] This disclosure provides images generated by a reconstruction process assuming sparsity, and techniques for improving the reliability of analysis results based on such images. [Means for solving the problem]

[0008] An image processing apparatus according to one aspect of the present disclosure includes a storage device that stores encoded information indicating the light transmission characteristics of an encoding mask including a plurality of optical filters with different light transmission characteristics arranged in two dimensions, and a signal processing circuit that generates a compressed image generated by imaging using the encoding mask and a reconstructed image based on the encoded information, estimates the error of the reconstructed image, and outputs a signal indicating the error.

[0009] The comprehensive or specific embodiments of this disclosure may be implemented as systems, apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable discs, or as any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. Computer-readable recording media may include, for example, non-volatile recording media such as CD-ROMs (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. If an apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. In this specification and in the claims, “apparatus” may mean not only a single device but also a system consisting of multiple devices. [Effects of the Invention]

[0010] According to one aspect of the present disclosure, it is possible to improve the reliability of an image generated by a restoration process assuming sparsity and an analysis result based on the image.

Brief Description of the Drawings

[0011] [Figure 1A] FIG. 1A is a diagram schematically showing a configuration example of an imaging system. [Figure 1B] FIG. 1B is a diagram schematically showing another configuration example of the imaging system. [Figure 1C] FIG. 1C is a diagram schematically showing yet another configuration example of the imaging system. [Figure 1D] FIG. 1D is a diagram schematically showing yet another configuration example of the imaging system. [Figure 2A] FIG. 2A is a diagram schematically showing an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of the spatial distribution of the transmittance of light for each of a plurality of wavelength bands W1, W2, ···, WN included in the target wavelength range. [[ID=2(7]] [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of the region A1 included in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of the region A1 included in the filter array shown in FIG. 2A. [Figure 3A] FIG. 3A is a diagram for explaining an example of the relationship between the target wavelength range W and a plurality of wavelength bands W1, W2, ···, WN included therein. [Figure 3B] FIG. 3B is a diagram for explaining another example of the relationship between the target wavelength range W and a plurality of wavelength bands W1, W2, ···, WN included therein. [Figure 4A] FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. x [Figure 4B] FIG. 4B is a diagram showing the result of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W1, W2, ···, WN. [Figure 5] FIG. 5 is a diagram schematically showing a configuration example of an inspection system. [Figure 6] FIG. 6 is a diagram showing a configuration example of an image processing apparatus that generates a restored image for each wavelength band and estimates an error thereof. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of a signal processing circuit. [Figure 8] FIG. 8 is a diagram showing an example of the relationship between the value of an evaluation function and a restoration error for three different types of subjects. [Figure 9] FIG. 9 is a block diagram showing another configuration example of an image processing apparatus that estimates a restoration error. [Figure 10] FIG. 10 is a diagram showing an example of a scene to be photographed. [Figure 11] FIG. 11 is a diagram showing another example of a scene to be photographed. [Figure 12] FIG. 12 is a diagram showing yet another example of a scene to be photographed. [Figure 13] FIG. 13 is a diagram showing yet another example of a scene to be photographed. [Figure 14] FIG. 14 is a diagram showing an example of a user interface (UI) having a function of displaying a warning when a restoration error exceeds a predetermined threshold value. [Figure 15] FIG. 15 is a block diagram showing a configuration example of a system that displays a warning. [Figure 16] FIG. 16 is a flowchart showing the operation of the signal processing circuit in the example of FIG. 14. [Figure 17] FIG. 17 is a diagram showing an example of a UI that displays the estimated error as a numerical value. [Figure 18] FIG. 18 is a diagram showing an example of a UI that displays the estimated error in a form superimposed on a spectrum. [Figure 19] FIG. 19 is a diagram showing an example of a UI having a function of predicting a time when a restoration error exceeds a threshold value based on the change over time of the restoration error. [Figure 20]Figure 20 shows another example of a UI that has a function to predict when the reconstruction error will exceed a threshold based on the change in the reconstruction error over time. [Figure 21] Figure 21 is a block diagram showing the configuration of an image processing device 200 that predicts when the reconstruction error will exceed a threshold based on the change in the reconstruction error over time. [Figure 22] Figure 22 shows an example of log information for the restoration error. [Modes for carrying out the invention]

[0012] The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement, position and connection configurations of components, steps, step order, and display screen layouts shown in the following embodiments are examples and are not intended to limit the technology of this disclosure. Components in the following embodiments that are not described in the independent claim indicating the highest-level concept are described as optional components. Each figure is a schematic diagram and is not necessarily a strict illustration. Furthermore, in each figure, substantially identical or similar components are denoted by the same reference numeral. Duplication of explanation may be omitted or simplified.

[0013] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large-scale integration (LSI). The LSI or IC may be integrated on a single chip or may be composed of multiple chips combined. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSIs or ICs, but the name may change depending on the degree of integration, and they may also be called system LSIs, VLSIs (very large-scale integrations), or ULSIs (ultra-large-scale integrations). Field-programmable gate arrays (FPGAs) that are programmed after the manufacture of the LSI, or reconfigurable logic devices that allow for the reconfiguration of junction relationships within the LSI or the setup of circuit compartments within the LSI, can also be used for the same purpose.

[0014] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROMs, optical disks, or hard disk drives, and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0015] (Knowledge that forms the basis of this disclosure) Before describing embodiments of this disclosure, we will explain the problem that this disclosure aims to solve: errors that occur in sparsity-based image restoration.

[0016] Sparsity is the property that the elements characterizing an observed object exist sparsely (sparsely) in a given space (for example, frequency space). Sparsity is widely observed in nature. By utilizing sparsity, it becomes possible to efficiently observe the necessary information. Sensing techniques that utilize sparsity are called compressed sensing techniques. By using compressed sensing techniques, it is possible to construct highly efficient devices and systems.

[0017] Specific examples of applications of compressed sensing technology include hyperspectral cameras with improved wavelength resolution, as disclosed in Patent Document 1, and imaging devices capable of improving resolution (i.e., super-resolution), as disclosed in Patent Document 2.

[0018] An imaging device utilizing compressed sensing includes, for example, an optical filter having random light transmission characteristics with respect to space and / or wavelength. Hereinafter, such an optical filter may be referred to as an "encoding mask." The encoding mask is placed on the optical path of light incident on the image sensor and transmits light incident from the subject with different light transmission characteristics depending on the region. This process by the encoding mask is referred to as "encoding." The image of light encoded by the encoding mask is captured by the image sensor. The image generated by imaging using the encoding mask is referred to as a "compressed image." Information indicating the light transmission characteristics of the encoding mask (hereinafter referred to as "mask information") is recorded in advance in a storage device. The processing unit of the imaging device performs a restoration process based on the compressed image and the mask information. The restoration process generates a restored image that has more information than the compressed image (for example, higher resolution image information or image information with more wavelengths). The mask information may, for example, be information indicating the spatial distribution of the transmission spectrum (also referred to as "spectral transmittance") of the encoding mask. By using such mask information for reconstruction, it is possible to reconstruct images for multiple wavelength bands from a single compressed image.

[0019] The reconstruction process includes estimation calculations that assume the sparsity of the observed data. This estimation calculation is sometimes called "sparse reconstruction." Because sparsity is assumed, estimation errors occur depending on the non-sparse components of the observed data. The calculations performed in sparse reconstruction may be, for example, estimation calculations of data by minimizing an evaluation function that incorporates a regularization term such as discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV), as disclosed in Patent Document 1. Alternatively, calculations using a convolutional neural network (CNN), as disclosed in Patent Document 3, may be performed. Patent Document 3 does not explicitly specify a function based on sparsity, but it discloses that calculations are performed using the sparsity of compressed sensing.

[0020] When performing calculations that generate more data than the observed information, regardless of the calculation method used, the calculation will involve solving an underdetermined system (i.e., a system with fewer variables than equations), resulting in estimation errors. These estimation errors lead to output images that differ from the actual observed object. Consequently, the reliability of the output image is compromised, and if image analysis is performed using the output image, the reliability of the analysis results is also compromised.

[0021] Based on the above considerations, the inventors have conceived of the configuration of the embodiments of the present disclosure described below. The outline of the embodiments of the present disclosure will be described below.

[0022] An image processing apparatus according to one embodiment of the present disclosure includes a storage device that stores encoded information indicating the light transmission characteristics of an encoding mask which includes a plurality of optical filters having different light transmission characteristics arranged in two dimensions, and a signal processing circuit that generates a compressed image generated by imaging using the encoding mask and a reconstructed image based on the encoded information, estimates the error of the reconstructed image, and outputs a signal indicating the error.

[0023] The above configuration allows for the output of errors in the reconstructed image. For example, an image showing the errors in the reconstructed image can be output to a display device. This makes it easy for the user to determine whether or not an accurate reconstructed image has been generated. If a reconstructed image with a large error is generated, it becomes easier to take corrective action such as redoing the reconstruction. As a result, the reliability of the reconstructed image and the analysis results based on the reconstructed image can be improved.

[0024] In one embodiment, the signal processing circuit estimates the error based on the compressed image, the reconstructed image, and the encoding information. For example, the signal processing circuit may estimate the error based on the value of an evaluation function based on the compressed image, the reconstructed image, and the encoding information. An example of an evaluation function will be described later.

[0025] The storage device may further store a reference image indicating the reference subject. The compressed image can be generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit may estimate the error based on a comparison between the reference image and the region indicating the reference subject in the restored image.

[0026] The storage device may further store reference information indicating the spectrum of the reference subject. The compressed image can be generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit may estimate the error based on the difference between the spectrum of the region corresponding to the reference subject in the restored image and the spectrum indicated by the reference information.

[0027] The storage device may further store reference information indicating the spatial frequency of the reference subject. The compressed image can be generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit may estimate the error based on the difference between the spatial frequency of the region corresponding to the reference subject in the restored image and the spatial frequency indicated by the reference information.

[0028] The reference information may be generated based on an image obtained by capturing a scene containing the reference subject using the encoding mask. The reference information may be recorded in a storage device during manufacturing, or it may be recorded in a storage device when a user performs an operation to capture the reference subject.

[0029] The signal processing circuit may output a warning if the magnitude of the error exceeds a threshold. The warning may be a signal that causes a display device, audio output device, or light source to emit an image, sound, or light warning. By outputting a warning, the user can be notified that the restoration error is large.

[0030] The signal processing circuit may record the estimated error, predict when the error will exceed a threshold based on the change in the error over time, and output a signal indicating the predicted time. This configuration is effective when repeatedly generating reconstructed images for similar objects, such as in product inspection. By recording the error each time the reconstruction process is performed, information on the change in the error over time can be obtained. Based on the change in the error over time, for example, the deterioration of the imaging device can be estimated, and the time when the error will exceed a threshold can be predicted as described above.

[0031] The plurality of optical filters may have different spectral transmittances. The reconstructed image may include image information for each of the plurality of wavelength bands. With such a configuration, image information for each of the plurality (e.g., four or more) wavelength bands can be reconstructed from the compressed image.

[0032] The signal processing circuit may cause the display device to display the error and the spatial variation of pixel values ​​in the reconstructed image in a manner that allows for distinction between them. This allows the user to easily understand the error and the spatial variation of pixel values ​​in the reconstructed image.

[0033] An imaging system according to another embodiment of the present disclosure comprises an image processing device according to an embodiment of the present disclosure, the encoding mask, and an image sensor. Such an imaging system can generate a compressed image by imaging using the encoding mask, generate a reconstructed image based on the compressed image and encoding information, and estimate the error of the reconstructed image.

[0034] A method according to yet another embodiment of the present disclosure is a method performed by a processor. The method includes: obtaining encoded information indicating the light transmission properties of an encoding mask comprising a plurality of optical filters having different light transmission properties arranged in two dimensions; obtaining a compressed image generated by imaging using the encoding mask; generating a reconstructed image based on the encoded information; estimating an error in the reconstructed image; and outputting a signal indicating the error.

[0035] A computer program according to yet another embodiment of the present disclosure is stored in a computer-readable non-temporary storage medium and executed by a processor. The computer program causes the processor to: acquire coding information indicating the light transmission properties of an encoding mask including a plurality of optical filters having different light transmission properties arranged in two dimensions; acquire a compressed image generated by imaging using the encoding mask; generate a reconstructed image based on the coding information; estimate the error of the reconstructed image; and output a signal indicating the error.

[0036] The following describes exemplary embodiments of this disclosure in more detail.

[0037] (Embodiment) First, an example configuration of an imaging system used in an exemplary embodiment of this disclosure will be described.

[0038] <1. Imaging System> Figure 1A is a schematic diagram showing an example of the configuration of an imaging system. This system comprises an imaging device 100 and an image processing device 200. The imaging device 100 has a configuration similar to the imaging device disclosed in Patent Document 1. The imaging device 100 comprises an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are arranged on the optical path of light incident from the object 70, which is the subject. In the example in Figure 1A, the filter array 110 is arranged between the optical system 140 and the image sensor 160.

[0039] Figure 1A shows an apple as an example of the object 70. The object 70 is not limited to an apple; it can be any object. The image sensor 160 generates compressed image 10 data, in which information from multiple wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 200 generates image data for each of the multiple wavelength bands included in a predetermined target wavelength range, based on the compressed image 10 data generated by the image sensor 160. This generated image data of multiple wavelength bands is sometimes referred to as a "hyperspectral (HS) data cube" or "hyperspectral image data." Here, the number of wavelength bands included in the target wavelength range is N (where N is an integer of 4 or more). In the following description, the generated image data of multiple wavelength bands will be referred to as restored images 20W1, 20W2, ..., 20W N These may be collectively referred to as "hyperspectral image 20" or "hyperspectral data cube 20". In this specification, data or signals representing an image, that is, a set of data or signals representing the pixel value of each pixel, may be simply referred to as "image".

[0040] In this embodiment, the filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters whose spectral transmittance, i.e., wavelength dependence of light transmittance, differs from one another. The filter array 110 modulates the intensity of incident light for each wavelength and outputs it. This process by the filter array 110 is called "coding," and the filter array 110 is sometimes referred to as an "coding element" or "coding mask."

[0041] In the example shown in Figure 1A, the filter array 110 is positioned near or directly above the image sensor 160. Here, "nearby" means that the image of light from the optical system 140 is formed on the surface of the filter array 110 in a reasonably clear state. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0042] The optical system 140 includes at least one lens. In Figure 1A, the optical system 140 is shown as a single lens, but the optical system 140 may be a combination of multiple lenses. The optical system 140 forms an image on the imaging plane of the image sensor 160 via the filter array 110.

[0043] The filter array 110 may be positioned away from the image sensor 160. Figures 1B to 1D show examples of the configuration of an imaging device 100 in which the filter array 110 is positioned away from the image sensor 160. In the example in Figure 1B, the filter array 110 is positioned between the optical system 140 and the image sensor 160, but away from the image sensor 160. In the example in Figure 1C, the filter array 110 is positioned between the object 70 and the optical system 140. In the example in Figure 1D, the imaging device 100 comprises two optical systems 140A and 140B, with the filter array 110 positioned between them. As in these examples, an optical system including one or more lenses may be positioned between the filter array 110 and the image sensor 160.

[0044] The image sensor 160 is a monochrome type photodetector having a plurality of photodetectors (also referred to as "pixels" in this specification) arranged in two dimensions. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The photodetectors include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may be used. By using a color type sensor, the amount of information regarding wavelength can be increased, and the accuracy of reconstructing the hyperspectral image 20 can be improved. The wavelength range to be acquired can be arbitrarily determined and is not limited to the visible wavelength range, but may also be the ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength range.

[0045] The image processing device 200 may be a computer comprising one or more processors and one or more storage media such as memory. Based on the compressed image 10 acquired by the image sensor 160, the image processing device 200 generates a plurality of reconstructed images 20W1, 20W2, ... 20W N Generate data.

[0046] Figure 2A is a schematic diagram showing an example of a filter array 110. The filter array 110 has multiple regions arranged in two dimensions. In this specification, these regions may be referred to as "cells". Each region is equipped with an optical filter having an individually set spectral transmittance. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take values ​​between 0 and 1, inclusive.

[0047] In the example shown in Figure 2A, the filter array 110 has 48 rectangular regions arranged in a 6x8 grid. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be, for example, roughly equivalent to the number of pixels in the image sensor 160. The number of filters included in the filter array 110 is determined according to the application, for example, ranging from tens of thousands to tens of millions.

[0048] Figure 2B shows multiple wavelength bands W1, W2, ..., W included in the target wavelength range. N This figure shows an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in Figure 2B, the difference in intensity in each region represents the difference in transmittance. Lighter regions have higher transmittance, and darker regions have lower transmittance. As shown in Figure 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0049] Figures 2C and 2D show examples of spectral transmittances for regions A1 and A2, respectively, included in the filter array 110 shown in Figure 2A. The spectral transmittances of region A1 and region A2 are different from each other. Thus, the spectral transmittance of the filter array 110 differs by region. However, it is not necessary for the spectral transmittances of all regions to be different. In the filter array 110, the spectral transmittances of at least some of the regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns for multiple regions included in the filter array 110 may be equal to or greater than the number of wavelength bands N included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0050] Figures 3A and 3B show the target wavelength range W and the multiple wavelength bands W1, W2, ..., W contained within it. NThis diagram illustrates the relationship. The target wavelength range W can be set to various ranges depending on the application. For example, the target wavelength range W may be the visible light wavelength range from about 400 nm to about 700 nm, the near-infrared wavelength range from about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range from about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. Thus, the wavelength range used is not limited to the visible light range. In this specification, "light" refers to all radiation, including infrared and ultraviolet rays, not just visible light.

[0051] In the example shown in Figure 3A, N is any integer greater than or equal to 4, and the target wavelength range W is divided into N equal parts, with each wavelength range being designated as wavelength bands W1, W2, ..., W N This is the case. However, it is not limited to this example. Multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be made non-uniform depending on the wavelength band. There may be gaps or overlaps between adjacent wavelength bands. In the example shown in Figure 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. Thus, the method of determining multiple wavelength bands is arbitrary.

[0052] Figure 4A is a diagram illustrating the spectral transmittance characteristics in a region of the filter array 110. In the example shown in Figure 4A, the spectral transmittance has multiple maximum values ​​P1 to P5 and multiple minimum values ​​with respect to wavelengths within the target wavelength range W. In the example shown in Figure 4A, the optical transmittance within the target wavelength range W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in Figure 4A, wavelength band W2 and wavelength band W N-1 The spectral transmittance has a maximum value in wavelength ranges such as W1 to W. N It can be designed to have maximum values ​​in at least two or more wavelength ranges. In the example in Figure 4A, the maximum values ​​P1, P3, P4, and P5 are greater than or equal to 0.5.

[0053] Thus, the light transmittance of each region varies with wavelength. Therefore, the filter array 110 transmits a larger proportion of the components in a certain wavelength range of the incident light and less of the components in other wavelength ranges. For example, for the light in k of the N wavelength bands, the transmittance can be greater than 0.5, and for the light in the remaining N - k wavelength ranges, the transmittance can be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that evenly contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of intensity peaks discrete with respect to wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0054] FIG. 4B shows, as an example, the spectral transmittance shown in FIG. 4A averaged for each of the wavelength bands W1, W2, ···, W N This is a diagram showing the result. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is taken as the transmittance in that wavelength band. In this example, in three wavelength ranges taking the maximum values P1, P3, and P5, the transmittance is prominently high. In particular, in two wavelength ranges taking the maximum values P3 and P5, the transmittance exceeds 0.8.

[0055] In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessarily a grayscale transmittance distribution. For example, a binary scale transmittance distribution in which the transmittance of each region can take either a value close to 0 or a value close to 1 may be adopted. In the binary scale transmittance distribution, each region transmits most of the light in at least two of the plurality of wavelength ranges included in the target wavelength range and does not transmit most of the light in the remaining wavelength ranges. Here, "most" generally refers to 80% or more.

[0056] A part of all the cells, for example, half of the cells, may be replaced with a transparent region. Such a transparent region transmits light from all the wavelength bands W1 to W included in the target wavelength range W with a similarly high transmittance, for example, a transmittance of 80% or more. In such a configuration, the plurality of transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the plurality of regions in the filter array 110, regions with light transmittances that vary depending on the wavelength and the transparent regions may be arranged alternately. N Data indicating the spatial distribution of the spectral transmittance of such a filter array 110 is acquired in advance based on design data or actual measurement calibration and stored in a storage medium provided in the image processing apparatus 200. This data is used for the arithmetic processing described later.

[0057]

[0058] The filter array 110 can be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, or a fine structure containing metal. When using a multilayer film, for example, a multilayer film including a dielectric multilayer film or a metal layer can be used. In this case, at least one of the thickness, material, and lamination order of each multilayer film is formed to be different for each cell. Thereby, different spectral characteristics can be realized for each cell. By using a multilayer film, sharp rises and falls in the spectral transmittance can be realized. Not limited to realizing sharp rises and falls in the spectral transmittance, a multilayer film may be used to realize various spectral transmittances. A configuration using an organic material can be realized by making the pigments or dyes contained in each cell different or by laminating different materials. A configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. When using a fine structure containing metal, it can be manufactured using spectroscopy based on the plasmon effect.

[0059] Next, an example of signal processing by the image processing device 200 will be described. The image processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, "multi-wavelength" means more wavelength ranges than, for example, the three RGB wavelength ranges acquired by a normal color camera. The number of these wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is called the "number of bands". Depending on the application, the number of bands may exceed 100.

[0060] The data we want is the hyperspectral image data 20, and we will call this data f. If the number of bands is N, then f is the image data for each band f1, f2, ..., f N This is integrated data. Here, as shown in Figure 3A, the horizontal direction of the image is the x-direction, and the vertical direction of the image is the y-direction. If the number of pixels in the x-direction of the image data to be obtained is m, and the number of pixels in the y-direction is n, then the image data f1, f2, ..., f N Each of these is two-dimensional data with n × m pixels. Therefore, data f is three-dimensional data with n × m × N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube". On the other hand, the number of elements in data g of the compressed image 10, which is obtained by encoding and multiplexing by the filter array 110, is n × m. Data g can be expressed by the following equation (1).

number

[0061] Here, f1, f2, ..., f N Each of these is data with n × m elements. Therefore, the vector on the right side is strictly an n × m × N x 1 1 dimensional vector. Vector g is transformed into an n × m x 1 1 dimensional vector and represented and computed. Matrix H is the vector f, with each component f1, f2, ..., f NThis represents a transformation in which the signal is encoded and intensity-modulated with different encoding information (hereinafter also referred to as "mask information") for each wavelength band, and then added together. Therefore, H is an n×m row n×m×N column matrix.

[0062] Given a vector g and a matrix H, it appears that f can be calculated by solving the inverse problem of equation (1). However, since the number of elements n × m × N of the data f to be obtained is greater than the number of elements n × m of the acquired data g, this problem is poorly set up and cannot be solved as is. Therefore, the image processing device 200 uses the redundancy of the image contained in the data f and employs a compressed sensing technique to find the solution. Specifically, the data f to be obtained is estimated by solving the following equation (2).

[0063]

number

[0064] Here, f' represents the estimated data for f. The first term in parentheses in the above equation represents the difference between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares may also be used as the residual term. The second term in parentheses is the regularization term or stabilization term. Equation (2) means finding the f that minimizes the sum of the first and second terms. The function in parentheses in equation (2) is called the evaluation function. The image processing device 200 can converge the solution through recursive iterative operations and calculate the f that minimizes the evaluation function as the final solution f'.

[0065] The first term in parentheses in equation (2) represents the operation of calculating the sum of squares of the differences between the acquired data g and Hf, which is obtained by transforming the estimation process f by matrix H. The second term, Φ(f), is a constraint in the regularization of f and is a function that reflects the sparse information of the estimated data. This function has the effect of making the estimated data smoother or more stable. The regularization term can be represented by, for example, the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation allows for obtaining stable estimated data that suppresses the effects of noise in the observed data g. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term may be selected that makes the texture of the object 70 more sparse in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the reduction of redundant data and the higher the compression rate. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to a moderate value that allows f to converge to a certain extent without overcompression.

[0066] In the configurations shown in Figures 1B and 1C, the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, by storing this blur information in advance and reflecting it in the matrix H mentioned above, the hyperspectral image 20 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree to which a point image spreads to surrounding pixels. For example, if a point image corresponding to one pixel on the image spreads to a k×k pixel region around that pixel due to blurring, the PSF can be defined as a set of coefficients, i.e., a matrix, that shows the influence on the pixel value of each pixel in that region. By reflecting the effect of blurring of the encoding pattern due to the PSF in the matrix H, the hyperspectral image 20 can be reconstructed. The position where the filter array 110 is placed is arbitrary, but a position can be selected where the encoding pattern of the filter array 110 does not spread too much and disappear.

[0067] Through the above process, the hyperspectral image 20 can be reconstructed from the compressed image 10 acquired by the image sensor 160.

[0068] <2. Inspection System> Next, we will explain an example of an inspection system that utilizes the imaging system described above.

[0069] Figure 5 is a schematic diagram showing an example configuration of the inspection system 1000. This inspection system 1000 comprises an imaging device 100, an image processing device 200, a display device 300, and a conveyor 400. In the example shown in Figure 5, the display device 300 is a display. Instead of the display, or in addition to the display, devices such as a speaker or a lamp may be provided. The conveyor 400 is a belt conveyor. In addition to the belt conveyor, a picking device for removing abnormal objects 70 may be provided.

[0070] The object 70 to be inspected is placed on the belt of the conveyor 400 and transported. The object 70 is any item, such as an industrial product or food. The inspection system 1000 acquires a hyperspectral image of the object 70 and determines the normality of the object 70 based on the image information. For example, the inspection system 1000 detects the presence or absence of foreign matter mixed in the object 70. The detected foreign matter may be any object, such as a specific metal, plastic, insect, dirt, or hair. The foreign matter is not limited to these objects, but may also be a part of the object 70 that has deteriorated in quality. For example, if the object 70 is food, a spoiled part of the food may be detected as foreign matter. If the inspection system 1000 detects foreign matter, it may output information indicating that foreign matter has been detected to an output device such as a display device 300 or a speaker, or it may remove the object 70 containing the foreign matter using a picking device.

[0071] The imaging device 100 is a camera capable of hyperspectral imaging as described above. The imaging device 100 generates the compressed image described above by photographing the object 70 as it moves continuously along the conveyor belt 400. The image processing device 200 may be any computer, such as a personal computer, a server computer, or a laptop computer. The image processing device 200 can generate restored images for each of the multiple wavelength bands by performing a restoration process based on the compressed image generated by the imaging device 100 according to the equation (2) described above. Based on these restored images, the image processing device 200 can determine the normality of the object 70 (for example, whether there is a foreign object or an abnormality) and output the determination result to the display device 300.

[0072] Note that the configuration of inspection system 1000 shown in Figure 5 is merely an example. The imaging system can be used for any purpose other than inspection to acquire spectral information of an object. The imaging system may be implemented in a mobile device such as a smartphone or tablet computer. The same functionality as the imaging system described above may also be achieved by combining a camera and processor built into a smartphone or tablet computer with an attachment equipped with a filter array that functions as an encoding mask.

[0073] <3. Estimation of Restoration Error> Next, we will describe an example of a configuration and operation for estimating the error of a reconstructed image. Note that the error estimation method described below is merely one example, and various modifications are possible.

[0074] <3-1. Error Estimation Based on Evaluation Function> Figure 6 shows an example configuration of an image processing device 200 that generates reconstructed images for each wavelength band and estimates their errors. The image processing device 200 is used in connection with an imaging device 100 and a display device 300. The image processing device 200 comprises a signal processing circuit 210 and a storage device 250. The signal processing circuit 210 includes an image reconstruction module 212 and an error estimation module 214. The image reconstruction module 212 generates a reconstructed image by performing calculations (for example, the calculation shown in equation (2) above) based on the compressed image generated by the imaging device 100. The error estimation module 214 estimates the error of the reconstructed image and outputs a signal indicating the estimation result. The signal indicating the estimation result is sent to the display device 300. The display device 300 displays information indicating the estimated error.

[0075] The signal processing circuit 210 includes one or more processors. The signal processing circuit 210 may separately include a processor that functions as an image restoration module 212 and a processor that functions as an error estimation module 214. Alternatively, the signal processing circuit 210 may have a single processor that has the functions of both the image restoration module 212 and the error estimation module 214. The signal processing circuit 210 may implement the functions of the image restoration module 212 and the error estimation module 214 by executing a computer program stored in the storage device 250.

[0076] The storage device 250 includes one or more storage media. Each storage media can be any storage media, such as semiconductor memory, magnetic storage media, or optical storage media. The storage device 250 pre-stores a reconstruction table that shows the light transmission characteristics of the filter array 110 provided by the imaging device 100. The reconstruction table is an example of encoded information that shows the light transmission characteristics of the filter array 110, which functions as an encoding mask. The reconstruction table may be, for example, data in the form of a table showing matrix H in equation (2). The storage device 250 also stores data of the reconstructed image generated by the image reconstruction module 212. Although not shown in Figure 6, the storage device 250 also stores data of the compressed image generated by the imaging device 100 and a computer program executed by the signal processing circuit 210. This data may be stored distributed across multiple storage media.

[0077] Figure 7 is a flowchart illustrating an example of the operation of the signal processing circuit 210. In this example, first, the image restoration module 212 in the signal processing circuit 210 acquires the compressed image generated by the imaging device 100 (step S110). The signal processing circuit 210 may acquire the compressed image directly from the imaging device 100 or via the storage device 250. Next, the image restoration module 212 performs a restoration process based on the compressed image and the restoration table stored in the storage device 250 to generate a restored image (step S120). The restoration process may include, for example, the operation shown in equation (2) above. The image restoration module 212 can generate a vector representing the restored image, f, which minimizes the evaluation function in parentheses in equation (2), by performing a recursive iterative operation. Next, the error estimation module 214 in the signal processing circuit 210 estimates the error of the restored image (step S130) and outputs a signal indicating the estimated error (hereinafter referred to as "restoration error") (step S140). At this time, a signal indicating the restored image may also be output. The error estimation module 214 estimates the reconstruction error based on the compressed image, the reconstructed image, and the reconstruction table (i.e., the encoded information). For example, the reconstruction error can be estimated based on the value of an evaluation function calculated based on the compressed image, the reconstructed image, and the reconstruction table. The evaluation function may be, for example, the function in parentheses in equation (2) above. The signal processing circuit 210 can estimate the reconstruction error based on the value of the evaluation function when the solution of the vector f representing the reconstructed image converges, i.e., the value of the evaluation function when the evaluation function is minimized. The evaluation function is not limited to the function in parentheses in equation (2) above, and may vary depending on the system configuration. The technology of this embodiment may also be applied to a system that reconstructs a high-resolution image based on an image synthesized from images acquired under multiple different imaging conditions, as shown in Patent Document 2. In that case, the function in parentheses in equation (9) of Patent Document 2 may be used as the evaluation function.

[0078] In reality, defining a "correct" image when actually photographing a subject is practically difficult, making it impossible to accurately determine the error, which represents the degree of deviation from the correct image. However, it is possible to infer to some extent how the calculation method, algorithm, and evaluation function used relate to the reconstruction error. For example, by preparing ideal images and correct images in software, the relationship between the evaluation function and the reconstruction error can be verified based on these images.

[0079] The error between the reconstructed image estimated by calculation and the ground truth image can be expressed, for example, by the Mean Squared Error (MSE). The MSE is expressed by the following equation (3).

number

[0080] In sparse reconstruction, data can be reconstructed by solving the minimization problem of an evaluation function as shown in equation (2) above or equation (9) in Patent Document 2. The reason such a method is used is that many subjects in nature have sparsity. For subjects with sparsity, the regularization terms of discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) become small. In reconstruction processes that utilize sparsity, the degree to which the minimization problem is solved accurately is one of the factors that determine the reconstruction error, and there is a strong correlation between the evaluation function used in the minimization problem and the reconstruction error. Therefore, the error can be estimated based on the value of the evaluation function.

[0081] Figure 8 shows an example of the relationship between the evaluation function value and the reconstruction error for three different types of subjects. In this example, the reconstruction calculation shown in equation (2) above was performed for the three types of subjects shown in Figure 8, with the number of wavelength bands set to 10. The function in parentheses in equation (2) is the evaluation function Q, and the total variation (TV) was adopted as the regularization term. The graph in Figure 8 shows the relationship between the evaluation function Q for each subject and the MSE, which represents the error between the reconstructed image and the ground truth image. The relationship between the evaluation function Q and MSE differs depending on the subject, but if Q is uniquely determined for any subject, the MSE can be uniquely determined.

[0082] Therefore, by pre-recording data for a function or table as shown in the graph of Figure 8, errors such as MSE can be calculated from the evaluation function Q based on that data. If there are multiple types of subjects to be inspected, data such as a table or function defining the relationship between the evaluation function Q and the reconstruction error such as MSE can be pre-recorded in the storage device 250 for each expected subject. The signal processing circuit 210 can calculate the error of the reconstructed image from the value of the evaluation function Q based on this data. The signal processing circuit 210 may express the error as an error evaluation index value such as MSE, or as symbols such as A, B, C, indications such as "high," "medium," "low," or words such as "good," "average," "bad."

[0083] The calculation of an evaluation function having a regularization term, as shown in equation (2) above or equation (9) in Patent Document 2, is possible when a compressed image acquired by an imaging device, a reconstructed image, and encoding information indicating the light transmission characteristics of the encoding mask are available. Alternatively, a CNN as disclosed in Patent Document 3 can also be used. Although Patent Document 3 does not explicitly define an evaluation function, an evaluation function can be defined to evaluate the likelihood of the estimated reconstructed image based on the compressed image, the reconstructed image, and encoding information (e.g., the measurement vector in Patent Document 3). In the reconstruction process, the error of the reconstructed image can be determined from the value of the defined evaluation function.

[0084] In this way, the error in the reconstructed image can be estimated based on an evaluation function that includes the compressed image obtained by imaging using an encoding mask, the reconstructed image estimated by the reconstruction process, and the encoding information indicating the light transmission characteristics of the encoding mask. According to this embodiment, even if spectral data or image data of the subject is not obtained in advance, the error that occurred in the reconstruction process can be estimated.

[0085] The signal processing circuit 210 may perform the estimation of the reconstructed image and the estimation of the error for the entire area of ​​the acquired image, or for only a portion of the image. The evaluation function shown in equation (2) above is uniquely determined for the acquired image or a portion of the image. Therefore, the estimation of the reconstructed image and the estimation of the error may be performed based only on a portion of the acquired image, for example, that corresponds to the subject being inspected. The error estimated from the evaluation function may contain spatial information, but it does not contain wavelength information. Therefore, in a certain area of ​​the image (i.e., an area composed of multiple pixels), a common error is estimated and output for all wavelength bands.

[0086] <3-2. Error estimation based on comparison with known transmission characteristics> Figure 9 is a block diagram showing another configuration example of the image processing device 200 for estimating the restoration error. In this example, a scene including the target subject to be restored and a reference subject having a known reflection spectrum is captured by the imaging device 100. The storage device 250 stores reference information (labeled "comparison spectrum" in Figure 9) that shows the spectrum of the reference subject. The imaging device 100 generates a compressed image by capturing the scene including the target subject and the reference subject using an encoding mask. After performing the restoration process, the signal processing circuit 210 reads the comparison spectrum recorded in the storage device 250 and estimates the error based on the difference between the spectrum of the region corresponding to the reference subject in the restored image and the comparison spectrum.

[0087] Figure 10 shows an example of a scene captured in the example shown in Figure 9. In this example, the target subject is a pen 50, and the reference subjects are the red region 60R, the green region 60G, and the blue region 60B in the color chart. Note that the reference subjects are not limited to the red region 60R, the green region 60G, and the blue region 60B, but may be any region of any color. One region of color may be specified as the reference subject, or multiple regions of colors may be specified. Reflection spectrum data of one or more regions used as reference subjects are pre-recorded in the storage device 250 as a comparison spectrum. The reflection spectrum data may be, for example, data of the values ​​of each of the multiple wavelength bands included in a pre-set target wavelength range. The signal processing circuit 210 extracts data of the region corresponding to the reference subject from the reconstructed image and calculates an error based on the difference between the values ​​of each wavelength band shown in the extracted data and the values ​​of the corresponding wavelength bands in the comparison spectrum pre-recorded in the storage device 250. For example, the error can be calculated by accumulating the square of the difference between the values ​​of each wavelength band shown by the data in the region of the reconstructed image corresponding to the reference subject and the values ​​of the corresponding wavelength bands in the comparison spectrum previously recorded in the storage device 250, across all wavelength bands. When multiple regions are used as the reference subject, as shown in Figure 10, the sum or average of the error values ​​calculated for each region may be used as the final error value.

[0088] In the above example, comparison spectrum data is pre-recorded as reference information showing the spectrum of the reference subject, but a reference image showing the reference subject may be recorded instead. The reference image may be a color image containing RGB color information, or an image containing information for four or more wavelength bands. The signal processing circuit 210 can estimate the error based on a comparison between the reference image and the region showing the reference subject in the reconstructed image. For example, error evaluation index values ​​such as MSE between the reference image and the corresponding region in the reconstructed image may be calculated for each band in which the reference image contains information, and the sum of these values ​​may be taken as the error.

[0089] The reference object is not limited to a color chart; it can be any object with a known spectrum. For example, a color sample, a whiteboard, or an opaque item whose spectrum has been measured using a measuring instrument such as a spectrophotometer can be widely used as a reference object. The color chart, color sample, or whiteboard used as a reference object may be sold as a set with the imaging device 100, the image processing device 200, or the software executed by the image processing device 200. The spectrum of the reference object, i.e., the comparison spectrum, may be stored in the storage device 250 as a factory-calibrated value when the imaging device 100, the image processing device 200, or the software for the image processing device 200 is sold. Alternatively, the comparison spectrum may be stored in the storage device 250 by the user when it is first started up.

[0090] Alternatively, as shown in Figure 11, the same processing may be performed using the background 64 as the reference subject. In this case, the reflection spectrum of the background 64 is measured and recorded in advance. The signal processing circuit 210 uses the spectral information of the background to estimate the error of the reconstructed image during the reconstruction process.

[0091] The error may be estimated using a reference subject with a known spatial frequency instead of the spectrum. In that case, the memory device 250 stores reference information indicating the spectrum of the reference subject. The signal processing circuit 210 estimates the error based on the difference between the spatial frequency of the region corresponding to the reference subject in the reconstructed image and the spatial frequency indicated by the reference information.

[0092] Figure 12 shows an example of a scene captured when estimating an error based on the difference in spatial frequencies between images. In this example, the imaging device 100 simultaneously captures the target subject, a pen 50, and the reference subject, a resolution chart 62. The spatial frequency of the resolution chart 62 is known, and reference information indicating that spatial frequency is pre-recorded in the storage device 250. The reference information recorded in this case may include, for example, one or more spatial frequency values ​​for at least one of the horizontal and vertical directions of the reference image showing the resolution chart. The spatial frequency values ​​can be obtained, for example, by performing a transformation process such as a Fast Fourier Transform on the reference image. The signal processing circuit 210 extracts the region corresponding to the reference subject from the reconstructed image and performs a transformation process such as a Fast Fourier Transform on the extracted region. This makes it possible to obtain one or more spatial frequency values ​​for at least one of the horizontal and vertical directions of the reconstructed image. The signal processing circuit 210 can estimate the error based on the difference between the spatial frequency value in the reference image and the corresponding spatial frequency value in the region corresponding to the reference subject in the reconstructed image. The signal processing circuit 210 may also estimate the error based on a comparison between the calculated modulation transfer function (MTF) for the reference image and the calculated MTF for the region corresponding to the reference subject in the reconstructed image.

[0093] Figure 13 shows another example of a scene captured when estimating the error based on the difference in spatial frequencies between images. In this example, a background 66 with a known spatial frequency is used as the reference subject. The background 66 shown in Figure 13 has a striped pattern with a constant spatial frequency in the horizontal direction. In this case, the signal processing circuit 210 estimates the reconstruction error based on the difference between the horizontal spatial frequency of the region corresponding to the background 66 in the reconstructed image and the horizontal spatial frequency of a pre-prepared reference image. In this case as well, the signal processing circuit 210 can estimate the error based on the difference between a calculated value such as MTF or FFT for the reference image showing the known background 66 and a calculated value such as MTF or FFT for the region corresponding to the background 66 in the reconstructed image.

[0094] In each of the above examples, an object or background with a known spectrum or spatial frequency (i.e., a reference subject) may always be placed or displayed within the imaging area, or it may be placed or displayed only at regular intervals or when necessary. For example, the reference subject may be photographed simultaneously with the target subject only when performing periodic maintenance or calibration, such as once a day or once a week, to check for the occurrence of reconstruction errors. Reconstruction errors may occur due to aging or malfunctions of the imaging device 100. Therefore, by performing the above reconstruction error estimation process during maintenance or calibration, it is possible to check whether or not there are any deteriorations or malfunctions in the imaging device 100.

[0095] <4. Example of displaying the restoration error> Next, we will describe some examples of how to display the reconstruction error.

[0096] <4-1. Warning display when the error exceeds the threshold> Figure 14 shows an example of a user interface (UI) that has a function to display a warning when the estimated reconstruction error exceeds a predetermined threshold. In the example in Figure 14, the compressed image is displayed in the upper left of the display device 300 screen, the reconstructed images (also called spectral images) for each of the multiple wavelength bands are displayed at the bottom of the screen, and spectral information is displayed in the upper right of the screen.

[0097] In this example, reconstructed images are displayed for each relatively wide band with a width of 50 nm. The width of the bands in the displayed reconstructed images is not limited to this example and can be set arbitrarily. The signal processing circuit 210 may perform reconstruction processing for each relatively narrow band, such as 5 nm, and combine the reconstructed images of multiple consecutive bands to generate a single reconstructed image of a relatively wide band, which is then displayed on the display device 300.

[0098] The spectral information in the example in Figure 14 shows the spectrum acquired for a rectangular designated region 90 within the compressed image. The designated region 90 can be specified, for example, by the user. In the example in Figure 14, the designated region 90 is a region containing multiple pixels, but a single point may be specified. The curve in the spectral information shows the average value of the pixel values ​​in the designated region 90 for each wavelength. The gray area in the spectral information shows the spatial variation of the spectrum, such as the range from the minimum to the maximum value of the pixel values ​​for each wavelength in the designated region 90, or the standard deviation. Spatial variation can occur, for example, due to lighting or the shape of the subject. In this embodiment, the reconstruction error is estimated independently of the spatial variation of the spectrum by one of the methods described above. If the estimated reconstruction error exceeds a threshold, the signal processing circuit 210 displays a warning 92, as shown in Figure 14, on the display device 300, prompting the user to remeasure or interrupt the process.

[0099] Figure 15 is a block diagram showing an example configuration of a system that displays a warning. In this example, the memory device 250 stores the maximum value of the acceptable restoration error as a threshold. The signal processing circuit 210 refers to this threshold and, if the restoration error is greater than the threshold, causes the display device 300 to display a warning. As in the example in Figure 9, if the error is estimated based on a comparison of spectra, the memory device 250 also stores the data of the comparison spectra. Furthermore, if the error is estimated based on a comparison of spatial frequencies, the memory device 250 also stores the spatial frequency data of the reference subject.

[0100] Figure 16 is a flowchart showing the operation of the signal processing circuit 210 in this example. Steps S110 to S130 are the same as the corresponding operations shown in Figure 7. In the example in Figure 16, after the reconstruction error estimation in step S130, the signal processing circuit 210 determines whether the reconstruction error exceeds a threshold (step S210). If the reconstruction error does not exceed the threshold, the signal processing circuit 210 outputs the reconstructed image (step S220). If the reconstruction error exceeds the threshold, the signal processing circuit 210 causes the display device 300 to display a warning (step S230). This operation allows the user to be prompted to remeasure or interrupt the process if the reconstruction error exceeds the threshold.

[0101] <4-2. Displaying the error numerically> Figure 17 shows an example of a UI that displays the estimated error as a numerical value. In this example, the signal processing circuit 210 displays the estimated reconstruction error, a numerical value 80, on the display device 300. In the example in Figure 17, the average value of the MSE for each wavelength band in the specified region 90 is displayed as the reconstruction error. The error may be expressed not only as MSE, but also as other error indicator values, or as symbols such as A, B, C, "high," "medium," "low," or as words such as "good," "average," or "bad." By displaying the estimated reconstruction error on the screen in this way, the user can evaluate the validity of the displayed reconstruction result.

[0102] <4-3. Displaying the error overlaid on the spectrum> Figure 18 shows an example of a UI that displays the estimated error superimposed on the spectrum. In this example, in the area where spectral information is displayed in the upper right of the screen, the spatial variation of the spectrum in the specified region 90 is displayed in gray, and the estimated reconstruction error is displayed as error bars 94. Instead of specifying a region 90 that includes multiple pixels, a single point in the compressed image may be specified. In that case, the gray display indicating the spatial variation of the spectrum will not be shown. However, even when a single point is specified, the reconstruction error is estimated and may be displayed in the form of error bars 94 or the like. The signal processing circuit 210 calculates the spatial spectral variation and the estimated error generated by the reconstruction calculation independently of each other. Regarding their relative magnitudes, one may be larger than the other, or the relative magnitudes may change for each wavelength band.

[0103] In this example, spatial spectral variability and estimation errors resulting from the reconstruction calculation are displayed separately. This allows the user to determine whether large spectral variability is caused by the illumination or measurement optics, or by errors in the reconstruction process.

[0104] The evaluation function is uniquely determined for the acquired compressed image or for a portion of the compressed image containing multiple pixels. Therefore, the error estimated from the evaluation function does not contain any information about wavelength. In the example shown in Figure 18, a common error value is calculated for all wavelength bands, so the width of the displayed error bar 94 is constant regardless of the band.

[0105] In the example in Figure 18, the estimation error is displayed using error bars 94, but the estimation error may also be displayed in other formats, such as using a semi-transparent area.

[0106] <5. Prediction of error occurrence timing based on changes in restoration error over time> Figure 19 shows an example of a UI that has a function to predict when the restoration error will exceed a threshold and be detected as an error based on the time-dependent change of the estimated restoration error. In this example, the signal processing circuit 210 records the estimated error in the storage device 250, predicts when the error will exceed the threshold based on the time-dependent change of the error, and outputs a signal indicating the predicted time. The signal processing circuit 210 may, for example, estimate the restoration error each time a restoration process is performed and record the estimated restoration error in the storage device 250 along with the date and time information. The signal processing circuit 120 may record the average value of the restoration error for a predetermined period, such as one day, along with the date information. The signal processing circuit 120 may predict the day on which the restoration error will exceed the threshold based on the trend of the time-dependent change of the restoration error, and display that day on the display device 300 as the error prediction day or the calibration recommendation day. In the example in Figure 19, a table is displayed that includes the measurement date, the MSE value calculated as the error, and the calibration recommendation day information. As in this example, by linking the estimated recovery error to the date or time of measurement, it is possible to record an error log and predict the date and time when the error will exceed a threshold. The error log and the predicted time when the error will exceed the threshold may also be displayed in graph format, as shown in Figure 20.

[0107] Figure 21 is a block diagram showing the configuration of an image processing device 200 that predicts the timing of an error based on the time-dependent change in the estimated error. In this example, the signal processing circuit 210 stores an error log in the storage device 250 each time it estimates an error. The error log may be recorded in a format that includes information such as the date and the estimated error (e.g., error index values ​​such as MSE, RMSE, and PSNR), as shown in Figure 22. Based on the trend of the time-dependent change in the error recorded daily, the signal processing circuit 210 predicts when the error will exceed a threshold and displays information indicating the predicted timing on the display device 300. The information indicating the predicted timing may also be notified to a computer such as a mobile terminal used by the user.

[0108] The following cases are also included in this disclosure. (1) As described in the embodiment, the number of elements in the data g of the compressed image 10 is n × m, and the number of elements in the data f of the restored image recovered from the compressed image 10 may be n × m × N. In other words, the restored image may contain more signals than the number of signals contained in the compressed image.

[0109] (2) The reconstruction error estimated by the signal processing circuit 210 may differ from the error calculated using the ground truth image. The ground truth image may be an image obtained by a method different from the method used to reconstruct the compressed image 10. For example, it may be an image obtained with a normal color camera, or it may be an image generated by simulation.

[0110] (3) Compressed and restored images may be generated by imaging in a manner different from imaging using an encoding mask that includes multiple optical filters.

[0111] For example, the imaging device 100 may be configured such that the image sensor 160 is modified to change the light-receiving characteristics of the image sensor for each pixel, and a compressed image may be generated by imaging using the modified image sensor 160. In other words, a compressed image may be generated by an imaging device in which the filter array 110 is essentially built into the image sensor. In this case, the encoded information will be information corresponding to the light-receiving characteristics of the image sensor.

[0112] Furthermore, the optical properties of the optical system 140 may be changed spatially and chromatically by introducing an optical element such as a metalens in at least a part of the optical system 140, thereby compressing spectral information, and a compressed image may be generated by an imaging device including such a configuration. In this case, the encoded information will be information corresponding to the optical properties of the optical element such as the metalens. Thus, by using an imaging device 100 with a configuration different from the configuration using the filter array 110, the intensity of the incident light may be modulated for each wavelength, a compressed image and a reconstructed image may be generated, and the reconstruction error of the reconstructed image may be estimated.

[0113] In other words, the present disclosure also includes an image processing apparatus comprising: a storage device for storing encoded information corresponding to the optical response characteristics of an imaging device including a plurality of light-receiving regions having different optical response characteristics; and a signal processing circuit that generates a reconstructed image containing more signals than the number of signals contained in the compressed image, estimates the error of the reconstructed image, and outputs a signal indicating the error, based on a compressed image generated by the imaging device and the encoded information.

[0114] As described above, the optical response characteristics may correspond to the light-receiving characteristics of the image sensor, or they may correspond to the optical characteristics of the optical element. [Industrial applicability]

[0115] The technology disclosed herein is useful, for example, in cameras and measuring instruments that acquire multi-wavelength or high-resolution images. The technology disclosed herein can also be applied, for example, to sensing for biomedical, cosmetic, foreign object and pesticide residue detection systems in food, remote sensing systems, and in-vehicle sensing systems. [Explanation of symbols]

[0116] 10 Compressed Images 20 Restored Images 50 Target subjects 60R Reference subject (red) 60G reference object (green) 60B Reference subject (blue) 62 Reference Subject (Resolution Chart) 64 Reference subject (background) 66 Reference subject (background) 70 Objects 100 Imaging device 110 filter array 140 Optical system 160 Image Sensors 200 Image Processing Devices 210 Signal Processing Circuit 212 Image Restoration Unit 214 Error estimation part 250 Storage device 300 display device 400 Conveyor 1000 Imaging System

Claims

1. A storage device that stores encoded information indicating the light transmission characteristics of an encoded mask, which includes multiple optical filters with different light transmission characteristics arranged in two dimensions, A signal processing circuit that generates a reconstructed image containing more signals than the number of signals contained in the compressed image, based on the compressed image generated by imaging using the encoding mask and the encoding information, and estimates the error of the reconstructed image. Equipped with, The signal processing circuit outputs at least one of a signal for issuing a warning based on the error and a signal for displaying error information related to the error. Image processing device.

2. The image processing apparatus according to claim 1, wherein the signal processing circuit estimates the error based on the compressed image, the reconstructed image, and the encoded information.

3. The image processing apparatus according to claim 1 or 2, wherein the signal processing circuit estimates the error based on the value of an evaluation function based on the compressed image, the restored image, and the encoded information.

4. The memory device further stores a reference image that represents the reference subject, The compressed image is generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit estimates the error based on a comparison between the reference image and the region in the reconstructed image that shows the reference subject. The image processing apparatus according to claim 1 or 2.

5. The memory device further stores reference information indicating the spectrum of the reference subject, The compressed image is generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit estimates the error based on the difference between the spectrum of the region corresponding to the reference subject in the reconstructed image and the spectrum indicated by the reference information. The image processing apparatus according to claim 1 or 2.

6. The memory device further stores reference information indicating the spatial frequency of the reference object, The compressed image is generated by capturing a scene including the reference subject and the target subject to be restored using the encoding mask. The signal processing circuit estimates the error based on the difference between the spatial frequency of the region corresponding to the reference subject in the reconstructed image and the spatial frequency indicated by the reference information. The image processing apparatus according to claim 1 or 2.

7. The image processing apparatus according to claim 5, wherein the reference information is generated based on an image obtained by capturing a scene including the reference subject using the encoding mask.

8. The image processing apparatus according to claim 1 or 2, wherein the signal processing circuit outputs the signal for issuing the warning.

9. The image processing apparatus according to claim 1 or 2, wherein the signal processing circuit records the estimated error, predicts the time when the error exceeds a threshold based on the change in the error over time, and the error information indicates the predicted time.

10. The plurality of optical filters have different spectral transmittances from each other. The reconstructed image includes image information for each of the multiple wavelength bands. The image processing apparatus according to claim 1 or 2.

11. The image processing apparatus according to claim 1 or 2, wherein the signal for displaying the error information is a signal for displaying the error and the spatial variation of pixel values ​​in the restored image in a manner that allows for distinction.

12. The image processing apparatus according to claim 1 or 2, wherein the estimated error is different from the error calculated using the ground truth image.

13. An image processing apparatus according to claim 1 or 2, The encoding mask and, Image sensor and An imaging system equipped with the following features.

14. A method that is executed by a processor, To obtain coded information that shows the light transmission characteristics of an encoding mask containing multiple optical filters with different light transmission characteristics arranged in two dimensions, The process involves obtaining a compressed image generated by imaging using the aforementioned encoding mask, Based on the encoding information, a reconstructed image is generated that contains more signals than the number of signals contained in the compressed image. To estimate the error of the aforementioned reconstructed image, Outputting at least one of a signal for issuing a warning based on the error, and a signal for displaying error information related to the error, A method that includes this.

15. A computer program executed by a processor, The aforementioned processor, To obtain coded information that shows the light transmission characteristics of an encoding mask containing multiple optical filters with different light transmission characteristics arranged in two dimensions, The process involves obtaining a compressed image generated by imaging using the aforementioned encoding mask, Based on the encoding information, a reconstructed image is generated that contains more signals than the number of signals contained in the compressed image. To estimate the error of the aforementioned reconstructed image, Outputting at least one of a signal for issuing a warning based on the error, and a signal for displaying error information related to the error, A computer program that executes something.

16. A storage device that stores encoded information corresponding to the optical response characteristics of an imaging device that includes multiple light-receiving regions having different optical response characteristics, A signal processing circuit generates a reconstructed image containing more signals than the number of signals contained in the compressed image, based on the compressed image generated by the imaging device and the encoded information, and estimates the error of the reconstructed image. Equipped with, The signal processing circuit outputs at least one of a signal for issuing a warning based on the error and a signal for displaying error information related to the error. Image processing device.

17. The imaging device receives light through an optical element, The image processing apparatus according to claim 16, wherein the encoded information is information corresponding to the optical characteristics of the optical element.

Images