Method used for image correction, and processing circuit for executing said method

Abstract

This method used for image correction includes: employing an image that is generated by imaging a first subject as a first image, employing an image that includes information pertaining to a plurality of wavelength bands and is generated by imaging a second subject as a second image, and acquiring the first image as a correction image for correcting the second image; and determining, on the basis of a pixel value for the first image, whether the first image satisfies a suitability condition as the correction image.

Description

Method for use in correcting an image and processing circuitry for carrying out the method - Patent Application 20070122997

[0001] The present disclosure relates to a method used to correct an image, and to a processing circuit that performs the method.

[0002] By utilizing spectral information from multiple narrow wavelength bands (hereinafter simply referred to as "bands"), e.g., several dozen bands, it becomes possible to grasp detailed physical properties of a subject that were not possible with conventional RGB images that contain information from three bands: red (R), green (G), and blue (B). Cameras that capture images in such multiple wavelength bands are called "hyperspectral cameras." These cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Document 1 discloses an image analysis device for analyzing the distribution of substances in biological tissue. The image analysis device acquires multiple sample images by illuminating the biological tissue with light in multiple wavelength bands selected from a predetermined wavelength range and capturing the images. Sample data based on the multiple sample images is compared with training data on the substances to generate distribution data of the substances in the tissue. Patent Document 1 discloses normalizing and correcting the intensity of light reflected from a sample based on the intensity of light reflected from a reference member such as a white board.

[0004] Patent Literature 2 discloses an example of a hyperspectral imaging device that uses compressed sensing technology. Compressed sensing technology is a technology that reconstructs more data than observed data by assuming that the data distribution of an observed object is sparse in a certain space (e.g., frequency space). The imaging device disclosed in Patent Literature 2 includes a coding mask, which is an array of multiple optical filters with different spectral transmittances, on an optical path connecting the object and the image sensor. The imaging device can generate images of multiple wavelength bands in one shot by performing reconstruction calculations based on compressed images acquired by imaging using the coding mask.

[0005] International Publication No. 2015 / 199067 U.S. Patent No. 9,599,511 JP 2006-153498 A

[0006] In the correction disclosed in Patent Document 1, if the object to be corrected, such as a reference member, is not appropriate, it is difficult to accurately obtain spectral information of the object to be analyzed due to inappropriate correction. Therefore, there is a need to prevent inappropriate correction and thereby obtain more accurate spectral information of the object to be analyzed.

[0007] A method used for correcting an image according to one aspect of the present disclosure includes acquiring an image generated by photographing a first subject as a first image, an image generated by photographing a second subject and including information on a plurality of wavelength bands as a second image, and acquiring the first image as a correction image for correcting the second image; and determining whether the first image satisfies suitability conditions for the correction image based on pixel values ​​of the first image.

[0008] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices.

[0009] According to one aspect of the present disclosure, spectral information of a subject to be analyzed can be obtained more accurately by preventing inappropriate correction.

[0010] FIG. 1A shows a target wavelength range W and a plurality of wavelength bands W included therein. １ , W ２、・・・、W ｉFIG. 1B is a diagram illustrating an example of a hyperspectral image. FIG. 2A is a block diagram illustrating a configuration of an imaging system according to an exemplary embodiment of the present disclosure that captures an image of a correction subject. FIG. 2B is a block diagram illustrating a configuration of an imaging system according to an exemplary embodiment of the present disclosure that captures an image of another correction subject. FIG. 2C is a block diagram illustrating a configuration of an imaging system according to an exemplary embodiment of the present disclosure that captures an image of an analysis target subject. FIG. 3 is a diagram illustrating a specific configuration of an imaging system according to an exemplary embodiment of the present disclosure. FIG. 4A is a diagram illustrating a schematic example of whiteboard correction. FIG. 4B is a diagram illustrating a schematic example of black subtraction. FIG. 5 is a flowchart illustrating a first example of processing operations performed by a processing circuit in the imaging system according to this embodiment. FIG. 6A is a diagram illustrating a first hyperspectral image and a reflectance spectrum when the correction subject is appropriate. FIG. 6B is a diagram illustrating a hyperspectral image and a reflectance spectrum when the correction subject is inappropriate. FIG. 7 shows a first hyperspectral image and an edge image obtained by edge detection on the first hyperspectral image when the correction subject is appropriate and when it is inappropriate. FIG. 8A shows a first hyperspectral image and a histogram of pixel values ​​of the first hyperspectral image when the correction subject is appropriate. FIG. 8B shows a first hyperspectral image and a histogram of pixel values ​​of the first hyperspectral image when the correction subject is inappropriate. FIG. 9 is a flowchart schematically showing a second example of processing operations performed by a processing circuit in the imaging system according to this embodiment. FIG. 10A shows a first compressed image and a histogram of pixel values ​​of the first compressed image when the correction subject is appropriate. FIG. 10B shows a first compressed image and a histogram of pixel values ​​of the first compressed image when the correction subject is inappropriate. FIG. 11 is a flowchart schematically showing a third example of processing operations performed by a processing circuit in the imaging system according to this embodiment. FIG. 12A is a diagram schematically showing an example of a UI of the display device. FIG. 12B is a diagram schematically showing another example of a UI of the display device.FIG. 12C is a diagram schematically illustrating yet another example of the UI of the display device. FIG. 13 is a flowchart schematically illustrating a summary of examples 1 to 3 of the processing operation performed by the processing circuit in the imaging system according to this embodiment. FIG. 14 is a diagram schematically illustrating an example of an image generated by capturing a scene including a correction subject and an analysis target subject using a camera. FIG. 15 is a flowchart schematically illustrating example 4 of the processing operation performed by the processing circuit in the imaging system according to this embodiment. FIG. 16A is a diagram schematically illustrating an example of the configuration of a compressed sensing hyperspectral camera. FIG. 16B is a diagram schematically illustrating another example of the configuration of a compressed sensing hyperspectral camera. FIG. 16C is a diagram schematically illustrating yet another example of the configuration of a compressed sensing hyperspectral camera. FIG. 16D is a diagram schematically illustrating yet another example of the configuration of a compressed sensing hyperspectral camera. FIG. 17A is a diagram schematically illustrating an example of a filter array. FIG. 17B illustrates a wavelength band W included in the target wavelength range. １ , W ２ , ..., W Ｎ 17C is a diagram showing an example of the spectral transmittance of a certain region included in the filter array shown in FIG. 17A . FIG. 17D is a diagram showing an example of the spectral transmittance of another region included in the filter array shown in FIG. 17A . FIG. 18A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. FIG. 18B is a diagram for explaining the spectral transmittance shown in FIG. 18A in the wavelength band W １ , W ２ , ..., W Ｎ FIG. 10 is a diagram showing the results of averaging for each pixel.

[0011] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection configurations, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0012] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices, which can reconfigure connections within the LSI or set up circuit partitions within the LSI, can also be used for the same purpose.

[0013] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section may be implemented by software processing, in which case the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices.

[0014] The system or apparatus may include one or more non-transitory recording media on which the software is recorded, a processor, and any necessary hardware devices, such as interfaces.

[0015] (Foundations on which the present disclosure is based) Before describing embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

[0016] First, with reference to FIGS. 1A and 1B , an example of a hyperspectral image generated by a hyperspectral camera will be briefly described. A hyperspectral image is image data that contains information on more wavelengths than a typical RGB image. An RGB image has values ​​for each of three bands, red (R), green (G), and blue (B), for each pixel. In contrast, a hyperspectral image has values ​​for more bands than the number of bands in an RGB image for each pixel. In this specification, a "hyperspectral image" refers to image data that includes multiple images corresponding to four or more bands included in a predetermined target wavelength range. In the following description, the value that each pixel has for each band is referred to as a "pixel value." The number of bands in a hyperspectral image is typically 10 or more, and in some cases may exceed 100. A "hyperspectral image" is also sometimes called a "hyperspectral data cube" or a "hyperspectral cube."

[0017] FIG. 1A shows a target wavelength range W and a plurality of wavelength bands W included therein. １ , W ２ , ..., W ｉ1 is a diagram for explaining the relationship between the wavelengths of visible light and near-infrared light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a mid-infrared or far-infrared wavelength range. In this manner, the wavelength range used is not limited to the visible light range. In this specification, for convenience, the term "light" is not limited to visible light, and electromagnetic waves with wavelengths outside the visible light wavelength range, such as ultraviolet and near-infrared rays, are also referred to as "light."

[0018] In the example shown in FIG. 1A, N is an arbitrary integer equal to or greater than 4, and each wavelength range obtained by dividing the target wavelength range W into N equal parts is called a wavelength band W １ , W ２ , ..., W Ｎ However, the present invention is not limited to this example. The number of wavelength bands included in the target wavelength range W may be set arbitrarily. Each wavelength band may be a wavelength range having a predetermined width, such as 5 nm, 10 nm, 20 nm, or 50 nm. The widths of the multiple wavelength bands may be the same or different. If the number of wavelength bands is four or more, more information can be obtained from a hyperspectral image than from an RGB image.

[0019] 1B is a diagram showing a schematic diagram of an example of a hyperspectral image. In the example shown in FIG. 1B, the subject is an apple. The hyperspectral image 36 is a set of wavelength bands W １ , W ２、 ..., W Ｎ The corresponding images are 36W. １ , 36W ２ , ..., 36W ＮEach of these images includes a plurality of pixels arranged two-dimensionally. FIG. 1B illustrates vertical and horizontal dashed lines indicating pixel divisions. The actual number of pixels per image can be as large as tens of thousands to tens of millions, for example. However, for ease of understanding, FIG. 1B illustrates pixel divisions as if the number of pixels is extremely small. Reflected light generated when an object is illuminated with light is detected by a plurality of photodetection elements included in the image sensor. A signal indicating the amount of light detected by each photodetection element represents the pixel value of the pixel corresponding to that photodetection element. Each pixel in the hyperspectral image 36 has a pixel value for each wavelength band. Therefore, spectral information of the object can be obtained by acquiring the hyperspectral image 36. Based on the spectral information of the object, it is possible to accurately analyze the light-related characteristics of the object.

[0020] Next, a correction method performed when a hyperspectral image of an object is acquired by photographing the object with a hyperspectral camera will be described. This method may include, for example, the following operations (1) to (3): (1) Obtaining a hyperspectral image for correction by photographing an object for correction with a hyperspectral camera. The hyperspectral image for correction includes a plurality of correction images corresponding to a plurality of wavelength bands. (2) Obtaining a hyperspectral image of the object to be analyzed by photographing the object to be analyzed with a hyperspectral camera. The hyperspectral image of the object to be analyzed includes a plurality of images of the object to be analyzed corresponding to a plurality of wavelength bands. (3) Correcting the hyperspectral image of the object to be analyzed based on the hyperspectral image for correction.

[0021] In (3), an example of correcting a hyperspectral image of an analysis target based on a hyperspectral image for correction is the correction disclosed in Patent Document 1, in which each of a plurality of images of an analysis target is normalized by a corresponding correction image from a plurality of correction images that share the same wavelength band. When the subject of correction is a white board, this type of correction is called "white board correction." White board correction is performed to reduce the effects of the spectral shape of the irradiated light, the illumination distribution during shooting, lens vignetting, uneven sensitivity of the image sensor, and the like.

[0022] In this specification, "normalizing image A by image B" means dividing the pixel value of each of a plurality of pixels in image A by the pixel value of a corresponding pixel among a plurality of pixels in image B, and multiplying the result by the maximum pixel value that the pixel can have. If the spatial distribution of pixel values ​​in image B is approximately constant, the pixel value of one representative pixel or the average of the pixel values ​​of two or more representative pixels may be used instead of the pixel value of the corresponding pixel. The maximum pixel value that the pixel can have is 255 for 8 bits and 4095 for 12 bits.

[0023] "Normalizing image A by image B" may be interpreted as calculating (pvmax × (pixel value pvA11 of pixel pA11 included in image A) / (pixel value pvB11 of pixel pB11 included in image B)), ..., (pvmax × (pixel value pvAmn of pixel pAmn included in image A) / (pixel value pvBmn of pixel pBmn included in image B)). Images A and B each contain m × n pixels, and the position of pixel pA11 in image A corresponds to pixel pB11 in image B, ..., and the position of pixel pAmn in image A corresponds to pixel pBmn in image B. The maximum value that each of the pixel values ​​pvA11, ..., pvAmn, pvB11, ..., pvBmn can take may be pvmax.

[0024] In this specification, pixel β in image B can be said to correspond to pixel α in image A when the position of pixel β is the same as the position of pixel α. Alternatively, pixel β in image B can be said to correspond to pixel α in image A when, in an image sensor that outputs an image signal, the position of the photodetector that outputs the signal for pixel β is the same as the position of the photodetector that outputs the signal for pixel α.

[0025] In addition, in (3), another example of correcting the hyperspectral image of the analysis target based on the correction hyperspectral image is to subtract from each of the multiple analysis target images a correction image that corresponds to the multiple correction images in that they share the same wavelength band. If the correction subject is a light-blocking lens cap attached to a hyperspectral camera, this type of correction is called "blackout subtraction" because the pixel values ​​of the correction image are nearly zero. Blackout subtraction is performed to reduce the effects of image sensor dark current, bright pixel points, fixed pattern noise, and fluctuations in sensor performance.

[0026] In this specification, "subtracting image B from image A" means subtracting the pixel value of a corresponding pixel among multiple pixels in image B from the pixel value of each of multiple pixels in image A. "Subtracting image B from image A" may also be interpreted as determining ((pixel value pvA11 of pixel pA11 included in image A) - (pixel value pvB11 of pixel pB11 included in image B)), ..., ((pixel value pvAmn of pixel pAmn included in image A) - (pixel value pvBmn of pixel pBmn included in image B)). Images A and B each contain m x n pixels, and the position of pixel pA11 in image A corresponds to pixel pB11 in image B, ..., and the position of pixel pAmn in image A corresponds to pixel pBmn in image B. In black subtraction, correction image B is subtracted from the image to be analyzed A. If the spatial distribution of pixel values ​​in image B is approximately constant, the pixel value of one representative pixel or the average of the pixel values ​​of two or more representative pixels may be used instead of the pixel value of the corresponding pixel.

[0027] By performing whiteboard correction and / or black subtraction, it is possible to obtain spectral information of the subject to be analyzed based on a plurality of corrected images corresponding to a plurality of wavelength bands.

[0028] However, the accuracy of whiteboard correction decreases if the whiteboard used as the correction subject is dirty or if an object other than a whiteboard is mistakenly photographed as the correction subject. In blackout correction, the accuracy decreases if the light-blocking lens cap used as the correction subject is not properly attached to the hyperspectral camera.

[0029] The present inventors have found the above-mentioned problems and have come up with a method for correcting an image according to an embodiment of the present disclosure to solve the problems. More specific embodiments of the present disclosure will be described below with reference to the drawings.

[0030] (Embodiments) [1. Imaging System] [1.1. Schematic Configuration Example of Imaging System] Below, a schematic configuration example of an imaging system according to an embodiment of the present disclosure will be described with reference to FIGS. 2A to 2C. FIGS. 2A to 2C are block diagrams schematically illustrating the configuration of an imaging system according to an exemplary embodiment of the present disclosure. FIG. 2A illustrates a white board as an example of the correction subject 10a1. The white board may be, for example, a standard white board with a uniform spatial distribution of reflectance and small wavelength dispersion of reflectance. FIG. 2B illustrates a light-blocking lens cap as an example of the correction subject 10a2, which can enable imaging in a light-blocking state. FIG. 2C illustrates an apple as an example of the analysis subject 10b. The subject 10b is not limited to an apple and may be any object. In this specification, the correction subject 10a1 or the subject 10a2 is also referred to as a "first subject," and the analysis subject 10b is also referred to as a "second subject."

[0031] The imaging system 100 shown in Figures 2A to 2C includes a light source 20, a camera 30, a display device 40, and a processing device 50. The processing device 50 includes a processing circuit 52, a memory 54, and a storage device 56. The thick arrowed lines shown in Figures 2A to 2C indicate the flow of signals.

[0032] The camera 30 may be, for example, a line scan or snapshot hyperspectral camera, as described below. In this case, a correction hyperspectral image is generated by capturing an image of the subject 10a1 or 10a2 using the camera 30, as shown in FIG. 2A or 2B . The correction hyperspectral image is generated when a user receives an instruction to capture an image of the subject 10a1 or 10a2. Similarly, a hyperspectral image to be analyzed is generated by capturing an image of the subject 10b using the camera 30, as shown in FIG. 2C . The analysis hyperspectral image is generated when a user receives an instruction to capture an image of the subject 10b.

[0033] Alternatively, the camera 30 may be a compressed sensing hyperspectral camera, which will be described later. In this case, as shown in FIG. 2A or 2B , a compressed image for correction is generated by photographing the subject 10a1 or the subject 10a2 with the camera 30, and a hyperspectral image for correction is generated based on the compressed image. The compressed image for correction is generated when a user receives an instruction to photograph the subject 10a1 or the subject 10a2. Similarly, as shown in FIG. 2C , a compressed image to be analyzed is generated by photographing the subject 10b with the camera 30, and a hyperspectral image to be analyzed is generated based on the compressed image. The compressed image to be analyzed is generated when a user receives an instruction to photograph the subject 10b.

[0034] In this specification, the hyperspectral image for correction or the compressed image for correction will also be referred to as the “first image,” and the hyperspectral image to be analyzed or the compressed image to be analyzed will also be referred to as the “second image.” Because a hyperspectral image includes multiple images corresponding to multiple wavelength bands and the compressed image compresses information from multiple wavelength bands, the first and second images can be said to include information from multiple wavelength bands.

[0035] As will be described in detail later, in the imaging system 100 according to this embodiment, a first image is acquired as a correction image for correcting a second image. When the first image is generated by photographing the subject 10a1 shown in Fig. 2A, the correction of the second image based on the first image is whiteboard correction. When the first image is generated by photographing the subject 10a2 shown in Fig. 2B, the correction of the second image based on the first image is black subtraction.

[0036] As described above, the accuracy of the white board correction decreases if the white board used as the subject 10a1 is dirty or if an object other than a white board is mistakenly used as the subject 10a1. In the blackout correction, the accuracy decreases if a light-blocking lens cap is not properly attached to the camera 30 as the subject 10a2.

[0037] Therefore, in the imaging system 100 according to this embodiment, whether the first image satisfies the suitability conditions for a correction image is determined based on the pixel values ​​of the first image. Therefore, even if the user believes that they have photographed the appropriate subject 10a1 or 10a2 in response to instructions, if the subject 10a1 or 10a2 is actually not appropriate, inappropriate correction can be prevented. As a result, appropriate correction can be performed to more accurately acquire spectral information about the subject 10b.

[0038] In this specification, "spectral information" refers to information about a spectrum that indicates the wavelength dependency of light intensity. Spectral information may be, for example, information about the spectrum itself or information from which the spectrum can be derived. The spectrum may be a reflection spectrum or a transmission spectrum.

[0039] In this specification, analysis may be, for example, determining characteristics of the subject 10b, such as sugar content and ripeness, or inspecting the subject 10b for defects and / or foreign matter. The analysis may include not only machine processing but also human evaluation.

[0040] Each component of the imaging system 100 will be described below.

[0041] <Light Source 20> The light source 20 emits illumination light for illuminating the subject 10a1 or the subject 10b. The illumination light includes light in multiple wavelength bands. The light source 20 may be, for example, an incandescent lamp, a halogen lamp, a mercury lamp, a fluorescent lamp, or an LED lamp that emits white light. The hollow arrows in Figures 2A and 2C represent the light emitted from the light source 20 and the light reflected from the subject 10a1 and the subject 10b.

[0042] It should be noted that the light source 20 is not necessarily required. If the photographing system 100 does not include the light source 20, the subject 10a1 or the subject 10b may be illuminated with sunlight or room light.

[0043] <Camera 30> The camera 30 photographs the subject 10a1 or the subject 10a2 as shown in Fig. 2A or 2B. Similarly, the camera 30 photographs the subject 10b as shown in Fig. 2C.

[0044] The target wavelength range W shown in FIG. 1A is a wavelength range in which the camera 30 can detect light. When the camera 30 includes an optical system and an image sensor, the target wavelength range W can be determined based on, for example, the transmission range of the optical system and the sensitivity range of the image sensor. When the transmission range of the optical system includes the sensitivity range of the image sensor, the target wavelength range W is determined by the sensitivity range of the image sensor. When the camera 30 further includes a band-pass filter, the target wavelength range W can be determined based on the transmission range of the band-pass filter in addition to the transmission range of the optical system and the sensitivity range of the image sensor. When the transmission range of the optical system includes the sensitivity range of the image sensor, and the sensitivity range of the image sensor includes the transmission range of the band-pass filter, the target wavelength range W is determined by the transmission range of the band-pass filter.

[0045] Examples of the camera 30 include a line scan type, a snapshot type, and a compressed sensing type hyperspectral camera. Representative components and operations of each hyperspectral camera are described below.

[0046] Line-scan hyperspectral camera When the camera 30 is a line-scan hyperspectral camera, the camera 30 includes a prism or diffraction grating, an image sensor, and a sliding mechanism for sliding the object to be photographed in one direction. The light source 20 emits a line beam extending in a direction perpendicular to the one direction. The object to be photographed is illuminated with the line beam emitted from the light source 20. The light generated by the illumination is separated into wavelength bands via the prism or diffraction grating and detected by the image sensor. In line scanning, such light detection is performed while the object to be photographed is moved in one direction by the sliding mechanism.

[0047] The camera 30 generates and outputs a hyperspectral image for correction by line scanning the subject 10a1 or 10a2. Similarly, the camera 30 generates and outputs a hyperspectral image to be analyzed by line scanning the subject 10b. A line-scan hyperspectral camera has high spatial and wavelength resolution, but the line scanning requires a long imaging time.

[0048] Snapshot-type hyperspectral camera When the camera 30 is a snapshot-type hyperspectral camera, the camera 30 includes a plurality of light-transmitting regions corresponding to a plurality of wavelength bands, respectively, and an image sensor. Each of the plurality of light-transmitting regions transmits light of the corresponding wavelength band among the plurality of wavelength bands in the target wavelength range. Light from the object to be photographed is detected by the image sensor via the plurality of light-transmitting regions.

[0049] Camera 30 captures a single shot of subject 10a1 or 10a2 to generate and output a hyperspectral image for correction. Similarly, camera 30 captures a single shot of subject 10b to generate and output a hyperspectral image to be analyzed. This is similar to the principle of a color camera capturing a single shot of an object to be captured through red, green, and blue color filters to generate and output red, green, and blue images. While snapshot-type hyperspectral cameras are capable of single-shot imaging, their sensitivity and spatial resolution are often insufficient.

[0050] Compressed Sensing Hyperspectral Camera When the camera 30 is a compressed sensing hyperspectral camera such as that disclosed in Patent Document 2, the camera 30 includes an encoding mask including multiple regions with different transmission spectra, an image sensor, and an image processing device. Light from an object to be photographed is detected by the image sensor via the encoding mask, and a compressed image is generated in which information from multiple wavelength bands is compressed. The image processing device generates a hyperspectral image of the object based on the compressed image.

[0051] Camera 30 photographs subject 10a1 or 10a2 to generate a compressed image for correction, and generates and outputs a hyperspectral image for correction based on the compressed image. Similarly, camera 30 photographs subject 10b to generate a compressed image of an object to be analyzed, and generates and outputs a hyperspectral image of the object to be analyzed based on the compressed image. A compressed sensing hyperspectral camera can generate a hyperspectral image for correction or analysis in one shot without reducing sensitivity or spatial resolution.

[0052] In a compressed sensing hyperspectral camera, the camera 30 may have a coding mask and an image sensor, but may not have an image processing device. In such a configuration, the camera 30 generates and outputs a compressed image, and the processing device 50 generates a hyperspectral image based on the compressed image.

[0053] Details of the compressed sensing hyperspectral camera will be described later.

[0054] <Display Device 40> The display device 40 displays an input user interface (UI) 42 and a display UI 44. The input UI 42 is used for a user to input information. The information input by the user to the input UI 42 is received by the processing circuitry 52. ​​The display UI 44 is used to display information generated by the processing circuitry 52.

[0055] The input UI 42 and the display UI 44 are displayed as a GUI (Graphical User Interface). It can be said that the information shown on the input UI 42 and the display UI 44 is displayed on the display device 40. The input UI 42 and the display UI 44 may be realized by a device capable of both input and output, such as a touch screen. In this case, the touch screen may function as the display device 40. When a keyboard and / or a mouse are used as the input UI 42, the input UI 42 is a device independent of the display device 40.

[0056] <Processing Device 50> The processing circuitry 52 included in the processing device 50 controls the operations of the light source 20, the camera 30, and the storage device 56. The processing circuitry 52 acquires the hyperspectral image for correction and the hyperspectral image to be analyzed generated by the camera 30, and performs processing based on these hyperspectral images. Alternatively, the processing circuitry 52 acquires the compressed image for correction and the compressed image to be analyzed generated by the camera 30, and performs processing based on these compressed images.

[0057] The memory 54 included in the processing device 50 stores a computer program executed by the processing circuit 52. The processing circuit 52 and the memory 54 may be integrated on a single circuit board or may be provided on separate circuit boards. The functions of the processing circuit 52 may also be distributed across multiple circuits. Part or all of the processing circuit 52 may be installed in a remote location away from the light source 20, the camera 30, and the storage device 56 and control the operations of these components via a wired or wireless communication network.

[0058] The storage device 56 included in the processing device 50 is a device including any storage medium such as a semiconductor storage medium or a magnetic storage medium. The storage device 56 is connected to the processing circuit 52 and stores the processing results of the processing circuit 52.

[0059] [1.2. Specific Configuration Example of Imaging System] Next, a specific configuration example of an imaging system according to an embodiment of the present disclosure will be described with reference to FIG. 3 . FIG. 3 is a diagram schematically illustrating a specific configuration of an imaging system according to an exemplary embodiment of the present disclosure. The imaging system 100 illustrated in FIG. 3 further includes a stage 60, a support 70, and an adjustment device 80 in addition to the light source 20, the camera 30, the display device 40, and the processing device 50 described above. In the example illustrated in FIG. 3 , the number of light sources 20 is two, but the number may be one, or three or more. The processing device 50 is connected to the light source 20, the camera 30, the display device 40, and the adjustment device 80 via a wire or wirelessly. The processing circuit 52 illustrated in FIGS. 2A to 2C and included in the processing device 50 controls the operation of the light source 20, the camera 30, the display device 40, and the adjustment device 80 in addition to the operation of the adjustment device 80.

[0060] The stage 60 has a flat support surface on which the subject 10a1 shown in Fig. 2A and the subject 10b shown in Fig. 2C are placed. The support 70 is fixed to the stage 60 and has a structure that extends in a direction perpendicular to the support surface of the stage 60, i.e., in the height direction. The support 70 supports the light source 20, the camera 30, and the adjustment device 80.

[0061] The adjustment device 80 includes a mechanism for independently moving the light source 20 and the camera 30 in a direction perpendicular to the support surface of the stage 60. The adjustment device 80 may include an actuator including one or more motors, such as a linear actuator. The actuator may be configured to change the distance between the light source 20 and the subject 10a1 or the distance between the light source 20 and the subject 10b, and the distance between the camera 30 and the subject 10a1 or the distance between the camera 30 and the subject 10b, using, for example, an electric motor, hydraulic pressure, or air pressure.

[0062] Since the adjustment device 80 can adjust the distance between the light source 20 and the subject 10a1 or the distance between the light source 20 and the subject 10b, it is possible to appropriately adjust the amount of light that is emitted from the light source 20, reflected by the subject 10a1 or the subject 10b, and then incident on the camera 30. This reduces the possibility that the hyperspectral image or compressed image will be too bright or too dark. Furthermore, since the adjustment device 80 can adjust the distance between the camera 30 and the subject 10a1 or the distance between the camera 30 and the subject 10b, it becomes easier to focus the camera 30.

[0063] The adjustment device 80 further includes measuring instruments for measuring the distance between the stage 60 and the light source 20, and the distance between the stage 60 and the camera 30. The support 150 is provided with a scale indicating the height of the stage 190 from the support surface. The positions of the light source 20 and the camera 30 in the height direction can be determined based on the scale.

[0064] 2. Issues with White Board Correction and Black Subtraction Issues with white board correction and black subtraction will be described below with reference to FIGS. 4A and 4B.

[0065] [2.1. White Board Correction] In white board correction, the hyperspectral image to be analyzed is normalized by the hyperspectral image for correction. More specifically, each of the multiple images included in the hyperspectral image to be analyzed is normalized by a corresponding image among the multiple images included in the hyperspectral image for correction, in that the corresponding image has the same wavelength band. In many cases, an appropriate white board has a reflectance equal to or greater than a certain level in the wavelength range included in the hyperspectral image. White board correction is performed to reduce the effects of the spectral shape of the irradiated light, the illumination distribution during shooting, lens vignetting, and uneven sensitivity of the image sensor. However, if there is no need to reduce such effects, white board correction is not necessarily required.

[0066] Fig. 4A is a diagram schematically illustrating an example of white board correction. The top three diagrams of Fig. 4A show examples of correct white board correction, while the middle three diagrams and the bottom three diagrams of Fig. 4A show examples of incorrect white board correction. Of the three diagrams in each row, the diagram on the left shows a hyperspectral image containing the subject 10b, the central diagram shows a hyperspectral image containing a white board, and the diagram on the right shows the hyperspectral image after white board correction. In Fig. 4A, an image corresponding to a certain wavelength band included in the hyperspectral image is illustrated as an example of the hyperspectral image.

[0067] When an appropriate white board is used as the subject 10a1, an appropriate hyperspectral image for correction is generated as shown in the center diagram in the top row. Since the hyperspectral image to be analyzed is corrected based on the appropriate hyperspectral image for correction, a correct hyperspectral image after white board correction is generated as shown in the right diagram in the top row.

[0068] In contrast, if dirt adheres to the white board, the dirt will appear in the correction hyperspectral image, as shown in the center diagram in the middle row. If the hyperspectral image to be analyzed is corrected based on such an inappropriate correction hyperspectral image, the hyperspectral image after white board correction will not be generated correctly, as shown in the right diagram in the middle row. In the correction hyperspectral image, the parts corresponding to the dirt will be darker than the other parts, so due to the normalization described above, the parts corresponding to the dirt will appear white in the hyperspectral image after white board correction.

[0069] Alternatively, if an object other than a white board is mistakenly used as the subject 10a1, the object will appear in the hyperspectral image for correction, as shown in the center diagram in the lower row. A fish is shown as an example of the object. If the hyperspectral image to be analyzed is corrected based on such an inappropriate hyperspectral image for correction, the hyperspectral image after white board correction will not be generated correctly, as shown in the right diagram in the lower row. Since the portion of the hyperspectral image for correction corresponding to the object will be darker than the other portions, the normalization described above will cause the portion corresponding to the object to appear white in the hyperspectral image after white board correction.

[0070] Therefore, if the white board correction is performed incorrectly, it is not easy to accurately obtain spectral information of the subject 10b based on the hyperspectral image after the white board correction.

[0071] [2.2. Blackout Correction] In blackout correction, a correction hyperspectral image is subtracted from the hyperspectral image to be analyzed. More specifically, from each of the multiple images included in the hyperspectral image to be analyzed, a corresponding image from the multiple images included in the correction hyperspectral image that has the same wavelength band is subtracted. Blackout correction is performed to reduce the effects of image sensor dark current, bright pixel points, fixed pattern noise, and fluctuations in sensor performance. However, if there is no need to reduce such effects, blackout correction is not necessarily required.

[0072] In the following description, the hyperspectral image to be analyzed may be read as the compressed image to be analyzed, and the hyperspectral image for correction may be read as the compressed image to be corrected.

[0073] 4B is a diagram schematically illustrating an example of black filtering. The top three diagrams of FIG. 4B show examples of correct black filtering, and the bottom three diagrams of FIG. 4B show examples of incorrect black filtering. Of the three diagrams in each row, the diagram on the left shows a hyperspectral image containing the subject 10b, the diagram in the middle shows a hyperspectral image representing a darkened image, and the diagram on the right shows the hyperspectral image after black filtering. In FIG. 4B, an example of a hyperspectral image is an image corresponding to a certain wavelength band contained in the hyperspectral image.

[0074] When a light-blocking lens cap is properly attached to the camera 30 as the subject 10a2, i.e., when the subject is photographed in a light-blocked state, an appropriate hyperspectral image for correction is generated, as shown in the center diagram in the top row. The hyperspectral image for correction, which is a light-blocked image, contains noise caused by the image sensor. The hyperspectral image to be analyzed contains the subject 10b with the noise superimposed, as shown in the left diagram in the top row. Because the hyperspectral image to be analyzed is corrected based on the appropriate hyperspectral image for correction, a correct hyperspectral image after blackout is generated, as shown in the right diagram in the top row. The hyperspectral image after blackout contains the subject 10b with the noise removed.

[0075] In contrast, if a light-blocking lens cap is forgotten to be attached to the camera 30 and a white board is photographed, a hyperspectral image showing the white board, rather than a light-blocked image, is generated as a correction hyperspectral image, as shown in the center diagram in the lower row. If the hyperspectral image to be analyzed is corrected based on such an inappropriate correction hyperspectral image, an incorrect hyperspectral image after black subtraction will be generated, as shown in the right diagram in the lower row.

[0076] Because the correction hyperspectral image, which is not a dark image, is subtracted from the hyperspectral image to be analyzed, more pixel values ​​than necessary are subtracted. This causes the image after incorrect black subtraction to be darker than the hyperspectral image after correct black subtraction. In the image after incorrect black subtraction, pixel values ​​may be negative. In such cases, the pixel values ​​may be output as zero. Therefore, the corrected hyperspectral image may differ from the image obtained by simply subtracting the correction hyperspectral image from the hyperspectral image to be analyzed.

[0077] Therefore, if the black subtraction is incorrect, it is not easy to accurately obtain spectral information of the subject 10b based on the corrected hyperspectral image.

[0078] As described above, if the subject 10a1 or the subject 10a2 is not suitable, it is not easy to accurately acquire the spectral information of the subject 10b. For example, if the subject 10a2 is not suitable, a light-blocking lens cap may not be properly attached to the camera 30, making it difficult to take a picture in a light-blocking state.

[0079] The present inventors have found this problem and have come up with a method used for correcting an image according to this embodiment that can solve the problem.

[0080] 5, an example of a method used to correct an image according to this embodiment when the camera 30 generates and outputs a hyperspectral image will be described. The image correction described below may be either whiteboard correction or black subtraction.

[0081] 5 is a flowchart showing an outline of Example 1 of the processing operation executed by the processing circuitry 52 in the imaging system according to this embodiment. The processing circuitry 52 executes the operations of steps S101 to S108 shown in Fig. 5. "HS image" shown in Fig. 5 represents a hyperspectral image.

[0082] <Step S101> The processing circuitry 52 displays an instruction to the user to photograph a whiteboard or an instruction to the user to photograph in a light-blocking state on the display device 40. Alternatively, if the imaging system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0083] Upon receiving the instruction, the user places subject 10a1 in front of camera 30 or attaches subject 10a2 to camera 30. Upon receiving input from the user via input UI 42, processing circuitry 52 causes camera 30 to capture subject 10a1 or subject 10a2 and generate a first hyperspectral image. In this manner, the first hyperspectral image is generated upon receiving an instruction from the user to capture subject 10a1 or subject 10a2.

[0084] Before photographing subject 10a1 or subject 10a2, processing circuitry 52 may receive input from a user via input UI 42 and adjust parameters related to camera 30. The parameters related to camera 30 may be, for example, exposure time, gain, number of integrations, and / or the distance between camera 30 and subject 10a1.

[0085] When photographing subject 10a1, processing circuitry 52 receives input from the user via input UI 42 before step S101 and causes light source 20 to emit illumination light for illuminating subject 10a1. When photographing subject 10a2, processing circuitry 52 does not need to cause light source 20 to emit illumination light before step S101.

[0086] Before causing the light source 20 to emit irradiation light, the processing circuit 52 may receive input from the user via the input UI 42 and adjust parameters related to the light source 20. The parameters related to the light source 20 may be, for example, the distance between the light source 20 and the subject 10a1, the current, voltage, and duty ratio of a PWM (Pulse Width Modulation) signal for driving the light source 20, and / or the attenuation rate of a Neutral Density (ND) filter (not shown) disposed between the light source 20 and the subject 10a1.

[0087] <Step S102> The processing circuitry 52 acquires a first hyperspectral image as a hyperspectral image for correction from the camera 30. The processing circuitry 52 may store the acquired first hyperspectral image in the storage device 56.

[0088] <Step S103> The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0089] Upon receiving the instruction, the user places the subject 10b in front of the camera 30. Upon receiving the input from the user via the input UI 42, the processing circuitry 52 causes the camera 30 to capture an image of the subject 10b and generate a second hyperspectral image. In this manner, the second hyperspectral image is generated upon receiving the instruction from the user to capture an image of the subject 10b.

[0090] When the subject 10a1 is photographed in step S101, the subject 10b is illuminated with the above-described illumination light. When the subject 10a2 is photographed in step S101, the processing circuit 52 causes the light source 20 to emit illumination light for illuminating the subject 10b before step S103.

[0091] <Step S104> The processing circuitry 52 acquires a second hyperspectral image as the hyperspectral image to be analyzed from the camera 30. The processing circuitry 52 may store the acquired second hyperspectral image in the storage device 56.

[0092] The processing circuitry 52 may perform the operations of steps S103 and S104 between steps S105 and S106.

[0093] <Step S105> Based on the pixel values ​​of the first hyperspectral image, the processing circuitry 52 determines whether the first hyperspectral image satisfies the suitability conditions for an image for use in whiteboard correction and black subtraction, which will be described later.

[0094] If the determination is Yes, the processing circuitry 52 executes the operation of step S106. If the determination is No, the processing circuitry 52 executes the operation of step S108.

[0095] <Step S106> The processing circuitry 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image, or performs black subtraction by subtracting the first hyperspectral image from the second hyperspectral image.

[0096] In the whiteboard correction, black subtraction may already have been performed on the second hyperspectral image.

[0097] <Step S107> The processing circuitry 52 stores the corrected hyperspectral image in the storage device 56.

[0098] <Step S108> The processing circuitry 52 causes the display device 40 to display an error indicating that there is an abnormality in the first hyperspectral image.

[0099] If part of the processing circuitry 52 is an external server installed in a remote location, the external server may perform the operations of steps S102 and S105.

[0100] The above-described method used for image correction according to the present embodiment can prevent inappropriate whiteboard correction when a user believes that he or she has photographed the appropriate subject 10a1 in response to instructions, but the subject 10a1 is actually inappropriate. Similarly, it can prevent inappropriate blackout when a user believes that he or she has photographed the appropriate subject 10a2 in response to instructions, but the subject 10a2 is actually inappropriate. As a result, spectral information of the subject 10b can be acquired more accurately than when the first hyperspectral image does not determine whether it satisfies the suitability conditions.

[0101] 6A and 6B , examples of the suitability conditions that the first hyperspectral image must satisfy to be an image for white board correction are described below. Here, whether the first hyperspectral image satisfies the suitability conditions for an image for white board correction is determined based on the spectral information acquired from the first hyperspectral image.

[0102] 6A is a diagram showing a first hyperspectral image and a reflectance spectrum when the subject 10a1 is appropriate. The first hyperspectral image is shown at the top of FIG. 6A. The first hyperspectral image includes five images corresponding to five wavelength bands. Wavelength bands W １ ~W ５ The smaller the subscript number, the shorter the central wavelength of the wavelength band. These images are smooth images without boundaries or structure. The whiter the image, the brighter the image, and the blacker the image.

[0103] The reflectance spectrum of the object 10a1 is shown at the bottom of Figure 6A. This reflectance spectrum was generated based on the first hyperspectral image. The reflectance intensity in each wavelength band is calculated by averaging the pixel values ​​of multiple pixels near the center of the image corresponding to that wavelength band. An appropriately selected object 10a1 reflects light in the wavelength band W. １ ~W ５ The reflectance of the subject 10a1 is approximately the same as the spectrum of the light emitted from the light source 20. In other words, the spectrum obtained when the camera 30 detects the reflected light is approximately the same as the spectrum obtained when the camera 30 directly detects the light emitted from the light source 20. The reflected light is light generated when the light emitted from the light source 20 is reflected by the subject 10a. In the example shown in FIG. 6A, the wavelength band W １ The reflection intensity is highest in the wavelength band W ２ The reflection intensity is lowest in the wavelength band W ３ ~W ５ The reflection intensity in the wavelength band W ２ The reflection intensity is higher than that in the wavelength band W １The reflection intensity is lower than that in the wavelength band W ４ The reflection intensity in the wavelength band W ３ and wavelength band W ５ The reflection intensity is higher than that at

[0104] 6B is a diagram schematically illustrating a first hyperspectral image and a reflectance spectrum when the subject 10a1 is not suitable. The upper and lower diagrams in FIG. 6B are as described above.

[0105] If the subject 10a1 is not suitable, the reflection spectrum of the subject 10a1 will differ from the spectrum of the light emitted from the light source 20. In the example shown in Figure 6B, a blue plate was used as the subject 10a1. In the example shown in Figure 6B, as the center wavelength of the wavelength band increases, the reflection intensity decreases and the image becomes darker.

[0106] Spectral information of the light from light source 20 is pre-stored in storage device 56. Processing circuitry 52 acquires the spectral information of the light from light source 20 from storage device 56 and determines whether the first hyperspectral image satisfies the suitability condition by comparing the spectral information acquired from the first hyperspectral image with the spectral information of the light from light source 20. If the shapes of the two spectra are similar, processing circuitry 52 determines that the first hyperspectral image satisfies the suitability condition.

[0107] For example, the Spectral Angular Mapping (SAM) method can be used to compare the two pieces of spectral information. When a spectrum includes N light intensities corresponding to N wavelength bands, the spectrum is expressed as an N-dimensional vector including the N light intensities as components. If a vector representing the spectrum acquired from the first hyperspectral image is denoted by u and a vector representing the spectrum of light from the light source 20 is denoted by v, the spectral angle formed by the vectors u and v can be calculated. The spectral angle is given by

[0108]

[0109] where u·v represents the dot product of vector u and vector v, |u| represents the magnitude of vector u, and |v| represents the magnitude of vector v.

[0110] When the spectral angle is zero, vectors u and v point in the same direction. In this case, the shapes of the two spectra are the same. Even if |u| is different from |v|, the shapes of the two spectra can be said to be the same if vectors u and v point in the same direction. Therefore, the SAM method can compare vectors u and v regardless of the amount of irradiated light used when generating the first hyperspectral image. If the spectral angle formed by vectors u and v is, for example, 1° or less, 5° or less, or 10° or less, or any angle in the range of 1° to 10°, the first hyperspectral image may be determined to satisfy the suitability condition.

[0111] Next, with reference to FIG. 7 , another example of the eligibility conditions for whiteboard correction that the first hyperspectral image must satisfy will be described. In practice, it is rare to mistakenly capture a board of a different color, such as blue, instead of a white board, as described above. Instead, it is common to mistakenly capture a scene containing one or more objects. Therefore, in many cases, if the spatial distribution of pixel values ​​in the first hyperspectral image is consistent, it is safe to consider the subject 10a1 to be appropriate.

[0112] In this method, the first hyperspectral image is subjected to edge detection, and based on the edge image obtained thereby, it is determined whether the first hyperspectral image satisfies the suitability condition. For example, the Sobel method can be used for edge detection.

[0113] FIG. 7 shows first hyperspectral images when the subject 10a1 is appropriate and when it is not appropriate, and an edge image obtained by edge detection on the first hyperspectral image. The upper left side of FIG. 7 shows the first hyperspectral image when the subject 10a1 is appropriate, and the upper right side of FIG. 7 shows an edge image obtained by edge detection on the first hyperspectral image. The lower left side of FIG. 7 shows the first hyperspectral image when a white board is not appropriately used as the subject 10a1, and the lower right side of FIG. 7 shows an edge image obtained by edge detection on the first hyperspectral image. In FIG. 7 , an image corresponding to a certain wavelength band included in the first hyperspectral image is illustrated as the first hyperspectral image. In the following description referring to FIG. 7 , the first hyperspectral image refers to an image corresponding to a certain wavelength band included in the first hyperspectral image.

[0114] 7, if the object 10a1 is appropriate, the first hyperspectral image is a smooth image without boundaries or structures, and even if edge detection is performed on the first hyperspectral image, no pixels are detected as edges.

[0115] In contrast, if the subject 10a1 is not appropriate, the first hyperspectral image will be an uneven image with boundaries and structures. When edge detection is performed on the first hyperspectral image, some pixels will be detected as edges. In the example shown in Figure 7, the first hyperspectral image contains multiple vegetables. Of the 65,535 pixels contained in the first hyperspectral image, 2,109 pixels were detected as edges.

[0116] Based on the above, the first hyperspectral image may be determined to satisfy the suitability condition if a condition for determining that the spatial distribution of pixel values ​​of the first hyperspectral image is constant is satisfied. The condition may be determined, for example, based on the number of pixels detected as edges among a plurality of pixels in an image corresponding to a certain wavelength band included in the first hyperspectral image. Specifically, the condition may be, for example, that the number of pixels detected as edges is equal to or less than a predetermined percentage. The number of images for which edge detection is performed among the plurality of images included in the first hyperspectral image may be one or more.

[0117] 8A and 8B , examples of the suitability conditions that the first hyperspectral image satisfies as an image for blackout are described below. Here, whether the first hyperspectral image satisfies the suitability conditions as an image for blackout is determined based on the pixel values ​​of the first hyperspectral image.

[0118] Fig. 8A is a diagram showing a first hyperspectral image and a histogram of pixel values ​​of the first hyperspectral image when the subject 10a2 is appropriate. The upper part of Fig. 8A shows the hyperspectral image, and the lower part of Fig. 8A shows a histogram of pixel values ​​of the first hyperspectral image. Fig. 8A illustrates an example of the first hyperspectral image, which is an image corresponding to a certain wavelength band included in the first hyperspectral image. In the following description referring to Fig. 8A, the first hyperspectral image refers to an image corresponding to a certain wavelength band included in the first hyperspectral image.

[0119] As shown in Figure 8A, the first hyperspectral image is a dark image. The pixel values ​​of all pixels in the first hyperspectral image are approximately zero. In the example shown in Figure 8A, the first hyperspectral image includes 256 x 256 pixels. In the histogram of pixel values ​​of the first hyperspectral image, the pixel values ​​are normalized by the maximum pixel value that the pixel can have. In the histogram of pixel values ​​of the first hyperspectral image, the pixels with the lowest pixel value are the most numerous, accounting for more than 97% of all pixels. The normalized minimum pixel value is 2.44 x 10 －４ The normalized pixel value is 1.0×10 －３ Pixels with this or higher value account for 0.07% of all pixels.

[0120] 8B is a diagram showing a first hyperspectral image and a histogram of pixel values ​​of the first hyperspectral image when the subject 10a2 is not suitable. The upper and lower figures in FIG. 8B are as described above. In the following description referring to FIG. 8B, the first hyperspectral image refers to an image corresponding to a certain wavelength band included in the first hyperspectral image.

[0121] As shown in Figure 8B, the first hyperspectral image is not a dark image but a landscape image. In the histogram of pixel values ​​of the first hyperspectral image, the pixel with the lowest pixel value accounts for 0.006% of all pixels. The normalized lowest pixel value is 0.0273. The normalized pixel value is 1.0 x 10 －３ Pixels that are equal to or greater than this account for 100% of all pixels.

[0122] From the above, if a condition for determining that the pixel values ​​of the first hyperspectral image are small is satisfied, the first hyperspectral image may be determined to satisfy the suitability condition. The condition may be determined, for example, based on the number of pixels having the lowest pixel value or the number of pixels whose pixel values ​​are greater than or equal to a threshold value in an image corresponding to a certain wavelength band included in the first hyperspectral image. Specifically, the condition may be that the number of pixels having the lowest pixel value is equal to or greater than a predetermined percentage, the number of pixels whose pixel values ​​are greater than or equal to a threshold value is equal to or less than a predetermined percentage, or the number of pixels whose pixel values ​​are equal to or less than a threshold value is equal to or greater than a predetermined percentage. Of the multiple images included in the first hyperspectral image, the number of images whose pixel value histograms are examined may be one or more.

[0123] Alternatively, as described with reference to Figure 7, similar to whiteboard correction, even with black subtraction, if the conditions for determining that the spatial distribution of pixel values ​​of the first hyperspectral image is constant are met, the first hyperspectral image may be determined to satisfy the suitability conditions.

[0124] 4. Method 2 Used for Image Correction 4.1. Processing Operation An example of a method used for image correction according to this embodiment when the camera 30 in a compressed sensing hyperspectral camera generates and outputs a compressed image will be described below with reference to Fig. 9. The image correction described below is whiteboard correction.

[0125] 9 is a flowchart schematically showing Example 2 of the processing operation executed by the processing circuitry 52 in the imaging system according to this embodiment. The processing circuitry 52 executes the operations of steps S201 to S210 shown in FIG.

[0126] <Step S201> The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph a whiteboard. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0127] Upon receiving the instruction, the user places the subject 10a1 in front of the camera 30. Upon receiving the input from the user via the input UI 42, the processing circuitry 52 causes the camera 30 to capture the subject 10a1 and generate a first compressed image. In this manner, the first compressed image is generated upon receiving the instruction from the user to capture the subject 10a1.

[0128] Before step S201, the processing circuitry 52 receives an input from the user via the input UI 42 and causes the light source 20 to emit irradiation light for irradiating the subject 10a1.

[0129] <Step S202> The processing circuitry 52 acquires a first compressed image from the camera 30. The processing circuitry 52 may store the acquired first compressed image in the storage device 56.

[0130] <Step S203> The processing circuitry 52 generates a first hyperspectral image based on the first compressed image. The processing circuitry 52 may store the generated first hyperspectral image in the storage device 56.

[0131] <Step S204> The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0132] Upon receiving the instruction, the user places the subject 10b in front of the camera 30. Upon receiving the input from the user via the input UI 42, the processing circuit 52 causes the camera 30 to capture the subject 10b and generate a second compressed image. In this manner, the second compressed image is generated upon receiving the instruction from the user to capture the subject 10b. The subject 10b is illuminated with the above-described illumination light.

[0133] <Step S205> The processing circuitry 52 acquires a second compressed image from the camera 30. The processing circuitry 52 may store the acquired second compressed image in the storage device 56.

[0134] <Step S206> The processing circuitry 52 generates a second hyperspectral image based on the second compressed image. The processing circuitry 52 may store the generated second hyperspectral image in the storage device 56.

[0135] The processing circuitry 52 may perform the operations of steps S203 to S206 between steps S207 and S208.

[0136] <Step S207> The processing circuitry 52 determines whether the first compressed image satisfies the suitability conditions for an image for whiteboard correction, based on the pixel values ​​of the first compressed image. The suitability conditions for an image for whiteboard correction will be described later.

[0137] If the determination is Yes, the processing circuit 52 executes the operation of step S208. If the determination is No, the processing circuit 52 executes the operation of step S210.

[0138] <Step S208> The processing circuitry 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image. The second hyperspectral image may have already been subjected to black subtraction.

[0139] <Step S209> The processing circuitry 52 stores the corrected hyperspectral image in the storage device 56.

[0140] <Step S210> The processing circuitry 52 causes the display device 40 to display an error indicating that there is an abnormality in the first compressed image.

[0141] If part of the processing circuitry 52 is an external server installed in a remote location, the external server may perform the operations of steps S202 and S207.

[0142] The method used for image correction according to the present embodiment prevents inappropriate whiteboard correction when the subject 10a1 is actually inappropriate, even though the user believes that he or she has photographed the appropriate subject 10a1 in response to instructions. As a result, spectral information of the subject 10b can be acquired more accurately than when the first compressed image is not determined to satisfy the suitability conditions for an image for whiteboard correction.

[0143] The processing circuitry 52 may execute the operations of S105 to S108 shown in FIG. 5 instead of the operations of S207 to S210 shown in FIG.

[0144] 10A and 10B, examples of suitability conditions that the first compressed image satisfies as an image for white board correction will be described below. Here, whether the first compressed image satisfies the suitability conditions as an image for white board correction is determined based on a histogram of pixel values ​​of the first compressed image.

[0145] 10A is a diagram showing a first compressed image and a histogram of pixel values ​​thereof when the subject 10a1 is appropriate, in which the first compressed image is shown in the upper part of Fig. 10A and the histogram of pixel values ​​of the first compressed image is shown in the lower part of Fig. 10A.

[0146] As shown in Fig. 10A, the first compressed image has an irregular distribution of light and dark pixel values, reflecting the spatial distribution of the transmittance of the encoding mask. The histogram of pixel values ​​of the first compressed image roughly shows a single peak. When the subject 10a1 is a white board, the peak width is narrower than when it is not. In the example shown in Fig. 10A, if the average pixel value is μ and the standard deviation of the pixel value is σ, then σ / μ = 0.2615.

[0147] 10B is a diagram showing the first compressed image and its pixel value histogram when the subject 10a1 is not appropriate. The upper and lower diagrams in FIG. 10B are as described above.

[0148] As shown in Figure 10B, the first compressed image has an irregular distribution of light and dark pixel values. Visually, the first compressed image shown in Figure 10B is similar to the first compressed image shown in Figure 10A. The histogram of pixel values ​​of the first compressed image roughly shows a single peak. If the object 10a1 is not a white board, the peak width will be wider. In the example shown in Figure 10B, σ / μ = 0.3016.

[0149] In view of the above, it may be possible to determine whether the first compressed image satisfies the suitability condition based on a histogram of pixel values ​​of the first compressed image. For example, if the σ / μ of the histogram of the first compressed image is equal to or less than a predetermined value, the first compressed image is determined to satisfy the suitability condition.

[0150] 5. Method 3 Used for Image Correction Another example of a method used for image correction according to this embodiment when the camera 30 in a compressed sensing hyperspectral camera generates and outputs a compressed image will be described below with reference to Fig. 11. The image correction described below is black subtraction.

[0151] 11 is a flowchart schematically showing Example 3 of the processing operation executed by the processing circuitry 52 in the imaging system according to this embodiment. The processing circuitry 52 executes the operations of steps S301 to S308 shown in FIG.

[0152] <Step S301> The processing circuitry 52 displays an instruction to the user to take a picture in a light-blocking state on the display device 40. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0153] Upon receiving the instruction, the user attaches the subject 10a2 to the camera 30. Upon receiving input from the user via the input UI 42, the processing circuitry 52 causes the camera 30 to capture an image of the subject 10a2 and generate a first compressed image. In this manner, the first compressed image is generated upon receiving an instruction from the user to capture an image of the subject 10a2.

[0154] <Step S302> The processing circuitry 52 acquires a first compressed image from the camera 30. The processing circuitry 52 may store the acquired first compressed image in the storage device 56.

[0155] <Step S303> The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

[0156] Upon receiving the instruction, the user places the subject 10b in front of the camera 30. Upon receiving the input from the user via the input UI 42, the processing circuit 52 causes the camera 30 to capture an image of the subject 10b and generate a second compressed image. In this manner, the second compressed image is generated upon receiving an instruction from the user to capture an image of the subject 10b.

[0157] Before step S303, the processing circuitry 52 receives an input from the user via the input UI 42 and causes the light source 20 to emit irradiation light for irradiating the subject 10b.

[0158] <Step S304> The processing circuitry 52 acquires a second compressed image from the camera 30. The processing circuitry 52 may store the acquired second compressed image in the storage device 56.

[0159] The processing circuitry 52 may perform the operations of steps S303 and S304 between steps S305 and S306.

[0160] <Step S305> The processing circuitry 52 determines whether the first compressed image satisfies the suitability conditions for an image for black subtraction based on the pixel values ​​of the first compressed image. The suitability conditions for an image for black subtraction correction that the first compressed image satisfies may be the same as the suitability conditions described with reference to Figures 8A and 8B. More specifically, whether the first compressed image satisfies the suitability conditions may be determined based on the number of pixels in the first compressed image that have the lowest pixel value or the number of pixels whose pixel values ​​are greater than or equal to a predetermined value or less than a predetermined value.

[0161] Alternatively, as described with reference to FIG. 7 , similar to whiteboard correction, even with black subtraction, if the conditions for determining that the spatial distribution of pixel values ​​of the first compressed image is constant are met, the first compressed image may be determined to satisfy the suitability conditions.

[0162] If the determination is Yes, the processing circuitry 52 executes the operation of step S306. If the determination is No, the processing circuitry 52 executes the operation of step S308.

[0163] <Step S306> The processing circuit 52 performs black subtraction by subtracting the first compressed image from the second compressed image.

[0164] <Step S307> The processing circuitry 52 stores the corrected compressed image in the storage device 56.

[0165] <Step S308> The processing circuitry 52 causes the display device 40 to display an error indicating that there is an abnormality in the first compressed image.

[0166] If part of the processing circuitry 52 is an external server installed in a remote location, the external server may perform the operations of steps S302 and S305.

[0167] The above-described method used for image correction according to the present embodiment can prevent inappropriate blackout when the subject 10a2 is actually inappropriate, even though the user believes that he or she has photographed the appropriate subject 10a2 in response to instructions. As a result, spectral information of the subject 10b can be acquired more accurately than when the first compressed image is not determined to satisfy the conditions for suitability as an image for blackout.

[0168] 12A to 12C are diagrams schematically illustrating examples of the UI of the display device 40 when the camera 30 in a compressed sensing hyperspectral camera generates and outputs a compressed image. The UI serves as both the input UI 42 and the display UI 44.

[0169] The upper left side of Fig. 12A shows a space for a compressed image of the subject 10b. The lower side of Fig. 12A shows a space for a corrected hyperspectral image of the subject 10b. The upper right side of Fig. 12A shows the shooting conditions, such as resolution, exposure time, gain, and number of integrations. The center right of Fig. 12A shows buttons for "whiteboard correction" and "blackout." When the user presses the "whiteboard correction" button, the processing circuitry 52 executes the operations of the flowchart shown in Fig. 9. When the user presses the "blackout" button, the processing circuitry 52 executes the operations of the flowchart shown in Fig. 11.

[0170] When the user presses the "Whiteboard Correction" button, if the first compressed image does not satisfy the suitability conditions, an error pop-up message is displayed on the UI of the display device 40, as shown in FIG. 12B. The error pop-up message reads, "An abnormality has been detected in the compressed image of the whiteboard. Please confirm that it is an appropriate whiteboard. Do you want to continue processing?" In this way, the error message may prompt the user to confirm whether the whiteboard has been correctly photographed as the correction subject 10a1. Note that the compressed image of the subject 10b generated during the processing is shown in the upper left corner.

[0171] When the user presses the "black out" button, if the first compressed image does not satisfy the suitability conditions, an error pop-up message is displayed on the UI of the display device 40, as shown in Fig. 12C. The error pop-up message reads, "An abnormality has been detected in the darkened image. Please check the darkened state. Do you want to continue processing?" In this way, the error message may prompt the user to check the darkened state when photographing the correction subject 10a2.

[0172] 12B and 12C, an error may be displayed and the process may be interrupted, or the process may be continued after confirmation from the user. Alternatively, the user may be requested to retake a compressed image of the subject 10a1 or the subject 10a2.

[0173] 7. Summary of Methods 1 to 3 Used for Image Correction Below, with reference to FIG. 13 , a summary of Methods 1 to 3 used for image correction according to this embodiment will be described, focusing on processing operations common to Methods 1 to 3.

[0174] 13 is a flowchart outlining examples 1 to 3 of the processing operation executed by the processing circuitry 52 in the imaging system according to this embodiment. The processing circuitry 52 executes the operations of steps S401 to S406 shown in FIG.

[0175] <Step S401> The processing circuitry 52 acquires a first image from the camera 30 as an image for correction.

[0176] In method 1 used to correct the image, the first image is a first hyperspectral image. In methods 2 and 3 used to correct the image, the first image is a first compressed image.

[0177] <Step S402 > The processing circuitry 52 acquires a second image from the camera 30 .

[0178] In methods 1 and 2 used to correct the image, the second image is a second hyperspectral image. In method 2 used to correct the image, processing circuitry 52 generates and obtains a second hyperspectral image based on the second compressed image. In method 3 used to correct the image, the second image is a second compressed image.

[0179] The processing circuitry 52 may perform the operation of step S402 between steps S403 and S404.

[0180] <Step S403> Based on the pixel values ​​of the first image, the processing circuitry 52 determines whether the first image satisfies the suitability conditions for a correction image for correcting the second image. If the determination is Yes, the processing circuitry 52 executes the operation of step S404. If the determination is No, the processing circuitry 52 executes the operation of step S406.

[0181] <Step S404> The processing circuitry 52 corrects the second image based on the first image.

[0182] In the first method used to correct the images, the processing circuitry 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image, or performs black subtraction by subtracting the first hyperspectral image from the second hyperspectral image.

[0183] In method 2 used to correct the image, the processing circuit 52 generates a first hyperspectral image based on the first compressed image and performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image.

[0184] In method 3 used to correct the image, processing circuitry 52 performs black subtraction by subtracting the first compressed image from the second compressed image.

[0185] <Step S405> The processing circuitry 52 stores the corrected image in the storage device 56.

[0186] <Step S406> The processing circuitry 52 causes the display device 40 to display an error indicating that there is an abnormality in the first image.

[0187] If part of the processing circuitry 52 is an external server installed in a remote location, the external server may perform the operations of steps S401 and S403.

[0188] The method used for correcting an image according to this embodiment prevents inappropriate correction when the subject 10a1 or 10a2 is actually not suitable, even though the user believes that the subject 10a1 or 10a2 has been photographed in accordance with instructions. As a result, spectral information of the subject 10b can be acquired more accurately than when the first image is not determined to satisfy the suitability conditions.

[0189] 8. Method 4 Used for Image Correction Method 4 used for image correction according to this embodiment will be described below with reference to Figs. 14 and 15. Here, an image generated by capturing a scene including subject 10a1 and subject 10b with camera 30 is used. The image correction described below is whiteboard correction.

[0190] Fig. 14 is a diagram illustrating an example of an image generated by capturing a scene including subjects 10a1 and 10b with camera 30. Image 38 shown in Fig. 14 is a hyperspectral image or a compressed image. Subjects 10a1 and 10b are captured in image 38.

[0191] 15 is a flowchart schematically showing Example 4 of the processing operation executed by the processing circuitry 52 in the imaging system according to this embodiment. The processing circuitry 52 executes the operations of steps S501 to S507 shown in FIG.

[0192] <Step S501> The processing circuitry 52 displays instructions to the user to photograph the subject for correction and the subject to be analyzed on the display device 40. Alternatively, if the imaging system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instructions as sound.

[0193] Upon receiving the instruction, the user positions subject 10a1 and subject 10b in front of camera 30. Upon receiving input from the user via input UI 42, processing circuitry 52 causes a scene including subject 10a1 and subject 10b to be captured, and generates image 38 containing subject 10a1 and subject 10b. In this manner, image 38 is generated upon receiving an instruction from the user to capture subject 10a1 and subject 10b.

[0194] <Step S502> The processing circuitry 52 acquires the image 38 from the camera 30.

[0195] <Step S503> The processing circuitry 52 generates, based on the image 38, a first sub-image corresponding to the object 10a1 and a second sub-image corresponding to the object 10a2.

[0196] If image 38 is a hyperspectral image, processing circuitry 52 extracts first and second sub-images from image 38, for example, by edge detection. Thus, the first sub-image is one portion of the hyperspectral image, and the second sub-image is another portion of the hyperspectral image. The first sub-image may be a region defined by the outline of object 10a1 in image 38, or may be a region including object 10a1 in image 38 and having any shape, such as a rectangle or a circle. The same applies to the second sub-image. Processing circuitry 52 may extract the second sub-image from the hyperspectral image between steps S504 and S505.

[0197] If image 38 is a compressed image, processing circuitry 52 generates a hyperspectral image based on the compressed image and extracts the first and second sub-images from the hyperspectral image, where the first sub-image is one portion of the hyperspectral image and the second sub-image is another portion of the hyperspectral image. Note that processing circuitry 52 may extract the second sub-image from the hyperspectral image between steps S504 and S505.

[0198] Alternatively, if image 38 is a compressed image, processing circuitry 52 extracts a first sub-image from the compressed image. Processing circuitry 52 then generates a hyperspectral image based on the compressed image and extracts a second sub-image from the hyperspectral image. Thus, the first sub-image is part of the compressed image, and the second sub-image is part of the hyperspectral image. Note that processing circuitry 52 may perform the operations of generating a hyperspectral image based on the compressed image and extracting the second sub-image from the hyperspectral image between steps S504 and S505.

[0199] <Step S504> The processing circuit 52 determines, based on the pixel values ​​of the first sub-image, whether or not the first sub-image satisfies the suitability conditions for being a correction image for correcting the second sub-image.

[0200] When image 38 is a hyperspectral image, the suitability conditions for an image for whiteboard correction are as described with reference to Figures 6A to 7. The same applies when image 38 is a compressed image and the first sub-image is part of a hyperspectral image.

[0201] When image 38 is a compressed image and the first sub-image is a part of the compressed image, the suitability conditions for the image to be used for whiteboard correction are as described with reference to FIGS. 10A and 10B.

[0202] If the determination is Yes, the processing circuitry 52 executes the operation of step S505. If the determination is No, the processing circuitry 52 executes the operation of step S507.

[0203] <Step S505> The processing circuit 52 corrects the second sub-image based on the first sub-image.

[0204] If image 38 is a hyperspectral image, processing circuitry 52 performs whiteboard correction by normalizing the second sub-image by the first sub-image, as well as if image 38 is a compressed image in which the first sub-image is one portion of the hyperspectral image and the second sub-image is another portion of the hyperspectral image.

[0205] If image 38 is a compressed image, the first sub-image is part of the compressed image, and the second sub-image is part of a hyperspectral image, processing circuit 52 performs whiteboard correction by extracting a sub-image corresponding to subject 10a1 from the hyperspectral image from which the second sub-image was extracted, and normalizing the second sub-image using the extracted sub-image.

[0206] <Step S506> The processing circuitry 52 stores the corrected sub-image in the storage device 56.

[0207] <Step S507> The processing circuitry 52 causes the display device 40 to display an error indicating that there is an abnormality in the first sub-image.

[0208] If part of the processing circuit 52 is an external server installed in a remote location, the external server may perform the operations of steps S502 to S504.

[0209] The above-described method used for image correction according to the present embodiment can prevent inappropriate white board correction when the user believes that he or she has photographed the appropriate subject 10a1 in response to instructions, but the subject 10a1 is actually inappropriate. As a result, spectral information of the subject 10b can be acquired more accurately than when determining whether the first sub-image satisfies the suitability conditions. Furthermore, the above-described method enables white board correction even when it is difficult to capture a white board in the entire image, such as when photographing outdoors.

[0210] 9. Compressed Sensing Hyperspectral Camera An example configuration of a compressed sensing hyperspectral camera will be described below with reference to FIGS. 16A to 18B. FIG. 16A is a diagram schematically illustrating an example configuration of a camera 30, which is a compressed sensing hyperspectral camera. Similar to the configuration disclosed in Patent Document 2, the camera 30 shown in FIG. 16A includes an optical system 31, a filter array 32, an image sensor 33, and an image processing device 34. The optical system 31 and the filter array 32 are disposed on the optical path of light incident from the subject 10b. In the example shown in FIG. 16A, the filter array 32 is disposed between the optical system 31 and the image sensor 33.

[0211] The image sensor 33 generates data of a compressed image 35 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 34 generates data representing a plurality of images corresponding one-to-one to the plurality of wavelength bands included in the target wavelength range based on the data of the compressed image 35 generated by the image sensor 33. Here, the number of wavelength bands included in the target wavelength range is assumed to be N (N is an integer of 4 or more). In the following description, the N images generated based on the compressed image 35 are referred to as a restored image 36W. １ , 36W ２ , ..., 36W Ｎ These may be collectively referred to as "hyperspectral images 36."

[0212] The filter array 32 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., different wavelength dependencies of light transmittance. The filter array 32 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 32 is called "encoding," and the filter array 32 is also called a "coding mask."

[0213] 16A, the filter array 32 is disposed near or directly above the image sensor 33. Here, "near" means close enough that a relatively clear image of light from the optical system 31 is formed on the surface of the filter array 32. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 32 and the image sensor 33 may be integrated.

[0214] The optical system 31 includes at least one lens. Although the optical system 31 is shown as a single lens in Fig. 16A, the optical system 31 may be a combination of multiple lenses. The optical system 31 forms an image on the imaging surface of the image sensor 33 via the filter array 32.

[0215] The filter array 32 may be disposed away from the image sensor 33. Figures 16B to 16D are diagrams showing configuration examples of the camera 30 in which the filter array 32 is disposed away from the image sensor 33. In the example shown in Figure 16B, the filter array 32 is disposed between the optical system 31 and the image sensor 33 and at a position away from the image sensor 33. In the example shown in Figure 16C, the filter array 32 is disposed between the subject 10b and the optical system 31. In the example shown in Figure 16D, the camera 30 includes two optical systems 31A and 31B, with the filter array 32 disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 32 and the image sensor 33.

[0216] The image sensor 33 is a monochrome photodetection device having a plurality of photodetection elements (also referred to herein as "pixels") arranged two-dimensionally. The image sensor 33 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or an infrared array sensor. The photodetection elements include, for example, photodiodes. The image sensor 33 does not necessarily have to be a monochrome sensor. For example, a color sensor may be used. A color sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light. A color sensor may further include a plurality of IR filters that transmit infrared light. Alternatively, a color sensor may include a plurality of transparent filters that transmit all red, green, and blue light. Using a color sensor can increase the amount of information related to wavelengths, thereby improving the accuracy of reconstruction of the hyperspectral image 36. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, but may also be the ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength range.

[0217] The image processing device 34 may be a computer including one or more processors and one or more storage media such as a memory. The image processing device 34 generates a decompressed image 36W based on the compressed image 35 acquired by the image sensor 33. １ , 36W ２ , ..., 36W Ｎ Generate data.

[0218] 17A is a diagram schematically illustrating an example of a filter array 32. The filter array 32 has a plurality of regions arranged two-dimensionally. In this specification, these regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0219] 17A, the filter array 32 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels of the image sensor 33, for example. The number of filters included in the filter array 32 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0220] FIG. 17B shows the wavelength band W included in the target wavelength range. １ , W ２ , ..., W Ｎ 17B is a diagram showing an example of the spatial distribution of the transmittance of each of the wavelength bands. In the example shown in FIG. 17B, the difference in the shading of each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 17B, the spatial distribution of the light transmittance differs depending on the wavelength band.

[0221] 17C and 17D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 32 shown in FIG. 17A . The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 32 varies depending on the region. However, it is not necessary for all regions to have different spectral transmittances. In the filter array 32, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 32 includes two or more filters with different spectral transmittances. In some examples, the number of spectral transmittance patterns of the multiple regions included in the filter array 32 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 32 may be designed so that the spectral transmittances of more than half of the regions are different.

[0222] 18A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 32. In the example shown in FIG. 18A, the spectral transmittance has multiple maximum values ​​P1 to P5 and multiple minimum values ​​for wavelengths within the target wavelength band W. In the example shown in FIG. 18A, the optical transmittance within the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 18A, the spectral transmittance within the wavelength band W２ , and the wavelength band W Ｎ－１ In this way, the spectral transmittance of each region is expressed as a wavelength band W １ , W ２ , ..., W Ｎ In the example of Fig. 18A, the maximum values ​​P1, P3, P4, and P5 are 0.5 or more.

[0223] As such, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 32 transmits a large amount of components in a certain wavelength range among the incident light, while not transmitting components in other wavelength ranges as much. For example, the transmittance of light in k wavelength bands out of N wavelength bands may be greater than 0.5, while the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light were white light that evenly contains all wavelength components of visible light, the filter array 32 would modulate the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and output this multi-wavelength light in a superimposed form.

[0224] FIG. 18B shows an example of the spectral transmittance shown in FIG. 18A in a wavelength band W １ , W ２ , ..., W Ｎ This figure shows the results of averaging the spectral transmittance for each wavelength band. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the transmittance value averaged for each wavelength band in this manner is referred to as the transmittance for that wavelength band. In this example, the transmittance is remarkably high in the three wavelength ranges that have maximum values ​​P1, P3, and P5. In particular, the transmittance exceeds 0.8 in the two wavelength ranges that have maximum values ​​P3 and P5.

[0225] In the examples shown in Figures 17A to 17D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, a grayscale transmittance distribution is not necessarily required. For example, a binary scale transmittance distribution may be employed in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two wavelength ranges out of multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most of the region" refers to approximately 80% or more.

[0226] A portion of all cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region is located within a wavelength band W included in the wavelength range W of interest. １ , W ２ , ..., W Ｎ The filter array 32 transmits light of each wavelength at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 32, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0227] Such data indicating the spatial distribution of the spectral transmittance of the filter array 32 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the image processing device 34. This data is used in the calculation processing described below.

[0228] The filter array 32 may be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, a microstructure containing metal, or a metasurface. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing metal layers may be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is different for each cell. This allows different spectral characteristics to be achieved for each cell. The use of a multilayer film allows for sharp rises and falls in the spectral transmittance. A configuration using organic materials can be achieved by containing different pigments or dyes in each cell or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each cell. A microstructure containing metal can be fabricated using plasmon effect-based spectral separation. A metasurface can be fabricated by microfabricating a dielectric material to a size smaller than the wavelength of the incident light. In this structure, the refractive index for the incident light is spatially modulated. Alternatively, incident light may be encoded by directly processing multiple pixels included in the image sensor 33 without using the filter array 32.

[0229] From the above, it can be said that camera 30 has a plurality of light-receiving regions with different photoresponse characteristics. When camera 30 includes filter array 32 including a plurality of filters with irregularly different light transmission characteristics, the plurality of light-receiving regions can be realized by image sensor 33 disposed adjacent to or directly above filter array 32. In this case, the photoresponse characteristics of the plurality of light-receiving regions are determined based on the light transmission characteristics of the plurality of filters included in filter array 32.

[0230] Alternatively, if the camera 30 does not include the filter array 32, the multiple light-receiving regions may be realized by, for example, an image sensor 33 in which multiple pixels are directly processed so that their photoresponse characteristics are irregularly different from one another. In this case, the photoresponse characteristics of the multiple light-receiving regions are determined based on the photoresponse characteristics of the multiple pixels included in the image sensor 33.

[0231] The above-mentioned multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface can encode incident light if it is configured so that the spectral transmittance varies depending on the position within a two-dimensional plane.Therefore, the above-mentioned multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface does not need to be configured so that multiple filters are arranged in an array.

[0232] Next, an example of signal processing by the image processing device 34 will be described. The image processing device 34 reconstructs a multi-wavelength hyperspectral image 36 based on the compressed image 35 output from the image sensor 33 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 32. Here, "multi-wavelength" means a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a normal color camera, for example. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the "number of bands." Depending on the application, the number of bands may exceed 100.

[0233] The data to be obtained is the data of the hyperspectral image 36, and this data is denoted as f. If the number of bands is N, f is the data f of the N image bands. １ , f ２ , ..., f Ｎ Here, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is m and the number of pixels in the y direction is n, then the image data f １ , f ２ , ..., f Ｎ Each of the elements has m×n pixel values. Therefore, the data f is data with m×n×N elements. On the other hand, the data g of the compressed image 35 obtained by encoding and multiplexing using the filter array 32 is two-dimensional data including m×n pixel values ​​corresponding to the m×n pixels. The data g can be expressed by the following equation (1):

[0234]

[0235] In equation (1), f represents the hyperspectral image data expressed as a one-dimensional vector. １ , f２ , ..., f Ｎ Each of these has m×n elements. Therefore, the vector on the right side is a one-dimensional vector with m×n×N rows and 1 column. In equation (1), the data g of the compressed image 35 is converted and expressed as a one-dimensional vector with m×n rows and 1 column. The matrix H is １ , f ２ , ..., f Ｎ represents a transformation in which each wavelength band is encoded with different encoding information, intensity-modulated, and then added together. Therefore, H is an m×n row and m×n×N column matrix. Equation (1) can also be expressed as follows:

[0236] g = (pg １１ ...pg １ｎ ...pg ｍ１ ...pg ｍｎ ) Ｔ = H(f １ ...f Ｎ ) Ｔ Here, pg ｉｊ represents the pixel value of the i-th row and j-th column of the compressed image 35.

[0237] Given the vector g and matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements m×n×N of the desired data f is greater than the number of elements m×n of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the image processing device 34 utilizes the sparsity of the image contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

[0238]

[0239] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means that f that minimizes the sum of the first and second terms is found. The function in the parentheses in Equation (2) is called the evaluation function. The image processing device 34 can converge the solution through recursive iterative calculations and calculate the f that minimizes the evaluation function as the final solution f'.

[0240] The first term in the parentheses in Equation (2) represents an operation to calculate the sum of squares of the difference between the acquired data g and Hf, which is obtained by transforming f in the estimation process using matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation can obtain stable estimated data that suppresses the influence of noise in the observed data g. The sparsity of the object 10b in the space of each regularization term varies depending on the texture of the object 10b. A regularization term that makes the texture of the object 10b sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0241] In the configurations of FIGS. 16B and 16C , the image encoded by the filter array 32 is acquired in a blurred state on the imaging surface of the image sensor 33. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned matrix H, a hyperspectral image 36 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to one pixel on an image spreads due to blurring to a k×k pixel region around the pixel, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the influence on the pixel values ​​of each pixel within that region. The hyperspectral image 36 can be reconstructed by reflecting the influence of blurring of the encoding pattern by the PSF in the matrix H. The filter array 32 may be positioned at any position, but a position where the encoding pattern of the filter array 32 does not become too diffused and disappear can be selected.

[0242] Through the above processing, a hyperspectral image 36 can be restored based on the compressed image 35 of the subject 10b acquired by the image sensor 33 via the filter array 32. Details of the method for restoring the hyperspectral image 36 are disclosed in Patent Document 2, the entire disclosure of which is incorporated herein by reference.

[0243] [10. Supplementary Notes] The above description of the embodiments discloses the following techniques.

[0244] (Technology 1) A method including: acquiring an image generated by photographing a first subject as a first image, an image generated by photographing a second subject as a second image containing information of a plurality of wavelength bands, and acquiring the first image as a correction image for correcting the second image; and determining whether the first image satisfies suitability conditions for the correction image based on pixel values ​​of the first image.

[0245] This method allows for more accurate acquisition of spectral information of the object being analyzed by preventing inappropriate corrections.

[0246] (Technology 2) The method according to Technology 1, wherein the correction of the second image based on the first image is black subtraction.

[0247] This method can prevent inappropriate blackouts when shooting in dark conditions.

[0248] (Technology 3) The method according to Technology 2, wherein determining whether the first image satisfies the suitability condition is determining that the first image satisfies the suitability condition when a condition for determining that a pixel value of the first image is small is satisfied.

[0249] This method makes it possible to determine whether or not an image was captured in a darkened state.

[0250] (Technology 4) The method according to Technology 1, wherein the correction of the second image based on the first image is whiteboard correction.

[0251] This method makes it possible to prevent inappropriate white board correction when the white board is not photographed correctly.

[0252] (Technology 5) The method according to Technology 4, wherein determining whether the first image satisfies the suitability condition is determining that the suitability condition is satisfied when a condition for determining that the spatial distribution of pixel values ​​of the first image is constant is satisfied.

[0253] This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

[0254] (Technology 6) The method according to Technology 4, wherein determining whether the first image satisfies the suitability condition comprises determining whether the first image satisfies the suitability condition by comparing spectral information acquired from the first image with pre-stored spectral information.

[0255] This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

[0256] (Technology 7) The method according to Technology 4, wherein the first image is a compressed image in which information of a plurality of wavelength bands about the first subject is compressed, and determining whether the first image satisfies the suitability condition is determining whether the first image satisfies the suitability condition based on a histogram of pixel values ​​of the compressed image.

[0257] This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

[0258] (Technology 8) The method according to Technology 2 or 4, wherein determining whether the first image satisfies the suitability condition is determining whether the first image satisfies the suitability condition by performing edge detection on the first image.

[0259] This method makes it possible to determine whether an image was captured in a light-blocking state or whether an image of a white board was captured correctly.

[0260] (Technology 9) The method according to any one of technologies 1 to 8, further comprising displaying an error on a display device if the first image does not satisfy the suitability condition.

[0261] In this way, the user can know that the first image does not meet the eligibility criteria.

[0262] (Technology 10) The method according to Technology 9, wherein the error prompts the user to confirm whether the whiteboard has been correctly photographed as the first subject.

[0263] This method allows the user to check whether the whiteboard is being photographed correctly.

[0264] (Technology 11) The method according to Technology 9, wherein the error prompts a user to check a light blocking state when photographing the first subject.

[0265] This method allows the user to check the light blocking state.

[0266] (Technology 12) The method according to any one of technologies 1 to 11, further comprising: acquiring the second image; and correcting the second image based on the first image if the first image satisfies the suitability condition.

[0267] This method allows the second image to be appropriately corrected based on the first image.

[0268] (Technology 13) A method comprising: acquiring an image containing information of multiple wavelength bands generated by photographing a scene including a first subject and a second subject; designating a sub-image corresponding to the first subject generated based on the image as a first sub-image, and a sub-image corresponding to the second subject generated based on the image as a second sub-image; and determining, based on pixel values ​​of the first sub-image, whether the first sub-image satisfies suitability conditions as a correction image for correcting the second sub-image.

[0269] This method allows for more accurate acquisition of spectral information of the object being analyzed by preventing inappropriate corrections.

[0270] (Technology 14) The method according to Technology 13, wherein the correction of the second sub-image based on the first sub-image is a whiteboard correction.

[0271] This method makes it possible to prevent inappropriate white board correction when the white board is not photographed correctly.

[0272] (Technology 15) A processing circuit for executing the method according to any one of techniques 1 to 14.

[0273] This processing circuitry allows for more accurate acquisition of spectral information of the object being analyzed by preventing inappropriate corrections.

[0274] (Other Form 1) An object other than a whiteboard may be used as the object to be corrected. The object may have spectral reflectance characteristics such that, for example, the reflectance of light in a specific wavelength range is high and the reflectance of light in another specific wavelength range is low. For example, the spectral reflectance characteristics of the object may be stored in memory 54. Based on the stored spectral reflectance characteristics, correction may be performed to reduce the influence of the spectral shape of the irradiated light, the irradiance distribution during shooting, lens vignetting, non-uniform sensitivity of the image sensor, and the like.

[0275] (Other Form 2) The first image may be an image including information on three or fewer wavelength bands, for example. For example, the second image including information on multiple wavelength bands may be an image generated based on a wavelength band corresponding to red, a wavelength band corresponding to green, and a wavelength band corresponding to blue.

[0276] The second image including information on multiple wavelength bands may be an image including information on three or fewer wavelength bands, and the corrected image generated by correcting the second image using the first image may be an image including information on three or fewer wavelength bands.

[0277] The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength or high-resolution images. The technology disclosed herein can also be applied to, for example, biomedical, cosmetic, and other sensing applications, food foreign matter and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

[0278] 10a1, 10a2, 10b Object 20 Light source 30 Hyperspectral camera 31, 31A, 31B Optical system 32 Filter array 33 Image sensor 34 Image processing device 35 Compressed image 36W １ ~36W Ｎ Reconstructed image 38 Image 40 Display device 42 Input UI 44 Display UI 50 Processing device 52 Processing circuit 54 Memory 56 Storage device 60 Stage 70 Support 80 Adjustment device

Claims

1. A method comprising: acquiring an image generated by photographing a first subject as a first image, an image generated by photographing a second subject as a second image including information of a plurality of wavelength bands, and acquiring the first image as a correction image for correcting the second image; and determining whether the first image satisfies suitability conditions for use as the correction image based on pixel values ​​of the first image.

2. The method according to claim 1, wherein the correction of the second image based on the first image is black subtraction.

3. The method according to claim 2, wherein determining whether the first image satisfies the suitability condition comprises determining that the first image satisfies the suitability condition if a condition for judging that the pixel value of the first image is small is satisfied.

4. The method of claim 1, wherein the correction of the second image based on the first image is a whiteboard correction.

5. The method according to claim 4, wherein determining whether the first image satisfies the suitability condition comprises determining that the first image satisfies the suitability condition if a condition is satisfied in which the spatial distribution of pixel values ​​of the first image is judged to be constant.

6. The method according to claim 4, wherein determining whether the first image satisfies the suitability condition comprises determining whether the first image satisfies the suitability condition by comparing spectral information acquired from the first image with pre-stored spectral information.

7. The method according to claim 4, wherein the first image is a compressed image in which information of multiple wavelength bands about the first subject is compressed, and determining whether the first image satisfies the suitability condition comprises determining whether the first image satisfies the suitability condition based on a histogram of pixel values ​​of the compressed image.

8. The method according to claim 2 or 4, wherein determining whether the first image satisfies the suitability condition comprises determining whether the first image satisfies the suitability condition by detecting edges in the first image.

9. The method of any one of claims 1 to 7, further comprising causing a display device to display an error if the first image does not satisfy the suitability condition.

10. The method according to claim 9, wherein the error prompts the user to check whether the whiteboard is correctly photographed as the first object.

11. The method of claim 9, wherein the error prompts a user to check for a shading condition when photographing the first object.

12. The method of any one of claims 1 to 7, further comprising: acquiring the second image; and, if the first image satisfies the suitability condition, correcting the second image based on the first image.

13. A method comprising: acquiring an image including information of a plurality of wavelength bands generated by photographing a scene including a first subject and a second subject; designating a sub-image corresponding to the first subject generated based on the image as a first sub-image, designating a sub-image corresponding to the second subject generated based on the image as a second sub-image, and determining, based on pixel values ​​of the first sub-image, whether the first sub-image satisfies suitability conditions for use as a correction image for correcting the second sub-image.

14. The method of claim 13, wherein the correction of the second sub-image based on the first sub-image is a whiteboard correction.

15. A processing circuit for carrying out the method according to any one of claims 1 to 7.

Images