Imaging system and imaging method

Abstract

This system provides an imaging system that can easily and accurately classify objects. [Solution] The local server 2 calibrates the local inference model 201 based on the optical system of the imaging device 1. The processor 22 inputs an encoded image 202, which is obtained by compressing multiple spectral images of the object taken in each of multiple wavelength channels, into the calibrated local inference model 201 and obtains a classification result that classifies the object.

Description

Technical Field

[0001] The present disclosure relates to an imaging system and an imaging method.

Background Art

[0002] In the field of inspection industries, in recent years, an imaging technique called hyperspectral imaging has attracted attention and is used for industrial material classification, agricultural monitoring, geological surveys, etc.

[0003] In hyperspectral imaging, for the reflected light from an object, a spectroscopic technique having high wavelength resolution in a wide wavelength range from the ultraviolet region to the near-infrared region is applied, and an image for each wavelength channel showing the object is generated as a hyperspectral image (hyperspectral data cube). Then, based on the hyperspectral image, the object is classified according to the presence or absence of abnormalities or the material, etc. For example, Patent Document 1 discloses a technique of inputting image data generated by decomposing and imaging the light from a monitoring object for each wavelength using a hyperspectral camera into an inference model and inferring the abnormal state of the monitoring object according to the algorithm of the inference model.

[0004] A hyperspectral image has a data size of H×W×C, where H is the number of pixels in the vertical direction of the image, W is the number of pixels in the horizontal direction of the image, and C is the number of wavelength channels. Therefore, in hyperspectral imaging, there is a problem that the amount of data becomes large. This problem becomes particularly prominent when high-definition images or many wavelength channels are required. For example, a hyperspectral image may have a data amount 100 times or more that of an RGB image acquired by a normal RGB camera.

[0005] Furthermore, hyperspectral imaging techniques such as Coded Aperture Snapshot Spectral Imaging (CASSI) and snapshot techniques using photonic crystal filters are also known. These techniques allow for the acquisition of multiple channel information related to multiple wavelength channels in a single image, improving image acquisition speed compared to conventional point scan or line scan hyperspectral imaging. However, since reconstructing a hyperspectral image from a coded image (which is an image with compressed multiple channel information) is necessary for object classification, reducing data volume remains a challenge. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-137973 [Overview of the project] [Problems that the invention aims to solve]

[0007] Hyperspectral images, with their rich information content, contribute to improved classification and recognition accuracy, but they also present a trade-off: increased system load due to the enormous amount of data. Therefore, there is a need to reduce the amount of data while leveraging the advantages of hyperspectral images with their rich information content.

[0008] Furthermore, while imaging devices that support hyperspectral imaging using snapshot technology have simpler hardware compared to conventional imaging devices that support hyperspectral imaging using point scan or line scan technology, their signal processing is more complex. Therefore, detailed calibration of imaging devices that support hyperspectral imaging using snapshot technology is necessary to accurately classify objects.

[0009] The purpose of this disclosure is to provide an imaging system and imaging method that can easily and accurately classify objects without increasing the amount of data. [Means for solving the problem]

[0010] An imaging system according to one aspect of the present disclosure includes an imaging device that acquires an encoded image obtained by compressing a plurality of spectral images obtained by capturing a subject in each of a plurality of wavelength channels, and an imaging device that classifies the subject based on the encoded image, wherein the imaging device includes a storage unit that stores a local inference model that takes the encoded image as input and outputs a classification result for classifying the subject, and a calculation unit that calibrates the local inference model based on the optical system of the imaging device, and inputs a target encoded image obtained by capturing the object to be classified as the object to be classified to the calibrated local inference model, which is the calibrated inference model, and acquires a classification result for classifying the object. [Effects of the Invention]

[0011] According to this disclosure, it will be possible to improve classification accuracy and recognition accuracy without increasing the amount of data. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows a hyperspectral imaging system according to a first embodiment of the present disclosure. [Figure 2] This is a disassembled perspective view of the imaging device. [Figure 3] This is a cross-sectional view taken along line AA in Figure 2. [Figure 4] This figure shows an example of a local server configuration. [Figure 5] This diagram shows an example of a cloud server configuration. [Figure 6] This figure shows an example of data stored on a local server and a cloud server. [Figure 7]This is a diagram for explaining a machine learning model and its inputs and outputs. [Figure 8] This is a diagram for explaining a local inference model. [Figure 9] This is a flowchart for explaining an example of a classification process for generating a classified image. [Figure 10] This is a diagram for explaining an example of a calibration method. [Figure 11] This is a flowchart for explaining an example of a calibration process. [Figure 12] This is a flowchart for explaining an example of a calibration process.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] (First Embodiment) FIG. 1 is a diagram showing a hyperspectral imaging system according to a first embodiment of the present disclosure. The hyperspectral imaging system shown in FIG. 1 includes an imaging device 1, a local server 2, and a cloud server 3. The local server 2 and the cloud server 3 are communicably connected to each other via a network 4. There may be a plurality of imaging devices 1 and local servers 2. In the example of FIG. 1, the local server 2 is provided for each imaging device 1, but one local server 2 may be provided for a plurality of imaging devices 1.

[0015] The imaging device 1 is a hyperspectral camera corresponding to snapshot technologies using coded aperture snapshot spectroscopy or photonic crystal filters. The imaging device 1 images an object 10 that is a subject to be classified, and acquires an encoded image that is a single image obtained by compressing a plurality of spectral images of the object 10 captured in each of a plurality of wavelength channels.

[0016] In this embodiment, the object 10 is conveyed by the conveyor belt 20, and the imaging device 1 acquires an encoded image by imaging the object 10 conveyed to a specific position by the conveyor belt 20. The encoded image may show a plurality of objects 10. Each of the plurality of imaging devices 1 may image a different type of object 10. Further, the wavelength channel is appropriately set according to the type of the object 10 or the like, and can be, for example, a wavelength included in the visible light region, the near infrared region, or the mid infrared region.

[0017] The local server 2 is a processing device that classifies the object 10 based on the encoded image acquired by the imaging device 1. The classification of the object 10 may be a classification based on the material or the material, or may be a classification based on various states such as an abnormal state.

[0018] Specifically, the local server 2 can classify the object without generating a hyperspectral image (hyperspectral data cube) that explicitly includes a plurality of spectral images from the encoded image. However, the local server 2 may generate a hyperspectral image from the encoded image. Even in this case, the hyperspectral image is not stored in the local server 2 but is stored in the cloud server 3.

[0019] Also, in this embodiment, the local server 2 generates a classification image that shows the object and assigns a label corresponding to the classification to the object as a classification result of classifying the object 10. The classification image is, for example, an image representing the object in a color corresponding to the classification.

[0020] The cloud server 3 functions as a management device that manages the local server 2.

[0021] FIG. 2 and FIG. 3 are diagrams schematically showing an example of the configuration of the imaging device 1. Specifically, FIG. 2 is an exploded perspective view of the imaging device 1, and FIG. 3 is a cross-sectional view taken along line AA in FIG. 2.

[0022] As shown in Figures 2 and 3, the imaging device 1 has a metasurface 11 and an image sensor 12.

[0023] The metasurface 11 is positioned closer to the object 10 than the image sensor 12. Light from the object 10 is incident on the metasurface 11 through the optical system (lens, etc.) of the imaging device 1.

[0024] The metasurface 11 has a plurality of optical filters 111 that transmit light of different wavelengths. Specifically, the plurality of optical filters 111 are arranged in a matrix on the metasurface 11. More specifically, filter units 112, each containing a plurality of optical filters 111 arranged in a matrix, are arranged in a matrix. The type of optical filter 111 is not particularly limited, but in this embodiment, it is a filter having a metaatomic structure in which structures (e.g., protrusions) with dimensions smaller than the wavelength of light transmitted by the optical filter 111 are arranged in a row.

[0025] In the image sensor 12, multiple pixels 121 that receive incident light and convert it into electrical signals are arranged in a matrix.

[0026] The metasurface 11 and the image sensor 12 are connected such that each pixel 121 of the image sensor 12 corresponds to each optical filter 111 of the metasurface 11, and each pixel 121 receives light that has passed through the optical filter 111 corresponding to itself. As a result, the image sensor 12 receives light with different wavelengths that has passed through each of the multiple optical filters 111 all at once, so the imaging device 1 can acquire an encoded image by compressing multiple spectral images with wavelength channels using the wavelengths of the transmitted light.

[0027] Figure 4 shows an example of the configuration of local server 2. As shown in Figure 4, local server 2 has a storage device 21, a processor 22, memory 23, a communication device 24, an input device 25, and a display device 26, which are connected via a bus 27.

[0028] The storage device 21 is a device that records data in a writable and readable manner, and is a storage unit that stores a program that defines the operation of the processor 22, and various information used and generated by that program. The processor 22 is an arithmetic unit that reads the program recorded in the storage device 21 into the memory 23 and uses the memory 23 to execute processing according to the program. The communication device 24 is connected to the external devices, the imaging device 1 and the cloud server 3, in a communicative manner, and transmits and receives various information with these external devices. The input device 25 receives various information from the operator of the local server 2, etc. The display device 26 displays various information.

[0029] Figure 5 shows an example of the configuration of the cloud server 3. As shown in Figure 5, the cloud server 3 has a storage device 31, a processor 32, memory 33, a communication device 34, an input device 35, and a display device 36, which are connected via a bus 37.

[0030] The storage device 31 is a device that records data in a writable and readable manner, and is a management storage unit that stores a program that defines the operation of the processor 32, and various information used and generated by that program. The processor 32 is a management unit that reads the program recorded in the storage device 31 into the memory 33 and uses the memory 33 to execute processing according to the program. The communication device 34 is connected to the external devices, the imaging device 1 and the local server 2, in a communicative manner, and transmits and receives various information with these external devices. The input device 35 receives various information from operators of the cloud server 3, etc. The display device 36 displays various information.

[0031] Figure 6 shows an example of data stored in the local server 2 and the cloud server 3. In this embodiment, this data is stored in the storage device 21 or 31, but at least a portion of it may be stored in memory 23 or 32.

[0032] Local server 2 has a local inference model 201, encoded image 202, classified image 203, effective transmittance matrix data 204, and reconstruction model 205. Cloud server 3 has a global inference model 301, hyperspectral image 302, and classified image 203.

[0033] The local inference model 201 is a pre-trained machine learning model that takes an encoded image 202, which is an image acquired by the imaging device 1, as input and outputs a classified image 203, which is the classification result of classifying the object 10.

[0034] The effective transmittance matrix data 204 is matrix data showing the effective transmittance of the optical system of the imaging device 1 for each pixel of the image sensor 12 of the imaging device 1, and is used to calibrate the local inference model 201.

[0035] The reconstruction model 205 is a pre-trained machine learning model that takes the encoded image 202 as input and outputs the hyperspectral image 302. The hyperspectral image 302 output from the reconstruction model 205 is stored on the cloud server 3, not on the local server 2. In this embodiment, it is not always necessary to output the hyperspectral image 302.

[0036] Furthermore, if multiple imaging devices 1 are provided for a single local server 2, a local inference model 201, effective transmittance matrix data 204, and reconstruction model 205 are prepared for each imaging device 1.

[0037] The global inference model 301 is a pre-trained machine learning model that takes a hyperspectral image 302 as input and outputs a classified image 303. A separate global inference model 301 is provided for each of the 10 types of objects.

[0038] Each of the machine learning models, the local inference model 201, the reconstruction model 205, and the global inference model 301, is implemented, for example, as a deep learning model.

[0039] Figure 7 illustrates the machine learning model used in the hyperspectral imaging system in this embodiment and the data that serves as the input and output for the machine learning model. Figure 8 illustrates the local inference model 201.

[0040] In this embodiment, as described above, the encoded image 202 is generated by the imaging device 1, which supports encoded aperture snapshot spectral imaging and snapshot technology using a photonic crystal filter.

[0041] As shown in Figures 7 and 8, the local inference model 201 stored on the local server 2 generates a classification image 203 that classifies the object 10 from the encoded image 202 without explicitly generating a hyperspectral image 302.

[0042] Furthermore, the hyperspectral image 302 can also be directly acquired by using an imaging device compatible with hyperspectral imaging using point scan or line scan technology, instead of the imaging device 1 of this embodiment which uses coded aperture snapshot spectral imaging or snapshot technology using a photonic crystal filter. However, the hyperspectral image 302 is an image composed of multiple spectral images 302a taken of the object 10 in each of multiple wavelength channels, and the amount of data is large, often several hundred MB. In contrast, the coded image 202 has a small amount of data, about a few MB. For this reason, in this embodiment, the amount of data stored in the local server 2 can be reduced. In addition, the hyperspectral image 302 can be converted to the coded image 202 by convolution that takes into account the optical system of the imaging device 1. This conversion is irreversible, but the reconstruction model 205 stored in the local server 2 can infer and generate the hyperspectral image 302 from the coded image 202 using machine learning.

[0043] Furthermore, as shown in Figure 7, the local inference model 201 can also be expressed as a combination of a reconstruction model 205 that generates a hyperspectral image 302 from an encoded image 202 and a global inference model 301 that generates a classification image 203 from the hyperspectral image 302, or as a combination of parts thereof. When the reconstruction model 205 and the global inference model 301 go through the low-dimensional feature representation of the hyperspectral image 302, the part of the reconstruction model 205 is the part that generates the feature representation from the encoded image 202, and the part of the global inference model 301 is the part that generates the classification image 203 from the feature representation.

[0044] The reconstruction model 205 is independent of the object 10 but depends on the optical system of the imaging device 1. On the other hand, the global inference model 301 is independent of the object 10 but does not depend on the optical system of the imaging device 1. Therefore, the local inference model 201 can be expressed as a combination of the reconstruction model 205 and the global inference model 301, and thus can be expressed as a combination of a part that depends on the optical system of the imaging device 1 and a part that depends on the object 10.

[0045] In this embodiment, the portion of the local inference model 201 that depends on the optical system of the imaging device 1 is calibrated according to the configuration of the optical system, and the portion of the local inference model 201 that depends on the object 10 is changed according to the type of object 10.

[0046] Figure 9 is a flowchart illustrating an example of a classification process that generates classified images.

[0047] First, the processor 22 of the local server 2 acquires the encoded image from the imaging device 1 via the communication device 24 and saves it as the encoded image 202 in the storage device 21 (step S101).

[0048] Next, the processor 22 reads the local inference model 201 from the storage device 21. The processor 22 inputs the encoded image 202 into the local inference model 201, obtains the classification image 203 output from the local inference model 201, and saves it to the storage device 21 (step S102).

[0049] Furthermore, the processor 22 reads the reconstructed model 205 from the storage device 21. The processor 22 inputs the encoded image 202 to the reconstructed model 205, obtains the hyperspectral image 302 output from the reconstructed model 205, sends the hyperspectral image 302 and the classification image 203 to the cloud server 3, stores the hyperspectral image 302 and the classification image 203 in the storage device 31 of the cloud server 3 (step S103), and terminates the classification process.

[0050] The process in step S103 may be performed under specific conditions. For example, the process in step S103 may be performed when instructed by the user, or when the imaging device 1 has been used for a certain period of time or longer. In this case, if the conditions are not specific, the processor 22 will skip the process in step S103 and terminate the classification process after completing the process in step S102. Furthermore, the hyperspectral images stored in the cloud server 3 can be used by any local server 2 or user as needed.

[0051] Next, we will explain the calibration of the local inference model 201. Since the local inference model 201 can be expressed as a combination of a part that depends on the optical system of the imaging device 1 and a part that depends on the object 10, we will first explain the calibration of the part that depends on the optical system of the imaging device 1.

[0052] In imaging devices using coded aperture patterns or photonic crystal filters, these are not changed during operation, but their characteristics can vary due to manufacturing variations. Therefore, the local inference model 201 is calibrated during the initial setup phase.

[0053] The calibration process proceeds as follows: an encoded image is captured using the encoded aperture pattern or photonic crystal filter; the parameters of the existing local inference model 201 are adjusted so that the encoded image outputs the desired inference result; and the local inference model 201 updated with the adjusted parameters is stored in the memory device 21, after which the calibration process is completed.

[0054] Furthermore, the classification accuracy can be improved by calibrating the local inference model 201 with the optical system (lenses, etc.) of the imaging device 1, particularly the point spread function (PSF) determined according to that optical system. The point spread function is a function that shows the impulse response of the optical system of the imaging device 1, that is, the response to a point light source, and more specifically, it is a function that shows how a single point in the subject is represented in the captured image. When the imaging device 1 is a hyperspectral camera as in this embodiment, the point spread function includes spatial parameters and wavelength channel parameters.

[0055] Figure 10 illustrates an example of a calibration method for calibrating the local inference model 201 using a point image distribution function.

[0056] The local inference model 201 is calibrated according to the point image distribution function of the optical system of the imaging device 1. In Figure 10, lenses A to C are shown as the optical system of the imaging device 1, and the point image distribution functions 401A to 401C corresponding to lenses A to C are shown. The point image distribution functions 401 to 401 may be calculated by measuring the response of lenses A to C to a point light source, but this is very time-consuming, so in this embodiment, they are calculated by simulating them using wave optics from a lens model that shows the configuration of lenses A to C (such as the dimensions and type of lenses).

[0057] The local inference model 201 is calibrated based on the point image distribution functions 401A to 401C. Specifically, the effective transmittance matrix data 204 corresponding to the configuration of the metasurface 11 of the imaging device 1 and the image sensor 12 is corrected by the point image distribution functions 401A to 401C, and the local inference model 201 is calibrated according to the corrected effective transmittance matrix data 204.

[0058] Figure 11 is a flowchart illustrating an example of a calibration process that calibrates the local inference model 201 using the point image distribution function.

[0059] In the calibration process, the processor 22 first acquires a lens model that shows the configuration of the optical system of the imaging device 1 (step S201). The lens model may be pre-stored in the storage device 21, or it may be input from an operator of the local server 2 via the communication device 24 or input device 25.

[0060] The processor 22 obtains the point image distribution function of the optical system of the imaging device 1 by simulating based on the lens model (step S202).

[0061] The processor 22 corrects the effective transmittance matrix data 204 based on the acquired point image distribution function and calibrates the local inference model 201 by adjusting its parameters based on the corrected effective transmittance matrix data 204 (step S203). At this time, the processor 22 may also calibrate the reconstruction model 205 based on the corrected effective transmittance matrix data 204 and calibrate the optical system-dependent portion of the local inference model 201 according to the calibrated reconstruction model 205.

[0062] The processor 22 stores the calibrated local inference model 201 in the memory device 21 (step S204) and terminates the calibration process.

[0063] Next, we will explain the calibration of the local inference model 201 with respect to the part that depends on the object 10. Figure 12 is a flowchart illustrating an example of a modification process that changes the local inference model 201 according to the type of object 10.

[0064] The part that depends on object 10 is based on the global inference model 301, which is a model that does not depend on the optical system of imaging device 1. Therefore, it is a common model that does not depend on the imaging device, and by managing it centrally, it becomes easy to maintain consistency between various models, such as version control.

[0065] In the change processing, when the cloud server 3's processor 32 receives notification from the user via the input device 35 or local server 2 that the type of object 10 to be classified by local server 2 has changed, it retrieves a global inference model 301 corresponding to the changed type from the storage device 31 and distributes it to local server 2 (step S301).

[0066] When the processor 22 of the local server 2 receives the global inference model 301 via the communication device 24, it modifies the part of the local inference model 201 that depends on the object 10 based on the global inference model 301 (step S302), and then terminates the modification process.

[0067] As described above, according to this embodiment, the processor 22 of the local server 2 calibrates the local inference model 201 based on the optical system of the imaging device 1. The processor 22 inputs the encoded image to the calibrated local inference model 201 and obtains a classification result that classifies the object 10. Since it is not necessary to generate hyperspectral images within the local server 2, the storage capacity of the local server 2 can be reduced.

[0068] Furthermore, according to this embodiment, the processor 22 inputs the encoded image to the reconstruction model 205 to obtain a hyperspectral image of the object 10 and stores it in the cloud server 3. This makes it possible to reduce the storage capacity of the local server 2 while making the hyperspectral image accessible when needed.

[0069] Furthermore, according to this embodiment, the processor 22 calculates the point image distribution function based on the configuration of the optical system of the imaging device 1. Therefore, the point image distribution function can be easily calculated, and by calibrating the local inference model 201, it becomes possible to obtain classification results with good accuracy.

[0070] Furthermore, in this embodiment, if the type of object 10 is changed, the local inference model 201 is updated based on the global inference model 301 corresponding to that type. Therefore, even if the type of object 10 is changed, the local inference model 201 can be easily updated according to that type.

[0071] Furthermore, in this embodiment, the imaging device 1 includes a metasurface 11 having a plurality of optical filters that transmit light of different wavelengths, and an image sensor 12 having a plurality of pixels that receive light transmitted through the metasurface 11. In this case, it becomes possible to easily manufacture the imaging device 1 to which spectral imaging using snapshot technology is applied.

[0072] The embodiments of the Disclosure described above are illustrative for illustrative purposes and are not intended to limit the scope of the Disclosure to those embodiments only. Those skilled in the art can implement the Disclosure in various other forms without departing from the scope of the Disclosure. [Explanation of symbols]

[0073] 1: Imaging device 2: Local server 3: Cloud server 4: Network 10: Object 11: Metasurface 12: Image sensor 20: Conveyor belt 21, 31: Storage device 22, 32: Processor 23, 33: Memory 24, 34: Communication device 25, 35: Input device 26, 36: Display device 111: Optical filter 112: Filter unit 121: Pixel 201: Local inference model 202: Encoded image 203: Classified image 204: Effective transmittance matrix data 205: Reconstruction model 301: Global inference model 302: Hyperspectral image

Claims

1. An imaging system comprising: an imaging device that acquires an encoded image obtained by compressing multiple spectral images of a subject captured in each of multiple wavelength channels; and a processing device that classifies the subject based on the encoded image, The aforementioned processing apparatus is A storage unit that stores a local inference model that takes the encoded image as input and outputs a classification result of classifying the subject, An imaging system comprising: a calculation unit that calibrates the local inference model based on the optical system of the imaging device, inputs the encoded image of the object to be classified, which is the subject to be classified, to the calibrated local inference model, which is the calibrated inference model, and obtains a classification result for classifying the object.

2. The imaging system according to claim 1, wherein the calculation unit calculates a point image distribution function of the optical system based on the configuration of the optical system of the imaging device, and calibrates the local inference model based on the point image distribution function.

3. The storage unit further stores a reconstruction model for reconstructing a hyperspectral image having the plurality of spectral images from the encoded image, The imaging system according to claim 1, wherein the calculation unit inputs the target encoded image to the reconstruction model, obtains the hyperspectral image of the target object, and stores it in a management device that is communicably connected to the processing device.

4. The imaging system according to claim 3, wherein the management device is a cloud server.

5. The imaging system according to claim 3, wherein the calculation unit calibrates the reconstruction model based on the optical system and calibrates the local inference model based on the calibrated reconstruction model.

6. The imaging system has the management device, The aforementioned control device is A management memory unit that stores a global inference model for classifying the subject from the hyperspectral image for each type of subject, The system includes a management unit that, when the type of object changes, distributes the global inference model corresponding to the type of object to the processing unit, The imaging system according to claim 3, wherein the calculation unit updates the local inference model based on the distributed global inference model and the reconstruction model.

7. The imaging device is A metasurface having multiple optical filters that transmit light of different wavelengths from each other, The imaging system according to claim 1, further comprising an image sensor having a plurality of pixels that receive light transmitted through the metasurface.

8. The imaging system according to claim 7, wherein the optical filter has a metaatomic structure in which structures with dimensions smaller than the wavelength of light transmitted by the optical filter are arranged in a row.

9. An imaging method using an imaging system comprising: an imaging device that acquires an encoded image obtained by compressing multiple spectral images of a subject captured in each of multiple wavelength channels; and a processing device that classifies the subject based on the encoded image, The processing device stores a local inference model that takes the encoded image as input and outputs a classification result that classifies the subject, The processing device calibrates the local inference model based on the optical system of the imaging device. An imaging method comprising: an imaging device inputting a target encoded image, which is an encoded image of an object that is the subject to be classified, to a calibrated inference model, which is a calibrated local inference model, and obtaining a classification result that classifies the object.

Images