Imaging system and imaging method

Abstract

Provided is an imaging system capable of easily and accurately classifying objects.　A local server calibrates a local inference model on the basis of the optical system of an imaging device. A processor inputs, to the calibrated local inference model, an encoded image in which a plurality of spectral images obtained by imaging an object on each of a plurality of wavelength channels are compressed, and acquires a classification result obtained by classifying the object.

Description

Imaging System and Imaging Method

[0001] The present disclosure relates to an imaging system and an imaging method.

[0002] In the field of inspection industries, in recent years, an imaging technology called hyperspectral imaging has attracted attention and is used in industrial material classification, agricultural monitoring, geological surveys, and the like.

[0003] In hyperspectral imaging, a spectroscopic technique with high wavelength resolution in a wide wavelength range from the ultraviolet region to the near-infrared region is applied to the reflected light from an object, and an image for each wavelength channel depicting the object is generated as a hyperspectral image (hyperspectral data cube). Then, based on the hyperspectral image, the object is classified according to the presence or absence of abnormalities or the material. For example, Patent Document 1 discloses a technique of inputting image data generated by decomposing and imaging the light from a monitoring object for each wavelength using a hyperspectral camera into an inference model and inferring the abnormal state of the monitoring object according to the algorithm of the inference model.

[0004] A hyperspectral image has a size of H×W×C, where H is the number of pixels in the vertical direction of the image, W is the number of pixels in the horizontal direction of the image, and C is the number of wavelength channels. Therefore, in hyperspectral imaging, there is a problem that the data volume becomes large. This problem becomes particularly prominent when high-definition images or many wavelength channels are required. For example, a hyperspectral image may have a data volume more than 100 times that of an RGB image acquired by a normal RGB camera.

[0005] Furthermore, hyperspectral imaging techniques such as Coded Aperture Snapshot Spectral Imaging (CASSI) and snapshot techniques using photonic crystal filters are also known. This technique allows for the acquisition of multiple channel information related to multiple wavelength channels in a single image, improving image acquisition speed compared to conventional hyperspectral imaging using point scan or line scan techniques. However, since it is necessary to reconstruct a hyperspectral image from the encoded image (which is an image with compressed multiple channel information) in order to classify objects, reducing the amount of data remains a challenge.

[0006] Japanese Patent Publication No. 2022-137973

[0007] Hyperspectral images, with their rich information content, contribute to improved classification and recognition accuracy, but they also present a trade-off: increased system load due to the enormous amount of data. Therefore, there is a need to reduce the amount of data while leveraging the advantages of hyperspectral images with their rich information content.

[0008] Furthermore, while imaging devices that support hyperspectral imaging using snapshot technology have simpler hardware compared to conventional imaging devices that support hyperspectral imaging using point scan or line scan technology, their signal processing is more complex. Therefore, detailed calibration of imaging devices that support hyperspectral imaging using snapshot technology is necessary to accurately classify objects.

[0009] The purpose of this disclosure is to provide an imaging system and imaging method that can easily and accurately classify objects without increasing the amount of data.

[0010] An imaging system according to one aspect of the present disclosure includes an imaging device that acquires an encoded image obtained by compressing a plurality of spectral images obtained by capturing a subject in each of a plurality of wavelength channels, and an imaging device that classifies the subject based on the encoded image, wherein the imaging device includes a storage unit that stores a local inference model that takes the encoded image as input and outputs a classification result for classifying the subject, and a calculation unit that calibrates the local inference model based on the optical system of the imaging device, and inputs a target encoded image obtained by capturing the object which is the subject to be classified to the calibrated local inference model, which is the calibrated inference model, to obtain a classification result for classifying the object.

[0011] According to this disclosure, it will be possible to improve classification accuracy and recognition accuracy without increasing the amount of data.

[0012] This figure shows a hyperspectral imaging system according to a first embodiment of the present disclosure. This is an exploded perspective view of the imaging device. This is a cross-sectional view taken along line AA in Figure 2. This figure shows an example of the configuration of a local server. This figure shows an example of the configuration of a cloud server. This figure shows an example of data stored on the local server and the cloud server. This figure illustrates a machine learning model and its input / output. This figure illustrates a local inference model. This is a flowchart illustrating an example of a classification process that generates a classified image. This figure illustrates an example of a calibration method. This is a flowchart illustrating an example of a calibration process. This is a flowchart illustrating an example of a calibration process.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] (First Embodiment) Figure 1 is a diagram showing a hyperspectral imaging system according to the first embodiment of the present disclosure. The hyperspectral imaging system shown in Figure 1 includes an imaging device 1, a local server 2, and a cloud server 3. The local server 2 and the cloud server 3 are connected to each other so as to be able to communicate with each other via a network 4. There may be multiple imaging devices 1 and local servers 2. In the example of Figure 1, a local server 2 is provided for each imaging device 1, but one local server 2 may be provided for multiple imaging devices 1.

[0015] The imaging device 1 is a hyperspectral camera that supports coded aperture snapshot spectral imaging and snapshot techniques using photonic crystal filters. The imaging device 1 images an object 10, which is the subject to be classified, and obtains a coded image, which is a single image obtained by compressing multiple spectral images of the object 10, each of which is captured in multiple wavelength channels.

[0016] In this embodiment, the object 10 is transported by a conveyor belt 20, and the imaging device 1 acquires an encoded image by imaging the object 10 as it is transported to a specific position on the conveyor belt 20. The encoded image may contain multiple objects 10. Each of the multiple imaging devices 1 may image a different type of object 10. The wavelength channel is set appropriately according to the type of object 10, and can be a wavelength included in the visible light region, near-infrared region, or mid-infrared region, for example.

[0017] Local server 2 is a processing device that classifies objects 10 based on encoded images acquired by imaging device 1. The classification of objects 10 may be based on material or composition, or on various conditions such as abnormal states.

[0018] Specifically, local server 2 can classify objects from encoded images without explicitly generating a hyperspectral image (hyperspectral data cube) containing multiple spectral images. However, local server 2 may generate a hyperspectral image from the encoded image. Even in this case, the hyperspectral image is not stored on local server 2 but on cloud server 3.

[0019] Furthermore, in this embodiment, the local server 2 generates a classification image as a result of classifying the object 10, which is a photograph of the object and has a label assigned to it according to its classification. The classification image is, for example, an image in which the object is represented by a color corresponding to its classification.

[0020] Cloud server 3 functions as a management device for managing local server 2.

[0021] Figures 2 and 3 schematically show an example of the configuration of the imaging device 1. Specifically, Figure 2 is an exploded perspective view of the imaging device 1, and Figure 3 is a cross-sectional view taken along line AA in Figure 2.

[0022] As shown in Figures 2 and 3, the imaging device 1 has a metasurface 11 and an image sensor 12.

[0023] The metasurface 11 is positioned on the object 10 side of the image sensor 12. Light from the object 10 is incident on the metasurface 11 through the optical system (lens, etc.) of the imaging device 1.

[0024] The metasurface 11 has a plurality of optical filters 111 that transmit light of different wavelengths. Specifically, the plurality of optical filters 111 are arranged in a matrix on the metasurface 11. More specifically, filter units 112, each having a plurality of optical filters 111 arranged in a matrix, are arranged in a matrix. The type of optical filter 111 is not particularly limited, but in this embodiment, it is a filter having a metaatomic structure in which structures (e.g., protrusions) with dimensions smaller than the wavelength of light transmitted by the optical filter 111 are arranged in a row.

[0025] In the image sensor 12, multiple pixels 121 that receive incident light and convert it into electrical signals are arranged in a matrix.

[0026] The metasurface 11 and the image sensor 12 are connected such that each pixel 121 of the image sensor 12 corresponds to each optical filter 111 of the metasurface 11, and each pixel 121 receives light that has passed through the optical filter 111 corresponding to itself. As a result, the image sensor 12 receives light with different wavelengths that has passed through each of the multiple optical filters 111 all at once, so the imaging device 1 can acquire an encoded image by compressing multiple spectral images with wavelength channels based on the wavelengths of the transmitted light.

[0027] Figure 4 shows an example of the configuration of the local server 2. As shown in Figure 4, the local server 2 has a storage device 21, a processor 22, memory 23, a communication device 24, an input device 25, and a display device 26, which are connected via a bus 27.

[0028] The storage device 21 is a device that records data in a writable and readable manner, and is a storage unit that stores a program that defines the operation of the processor 22, and various information used and generated by that program. The processor 22 is an arithmetic unit that reads the program recorded in the storage device 21 into the memory 23 and uses the memory 23 to execute processing according to the program. The communication device 24 is connected to the external devices, the imaging device 1 and the cloud server 3, in a communicative manner, and transmits and receives various information with these external devices. The input device 25 receives various information from the operator of the local server 2, etc. The display device 26 displays various information.

[0029] Figure 5 shows an example of the configuration of the cloud server 3. As shown in Figure 5, the cloud server 3 has a storage device 31, a processor 32, memory 33, a communication device 34, an input device 35, and a display device 36, which are connected via a bus 37.

[0030] The storage device 31 is a device that records data in a writable and readable manner, and is a management storage unit that stores a program that defines the operation of the processor 32, and various information used and generated by that program. The processor 32 is a management unit that reads the program recorded in the storage device 31 into the memory 33 and uses the memory 33 to execute processing according to the program. The communication device 34 is connected to the external devices, the imaging device 1 and the local server 2, in a communicative manner, and transmits and receives various information with these external devices. The input device 35 receives various information from the operator of the cloud server 3, etc. The display device 36 displays various information.

[0031] Figure 6 shows an example of data stored in the local server 2 and the cloud server 3. In this embodiment, this data is stored in the storage device 21 or 31, but at least a portion of it may be stored in memory 23 or 32.

[0032] Local server 2 includes a local inference model 201, an encoded image 202, a classified image 203, effective transmittance matrix data 204, and a reconstruction model 205. Cloud server 3 includes a global inference model 301, a hyperspectral image 302, and a classified image 203.

[0033] The local inference model 201 is a pre-trained machine learning model that takes an encoded image 202, which is an image acquired by the imaging device 1, as input and outputs a classified image 203, which is the classification result of classifying the object 10.

[0034] The effective transmittance matrix data 204 is matrix data showing the effective transmittance of the optical system of the imaging device 1 for each pixel of the image sensor 12 of the imaging device 1, and is used to calibrate the local inference model 201.

[0035] The reconstruction model 205 is a pre-trained machine learning model that takes an encoded image 202 as input and outputs a hyperspectral image 302. The hyperspectral image 302 output from the reconstruction model 205 is stored on the cloud server 3, not on the local server 2. In this embodiment, it is not always necessary to output the hyperspectral image 302.

[0036] Furthermore, if multiple imaging devices 1 are provided for a single local server 2, the local inference model 201, effective transmittance matrix data 204, and reconstruction model 205 are prepared for each imaging device 1.

[0037] The global inference model 301 is a pre-trained machine learning model that takes a hyperspectral image 302 as input and outputs a classified image 303. A separate global inference model 301 is prepared for each of the 10 types of objects.

[0038] Each of the machine learning models, the local inference model 201, the reconstruction model 205, and the global inference model 301, is implemented, for example, as a deep learning model.

[0039] Figure 7 illustrates the machine learning model used in the hyperspectral imaging system in this embodiment and the data that serves as the input and output for the machine learning model. Figure 8 illustrates the local inference model 201.

[0040] In this embodiment, as described above, the encoded image 202 is generated by the imaging device 1, which supports encoded aperture snapshot spectral imaging and snapshot technology using a photonic crystal filter.

[0041] As shown in Figures 7 and 8, the local inference model 201 stored on the local server 2 generates a classification image 203 that classifies the object 10 from the encoded image 202 without explicitly generating a hyperspectral image 302.

[0042] Furthermore, the hyperspectral image 302 can also be directly acquired by using an imaging device compatible with hyperspectral imaging using point scan or line scan technology, instead of the imaging device 1 of this embodiment which uses coded aperture snapshot spectral imaging or snapshot technology using a photonic crystal filter. However, the hyperspectral image 302 is an image composed of multiple spectral images 302a taken of the object 10 in each of multiple wavelength channels, and the amount of data is large, often several hundred MB. In contrast, the coded image 202 has a small amount of data, about a few MB. For this reason, in this embodiment, the amount of data stored in the local server 2 can be reduced. In addition, the hyperspectral image 302 can be converted to the coded image 202 by convolution that takes into account the optical system of the imaging device 1. This conversion is irreversible, but the reconstruction model 205 stored in the local server 2 can infer and generate the hyperspectral image 302 from the coded image 202 using machine learning.

[0043] Furthermore, as shown in Figure 7, the local inference model 201 can also be expressed as a combination of a reconstruction model 205 that generates a hyperspectral image 302 from an encoded image 202 and a global inference model 301 that generates a classification image 203 from the hyperspectral image 302, or as a combination of parts thereof. When the reconstruction model 205 and the global inference model 301 pass through the low-dimensional feature representation of the hyperspectral image 302, the part of the reconstruction model 205 is the part that generates the feature representation from the encoded image 202, and the part of the global inference model 301 is the part that generates the classification image 203 from the feature representation.

[0044] The reconstruction model 205 is a model that does not depend on the object 10 but depends on the optical system of the imaging device 1. On the other hand, the global inference model 301 is a model that depends on the object 10 but does not depend on the optical system of the imaging device 1. Therefore, since the local inference model 201 can be expressed as a combination of the reconstruction model 205 and the global inference model 301, it can be expressed as a combination of a part that depends on the optical system of the imaging device 1 and a part that depends on the object 10.

[0045] In this embodiment, the part of the local inference model 201 that depends on the optical system of the imaging device 1 is calibrated according to the configuration of the optical system, and the part of the local inference model 201 that depends on the object 10 is changed according to the type of the object 10.

[0046] FIG. 9 is a flowchart for explaining an example of a classification process for generating a classified image.

[0047] First, the processor 22 of the local server 2 acquires an encoded image from the imaging device 1 via the communication device 24 and stores it as an encoded image 202 in the storage device 21 (step S101).

[0048] Subsequently, the processor 22 reads out the local inference model 201 from the storage device 21. The processor 22 inputs the encoded image 202 into the local inference model 201, acquires the classified image 203 output from the local inference model 201, and stores it in the storage device 21 (step S102).

[0049] Furthermore, the processor 22 reads out the reconstruction model 205 from the storage device 21. The processor 22 inputs the encoded image 202 into the reconstruction model 205, acquires the hyperspectral image 302 output from the reconstruction model 205, transmits the hyperspectral image 302 and the classified image 203 to the cloud server 3, and stores the hyperspectral image 302 and the classified image 203 in the storage device 31 of the cloud server 3 (step S103), and ends the classification process.

[0050] The process in step S103 may be performed under specific conditions. For example, the process in step S103 may be performed when instructed by the user, or when the imaging device 1 has been used for a certain period of time or longer. In this case, if the conditions are not specific, the processor 22 will skip the process in step S103 and terminate the classification process after completing the process in step S102. Furthermore, the hyperspectral images stored in the cloud server 3 can be used by any local server 2 or user as needed.

[0051] Next, we will explain the calibration of the local inference model 201. Since the local inference model 201 can be expressed as a combination of a part that depends on the optical system of the imaging device 1 and a part that depends on the object 10, we will first explain the calibration of the part that depends on the optical system of the imaging device 1.

[0052] In imaging devices using coded aperture patterns or photonic crystal filters, these are not changed during operation, but their characteristics can vary due to manufacturing variations. Therefore, the local inference model 201 is calibrated during the initial setup phase.

[0053] The calibration process proceeds as follows: an encoded image is captured using the encoded aperture pattern or photonic crystal filter; the parameters of the existing local inference model 201 are adjusted so that the encoded image outputs the desired inference result; and the local inference model 201 updated with the adjusted parameters is stored in the storage device 21, after which the calibration process is completed.

[0054] Furthermore, the classification accuracy can be improved by calibrating the local inference model 201 with the optical system (lenses, etc.) of the imaging device 1, particularly the point spread function (PSF) determined according to that optical system. The point spread function is a function that shows the impulse response of the optical system of the imaging device 1, that is, the response to a point light source, and more specifically, it is a function that shows how a single point in the subject is represented in the captured image. When the imaging device 1 is a hyperspectral camera as in this embodiment, the point spread function includes spatial parameters and wavelength channel parameters.

[0055] Figure 10 illustrates an example of a calibration method for calibrating the local inference model 201 using a point image distribution function.

[0056] The local inference model 201 is calibrated according to the point image distribution function of the optical system of the imaging device 1. In Figure 10, lenses A to C are shown as the optical system of the imaging device 1, and the point image distribution functions 401A to 401C corresponding to lenses A to C are shown. The point image distribution functions 401 to 401 may be calculated by actually measuring the response of lenses A to C to a point light source, but this is very time-consuming, so in this embodiment, they are calculated by simulating them using wave optics from a lens model that shows the configuration of lenses A to C (such as the dimensions and type of lenses).

[0057] The local inference model 201 is calibrated based on the point image distribution functions 401A to 401C. Specifically, the effective transmittance matrix data 204 corresponding to the configuration of the metasurface 11 of the imaging device 1 and the image sensor 12 is corrected by the point image distribution functions 401A to 401C, and the local inference model 201 is calibrated according to the corrected effective transmittance matrix data 204.

[0058] Figure 11 is a flowchart illustrating an example of a calibration process for calibrating the local inference model 201 using a point image distribution function.

[0059] In the calibration process, first, the processor 22 acquires a lens model that shows the configuration of the optical system of the imaging device 1 (step S201). The lens model may be pre-stored in the storage device 21, or it may be input from an operator of the local server 2 via the communication device 24 or input device 25.

[0060] The processor 22 obtains the point image distribution function of the optical system of the imaging device 1 by simulating based on the lens model (step S202).

[0061] The processor 22 corrects the effective transmittance matrix data 204 based on the acquired point image distribution function and calibrates the local inference model 201 by adjusting the parameters of the local inference model 201 based on the corrected effective transmittance matrix data 204 (step S203). At this time, the processor 22 may also calibrate the reconstruction model 205 based on the corrected effective transmittance matrix data 204 and calibrate the optical system-dependent portion of the local inference model 201 according to the calibrated reconstruction model 205.

[0062] The processor 22 stores the calibrated local inference model 201 in the storage device 21 (step S204) and terminates the calibration process.

[0063] Next, we will explain the calibration of the local inference model 201 with respect to the part that depends on the object 10. Figure 12 is a flowchart illustrating an example of a modification process that changes the local inference model 201 according to the type of object 10.

[0064] The part that depends on the object 10 is based on the global inference model 301, which is a model that does not depend on the optical system of the imaging device 1. Therefore, it is a common model that does not depend on the imaging device, and by managing it centrally, it becomes easy to maintain consistency between various models, such as version control.

[0065] In the change processing, when the cloud server 3's processor 32 receives notification from the user via the input device 35 or local server 2 that the type of object 10 to be classified by the local server 2 has changed, it retrieves a global inference model 301 corresponding to the changed type from the storage device 31 and distributes it to the local server 2 (step S301).

[0066] When the processor 22 of the local server 2 receives the global inference model 301 via the communication device 24, it modifies the part of the local inference model 201 that depends on the object 10 based on the global inference model 301 (step S302), and then terminates the modification process.

[0067] As described above, according to this embodiment, the processor 22 of the local server 2 calibrates the local inference model 201 based on the optical system of the imaging device 1. The processor 22 inputs the encoded image to the calibrated local inference model 201 and obtains a classification result that classifies the object 10. Since it is not necessary to generate hyperspectral images within the local server 2, the storage capacity of the local server 2 can be reduced.

[0068] Furthermore, according to this embodiment, the processor 22 inputs the encoded image to the reconstruction model 205 to obtain a hyperspectral image of the object 10 and stores it in the cloud server 3. This makes it possible to reduce the storage capacity of the local server 2 while making the hyperspectral image accessible when needed.

[0069] Furthermore, according to this embodiment, the processor 22 calculates the point image distribution function based on the configuration of the optical system of the imaging device 1. Therefore, the point image distribution function can be easily calculated, and by calibrating the local inference model 201, it becomes possible to obtain classification results with good accuracy.

[0070] Furthermore, in this embodiment, if the type of object 10 is changed, the local inference model 201 is updated based on the global inference model 301 corresponding to that type. Therefore, even if the type of object 10 is changed, the local inference model 201 can be easily updated according to that type.

[0071] Furthermore, in this embodiment, the imaging device 1 includes a metasurface 11 having a plurality of optical filters that transmit light of different wavelengths, and an image sensor 12 having a plurality of pixels that receive light transmitted through the metasurface 11. In this case, it becomes possible to easily manufacture an imaging device 1 to which spectral imaging using snapshot technology is applied.

[0072] The embodiments of the Disclosure described above are illustrative for illustrative purposes and are not intended to limit the scope of the Disclosure to those embodiments only. Those skilled in the art can implement the Disclosure in various other forms without departing from the scope of the Disclosure.

[0073] 1: Imaging device 2: Local server 3: Cloud server 4: Network 10: Object 11: Metasurface 12: Image sensor 20: Conveyor belt 21, 31: Storage device 22, 32: Processor 23, 33: Memory 24, 34: Communication device 25, 35: Input device 26, 36: Display device 111: Optical filter 112: Filter unit 121: Pixel 201: Local inference model 202: Encoded image 203: Classified image 204: Effective transmittance matrix data 205: Reconstruction model 301: Global inference model 302: Hyperspectral image

Claims

1. An imaging system comprising: an imaging device that acquires an encoded image obtained by compressing a plurality of spectral images of a subject taken in each of a plurality of wavelength channels; and a processing device that classifies the subject based on the encoded image, wherein the processing device comprises: a storage unit that stores a local inference model which takes the encoded image as input and outputs a classification result of classifying the subject; and a calculation unit that calibrates the local inference model based on the optical system of the imaging device, and inputs a target encoded image which is an encoded image of the subject to be classified to the calibrated local inference model, which is the calibrated inference model, and acquires a classification result of classifying the object.

2. The imaging system according to claim 1, wherein the calculation unit calculates a point image distribution function of the optical system based on the configuration of the optical system of the imaging device, and calibrates the local inference model based on the point image distribution function.

3. The imaging system according to claim 1, wherein the storage unit further stores a reconstruction model for reconstructing a hyperspectral image having a plurality of spectral images from the encoded image, and the calculation unit inputs the target encoded image to the reconstruction model to obtain the hyperspectral image of the target object and stores it in a management device that is communicably connected to the processing device.

4. The imaging system according to claim 3, wherein the management device is a cloud server.

5. The imaging system according to claim 3, wherein the calculation unit calibrates the reconstruction model based on the optical system and calibrates the local inference model based on the calibrated reconstruction model.

6. The imaging system according to claim 3, wherein the imaging system has the management device, the management device having a management storage unit that stores a global inference model for classifying the subject from the hyperspectral image for each type of subject, and a management unit that, when the type of object is changed, distributes the global inference model corresponding to the type of object to the processing device, and the calculation unit updates the local inference model based on the distributed global inference model and the reconstruction model.

7. The imaging system according to claim 1, comprising: a metasurface having a plurality of optical filters that transmit light of different wavelengths; and an image sensor having a plurality of pixels that receive light transmitted through the metasurface.

8. The imaging system according to claim 7, wherein the optical filter has a metaatomic structure in which structures with dimensions smaller than the wavelength of light transmitted by the optical filter are arranged in a row.

9. An imaging method using an imaging system comprising: an imaging device that acquires an encoded image obtained by compressing a plurality of spectral images of a subject taken in each of a plurality of wavelength channels; and a processing device that classifies the subject based on the encoded image, wherein the processing device stores a local inference model that takes the encoded image as input and outputs a classification result of classifying the subject; the processing device calibrates the local inference model based on the optical system of the imaging device; and the processing device inputs a target encoded image, which is an encoded image of the subject to be classified, to the calibrated local inference model, which is the calibrated inference model, and acquires a classification result of classifying the object.

Images