Imaging system and imaging method

Abstract

This system provides an imaging system capable of generating hyperspectral images with greater accuracy. [Solution] The imaging system comprises an imaging device that images an object and outputs a compressed image in which spectral information is compressed; a sensor that detects the movement of the imaging device when the compressed image is taken and outputs parameters related to the movement; and a processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, wherein the processing circuit edits matrix data used in the hyperspectral image reconstruction process based on the parameters before generating the hyperspectral image.

Description

[Technical Field]

[0001] This disclosure relates to an imaging system and an imaging method. [Background technology]

[0002] Compressed sensing is a technique that recovers more data than observed by assuming that the data distribution of the observed object is sparse in a certain space, such as frequency space. Compressed sensing can be applied, for example, to imaging devices that recover images containing more information from a small amount of observed data. An imaging device to which compressed sensing is applied generates a hyperspectral image containing multiple images corresponding to multiple wavelength bands through calculation from an image in which the spectral information of the object has been compressed. As a result, it becomes possible to obtain various effects such as higher resolution, multi-wavelength capabilities, reduced imaging time, or increased sensitivity of the image. Patent Document 1 discloses an example of applying compressed sensing technology to a camera that acquires hyperspectral images, i.e., a hyperspectral camera. Patent Document 2 discloses an example of a method for correcting image blur caused by camera shake during imaging with a normal camera. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] U.S. Patent No. 9599511 [Patent Document 2] Japanese Patent Publication No. 2009-177332 [Overview of the project] [Problems that the invention aims to solve]

[0004] This system provides an imaging system capable of generating hyperspectral images with greater accuracy. [Means for solving the problem]

[0005] An imaging system according to one aspect of the present disclosure includes: an imaging device that images an object and outputs a compressed image in which spectral information is compressed; a sensor that detects the movement of the imaging device when the compressed image is captured and outputs parameters related to the movement; and a processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, wherein the processing circuit edits matrix data used in the hyperspectral image reconstruction process based on the parameters before generating the hyperspectral image.

[0006] The comprehensive or specific embodiments of this disclosure may be implemented as systems, apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable discs, or as any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. Computer-readable recording media may include, for example, non-volatile recording media such as CD-ROMs (Compact Disc-Read Only Memory). An apparatus may consist of one or more devices. If an apparatus consists of two or more devices, these two or more devices may be located in a single device or in two or more separate devices. In this specification and in the claims, “apparatus” may mean not only a single device but also a system consisting of multiple devices. [Effects of the Invention]

[0007] The technology disclosed herein enables the realization of an imaging system capable of generating hyperspectral images with greater accuracy. [Brief explanation of the drawing]

[0008] [Figure 1A] Figure 1A is a schematic diagram showing an example of the configuration of an imaging system. [Figure 1B] Figure 1B schematically shows another example configuration of the imaging system. [Figure 1C] Figure 1C schematically shows yet another example of the imaging system configuration. [Figure 1D] FIG. 1D is a diagram schematically showing yet another configuration example of the imaging system. [Figure 2A] FIG. 2A is a diagram schematically showing an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of the spatial distribution of the transmittance of light of each of a plurality of wavelength bands included in the target wavelength range. [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of region A1 included in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of region A2 included in the filter array shown in FIG. 2A. [Figure 3] FIG. 3 is a diagram for explaining an example of the relationship between the target wavelength range and a plurality of wavelength bands included therein. [Figure 4A] FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. [Figure 4B] FIG. 4B is a diagram showing the result of averaging the spectral transmittance shown in FIG. 4A for each wavelength band. [Figure 5] FIG. 5 is a block diagram schematically showing the configuration of an imaging system according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram schematically showing an example of the data format of the restoration table stored in the storage device. [Figure 7] FIG. 7 is a flowchart schematically showing an example of the processing operation executed by the processing circuit in the present embodiment when generating a hyperspectral image. [Figure 8A] FIG. 8A is a flowchart schematically showing an example of the processing operation executed by the processing circuit in the present embodiment when storing a compressed image and its related information in the storage device. [Figure 8B] FIG. 8B is a flowchart schematically showing another example of the processing operation executed by the processing circuit in the present embodiment when storing a compressed image and its related information in the storage device. [Figure 9]Figure 9 is a schematic block diagram illustrating another configuration of the imaging system according to an exemplary embodiment of the present disclosure. [Figure 10A] Figure 10A schematically shows examples of the data formats for the recovery table, the first PSF group, and the second PSF group. [Figure 10B] Figure 10B schematically shows an example of the edited restoration table based on the first PSF group and the data format of the second PSF group. [Figure 10C] Figure 10C schematically shows an example of the edited restoration table based on the second PSF group and the data format of the first PSF group. [Figure 10D] Figure 10D schematically shows an example of the data format of the reconstructed table after editing, based on the first and second PSF groups. [Figure 11] Figure 11 is a flowchart illustrating an example of the processing operations performed by the processing circuit in this embodiment when generating a hyperspectral image. [Figure 12] Figure 12 is a flowchart illustrating an example of the processing operations performed by the processing circuit in this embodiment when generating a hyperspectral image. [Figure 13] Figure 13 is a schematic block diagram showing another configuration of an imaging device according to an exemplary embodiment of the present disclosure for correcting blur. [Figure 14] Figure 14 is a sequence diagram illustrating an example of the processing operation for acquiring one or more first PSF groups and one or more second PSF groups corresponding to the lens system model number in the imaging system according to this embodiment. [Figure 15] Figure 15 is a schematic block diagram illustrating yet another configuration of an imaging system according to an exemplary embodiment of the present disclosure. [Figure 16] Figure 16 is a flowchart illustrating an example of the processing operations performed by the processing circuit 210 in this embodiment when generating a hyperspectral image. [Figure 17A] Figure 17A is a schematic diagram illustrating Example 1 of the output data format. [Figure 17B]Figure 17B schematically shows Example 2 of the output data format. [Figure 17C] Figure 17C is a schematic diagram illustrating Example 3 of the output data format. [Figure 17D] Figure 17D schematically shows Example 4 of the output data format. [Figure 17E] Figure 17E schematically shows Example 5 of the output data format. [Figure 17F] Figure 17F is a schematic diagram illustrating Example 6 of the output data format. [Figure 17G] Figure 17G schematically shows Example 7 of the output data format. [Figure 17H] Figure 17H ​​schematically shows Example 8 of the output data format. [Modes for carrying out the invention]

[0009] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or a large-scale integration (LSI). The LSI or IC may be integrated on a single chip or may be composed of multiple chips combined. For example, functional blocks other than memory elements may be integrated on a single chip. Here, we refer to them as LSIs or ICs, but the name may change depending on the degree of integration, and they may also be called system LSIs, VLSIs (very large-scale integrations), or ULSIs (ultra-large-scale integrations). Field-programmable gate arrays (FPGAs) that are programmed after the manufacture of the LSI, or reconfigurable logic devices that allow for the reconfiguration of junction relationships within the LSI or the setup of circuit compartments within the LSI, can also be used for the same purpose.

[0010] Furthermore, the functions or operations of all or part of a circuit, unit, device, component, or part can be performed by software processing. In this case, the software is recorded on one or more non-temporary recording media such as ROMs, optical disks, or hard disk drives, and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. The system or device may include one or more non-temporary recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0011] In this disclosure, "light" means electromagnetic waves including not only visible light (wavelengths of approximately 400 nm to 700 nm) but also ultraviolet light (wavelengths of approximately 10 nm to 400 nm) and infrared light (wavelengths of approximately 700 nm to 1 mm).

[0012] The following describes exemplary embodiments of this disclosure. The embodiments described below are either comprehensive or specific examples. The numerical values, shapes, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in the independent claim representing the highest-level concept are described as optional components. Also, the figures are schematic diagrams and not necessarily strictly accurate representations. In addition, substantially identical components are denoted by the same reference numerals in the figures, and redundant explanations may be omitted or simplified.

[0013] Before describing embodiments of this disclosure, we will briefly explain compressed sensing techniques that utilize sparsity.

[0014] Sparsity is the property that the elements characterizing an observed object exist sparsely (sparsely) in a certain space, such as frequency space. Sparsity is widely observed in nature. By utilizing sparsity, it becomes possible to efficiently observe the necessary information. Sensing technology that utilizes sparsity is called compressed sensing technology. By using compressed sensing technology, it is possible to construct highly efficient devices and systems.

[0015] Patent Document 1 discloses a hyperspectral camera that improves wavelength resolution using compressed sensing technology. The hyperspectral camera of Patent Document 1 includes, for example, an optical filter having irregular light transmission characteristics with respect to space and wavelength. Such an optical filter is also called an "encoding mask." The encoding mask is placed on the optical path of light incident on the image sensor and transmits light incident from an object with different light transmission characteristics depending on the region. This process by the encoding mask is called "encoding."

[0016] The image of an object obtained through an encoding mask has its spectral information compressed. This image is called a "compressed image." Mask information, which reflects the spatial distribution of the light transmission characteristics of the encoding mask, is pre-stored in a memory device as a reconstruction table. The reconstruction table can be generated, for example, by detecting spatially uniform light through an image sensor while changing the wavelength of the light via the encoding mask.

[0017] The hyperspectral camera described in Patent Document 1 can generate a hyperspectral image by calculations based on a compressed image and a reconstruction table. The hyperspectral image includes multiple images, each corresponding to a different wavelength band. These multiple images contain more information than the compressed image, such as higher resolution image information or image information for more wavelengths.

[0018] On the other hand, if the hyperspectral camera moves due to camera shake during shooting, the hyperspectral image may be blurred or out of focus. Furthermore, the hyperspectral image may also be blurred depending on the adjustment of the optical system, such as the lens and / or aperture, included in the hyperspectral camera. To correct blur and / or out-of-focus images caused by camera movement, the image is edited by applying a Point Spread Function (PSF) that reflects these blurs and / or out-of-focus images. To correct out-of-focus images caused by the optical system, the image is edited by applying a PSF that reflects these out-of-focus images.

[0019] PSF is a function that describes the effect that light incident on each pixel of an image sensor has on its surrounding pixels. PSF assumes that the light transmission characteristics of the optical filters and / or lenses corresponding to each pixel are equivalent. That is, under the condition that the light transmission characteristics due to the filters and / or lenses are equivalent between pixels, light transmitted through the optical filters and / or lenses is incident on each pixel. Since the appropriate PSF for correction varies with wavelength, applying PSF is not easy if there are differences in the light transmission characteristics of the filters transmitted between a given pixel and its surrounding pixels.

[0020] For example, in a typical camera such as the one disclosed in Patent Document 2, multiple pixels in the image sensor detect light from an object with the same light transmission characteristics, making the application of PSF (Photosensor Filter) easy. Each pixel on an image sensor for color images is provided with one of three types of color filters: red (R), green (G), and blue (B). Therefore, an image sensor for color images includes three types of pixels, each having the three types of light transmission characteristics of RGB. Each RGB color filter is a filter that transmits only a specific band of RGB, and their light transmission characteristics can be expressed by spectral bands and transmittance. Pixels provided with the same type of color filter have the same transmission band and transmittance. The positions of the same type of color filter on the image sensor can also be identified. Therefore, by preparing PSFs for the three types of pixels mentioned above, it becomes possible to apply PSF to the image.

[0021] However, in a hyperspectral camera, light from an object enters the image sensor via an encoding mask, so multiple pixels in the image sensor detect light from the object with different light transmission characteristics. As mentioned above, the encoding mask has irregular light transmission characteristics in space and wavelength, so the light transmission characteristics of each pixel can differ irregularly. Therefore, it is not practical to prepare a PSF for each light transmission characteristic. For these reasons, it is not easy to accurately generate a hyperspectral image by applying a PSF to a compressed image and editing the compressed image.

[0022] The present inventors have considered the above problems and have come up with an imaging system and an imaging method performed in the imaging system according to this embodiment that can solve these problems. In the imaging system according to this embodiment, a reconstructed table may be compiled, rather than a compressed image, based, for example, on the movement of the imaging device and / or the optical system included in the imaging device. The imaging system and imaging method according to embodiments of the present disclosure are described below.

[0023] (Embodiment) The following describes, first, an imaging system comprising an imaging device that generates compressed images and an image processing device that performs image processing. Next, an imaging system according to this embodiment that corrects blur and / or blur caused by the movement of the imaging device is described. Furthermore, an imaging system according to this embodiment that corrects blur caused by the optical system included in the imaging device is described.

[0024] [1. Imaging System] Figure 1A is a schematic diagram showing an example of the configuration of an imaging system. The system shown in Figure 1A comprises an imaging device 100 and an image processing device 200. The imaging device 100 has a configuration similar to the imaging device disclosed in Patent Document 1. The imaging device 100 comprises an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are arranged on the optical path of light incident from the object 70, which is the subject. In the example in Figure 1A, the filter array 110 is arranged between the optical system 140 and the image sensor 160.

[0025] Figure 1A shows an apple as an example of the object 70. The object 70 is not limited to an apple; it can be any object. The image sensor 160 generates compressed image 10 data in which information from multiple wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 200 generates data showing multiple images that correspond one-to-one with multiple wavelength bands included in a predetermined target wavelength range, based on the compressed image 10 data generated by the image sensor 160. Here, the number of wavelength bands included in the target wavelength range is N (N is an integer of 4 or more). In the following description, the N images generated based on the compressed image 10 are referred to as restored images 20W1, 20W2, ..., 20W N These are sometimes referred to as "hyperspectral images 20".

[0026] In this embodiment, the filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters whose spectral transmittance, i.e., wavelength dependence of light transmittance, differs from one another. The filter array 110 modulates the intensity of the incident light for each wavelength and outputs it. This process by the filter array 110 is called "coding," and the filter array 110 is also called an "encoding mask."

[0027] In the example shown in Figure 1A, the filter array 110 is positioned near or directly above the image sensor 160. Here, "nearby" means that the image of light from the optical system 140 is formed on the surface of the filter array 110 in a reasonably clear state. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0028] The optical system 140 includes at least one lens. In Figure 1A, the optical system 140 is shown as a single lens, but the optical system 140 may be a combination of multiple lenses. The optical system 140 forms an image on the imaging plane of the image sensor 160 via the filter array 110.

[0029] The filter array 110 may be positioned away from the image sensor 160. Figures 1B to 1D show examples of the configuration of an imaging device 100 in which the filter array 110 is positioned away from the image sensor 160. In the example in Figure 1B, the filter array 110 is positioned between the optical system 140 and the image sensor 160, but away from the image sensor 160. In the example in Figure 1C, the filter array 110 is positioned between the object 70 and the optical system 140. In the example in Figure 1D, the imaging device 100 comprises two optical systems 140A and 140B, with the filter array 110 positioned between them. As in these examples, an optical system including one or more lenses may be positioned between the filter array 110 and the image sensor 160.

[0030] The image sensor 160 is a monochrome type photodetector having a plurality of photodetectors (also referred to herein as "pixels") arranged in two dimensions. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The photodetectors include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor may be used. A color type sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light. A color type sensor may further include a plurality of IR filters that transmit infrared light. A color type sensor may also include a plurality of transparent filters that transmit all red, green, and blue light. By using a color type sensor, the amount of information regarding wavelength can be increased, and the accuracy of generating the hyperspectral image 20 can be improved. The wavelength range to be acquired can be determined arbitrarily and is not limited to the visible wavelength range; it may also include ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength ranges.

[0031] The image processing device 200 may be a computer comprising one or more processors and one or more storage media such as memory. Based on the compressed image 10 acquired by the image sensor 160, the image processing device 200 restores images 20W1, 20W2, ... 20W N Generate data.

[0032] Figure 2A is a schematic diagram showing an example of a filter array 110. The filter array 110 has multiple regions arranged in two dimensions. In this specification, these regions may be referred to as "cells". Each region is equipped with an optical filter having an individually set spectral transmittance. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take values ​​between 0 and 1, inclusive.

[0033] In the example shown in Figure 2A, the filter array 110 has 48 rectangular regions arranged in a 6x8 grid. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be, for example, roughly equivalent to the number of pixels in the image sensor 160. The number of filters included in the filter array 110 is determined according to the application, for example, ranging from tens of thousands to tens of millions.

[0034] Figure 2B shows the wavelength bands W1, W2, ..., W included in the target wavelength range. N This figure shows an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in Figure 2B, the difference in intensity in each region represents the difference in transmittance. Lighter regions have higher transmittance, and darker regions have lower transmittance. As shown in Figure 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0035] Figures 2C and 2D show examples of spectral transmittances for regions A1 and A2, respectively, included in the filter array 110 shown in Figure 2A. The spectral transmittances of region A1 and region A2 are different from each other. Thus, the spectral transmittance of the filter array 110 differs by region. However, it is not necessary for the spectral transmittances of all regions to be different. In the filter array 110, the spectral transmittances of at least some of the regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns for multiple regions included in the filter array 110 may be equal to or greater than the number of wavelength bands N included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0036] Figure 3 shows the target wavelength range W and the wavelength bands W1, W2, ..., W contained within it. NThis diagram illustrates the relationship. The target wavelength range W can be set to various ranges depending on the application. For example, the target wavelength range W may be the visible light wavelength range from about 400 nm to about 700 nm, the near-infrared wavelength range from about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range from about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. Thus, the wavelength range used is not limited to the visible light range. In this specification, "light" refers to all radiation, including infrared and ultraviolet rays, not just visible light.

[0037] In the example shown in Figure 3, N is any integer greater than or equal to 4, and the target wavelength range W is divided into N equal parts, with each wavelength range being designated as wavelength bands W1, W2, ..., W N This is the general rule. However, the examples are not limited to this. Multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be made non-uniform depending on the wavelength band.

[0038] Figure 4A is a diagram illustrating the spectral transmittance characteristics in a region of the filter array 110. In the example shown in Figure 4A, the spectral transmittance has multiple maximum values ​​P1 to P5 and multiple minimum values ​​with respect to wavelengths within the target wavelength range W. In the example shown in Figure 4A, the optical transmittance within the target wavelength range W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in Figure 4A, wavelength band W2 and wavelength band W N-1 The spectral transmittance has a maximum value in the wavelength ranges such as W1, W2, ..., W N It can be designed to have maximum values ​​in at least two or more wavelength ranges. In the example in Figure 4A, the maximum values ​​P1, P3, P4, and P5 are greater than or equal to 0.5.

[0039] Thus, the light transmittance of each region varies with wavelength. Therefore, the filter array 110 transmits more components of a certain wavelength range of the incident light and less components of other wavelength ranges. For example, for the light of k wavelength bands out of N wavelength bands, the transmittance may be greater than 0.5, and for the light of the remaining N - k wavelength ranges, the transmittance may be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that equally contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of intensity peaks discrete with respect to wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0040] FIG. 4B shows, as an example, the result of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W1, W2, ···, W N This is a diagram showing the result of averaging the spectral transmittance T(λ) for each wavelength band and dividing it by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is defined as the transmittance in that wavelength band. In this example, the transmittance is prominently high in three wavelength ranges taking the maximum values P1, P3, and P5. In particular, in the two wavelength ranges taking the maximum values P3 and P5, the transmittance exceeds 0.8.

[0041] In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessarily required to have a grayscale transmittance distribution. For example, a binary-scale transmittance distribution in which the transmittance of each region can take either a value close to 0 or a value close to 1 may be adopted. In the binary-scale transmittance distribution, each region transmits most of the light in at least two of the plurality of wavelength ranges included in the target wavelength range and does not transmit most of the light in the remaining wavelength ranges. Here, "most" generally refers to 80% or more.

[0042] A part of all cells, for example, half of the cells, may be replaced with a transparent region. Such a transparent region transmits light of each of the wavelength bands W1, W2, ···, W included in the target wavelength range W with a transmittance of about the same high level, for example, a transmittance of 80% or more. In such a configuration, the plurality of transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the plurality of regions in the filter array 110, regions with different light transmittances depending on the wavelength and the transparent regions may be alternately arranged. N In such a configuration, data indicating the spatial distribution of the spectral transmittance of such a filter array 110 is acquired in advance based on design data or actual measurement calibration and stored in a storage medium included in the image processing apparatus 200. This data is used for arithmetic processing described later.

[0043] The filter array 110 can be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, or a fine structure including a metal. When using a multilayer film, for example, a multilayer film including a dielectric multilayer film or a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is formed to be different for each cell. Thereby, different spectral characteristics can be realized for each cell. By using a multilayer film, a sharp rise and fall in spectral transmittance can be realized. A configuration using an organic material can be realized by making the pigments or dyes contained in each cell different or by laminating different materials. A configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. When using a fine structure including a metal, it can be manufactured using spectroscopy based on the plasmon effect. <000025​​​​​Alternatively, instead of providing a separate filter array 110 in the imaging device 100, the image sensor 160 may be modified to change the light-receiving characteristics of the image sensor 160 for each pixel. Compressed images 10 can also be generated by imaging using such an image sensor 160. In other words, compressed images 10 may be generated by an imaging device in which the filter array 110 is built into the image sensor 160. In this case, the encoded information corresponds to the light-receiving characteristics of the image sensor 160.

[0046] Alternatively, in the imaging device 100, an optical element such as a metalens may be introduced into at least a part of the optical system 140. Since the optical properties of the optical system 140 change spatially and chromatically, a compressed image 10 can also be generated by imaging using such an optical system 140. In this case, the encoded information will be information corresponding to the optical properties of the optical element such as a metalens.

[0047] From the above, it can be said that the imaging device 100 has multiple light-receiving regions with different light transmission characteristics, regardless of whether it has a filter array 110 or a configuration different from the filter array 110. An example of light transmission characteristics is the light transmission spectrum.

[0048] Next, an example of signal processing by the image processing device 200 will be described. Based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110, the image processing device 200 generates a multi-wavelength hyperspectral image 20 by the following reconstruction process. Here, "multi-wavelength" means more wavelength ranges than, for example, the three RGB wavelength ranges acquired by a normal color camera. The number of these wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is called the "number of bands". Depending on the application, the number of bands may exceed 100.

[0049] The data we want is the data from the hyperspectral image 20, and we will call this data f. If the number of bands is N, then f is the data for N bands f1, f2, ..., fN This is integrated data. Here, the horizontal direction of the image is the x-direction, and the vertical direction of the image is the y-direction. If the number of pixels in the x-direction of the image data to be obtained is m, and the number of pixels in the y-direction is n, then the image data f1, f2, ..., f N Each of these has n × m pixel values. Therefore, data f is data with n × m × N elements. On the other hand, the number of elements in data g of the compressed image 10, which is obtained by encoding and multiplexing by the filter array 110, is n × m. Data g can be expressed by the following equation (1).

number

[0050] In equation (1), f represents the data of the hyperspectral image, expressed as a one-dimensional vector: f1, f2, ..., f N Each of these has n × m elements. Therefore, the vector on the right side is strictly an n × m × N x 1 1 dimensional vector. The compressed image data g is calculated as an n × m x 1 1 dimensional vector. The matrix H is the vector f, with each component f1, f2, ..., f N This represents a transformation that encodes and intensity modulates each wavelength band with different encoding information and then adds them together. Therefore, H is an n×m row n×m×N column matrix. Equation (1) can also be expressed as follows: g=(pg 11 ...pg 1m ...pg n1 ...pg nm ) T =H(f1···f N ) T Here, pg ij This represents the pixel value of the i-th row and j-th column of the compressed image 10.

[0051] Given a vector g and a matrix H, it appears that data f can be calculated by solving the inverse problem of equation (1). However, since the number of elements n × m × N of the data f to be obtained is greater than the number of elements n × m of the acquired data g, this problem is poorly set up and cannot be solved as is. Therefore, the image processing device 200 uses the sparsity of the image contained in data f and employs a compressed sensing technique to find the solution. Specifically, the data f to be obtained is estimated by solving the following equation (2).

number

[0052] Here, f' represents the estimated data for f. The first term in parentheses in the above equation represents the difference between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares may also be used as the residual term. The second term in parentheses is the regularization term or stabilization term. Equation (2) means finding the f that minimizes the sum of the first and second terms. The function in parentheses in equation (2) is called the evaluation function. The image processing device 200 can converge the solution through recursive iterative operations and calculate the f that minimizes the evaluation function as the final solution f'. In this way, the compressed image 10 can be reconstructed to generate a hyperspectral image.

[0053] The first term in parentheses in equation (2) represents the operation of calculating the sum of squares of the differences between the acquired data g and Hf, which is obtained by transforming the estimation process f by matrix H. The second term, Φ(f), is a constraint in the regularization of f and is a function that reflects the sparse information of the estimated data. This function has the effect of making the estimated data smoother or more stable. The regularization term can be represented by, for example, the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, using total variation allows for obtaining stable estimated data that suppresses the effects of noise in the observed data g. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term may be selected that makes the texture of the object 70 more sparse in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the reduction of redundant data and the higher the compression rate. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to a moderate value that allows f to converge to a certain extent without overcompression.

[0054] Through the above process, a hyperspectral image 20 can be generated by a reconstruction process based on the compressed image 10 acquired by the image sensor 160. Details of the method for generating the hyperspectral image 20 are disclosed in Patent Document 1. The entirety of the disclosure in Patent Document 1 is incorporated herein by reference.

[0055] [2. Imaging system that corrects blur and / or defocus caused by movement of the imaging device] [2.1. Example configuration of an imaging system] Below, with reference to Figure 5, an example of an imaging system for correcting blur and / or blur caused by the movement of the imaging device 100 is described. The movement of the imaging device 100 may be caused, for example, by camera shake. Figure 5 is a schematic block diagram showing the configuration of an imaging device according to an exemplary embodiment of the present disclosure for correcting blur and / or blur caused by the movement of the imaging device 100. The imaging system shown in Figure 5 comprises an imaging device 100, an image processing device 200, a display device 300, and a vibration sensor 400.

[0056] The imaging device 100 captures an image of the object 70 and outputs a compressed image 10 in which spectral information has been compressed. The imaging device 100 includes an optical system 140 including a lens 142 and an aperture 144, a filter array 110, an image sensor 160, and a control circuit 170.

[0057] Lens 142 is at least one lens with a fixed magnification. Aperture 144 is an aperture with a fixed depth of field. The optical system 140 may be a lens system comprising a pair of lenses 142 and aperture 144. The filter array 110 includes a plurality of filters with different transmittances depending on the wavelength band, and is arranged irregularly. The image sensor detects light from the object 70 via the optical system 140 and the filter array 110 and outputs a compressed image 10.

[0058] The image sensor 160 may have, for example, a global shutter that detects light at the same timing in multiple pixels it contains. Alternatively, the image sensor 160 may have a rolling shutter that detects light at different timings in multiple pixels it contains, from the top row to the bottom row.

[0059] The control circuit 170 controls the shutter operation of the image sensor 160. The control circuit 170 may also control the operation of components (not shown) included in the imaging device 100.

[0060] The image processing apparatus 200 comprises a processing circuit 210 and a storage device 220. The processing circuit 210 generates a hyperspectral image based on the compressed image 10 by a reconstruction process. The storage device 220 stores a reconstruction table used in the hyperspectral image reconstruction process. The reconstruction table includes encoded information that reflects the light transmission spectrum of the filter array 110. In this specification, the reconstruction table is also simply referred to as "matrix data".

[0061] The display device 300 displays the hyperspectral image output from the processing circuit 210. The vibration sensor 400 detects the movement of the imaging device 100 when the compressed image 10 is captured and outputs parameters related to this movement. These parameters include, for example, at least one of displacement, velocity, and acceleration. The vibration sensor 400 may be mounted, for example, inside or outside the imaging device 100. In this specification, the vibration sensor 400 is also referred to as "other device".

[0062] [2.2. Example of data format for the recovery table] The reconstruction table contains multiple mask matrices, each corresponding to a multiple wavelength band. The number of multiple wavelength bands in the reconstruction table is the same as the number of four or more wavelength bands in the hyperspectral image. Each mask matrix contains, as multiple matrix elements, the pixel values ​​of multiple pixels contained in the image sensor 160 when spatially uniform light of the corresponding wavelength band is detected by the imaging device 100. Each of the multiple matrix elements corresponds to one of the multiple pixels contained in the compressed image 10.

[0063] Next, an example of the data format of the recovery table will be explained with reference to Figure 6. Figure 6 is a schematic diagram showing an example of the data format of the recovery table stored in the storage device 220. The recovery table shown in Figure 6 is represented as a three-dimensional matrix in which the depth represents the wavelength band and the height and width represent the pixel values ​​of each mask matrix. The number of data points in the recovery table stored in the storage device 220 is the product of the number of pixels in two dimensions and the number of wavelength bands.

[0064] In each mask matrix, the pixel values ​​can be normalized, for example, by the maximum grayscale value corresponding to the number of bits, and can be represented in the range of 0 to 1, as shown in Figure 6. Alternatively, the pixel values ​​may be represented by the grayscale range corresponding to the number of bits. In the case of 8 bits, the pixel values ​​are between 0 and 255, and the maximum grayscale value is 255.

[0065] As shown in Figure 6, each mask matrix is ​​an n x m two-dimensional matrix representing the two-dimensional distribution of multiple numerical values. This data format makes it easier to understand how pixel values ​​are distributed two-dimensionally for a given wavelength band.

[0066] [2.3. Example of processing operation for generating hyperspectral images] Next, with reference to Figure 7, an example of the processing operation by the processing circuit 210 to generate a hyperspectral image will be described. Figure 7 is a flowchart schematically showing an example of the processing operation performed by the processing circuit 210 in this embodiment when generating a hyperspectral image. The processing circuit 210 performs the operations of steps S101 to S107 shown in Figure 7.

[0067] <Step S101> The processing circuit 210 causes the vibration sensor 400 to start a detection operation to detect the movement of the imaging device 100.

[0068] <Step S102> The processing circuit 210 causes the imaging device 100 to capture a compressed image 10. More specifically, the processing circuit 210, via the control circuit 170, causes the image sensor 160 to detect light from the object 70. The processing circuit 210 further acquires the compressed image 10 output from the imaging device 100, more specifically from the image sensor 160.

[0069] <Step S103> The processing circuit 210 obtains parameters from the vibration sensor 400 regarding the movement of the imaging device 100 during the exposure time when the compressed image 10 is captured.

[0070] <Step S104> The processing circuit 210 generates a PSF that reflects the blur and / or blur caused by the movement of the imaging device 100, based on parameters related to the movement of the imaging device 100. The PSF is obtained by integrating the motion trajectory over time. The PSF is a function that represents the amount of attenuation depending on the distance from a particular pixel. The PSF can be represented as a two-dimensional array matrix data that indicates the amount of light diffusing from a particular pixel or the amount of light attenuation. The size of the PSF array may be smaller than or the same as the size of the pixel array in the image sensor 160.

[0071] If the image sensor 160 has a global shutter, the PSF is independent of the pixel position in the image sensor 160. If the image sensor 160 has a rolling shutter, the processing circuit 210 generates a PSF for each pixel based on parameters relating to the movement of the imaging device 100 during the exposure time for each pixel. In either case, the above PSF is independent of the wavelength band and is independent of the wavelength band.

[0072] <Step S105> The processing circuit 210 obtains the restoration table from the storage device 220 and convolves the PSF into the restoration table. More specifically, the processing circuit 210 convolves the PSF into each of the multiple mask matrices included in the restoration table. If the image sensor 160 has a rolling shutter, the processing circuit 210 convolves a different PSF for each pixel into each mask matrix.

[0073] From the above, the following can be said: The processing circuit 210 edits the reconstruction table by performing a convolution process on the reconstruction table based on parameters related to the movement of the imaging device 100. More specifically, the processing circuit 210 edits the reconstruction table based on the PSF. The PSF is generated based on parameters related to the movement of the imaging device 100. In this way, the edited reconstruction table becomes a reconstruction table that corresponds to the parameters related to the movement of the imaging device 100.

[0074] The convolution process allows information about the movement of the imaging device 100 to be incorporated into the reconstruction table without changing the amount of data in the reconstruction table. Therefore, there is no need to change the reconstruction process itself for generating hyperspectral images.

[0075] Convolution is described, for example, in section 3.3.2.Linear filtering of section 3.Image processing in Richard Szeliski, “Computer Vision Algorithms and Applications”, Springer 2nd ed.

[0076] <Step S106> The processing circuit 210 uses the edited reconstruction table to generate a hyperspectral image based on the compressed image 10 through a reconstruction process. Since the edited reconstruction table corresponds to parameters related to the movement of the imaging device 100, blur and / or out-of-focus images can be reduced, making it possible to generate a more accurate hyperspectral image.

[0077] <Step S107> The processing circuit 210 outputs a hyperspectral image. The processing circuit 210 may also display the outputted hyperspectral image on the display device 300.

[0078] As described above, in the imaging system according to this embodiment, the reconstruction table, rather than the compressed image 10, is edited based on parameters related to the movement of the imaging device 100 during shooting. Therefore, in the compressed image 10, which is captured with different transmittances for each pixel according to the wavelength band, there is no need to change the pixel values ​​of the pixels. Furthermore, the influence of the parameters related to the movement of the imaging device 100 during shooting can be added to the reconstruction table, which includes information on the transmittance of each pixel according to the wavelength band. As a result, blurring and / or blurring caused by the movement of the imaging device 100 during shooting can be reduced, and hyperspectral images can be generated more accurately. Even if the reconstruction table is edited, the amount of data in the edited reconstruction table does not change, and there is no change in the processing operation of the reconstruction process, so the processing time of the reconstruction process does not increase significantly.

[0079] When generating multiple hyperspectral images during continuous shooting and video recording, the processing circuit 210 repeatedly performs the operations in steps S101 to S107. Each time the imaging device 100 captures a compressed image 10, i.e., each frame, the processing circuit 210 acquires parameters related to the movement of the imaging device 100 during the exposure time of that capture, and edits a reconstruction table based on these parameters. Therefore, a hyperspectral image can be generated more accurately for each frame.

[0080] During continuous shooting and video recording, the processing circuit 210 performs the following processing operations. The processing circuit 210 acquires a first parameter relating to the movement of the imaging device 100 during the exposure time when the first compressed image is captured, and edits the reconstruction table based on the first parameter. The processing circuit 210 further acquires a second parameter relating to the movement of the imaging device 100 during the exposure time when the second compressed image is captured, and edits the reconstruction table based on the second parameter. The first compressed image is the compressed image 10 in a certain frame, and the second compressed image is the compressed image 10 in the next frame. The second parameter is different from the first parameter.

[0081] [2.4. Example of a process operation for storing compressed images and related information in a storage device] When generating multiple hyperspectral images during continuous shooting and video recording, the processing circuit 210 does not need to generate a hyperspectral image for each frame. For example, the processing circuit 210 may output data containing multiple compressed images and multiple pieces of related information acquired during continuous shooting and video recording, and store the output data in the storage device 220. Each piece of related information is information about the PSF at the time of shooting the corresponding compressed image 10. Subsequently, the processing circuit 210 may generate multiple hyperspectral images all at once. Alternatively, the processing circuit 210 may transmit the above output data to an external device of the imaging system. The external device may generate multiple hyperspectral images all at once based on the output data.

[0082] Next, with reference to Figures 8A and 8B, an example of processing operation in which the processing circuit 210 outputs data containing a compressed image and related information will be described. Figure 8A is a flowchart schematically showing an example of processing operation performed by the processing circuit 210 in this embodiment when outputting data containing a compressed image and related information. The processing circuit 210 performs the operations of steps S101 to S105 and S108 shown in Figure 8A. The operations of steps S101 to S105 are as described with reference to Figure 7.

[0083] <Step S108> The processing circuit 210 outputs the compressed image 10 and the edited reconstruction table obtained by convolving the PSF in association. The processing circuit 210 may store the output data, which has been output in association, in the storage device 220. In that case, the processing circuit 210 can later retrieve the output data stored in the storage device 220 and easily perform other processing operations that require data consistency with the edited reconstruction table. Alternatively, the processing circuit 210 may transmit the output data, which has been output in association, to an external device of the imaging system.

[0084] Figure 8B is a flowchart schematically showing another example of the processing operations performed by the processing circuit 210 in this embodiment when outputting data including a compressed image and related information. The processing circuit 210 performs the operations of steps S101 to S104 and S109 shown in Figure 8B. The operations of steps S101 to S104 have been described with reference to Figure 7.

[0085] <Step S109> The processing circuit 210 outputs the compressed image 10 and the PSF in association. Since the amount of data in the PSF is significantly less than the amount of data in the reconstructed table after editing, the amount of data stored in the storage device 220 can be significantly reduced. The processing circuit 210 may store the output data that has been output in association in the storage device 220, or it may transmit it to an external device of the imaging system.

[0086] As described above, in the imaging device according to this embodiment, the compressed image 10 and related information may be stored in the storage device 220 in association with each other. As a result, in continuous shooting and video shooting, it is not necessary to generate a hyperspectral image for each frame, but rather multiple hyperspectral images can be generated more accurately at once later.

[0087] [3. Imaging system 1 for correcting blur caused by the lens and aperture] [3.1. Example configuration of an imaging system] An example of an imaging system for correcting blur caused by the optical system 140 is described below with reference to Figure 9. Figure 9 is a schematic block diagram showing the configuration of an imaging system according to an exemplary embodiment of the present disclosure for correcting blur caused by the optical system 140. The imaging system shown in Figure 9 comprises an imaging device 100, an image processing device 200, a display device 300, and an input device 500.

[0088] The imaging device 100 captures an image of the object 70 and outputs a compressed image 10 in which spectral information has been compressed. The imaging device 100 includes an optical system 140 including a lens 142 and an aperture 144, a filter array 110, an image sensor 160, and a control circuit 170.

[0089] Lens 142 is a lens with a changeable focal length, i.e., a variable-focus lens. Aperture 144 is an aperture whose aperture value, which determines the depth of field, can be changed. The optical system 140 may be a lens system consisting of lens 142 and aperture 144 as a pair. The filter array 110 and image sensor 160 are described with reference to, for example, Figure 5.

[0090] The control circuit 170 obtains shooting parameters for the imaging device 100, more specifically, shooting parameters related to the optical system 140 included in the imaging device 100, from the input device 500 and adjusts the optical system 140. The shooting parameters may include, for example, information regarding the focal length of the lens 142 and / or the aperture value of the aperture 144. The control circuit 170 further transmits the shooting parameters related to the optical system 140 to the processing circuit 210. The control circuit 170 may further control the operation of components not shown included in the imaging device 100. The control circuit 170 may obtain the shooting parameters from the optical system 140 instead of from the input device 500.

[0091] The input device 500 acquires shooting parameters related to the optical system 140 based on input from the user or from an external system, and inputs these shooting parameters to the control circuit 170.

[0092] The image processing device 200 comprises a processing circuit 210 and a storage device 220. The processing circuit 210 generates a hyperspectral image by a reconstruction process based on the compressed image 10. The storage device 220 stores a reconstruction table used in the hyperspectral image reconstruction process and shooting parameters corresponding to the reconstruction table, which are the shooting parameters used when the reconstruction table was generated. The storage device 220 further stores one or more first PSF groups corresponding to one or more focal lengths of the lens 142, and one or more second PSF groups corresponding to one or more aperture values ​​of the aperture 144. The first PSF groups and second PSF groups are described below.

[0093] [3.2. Examples of data formats for the recovery table, the first PSF group, and the second PSF group] [3.2.1. Example of Data Format 1] The following describes examples of the data formats for the recovery table, one or more first PSF groups, and one or more second PSF groups stored in the storage device 220, with reference to Figure 10A. Figure 10A is a schematic diagram showing examples of the data formats for the recovery table, the first PSF groups, and the second PSF groups. The upper part of Figure 10A shows an example of the data format for the recovery table. The example of the data format for the recovery table is as explained with reference to Figure 6. The middle part of Figure 10A shows an example of the data format for the first PSF group. The lower part of Figure 10A shows an example of the data format for the second PSF group.

[0094] As shown in the center diagram of Figure 10A, one first PSF group includes multiple PSF parameter matrices corresponding to multiple wavelength bands, depending on the focal length of the lens 142. Each PSF parameter matrix includes multiple matrix elements, each containing two-dimensional data of multiple PSF parameters corresponding to multiple pixels, depending on the focal length of the lens 142. Each of the matrix elements corresponds to one of the multiple pixels contained in the compressed image 10. These PSF parameters are used to generate a first PSF that reflects the blur caused by the lens 142. The first PSF depends on the pixel position in the image sensor 160 and on the wavelength band.

[0095] As shown in the lower diagram of Figure 10A, the second PSF group includes multiple PSF parameter matrices corresponding to multiple wavelength bands. Each PSF parameter matrix contains multiple matrix elements, each containing two-dimensional data of multiple PSF parameters corresponding to multiple pixels, depending on the aperture value of aperture 144. Each of the matrix elements corresponds to one of the multiple pixels contained in the compressed image 10. These PSF parameters are used to generate a second PSF that reflects the blur caused by aperture 144. The second PSF depends on the pixel position in the image sensor 160 and on the wavelength band.

[0096] PSF parameters can be, for example, damping constants. When multiple PSF parameters are used to generate a single PSF, there are multiple sets of multiple PSF parameter matrices, as shown in the center figure of Figure 10A. The number of sets is equal to the number of PSF parameters. Similarly, there are multiple sets of multiple PSF parameter matrices, as shown in the lower figure of Figure 10A.

[0097] The number of data points in one or more first PSF groups stored in the memory device 220 is the product of the number of PSF parameters used to generate one PSF, the number of pixels in two dimensions, the number of wavelength bands, and the number of focal lengths. Similarly, the number of data points in one or more second PSF groups stored in the memory device 220 is the product of the number of PSF parameters used to generate one PSF, the number of pixels in two dimensions, the number of wavelength bands, and the number of aperture values.

[0098] In the subsequent processing operation for generating hyperspectral images, a first PSF group corresponding to a desired focal length is selected from one or more first PSF groups stored in the memory device 220. Similarly, a second PSF group corresponding to a desired aperture value is selected from one or more second PSF groups stored in the memory device 220.

[0099] On the other hand, in one or more first PSF groups stored in the memory device 220, there may be cases where multiple wavelength bands in each first PSF group do not match multiple wavelength bands in the reconstruction table. In such cases, a PSF parameter matrix corresponding to each wavelength band may be generated by linear interpolation using the PSF parameter matrix corresponding to the wavelength band closest to each wavelength band in the reconstruction table, and the PSF parameter matrix corresponding to the second closest wavelength band. Linear interpolation is one example of interpolation.

[0100] Similarly, in one or more second PSF groups stored in the memory device 220, there may be cases where multiple wavelength bands in each second PSF group do not match multiple wavelength bands in the recovery table. In such cases, a PSF parameter matrix corresponding to each wavelength band may be generated by linear interpolation similar to that described above.

[0101] In one or more first PSF groups stored in the memory device 220, there may be no first PSF group corresponding to a desired focal length. In that case, a first PSF group corresponding to the desired focal length may be generated by linear interpolation using the first PSF group corresponding to the focal length closest to the desired focal length and the first PSF group corresponding to the second closest focal length. More specifically, a PSF parameter matrix corresponding to a certain wavelength band may be generated by linear interpolation using a PSF parameter matrix corresponding to a certain wavelength band included in one first PSF group and a PSF parameter matrix corresponding to the same wavelength band included in the other first PSF group.

[0102] Similarly, in one or more second PSF groups stored in the memory device 220, there may be no second PSF group corresponding to the desired aperture value 144. In that case, a second PSF group corresponding to the desired aperture value may be generated by linear interpolation as described above.

[0103] [3.2.2. Example of Data Format 2] The memory device 220 may store an edited reconstruction table based on one or more first PSF groups, and one or more second PSF groups. The edited reconstruction table is obtained by convolving the first PSFs, which reflect the blur caused by the lens 142, into the reconstruction table.

[0104] Next, with reference to Figure 10B, examples of the data formats for the reconstructed table after editing based on one or more first PSF groups, and one or more second PSF groups, will be described. Figure 10B is a schematic diagram showing examples of the data formats for the reconstructed table after editing based on the first PSF groups, and second PSF groups. The upper part of Figure 10B shows an example of the data format for the reconstructed table after editing. The lower part of Figure 10B shows an example of the data format for the second PSF groups. The data format for the second PSF groups is as described with reference to Figure 10A.

[0105] As shown in the upper diagram of Figure 10B, the edited reconstruction table contains multiple edited mask matrices corresponding to multiple wavelength bands. Each edited mask matrix is ​​obtained by convolving the first PSF, which reflects the blur caused by the lens 142, into the mask matrix pixel by pixel, depending on the focal length. The number of data points in the edited reconstruction table stored in the memory device 220 is the product of the number of pixels in two dimensions, the number of wavelength bands, and the number of focal lengths.

[0106] In the subsequent processing operation for generating hyperspectral images, an edited reconstruction table corresponding to the desired focal length is selected from one or more edited reconstruction tables based on a group of first PSFs stored in the storage device 220.

[0107] On the other hand, in the edited reconstruction tables based on one or more first PSF groups stored in the memory device 220, there may be cases where no reconstruction table corresponding to the desired focal length exists. In such cases, the processing circuit 210 may generate an edited reconstruction table corresponding to the desired focal length by linear interpolation using the edited reconstruction table corresponding to the focal length closest to the desired focal length and the edited reconstruction table corresponding to the second closest focal length. More specifically, an edited mask matrix corresponding to a certain wavelength band may be generated by linear interpolation using an edited mask matrix corresponding to a certain wavelength band included in one edited reconstruction table and an edited mask matrix corresponding to the same wavelength band included in the other edited reconstruction table.

[0108] [3.2.3. Example of Data Format 3] The memory device 220 may store an edited reconstruction table based on one or more second PSF groups, and one or more first PSF groups. The edited reconstruction table is obtained by convolving the second PSFs, which reflect the blur caused by aperture 144, into the reconstruction table.

[0109] Next, with reference to Figure 10C, examples of the data formats for the reconstructed table after editing based on one or more second PSF groups, and one or more first PSF groups, will be described. Figure 10C is a schematic diagram showing examples of the reconstructed table after editing based on second PSF groups, and examples of the data formats for first PSF groups. The upper part of Figure 10C shows an example of the data format for the reconstructed table after editing. The lower part of Figure 10C shows an example of the data format for first PSF groups. The data format for first PSF groups is as described with reference to Figure 10A.

[0110] As shown in the upper part of Figure 10C, the edited reconstruction table contains multiple edited mask matrices corresponding to multiple wavelength bands. Each edited mask matrix is ​​obtained by convolving a second PSF, which reflects the blur caused by aperture 144, into the mask matrix pixel by pixel, depending on the aperture value. The number of data points in the edited reconstruction table stored in the memory device 220 is the product of the number of pixels in two dimensions, the number of wavelength bands, and the number of aperture values.

[0111] In the subsequent processing operation for generating hyperspectral images, an edited reconstruction table corresponding to a desired aperture value is selected from one or more edited reconstruction tables based on a second PSF group stored in the memory device 220.

[0112] On the other hand, in the edited restoration tables based on one or more second PSF groups stored in the memory device 220, there may be cases where no restoration table corresponding to the desired aperture value exists. In such cases, an edited restoration table corresponding to the desired aperture value may be generated by linear interpolation using the edited restoration table corresponding to the aperture value closest to the desired aperture value and the edited restoration table corresponding to the second closest aperture value. More specifically, an edited mask matrix corresponding to a certain wavelength band may be generated by linear interpolation using an edited mask matrix corresponding to a certain wavelength band included in one edited restoration table and an edited mask matrix corresponding to the same wavelength band included in the other edited restoration table.

[0113] [3.2.4. Example of Data Format 4] The storage device 220 may store an edited reconstruction table based on one or more first PSF groups and one or more second PSF groups. The edited reconstruction table is obtained by convolving the first PSF, which reflects the blur caused by the lens 142, and the second PSF, which reflects the blur caused by the aperture 144, into the reconstruction table.

[0114] Next, with reference to Figure 10D, an example of the data format of the edited recovery table based on one or more first PSF groups and one or more second PSF groups will be described. Figure 10D is a schematic diagram showing an example of the data format of the edited recovery table based on the first PSF groups and second PSF groups. The storage device 220 has focal lengths f1, f2, ..., f k and aperture values ​​F1, F2, ..., F l Multiple edited restoration tables corresponding to multiple combinations of the above are stored. Each edited restoration table contains multiple edited mask matrices corresponding to multiple wavelength bands. Each edited mask matrix is ​​obtained by convolving a first PSF reflecting the blur caused by lens 142 and a second PSF reflecting the blur caused by aperture 144 into the mask matrix, depending on the focal length and aperture value.

[0115] In the subsequent processing operation for generating hyperspectral images, an edited reconstruction table corresponding to a desired focal length and desired aperture value is selected from an edited reconstruction table based on one or more first PSF groups and one or more second PSF groups stored in the storage device 220.

[0116] On the other hand, in the edited reconstruction table based on one or more first PSF groups and one or more second PSF groups stored in the memory device 220, there may be cases where an edited reconstruction table corresponding to the desired focal length and the desired aperture value does not exist. In that case, since linear interpolation is not easy, the edited reconstruction table corresponding to the focal length closest to the desired focal length and the aperture value closest to the desired aperture value may be used as the edited reconstruction table corresponding to the desired focal length and the desired aperture value. To find a focal length as close as possible to the desired focal length, focal lengths f1, f2, ..., f k Regarding this, the interval between focal lengths may be made smaller, and the number of focal lengths may be increased. Similarly, to find an aperture value as close as possible to the desired aperture value, aperture values ​​F1, F2, ..., F l Regarding this, the interval between aperture values ​​can be made smaller, and the number of aperture values ​​can be increased.

[0117] [3.3. Example of processing operation for generating hyperspectral images] Next, with reference to Figure 11, an example of the processing operation by which the processing circuit 210 generates a hyperspectral image will be described. Here, the storage device 220 stores a recovery table, one or more first PSF groups, and one or more second PSF groups, as shown in Figure 10A. Figure 11 is a flowchart schematically showing an example of the processing operation performed by the processing circuit 210 in this embodiment when generating a hyperspectral image. The processing circuit 210 performs the operations of steps S201 to S209 shown in Figure 11.

[0118] <Step S201> The processing circuit 210 acquires imaging parameters related to the optical system 140 from the imaging device 100, or more specifically, from the control circuit 170 included in the imaging device 100.

[0119] <Step S202> The processing circuit 210 causes the imaging device 100 to capture a compressed image 10. More specifically, the processing circuit 210, via the control circuit 170, causes the image sensor 160 to detect light from the object 70. The processing circuit 210 further acquires the compressed image 10 output from the imaging device 100, more specifically from the image sensor 160.

[0120] The processing circuit 210 may perform the operation of step S202 before step S201.

[0121] <Step S203> The processing circuit 210 determines whether the shooting parameters at the time of shooting are different from the shooting parameters at the time of generating the reconstruction table. If the determination is yes, the processing circuit 210 performs the operation of step S204. If the determination is no, the processing circuit 210 performs the operation of step S209. The processing circuit 210 may, for example, be configured before this determination to generate a hyperspectral image by a reconstruction process based on the compressed image 10 using the reconstruction table stored in the storage device 220. In this specification, the shooting parameters at the time of generating the reconstruction table are also referred to as "first shooting parameters," and the shooting parameters at the time of shooting are also referred to as "second shooting parameters."

[0122] <Step S204> The processing circuit 210 obtains a first PSF group from the memory device 220 that corresponds to the focal length of the lens 142 at the time of shooting. Based on this first PSF group, the processing circuit 210 generates multiple first PSFs for each of the multiple mask matrices in the reconstruction table, to be convolved into multiple pixels. The first PSFs reflect the blur caused by the lens 142.

[0123] <Step S205> The processing circuit 210 obtains a second PSF group from the memory device 220 that corresponds to the aperture value of aperture 144 at the time of shooting. Based on this second PSF group, the processing circuit 210 generates multiple second PSFs for each of the multiple mask matrices in the reconstruction table, to be convolved into multiple pixels. The second PSFs reflect the blur caused by aperture 144.

[0124] <Step S206> The processing circuit 210 obtains a reconstruction table from the memory device 220 and convolves a PSF that reflects the blur caused by the optical system 140 onto this reconstruction table. More specifically, the processing circuit 210 convolves a first PSF onto each of the multiple mask matrices included in the reconstruction table, pixel by pixel, and then convolves a second PSF onto each of those pixels.

[0125] Alternatively, if the storage device 220 stores an edited restoration table based on one or more first PSF groups and one or more second PSF groups, as shown in Figure 10B, the processing circuit 210 may not perform the operation in step S204, but in step S206, obtain the edited restoration table based on the first PSF group corresponding to the focal length of the lens 142 at the time of shooting from the storage device 220. In that case, the processing circuit 210 convolves the second PSFs pixel by pixel into each of the multiple mask matrices included in the edited restoration table based on the first PSF group.

[0126] Alternatively, if the storage device 220 stores an edited restoration table based on one or more second PSF groups and one or more first PSF groups, as shown in Figure 10C, the processing circuit 210 may not perform the operation in step S205, but in step S206, obtain the edited restoration table corresponding to the aperture value of aperture 144 at the time of shooting from the storage device 220. In that case, the processing circuit 210 convolves the first PSF pixel by pixel into each of the multiple mask matrices included in the edited restoration table.

[0127] Alternatively, if the storage device 220 stores an edited restoration table based on one or more first PSF groups and one or more second PSF groups, as shown in Figure 10D, the processing circuit 210 may not perform the operations in steps S204 and S205, but instead in step S206 obtain the edited restoration table corresponding to the focal length of the lens 142 and the aperture value of the aperture 144 at the time of shooting from the storage device 220. In this case, since it is not necessary to edit the restoration table in step S206, the processing load and time can be reduced.

[0128] From the above, the following can be said: The processing circuit 210 obtains an edited restoration table by editing the restoration table based on the shooting parameters at the time of shooting. More specifically, the processing circuit 210 obtains an edited restoration table by editing the restoration table by convolving the first PSF and second PSF corresponding to the shooting parameters at the time of shooting into the restoration table. In this way, the edited restoration table becomes a restoration table corresponding to the shooting parameters at the time of shooting.

[0129] As shown in Figure 10A, the storage device 220 stores multiple first PSF groups corresponding to multiple shooting parameters and multiple second PSF groups corresponding to multiple shooting parameters. The processing circuit 210 selects a first PSF group from the multiple first PSF groups that corresponds to the shooting parameters at the time of shooting, and selects a second PSF group from the multiple second PSF groups that corresponds to the shooting parameters at the time of shooting. The processing circuit 210 further edits the restoration table based on the selected first and second PSF groups to obtain the edited restoration table.

[0130] In some cases, among the multiple first PSF groups stored in the memory device 220, there may not be a first PSF group that matches the shooting parameters at the time of shooting, or among the multiple second PSF groups stored in the memory device 220, there may not be a second PSF group that matches the shooting parameters at the time of shooting. In such cases, the processing circuit 210 treats the first PSF group generated by interpolation using the multiple first PSF groups stored in the memory device 220 as the first PSF group corresponding to the shooting parameters at the time of shooting. Similarly, the processing circuit 210 treats the second PSF group generated by interpolation using the multiple second PSF groups stored in the memory device 220 as the second PSF group corresponding to the shooting parameters at the time of shooting.

[0131] Alternatively, the storage device 220 stores multiple post-edited reconstruction tables corresponding to multiple imaging parameters, as shown in Figures 10B to 10D. These multiple post-edited reconstruction tables are used to generate a hyperspectral image through a reconstruction process. Each post-edited reconstruction table is based on a first PSF group and / or a second PSF group. The processing circuit 210 obtains the post-edited reconstruction tables based on the first PSF group and the second PSF group from among the multiple post-edited reconstruction tables stored in the storage device 220, based on the post-edited reconstruction table corresponding to the imaging parameters at the time of imaging.

[0132] In some cases, among the multiple post-edited restoration tables stored in the storage device 220, there may not be a post-edited restoration table that matches the shooting parameters at the time of shooting. In such cases, as shown in the examples in Figures 10B and 10C, the processing circuit 210 treats the post-edited restoration table generated by interpolation using the multiple post-edited restoration tables stored in the storage device 220 as the post-edited restoration table corresponding to the shooting parameters at the time of shooting. In the example shown in Figure 10D, the processing circuit 210 treats the post-edited restoration table that is closest to the shooting parameters at the time of shooting among the multiple post-edited restoration tables stored in the storage device 220 as the post-edited restoration table corresponding to the shooting parameters at the time of shooting.

[0133] <Step S207> The processing circuit 210 generates a hyperspectral image based on the compressed image 10 through a reconstruction process using an edited reconstruction table based on the first and second PSF groups. Since the edited reconstruction table corresponds to the shooting parameters at the time of shooting, blurring caused by the optical system 140 can be reduced, making it possible to generate a hyperspectral image more accurately.

[0134] <Step S208> The processing circuit 210 outputs a hyperspectral image. The processing circuit 210 may also display the outputted hyperspectral image on the display device 300.

[0135] <Step S209> The processing circuit 210 uses the restoration table stored in the memory device 220 to generate a hyperspectral image based on the compressed image 10 through a reconstruction process.

[0136] In steps S204 to S207, the focal length of lens 142 at the time of shooting differs from the focal length of lens 142 when the restoration table is generated, and the aperture value of aperture 144 at the time of shooting differs from the aperture value of aperture 144 when the restoration table is generated. If the focal length of lens 142 at the time of shooting is the same as the focal length of lens 142 when the restoration table is generated, it is not necessary to convolve the first PSF into the restoration table. If the aperture value of aperture 144 at the time of shooting is the same as the aperture value of aperture 144 when the restoration table is generated, it is not necessary to convolve the second PSF into the restoration table.

[0137] As described above, in the imaging system according to this embodiment, the reconstruction table, rather than the compressed image 10, is edited based on the shooting parameters related to the optical system 140 at the time of shooting. Therefore, in the compressed image 10, which is captured with different transmittances for each pixel according to the wavelength band, there is no need to change the pixel values ​​of the pixels. Furthermore, the effect of the shooting parameters related to the optical system 140 at the time of shooting can be added to the reconstruction table, which includes information on the transmittance of each pixel according to the wavelength band. As a result, blurring caused by the optical system 140 at the time of shooting can be reduced, and hyperspectral images can be generated more accurately.

[0138] In this specification, the unedited restoration table is also referred to as the "first matrix data," and the edited restoration table is also referred to as the "second matrix data." The first PSF group, the second PSF group, and the edited restoration table based on the first PSF group and / or the second PSF group stored in the storage device 220 are also referred to as the "third matrix data."

[0139] [3.4. Other examples of processing operations that generate hyperspectral images] When changing the shooting parameters related to the optical system 140 during continuous shooting and video shooting, the processing circuit 210 may generate multiple hyperspectral images as follows.

[0140] The following describes another example of the processing operation by which the processing circuit 210 generates a hyperspectral image, with reference to Figure 12. Figure 12 is a flowchart schematically showing an example of the processing operation performed by the processing circuit 210 in this embodiment when generating a hyperspectral image. The processing circuit 210 performs the operations of steps S301 to S310 shown in Figure 12.

[0141] <Step S301> The processing circuit 210 determines whether or not to terminate the shooting. If the determination is yes, the processing circuit 210 executes the operation in step S309. If the determination is no, the processing circuit 210 executes the operation in step S302.

[0142] <Step S302> The processing circuit 210 determines whether or not to change the focal length of the lens 142 during shooting. If the determination is yes, the processing circuit 210 executes the operation in step S303. If the determination is no, the processing circuit 210 executes the operation in step S304.

[0143] <Step S303> The processing circuit 210 obtains the first PSF group corresponding to the changed focal length of the lens 142 from the storage device 220.

[0144] <Step S304> The processing circuit 210 determines whether or not to change the aperture value of aperture 144 during shooting. If the determination is yes, the processing circuit 210 executes the operation in step S305. If the determination is no, the processing circuit 210 executes the operation in step S306.

[0145] <Step S305> The processing circuit 210 acquires the second PSF group corresponding to the changed aperture value of aperture 144 from the storage device 220.

[0146] <Step S306> The processing circuit 210 causes the imaging device 100 to capture the compressed image 10.

[0147] <Step S307> The processing circuit 210 obtains a reconstruction table from the memory device 220 and convolves the PSF, which reflects the blur caused by the optical system 140, into this reconstruction table.

[0148] <Step S308> The processing circuit 210 associates the compressed image 10 with the edited table obtained by convolving the PSF and stores it in the memory device 220.

[0149] <Step S309> The processing circuit 210 retrieves multiple compressed images 10 and multiple edited tables from the storage device 220, and uses the multiple edited restoration tables to generate multiple hyperspectral images based on the multiple compressed images 10 through a reconstruction process. Each hyperspectral image is generated by the reconstruction process based on the corresponding compressed image 10 using the corresponding edited restoration table.

[0150] <Step S310> The processing circuit 210 outputs multiple hyperspectral images. The processing circuit 210 may also display the outputted multiple hyperspectral images on the display device 300.

[0151] As described above, in the imaging system according to this embodiment, in continuous shooting and video shooting, after storing multiple compressed images and multiple edited restoration tables in the storage device 220, multiple hyperspectral images can be generated more accurately all at once. When generating a hyperspectral image for each frame, the processing circuit 210 is required to have the processing capability to generate a hyperspectral image in one frame. In contrast, when generating multiple hyperspectral images all at once, the processing circuit 210 does not need to have such processing capability.

[0152] In the example shown in Figure 12, the processing circuit 210 generates and outputs multiple hyperspectral images in steps S309 and S310 after a series of imaging in steps S301 to S308, but is not limited to this example. The processing circuit 210 may terminate its processing operation after a series of imaging in steps S301 to S308. After that, the processing circuit 210 may resume its processing operation and generate and output multiple hyperspectral images in steps S309 and S310.

[0153] [4. Imaging system 2 for correcting blur caused by the lens and aperture] In the imaging system shown in Figure 9, if the optical system 140 included in the imaging device 100 is a single interchangeable lens system, one or more first PSF groups and one or more second PSF groups may be acquired from an external server when the lens system is replaced or during imaging.

[0154] [4.1. Example configuration of an imaging system] An example of an imaging system for correcting blur caused by the optical system 140 is described below with reference to Figure 13. Figure 13 is a schematic block diagram showing another configuration of an imaging device according to an exemplary embodiment of the present disclosure for correcting blur. The imaging system shown in Figure 13 differs from the imaging system shown in Figure 9 in that it further comprises a communication device 600. An external server 700 is also shown in Figure 13. The external server 700 comprises a communication device 710 and a storage device 720. The storage device 720 stores one or more first PSF groups and one or more second PSF groups for each of a plurality of lens systems.

[0155] [4.2. Example of operation to acquire one or more first PSF groups and one or more second PSF groups corresponding to the lens system model number] Next, with reference to Figure 14, an example of the processing operation for acquiring one or more first PSF groups and one or more second PSF groups corresponding to the lens system model number will be described. Figure 14 is a sequence diagram illustrating an example of the processing operation for acquiring one or more first PSF groups and one or more second PSF groups corresponding to the lens system model number in the imaging system according to this embodiment. The control circuit 170 included in the imaging system performs the operations in steps S401 and S402. The processing circuit 210 included in the imaging system performs the operations in steps S403 to S405, S409, and S410. The communication device 710 included in the external server 700 performs the operations in steps S406 to S408.

[0156] <Step S401> When a lens system is mounted as optical system 140, or when the imaging device 100 is started with this lens system mounted, the control circuit 170 obtains the model number that identifies the lens system from optical system 140.

[0157] <Step S402> The control circuit 170 transmits the lens system model number to the processing circuit 210.

[0158] <Step S403> The processing circuit 210 obtains the model number of the lens system.

[0159] <Step S404> The processing circuit 210 compares the acquired lens system model number with the model numbers of one or more first PSF groups and one or more second PSF groups of lens systems stored in the storage device 220 to determine whether the acquired lens system model number is different from all the lens system model numbers stored in the storage device 220. If the determination is yes, the processing circuit 210 executes the operation in step S405. If the determination is no, the processing circuit 210 terminates the processing operation.

[0160] <Step S405> The processing circuit 210 transmits the acquired lens system model number to the communication device 710 of the external server 700 via the communication device 600 in order to request the acquired lens system model number from the external server 700.

[0161] <Step S406> The communication device 710 receives the model number of the lens system.

[0162] <Step S407> The communication device 710 obtains one or more first PSF groups and one or more second PSF groups of the requested model numbers from the processing circuit 210 from the storage device 220.

[0163] <Step S408> The communication device 710 transmits one or more first PSF groups and one or more second PSF groups of the requested model numbers from the processing circuit 210 to the communication device 600 of the imaging system.

[0164] <Step S409> The processing circuit 210 receives one or more first PSF groups and one or more second PSF groups via the communication device 600.

[0165] <Step S410> When the storage device 220 stores the data shown in Figure 10A, the processing circuit 210 causes one or more first PSF groups and one or more second PSF groups to be stored in the storage device 220. When the storage device 220 stores the data shown in Figure 10B, the processing circuit 210 causes the storage device 220 to store an edited restoration table based on one or more first PSF groups and one or more second PSF groups. When the storage device 220 stores the data shown in Figure 10C, the processing circuit 210 causes the storage device 220 to store an edited restoration table based on one or more second PSF groups and one or more first PSF groups. When the storage device 220 stores the data shown in Figure 10D, the processing circuit 210 causes the storage device 220 to store an edited restoration table based on one or more first PSF groups and one or more second PSF groups.

[0166] [5. Imaging system that corrects blur and / or blur caused by movement of the imaging device, as well as blur caused by the lens and / or aperture.] [5.1. Example configuration of an imaging system] Below, with reference to Figure 15, an example of an imaging system that corrects blur and / or blur caused by the movement of the imaging device 100, as well as blur caused by the optical system 140, is described. Figure 15 is a schematic block diagram showing the configuration of an imaging system according to an exemplary embodiment of the present disclosure that corrects blur and / or blur caused by the movement of the imaging device 100, as well as blur caused by the optical system 140. The imaging system shown in Figure 15 comprises an imaging device 100, an image processing device 200, a display device 300, a vibration sensor 400, and an input device 500. These components have been described with reference to Figures 5 and 9.

[0167] [5.2. Example of processing operation for generating hyperspectral images] Next, with reference to Figure 16, an example of the processing operation by which the processing circuit 210 generates a hyperspectral image will be described. Figure 16 is a flowchart schematically showing an example of the processing operation performed by the processing circuit 210 in this embodiment when generating a hyperspectral image. The processing circuit 210 performs the operations of steps S501 to S510 shown in Figure 16.

[0168] <Step S501> The processing circuit 210 acquires imaging parameters related to the optical system 140 from the imaging device 100, or more specifically, from the control circuit 170 included in the imaging device 100.

[0169] <Step S502> The processing circuit 210 causes the vibration sensor 400 to start a detection operation to detect the movement of the imaging device 100.

[0170] <Step S503> The processing circuit 210 causes the imaging device 100 to capture the compressed image 10.

[0171] <Step S504> The processing circuit 210 determines whether the shooting parameters at the time of shooting are different from the shooting parameters at the time of generating the restoration table. If the determination is yes, the processing circuit 210 executes the operation in step S505. If the determination is no, the processing circuit 210 executes the operation in step S507.

[0172] <Step S505> The processing circuit 210 retrieves a first PSF group from the storage device 220 that corresponds to the acquired focal length of the lens 142. Based on this first PSF group, the processing circuit 210 generates multiple first PSFs for each of the multiple mask matrices in the reconstruction table, to be convolved into multiple pixels. The first PSFs reflect the blur caused by the lens 142.

[0173] If the focal length of lens 142 at the time of shooting is the same as the focal length of lens 142 when generating the restoration table, the processing circuit 210 does not perform the operation in step S505.

[0174] <Step S506> The processing circuit 210 retrieves a second PSF group from the storage device 220 that corresponds to the aperture value of aperture 144 that was acquired. Based on this second PSF group, the processing circuit 210 generates multiple second PSFs for each of the multiple mask matrices in the reconstruction table, to be convolved into multiple pixels. The second PSFs reflect the blur caused by aperture 144.

[0175] If the aperture value of aperture 144 at the time of shooting is the same as the aperture value of aperture 144 at the time of generating the restoration table, the processing circuit 210 does not perform the operation in step S506.

[0176] <Step S507> The processing circuit 210 generates a third PSF that reflects the blur and / or blur caused by the movement of the imaging device 100.

[0177] <Step S508> The processing circuit 210 obtains a recovery table from the storage device 220 and convolves a PSF into this recovery table. If the processing circuit 210 performs the operations in steps S505 and S506, it convolves three types of PSFs into the recovery table. If the processing circuit 210 performs one of the operations in steps S505 and S506 and does not perform the other, it convolves two types of PSFs into the recovery table. If the processing circuit 210 does not perform the operations in steps S505 and S506, it convolves one type of PSF into the recovery table.

[0178] <Step S509> The processing circuit 210 uses the edited restoration table to generate a hyperspectral image based on the compressed image 10 through a reconstruction process.

[0179] <Step S510> The processing circuit 210 outputs a hyperspectral image. The processing circuit 210 may also display the outputted hyperspectral image on the display device 300.

[0180] As described above, in the imaging system according to this embodiment, the restoration table is edited based on parameters related to the movement of the imaging device 100 during shooting and the shooting parameters related to the optical system 140 during shooting, rather than the compressed image 10. Therefore, blur and / or blur caused by the movement of the imaging device 100 during shooting, as well as blur caused by the optical system 140 during shooting, can be reduced, making it possible to generate hyperspectral images more accurately.

[0181] In the imaging system according to this embodiment, the restoration table may be edited based on at least one of the parameters related to the movement of the imaging device 100 during shooting and the shooting parameters related to the optical system 140 during shooting. At least one of the parameters is obtained from at least one of the imaging device 100 and the vibration sensor 400 attached to the imaging device 100.

[0182] [6. Example of output data format] The following describes examples of the data format of the output data output from the processing circuit 210, which is stored in the storage device 220 so that a hyperspectral image can be generated later, with reference to Figures 17A to 17H.

[0183] Figure 17A schematically shows Example 1 of the output data format. The output data shown in Figure 17A includes a compressed image 10 and a reconstruction table. Therefore, if there is no movement of the imaging device 100 during shooting, and the shooting parameters for the optical system 140 during shooting are the same as the shooting parameters when the reconstruction table is generated, then in the imaging systems shown in Figures 5, 9, and 15, a hyperspectral image can be generated based on the output data, regardless of whether the storage device 220 stores the reconstruction table or not.

[0184] Figure 17B schematically shows Example 2 of the output data format. The output data shown in Figure 17B includes a compressed image 10 and a PSF that reflects blur and / or blur caused by the movement of the imaging device 100 during shooting. Alternatively, the output data may include the compressed image 10 and an edited reconstruction table into which the above PSF is convolved. In the imaging system shown in Figure 5, even if there is blur and / or blur caused by the movement of the imaging device 100, a hyperspectral image can be generated more accurately based on the output data.

[0185] If the storage device 220 stores the recovery table, the output data does not need to include the recovery table. Conversely, if the storage device 220 does not store the recovery table, the output data includes the recovery table in addition to the compressed image 10 and the PSF mentioned above. Alternatively, the output data includes the compressed image 10 and the edited recovery table into which the PSF mentioned above is folded. The same applies to the output data described below.

[0186] Figure 17C schematically illustrates Example 3 of the output data format. Unlike the output data shown in Figure 17B, the output data shown in Figure 17C includes the parameter set of the PSF used to generate the PSF, instead of the PSF that reflects the blur and / or blur at the time of shooting of the compressed image 10. Thus, since the output data does not need to include the PSF itself, the amount of data in the output data can be significantly reduced.

[0187] Figure 17D schematically shows Example 4 of the output data format. The output data shown in Figure 17D includes the compressed image 10 and the first and second PSF groups at the time of acquisition. Alternatively, the output data may include the compressed image 10, an edited reconstruction table based on the first PSF group at the time of acquisition, and the second PSF group at the time of acquisition. The output data may include the compressed image 10, an edited reconstruction table based on the second PSF group at the time of acquisition, and the first PSF group at the time of acquisition. The output data may include the compressed image 10 and an edited reconstruction table based on the first and second PSF groups at the time of acquisition. In the imaging system shown in Figure 9, even if there is blurring caused by the optical system 140, a hyperspectral image can be generated more accurately based on the output data.

[0188] Figure 17E is a schematic diagram illustrating Example 5 of the output data format. The output data shown in Figure 17E includes a compressed image 10 and PSF IDs to identify the first and second PSF groups at the time of shooting. The storage device 220 stores one or more IDs, one or more first PSF groups, and one or more second PSF groups. Each ID is associated with a first PSF group corresponding to a certain focal length and a second PSF group corresponding to a certain aperture value. Instead of the first PSF group, an edited restoration table based on the first PSF group may be used, or instead of the second PSF group, an edited restoration table based on the second PSF group may be used. Alternatively, instead of the first and second PSF groups, an edited restoration table based on the first and second PSF groups may be used.

[0189] Based on the PSF IDs included in the output data, a specific group of first and second PSFs can be selected from one or more first PSF groups and one or more second PSF groups stored in the storage device 220. In the imaging system shown in Figure 9, even if there is blurring caused by the optical system 140, a hyperspectral image can be generated more accurately based on the output data.

[0190] When using the interpolated first and second PSF groups, the output data shown in Figure 17E may include the compressed image 10, the IDs of multiple PSFs, and information for interpolation. Each ID of a multiple PSF is associated with multiple first PSF groups and multiple second PSF groups used for interpolation. Each PSF ID is associated with one first PSF group and one second PSF group. The information for interpolation may be, for example, information about the interpolation coefficients. Instead of the first PSF group, an edited reconstruction table based on the first PSF group may be used, or instead of the second PSF group, an edited reconstruction table based on the second PSF group may be used. The same applies to the output data described below.

[0191] Figure 17F is a schematic diagram illustrating Example 6 of the output data format. In continuous shooting and video shooting, the output data shown in Figure 17F includes a header, a series of multiple compressed images 10, and multiple PSF IDs corresponding to each of the multiple compressed images 10. The multiple compressed images 10 may be arranged, for example, in order of shooting time.

[0192] The header includes one or more IDs, one or more first PSF groups corresponding to the lens set used during shooting, and one or more second PSF groups corresponding to the lens set used during shooting. Each ID in the header is associated with a first PSF group corresponding to a certain focal length and a second PSF group corresponding to a certain aperture value. Instead of the first PSF groups, an edited reconstruction table based on the first PSF groups may be used, or instead of the second PSF groups, an edited reconstruction table based on the second PSF groups may be used. Alternatively, instead of the first and second PSF groups, an edited reconstruction table based on the first and second PSF groups may be used. The header may further include, for example, information about the shooting time, number of frames, and the lens set used during shooting. The same applies to the output data described below.

[0193] Based on the PSF IDs included in the output data, a specific group of first and second PSFs can be selected from one or more first PSF groups and one or more second PSF groups included in the header. In the imaging system shown in Figure 9, even if there is blurring caused by the optical system 140, multiple hyperspectral images can be generated more accurately based on the output data.

[0194] Figure 17G schematically shows Example 7 of the output data format. In continuous shooting and video shooting, the output data shown in Figure 17G includes a header, a series of multiple compressed images 10, and multiple PSF IDs corresponding to each of the multiple compressed images 10. The multiple compressed images 10 may be arranged, for example, in order of shooting time.

[0195] The header contains the ID of the lens set used during shooting. The storage device 220 contains the IDs of one or more lens sets, and one or more first PSF groups and one or more second PSF groups associated with the ID of each lens set.

[0196] Based on the lens set ID and PSF ID included in the output data, a specific first PSF group and a specific second PSF group can be selected from one or more first PSF groups and one or more second PSF groups stored in the storage device 220. In the imaging system shown in Figure 9, even if there is blurring caused by the optical system 140, multiple hyperspectral images can be generated more accurately based on the output data.

[0197] Figure 17H ​​schematically shows Example 8 of the output data format. In continuous shooting and video shooting, the output data shown in Figure 17H ​​includes a header, a series of multiple compressed images 10, and IDs of multiple PSF parameter sets corresponding to each of the multiple compressed images 10. The multiple compressed images 10 may be arranged, for example, in order of shooting time.

[0198] The header includes one or more IDs and one or more PSF parameter sets. Each ID in the header is associated with a PSF parameter set for generating a PSF that reflects blur and / or blur caused by the movement of the imaging device 100 during shooting. If the PSF reflecting blur and / or blur can be associated with two or more compressed images 10, the amount of data in the output data can be reduced.

[0199] Based on the PSF ID included in the output data, a specific PSF parameter set can be selected from one or more PSF parameter sets included in the header. In the imaging system shown in Figure 5, even if there is blur and / or blurring due to the movement of the imaging device 100, multiple hyperspectral images can be generated more accurately based on the output data.

[0200] As shown in Figures 17B and 17C, the processing circuit 210 may output the compressed image 10 in association with at least one of the parameters and PSF related to the movement of the imaging device 100 during its capture. As shown in Figure 17D, the processing circuit 210 may output the compressed image 10 in association with a first PSF group and a second PSF group corresponding to the shooting parameters of the optical system 140 during its capture. Instead of the first PSF group and / or the second PSF group, an edited restoration table based on the first PSF group and / or the second PSF group may be used.

[0201] As shown in Figures 17C and 17E, the processing circuit 210 may output the compressed image 10 in association with information on which a PSF can be obtained. This information may be, for example, an ID. In the example shown in Figures 17C and 17E, the PSF is obtained from the storage device 220 based on this information, but the PSF may also be obtained from an external server. Instead of the PSF, for example, a first PSF group, a second PSF group, or an edited restoration table may be used.

[0202] As shown in Figures 17F to 17G, the processing circuit 210 outputs data containing multiple compressed images as output data. The output data includes in the header one or more PSFs, or one or more first pieces of information from which one or more PSFs can be obtained. The output data further includes second pieces of information from which the PSF corresponding to each of the multiple compressed images 10 can be obtained. The first and second pieces of information may be, for example, IDs. Instead of PSFs, for example, a first group of PSFs, a second group of PSFs, or an edited restoration table may be used.

[0203] [Note] The following technologies are disclosed based on the above description of embodiments.

[0204] [Item 1] An imaging device that captures an image of an object and outputs a compressed image in which spectral information is compressed, A sensor that detects the movement of the imaging device during the capture of the compressed image and outputs parameters related to the movement, A processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, Equipped with, The processing circuit edits matrix data used in the hyperspectral image reconstruction process based on the parameters before generating the hyperspectral image. Imaging system.

[0205] This imaging system can generate hyperspectral images with greater accuracy.

[0206] [Item 2] The aforementioned processing circuit is Each time the imaging device captures the compressed image, it acquires the parameters at the time of capturing the compressed image. Based on the aforementioned parameters, the matrix data is edited. The imaging system described in item 1.

[0207] This imaging system can generate multiple hyperspectral images more accurately during continuous shooting and video recording.

[0208] [Item 3] The processing circuit edits the matrix data by performing a convolution operation on the matrix data based on the parameters. The imaging system described in item 1 or 2.

[0209] This imaging system can reflect blur and / or blur caused by the movement of the imaging device in the matrix data.

[0210] [Item 4] The processing circuit generates a point spread function based on the parameters and edits the matrix data based on the point spread function. An imaging system as described in any of items 1 to 3.

[0211] In this imaging system, the point spread function allows blurring and / or blurring caused by the movement of the imaging device to be reflected in the matrix data.

[0212] [Item 5] The processing circuit outputs the compressed image in association with at least one of the parameters and point spread function used when capturing the compressed image. An imaging system as described in any of items 1 through 4.

[0213] This imaging system can generate hyperspectral images based on the output data.

[0214] [Item 6] An imaging device that captures an image of an object and outputs a compressed image in which spectral information is compressed, A processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, A storage device that stores first matrix data used in the hyperspectral image reconstruction process, and first imaging parameters corresponding to the first matrix data, Equipped with, The processing circuit determines that the second shooting parameter, which is the shooting parameter at the time of capturing the compressed image, is different from the first shooting parameter. The second matrix data corresponding to the second imaging parameter is obtained, Using the second matrix data, the hyperspectral image is generated by a reconstruction process based on the compressed image. Imaging system.

[0215] This imaging system can generate hyperspectral images with greater accuracy.

[0216] [Item 7] The processing circuit, when configured to generate the hyperspectral image using the first matrix data, and when it determines that the second imaging parameter is different from the first imaging parameter, Obtain the second matrix data, Generate the hyperspectral image based on the second matrix data, The imaging system according to item 6.

[0217] In this imaging system, a hyperspectral image can be generated more accurately.

[0218] [Item 8] Each of the first imaging parameter and the second imaging parameter includes information related to at least one of the focal length of the lens and the aperture value of the aperture included in the imaging device. The imaging system according to item 6 or 7.

[0219] In this imaging system, the second matrix data can be obtained based on at least one of the focal length of the lens and the aperture value of the aperture.

[0220] [Item 9] The processing circuit obtains the second matrix data by editing the first matrix data based on the second imaging parameter. The imaging system according to any one of items 6 to 8.

[0221] In this imaging system, it is possible to reflect the blur caused by, for example, the optical system of the imaging device in the first matrix data to obtain the second matrix.

[0222] [Item 10] The processing circuit obtains the second matrix data by editing the first matrix data by performing a process of convolution of third matrix data corresponding to the second imaging parameter on the first matrix data. The imaging system according to any one of items 6 to 9.

[0223] In this imaging system, it is possible to reflect the blur caused by, for example, the optical system of the imaging device in the first matrix data to obtain the second matrix.

[0224] [Item 11] The memory device stores multiple third matrix data corresponding to multiple shooting parameters, The aforementioned processing circuit is From the plurality of third matrices, select the third matrix data corresponding to the second shooting parameters at the time of capturing the compressed image. The second matrix data is obtained by editing the first matrix data based on the selected third matrix data. An imaging system as described in any of items 6 through 10.

[0225] In this imaging system, based on one third matrix data point included in a plurality of third matrices stored in a memory device, a second matrix can be obtained by reflecting, for example, blur caused by the optical system of the imaging device in the first matrix data.

[0226] [Item 12] The memory device stores multiple third matrix data corresponding to multiple shooting parameters, If the processing circuit does not have a third matrix data among the plurality of third matrix data that matches the second shooting parameters at the time of shooting, it obtains the second matrix data by editing the first matrix data based on the third matrix data generated by interpolation using the plurality of third matrix data. An imaging system as described in any of items 6 through 10.

[0227] In this imaging system, a second matrix can be obtained by reflecting, for example, blur caused by the optical system of the imaging device, in the first matrix data based on third matrix data generated by interpolation using multiple third matrices stored in a memory device.

[0228] [Item 13] The memory device stores a plurality of third matrix data, each corresponding to a plurality of imaging parameters, for generating the hyperspectral image through a reconstruction process. The processing circuit acquires the second matrix data based on the third matrix data among the plurality of third matrix data that corresponds to the second shooting parameters at the time of capturing the compressed image. An imaging system as described in any of items 6 through 10.

[0229] This imaging system can acquire second matrix data based on one of several third matrix data stored in a memory device.

[0230] [Item 14] The processing circuit outputs the compressed image in association with the second matrix data corresponding to the second shooting parameters at the time of capturing the compressed image, or information that allows the acquisition of the second matrix data.

[0231] An imaging system as described in any of items 6 through 13.

[0232] This imaging system can generate hyperspectral images based on the output data.

[0233] [Item 15] The processing circuit outputs data containing multiple compressed images as output data. The output data includes one or more third matrix data or one or more first pieces of information from which each of the one or more third matrix data can be obtained, and further includes second pieces of information from which third matrix data corresponding to each of the plurality of compressed images can be obtained. An imaging system as described in any of items 6 through 13.

[0234] This imaging system can generate multiple hyperspectral images based on the output data.

[0235] [Item 16] The imaging device captures an image of the target object and outputs a compressed image in which the spectral information has been compressed. Cause the sensor to detect the movement of the imaging device during the shooting of the compressed image and output parameters related to the movement. Before generating a hyperspectral image by a reconstruction process based on the compressed image, edit the matrix data used in the reconstruction process of the hyperspectral image based on the parameters. Including Method.

[0236] <0> By this method, a hyperspectral image can be generated more accurately.

[0237] [Item 17] A method executed in an imaging system including an imaging device and a storage device, The storage device stores first matrix data used in a reconstruction process of a hyperspectral image generated based on a compressed image in which spectral information is compressed, and first shooting parameters corresponding to the first matrix data. The method includes Causing the imaging device to image an object and output the compressed image. When it is determined that a second shooting parameter, which is a shooting parameter at the time of shooting the compressed image, is different from the first shooting parameter, Obtaining second matrix data corresponding to the second shooting parameter. Generating the hyperspectral image by a reconstruction process based on the compressed image using the second matrix data. Including Method.

[0238] By this method, a hyperspectral image can be generated more accurately.

[0239] [Item 18] An imaging device that images an object and outputs a compressed image in which spectral information is compressed, A processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image. Equipped with, The aforementioned processing circuit is At least one parameter is obtained from the imaging device and at least one of the other devices attached to the imaging device, Before generating the hyperspectral image, edit the first matrix data used in the hyperspectral image reconstruction process based on the at least one parameter, or obtain a second matrix data obtained by editing the first matrix data based on the at least one parameter. Imaging system.

[0240] This imaging system can generate hyperspectral images with greater accuracy.

[0241] [Item 19] The imaging device captures an image of the target object and outputs a compressed image in which the spectral information has been compressed. Acquiring at least one parameter from at least one of the imaging device and other devices attached to the imaging device, Before generating a hyperspectral image by a reconstruction process based on the compressed image, the process involves editing a first matrix data used in the hyperspectral image reconstruction process based on at least one parameter, or obtaining a second matrix data obtained by editing the first matrix data based on at least one parameter. including, method.

[0242] This method allows for the more accurate generation of hyperspectral images. [Industrial applicability]

[0243] The technology disclosed herein is useful, for example, in cameras and measuring instruments that acquire multi-wavelength or high-resolution images. The technology disclosed herein can also be applied, for example, to sensing for biomedical, cosmetic, foreign object and pesticide residue detection systems in food, remote sensing systems, and in-vehicle sensing systems. [Explanation of Symbols]

[0244] 10 Compressed Images 20 Hyperspectral images 20W1, 20W2, ..., 20W N Restored image 70 Objects 100 Imaging device 110 filter array 140, 140A, 140B optical system 142 lenses 144 aperture 160 Image Sensors 200 Image Processing Devices 210 Processing Circuit 220 Storage device 300 display device 400 vibration sensor 500 Input Devices 600 Communication devices 700 External Servers 710 Communication equipment 720 Storage device

Claims

1. An imaging device that captures an image of an object and outputs a compressed image in which spectral information is compressed, A sensor that detects the movement of the imaging device during the capture of the compressed image and outputs parameters related to the movement, A processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, Equipped with, The processing circuit edits matrix data used in the hyperspectral image reconstruction process based on the parameters before generating the hyperspectral image. Imaging system.

2. The aforementioned processing circuit is Each time the imaging device captures the compressed image, it acquires the parameters at the time of capturing the compressed image. Based on the aforementioned parameters, the matrix data is edited. The imaging system according to claim 1.

3. The processing circuit edits the matrix data by performing a convolution operation on the matrix data based on the parameters. The imaging system according to claim 1 or 2.

4. The processing circuit generates a point spread function based on the parameters and edits the matrix data based on the point spread function. The imaging system according to claim 1 or 2.

5. The processing circuit outputs the compressed image in association with at least one of the parameters and point spread function used when capturing the compressed image. The imaging system according to claim 1 or 2.

6. An imaging device that captures an image of an object and outputs a compressed image in which spectral information is compressed, A processing circuit that generates a hyperspectral image by a reconstruction process based on the compressed image, A storage device that stores first matrix data used in the hyperspectral image reconstruction process and first imaging parameters corresponding to the first matrix data, Equipped with, The processing circuit determines that the second shooting parameter, which is the shooting parameter at the time of capturing the compressed image, is different from the first shooting parameter. The second matrix data corresponding to the second imaging parameter is obtained, Using the second matrix data, the hyperspectral image is generated by a reconstruction process based on the compressed image. Imaging system.

7. The processing circuit, when configured to generate the hyperspectral image using the first matrix data, and when it determines that the second imaging parameter is different from the first imaging parameter, Obtain the second matrix data, Based on the second matrix data, the hyperspectral image is generated. The imaging system according to claim 6.

8. Each of the first and second shooting parameters includes information regarding at least one of the focal length and aperture value of the lens included in the imaging device. The imaging system according to claim 6 or 7.

9. The processing circuit obtains the second matrix data by editing the first matrix data based on the second shooting parameters. The imaging system according to claim 6 or 7.

10. The processing circuit obtains the second matrix data by editing the first matrix data by performing a process in which the third matrix data corresponding to the second imaging parameter is convolved with the first matrix data. The imaging system according to claim 6 or 7.

11. The memory device stores multiple third matrix data corresponding to multiple shooting parameters, The aforementioned processing circuit is From the plurality of third matrices, select the third matrix data corresponding to the second shooting parameters at the time of capturing the compressed image. The second matrix data is obtained by editing the first matrix data based on the selected third matrix data. The imaging system according to claim 6 or 7.

12. The memory device stores multiple third matrix data corresponding to multiple shooting parameters, If there is no third matrix data among the plurality of third matrix data that matches the second shooting parameters at the time of shooting, the processing circuit obtains the second matrix data by editing the first matrix data based on the third matrix data generated by interpolation using the plurality of third matrix data. The imaging system according to claim 6 or 7.

13. The storage device stores a plurality of third matrix data, each corresponding to a plurality of imaging parameters, for generating the hyperspectral image through a reconstruction process. The processing circuit acquires the second matrix data based on the third matrix data among the plurality of third matrix data that corresponds to the second shooting parameters at the time of capturing the compressed image. The imaging system according to claim 6 or 7.

14. The processing circuit outputs the compressed image in association with the second matrix data corresponding to the second shooting parameters at the time the compressed image was captured, or information that allows the acquisition of the second matrix data. The imaging system according to claim 6 or 7.

15. The processing circuit outputs data containing multiple compressed images as output data. The output data includes one or more third matrix data or one or more first pieces of information from which each of the one or more third matrix data can be obtained, and further includes second pieces of information from which each of the plurality of compressed images can be obtained. The imaging system according to claim 6 or 7.

16. The imaging device captures an image of the target object and outputs a compressed image in which the spectral information has been compressed. The sensor is made to detect the movement of the imaging device during the capture of the compressed image and to output parameters related to the movement. Before generating a hyperspectral image by a reconstruction process based on the compressed image, the matrix data used in the hyperspectral image reconstruction process is edited based on the parameters, including, method.

17. A method performed in an imaging system comprising an imaging device and a storage device, The storage device stores first matrix data used in the reconstruction process of a hyperspectral image generated based on a compressed image in which spectral information has been compressed, and first imaging parameters corresponding to the first matrix data. The aforementioned method, The imaging device is used to capture an object and output the compressed image, If it is determined that the second shooting parameter, which is the shooting parameter at the time of capturing the compressed image, is different from the first shooting parameter, To obtain the second matrix data corresponding to the second imaging parameter, Using the second matrix data, the hyperspectral image is generated by a reconstruction process based on the compressed image, including, method.

Images