Method and device of processing data used for processing of generating spectral images

Abstract

To reduce an error accompanying the generation of images of a plurality of wavelength bands.SOLUTION: A method includes: acquiring first information indicating a relative position between an encoding element including a plurality of regions having spectral transmittance different from each other and an image sensor that detects light transmitted through the encoding element; and correcting data including second information about the spatial distribution of the spectral transmittance on the basis of the first information. The data is used for processing of generating four or more spectral images on the basis of an image acquired by causing the image sensor to detect the light transmitted through the encoding element. Each of the four or more spectral images indicates an image corresponding to one wavelength band of the four or more wavelength bands.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to methods and apparatus for processing data used in processes that generate spectroscopic images. [Background technology]

[0002] By utilizing spectral information from multiple narrow wavelength bands, for example, several dozen bands, it becomes possible to understand the detailed physical properties of an object, which was not possible with conventional RGB images. A camera that captures such multi-wavelength information is called a "hyperspectral camera." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis.

[0003] Patent Document 1 discloses an example of a hyperspectral imaging device that uses compressed sensing. The imaging device includes an encoding element, which is an array of multiple optical filters with different wavelength-dependent light transmittances, an image sensor that detects light transmitted through the encoding element, and a signal processing circuit. The encoding element is disposed on an optical path connecting the subject and the image sensor. The image sensor simultaneously detects light in which components of multiple wavelength bands are superimposed for each pixel to acquire a single wavelength-multiplexed image. The signal processing circuit reconstructs image data for each of the multiple wavelength bands by applying compressed sensing to the acquired wavelength-multiplexed image using information on the spatial distribution of the spectral transmittance of the encoding element. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,599,511 Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides techniques for reducing errors associated with reconstructing images of multiple wavelength bands. [Means for solving the problem]

[0006] An imaging device according to one aspect of the present disclosure includes an image sensor, a filter array disposed on an optical path from an object to the image sensor and including a plurality of optical filters arranged two-dimensionally, the plurality of optical filters including a plurality of types of optical filters with different spectral transmittances, and a processing circuit configured to generate four or more spectral image data based on an image acquired by the image sensor. Each of the four or more spectral image data represents an image corresponding to one of four or more wavelength bands. The filter array includes one or more features. The processing circuit detects a relative position between the filter array and the image sensor based on the one or more features in the image acquired by the image sensor, and, when a deviation between the relative position and a predetermined relative position is detected, compensates for the deviation.

[0007] An imaging device according to another aspect of the present disclosure includes an image sensor and a filter array arranged on an optical path from an object to the image sensor and including a plurality of optical filters arranged two-dimensionally, the plurality of optical filters including a plurality of types of optical filters with different spectral transmittances, and an imaging device that performs image processing to obtain four or more images based on an image acquired by the image sensor and data indicating the spatial distribution of the spectral transmittances of the plurality of optical filters. and a processing circuit for generating spectral image data, each of the four or more sets of spectral image data representing an image corresponding to one of the four or more wavelength bands. The processing circuit performs (a) a first operation of generating multiple sets of the four or more sets of spectral image data by repeating the process of generating the four or more sets of spectral image data multiple times while changing the relative position of the filter array and the image sensor, or (b) a second operation of generating multiple sets of the four or more sets of spectral image data by repeating the process of generating the four or more sets of spectral image data multiple times while changing the coordinates of the image acquired by the image sensor, and generates output data by combining the multiple sets of the four or more sets of spectral image data.

[0008] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, errors associated with reconstructing images of multiple wavelength bands can be reduced. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1A is a schematic diagram of an exemplary imaging device. [Figure 1B]FIG. 1B is a diagram showing another example of the configuration of the imaging device. [Figure 1C] FIG. 1C is a diagram showing another example of the configuration of the imaging device. [Figure 2A] FIG. 2A is a diagram schematically illustrating an example of a filter array. [Figure 2B] FIG. 2B is a diagram showing an example of the spatial distribution of the transmittance of light in each of a plurality of wavelength bands included in the target wavelength range. [Figure 2C] FIG. 2C is a diagram showing an example of the spectral transmittance of the region A1 included in the filter array shown in FIG. 2A. [Figure 2D] FIG. 2D is a diagram showing an example of the spectral transmittance of the region A2 included in the filter array shown in FIG. 2A. [Figure 3A] FIG. 3A is a diagram for explaining the relationship between a target wavelength range and a plurality of wavelength bands included therein. [Figure 3B] FIG. 3B is a diagram for explaining the relationship between the target wavelength range and the multiple wavelength bands included therein. [Figure 4A] FIG. 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array. [Figure 4B] FIG. 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each wavelength range. [Figure 5] FIG. 5 is a diagram schematically illustrating the configuration of an imaging device according to the first exemplary embodiment of the present disclosure. [Figure 6A] FIG. 6A is a diagram illustrating a filter array according to the first embodiment. [Figure 6B] FIG. 6B is a diagram showing an example of the spatial distribution of the transmittance of the filter array for N wavelength bands included in the target wavelength range. [Figure 6C] FIG. 6C shows a modified example of a filter array. [Figure 6D] FIG. 6D shows another modification of the filter array. [Figure 7] FIG. 7 is an image showing an example of alignment marks formed on a filter array. [Figure 8A] FIG. 8A is a diagram showing an example of the rate of change (unit: %) in the transmittance of the filter array when the relative position of the filter array as viewed from the image sensor is shifted by one pixel in the x and y directions. [Figure 8B] FIG. 8B is a diagram showing an example of the error in average brightness between the generated image and the correct image. [Figure 9A] FIG. 9A is a diagram showing an example of an image generated for each wavelength band. [Figure 9B] FIG. 9B is a diagram showing an example of an image for each wavelength band that is generated when no correction is performed. [Figure 10] FIG. 10 is a flowchart showing an outline of the processing executed by the processing circuit. [Figure 11] FIG. 11 is a diagram schematically illustrating the configuration of an imaging device according to a modification of the first embodiment. [Figure 12] FIG. 12 is a diagram schematically showing the configuration of an imaging device according to another modification of the first embodiment. [Figure 13] FIG. 13 is a schematic diagram illustrating a configuration of an imaging device according to the second exemplary embodiment of the present disclosure. [Figure 14A] FIG. 14A is a first diagram for explaining the effect of the second embodiment. [Figure 14B] FIG. 14B is a second diagram for explaining the effect of the second embodiment. [Figure 14C] FIG. 14C is a third diagram for explaining the effect of the second embodiment. [Figure 15A] FIG. 15A is a diagram schematically illustrating an example of a filter array according to the third exemplary embodiment of the present disclosure. [Figure 15B] FIG. 15B is a diagram schematically illustrating a filter array according to a modification of the third embodiment. [Figure 16] FIG. 16 is a diagram for explaining signal processing in the fourth embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of signal processing in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection configurations, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0012] In this disclosure, all or part of a circuit, unit, device, member, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or IC may be integrated into a single chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into a single chip. Although referred to as an LSI or IC here, the term may be changed depending on the degree of integration, and may be called a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). Field programmable gate arrays (FETs) that are programmed after the LSI is manufactured may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). A field programmable gate array (FPGA), or a reconfigurable logic device that can reconfigure the connections within an LSI or set up circuit partitions within an LSI, can also be used for the same purpose.

[0013] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0014] A configuration example of a hyperspectral imaging device according to an embodiment of the present disclosure and findings discovered by the present inventors will be described.

[0015] FIG. 1A is a schematic diagram illustrating an exemplary imaging device. This imaging device has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device includes an optical system 40, a filter array 100, an image sensor 60, and a processing circuit 200. The filter array 100 has a structure and function similar to that of the "encoding element" disclosed in Patent Document 1. The optical system 40 and the filter array 100 are arranged on the optical path of light incident from an object 70, which is a subject. The filter array 100 is arranged between the optical system 40 and the image sensor 60 and at a position distant from the image sensor 60. FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple and may be any object. Based on the image data generated by the image sensor 60, the processing circuit 200 generates spectral image data for each of multiple wavelength bands included in a specific wavelength range to be detected (hereinafter, sometimes referred to as the "target wavelength range"). In the following description, the generated spectral image data of the multiple wavelength bands will be referred to as separated images 220W1, 220W2, . . . , 220W NThese are collectively referred to as a separated image 220. In this specification, a signal representing an image (that is, a set of signals representing the pixel values ​​of each pixel) may be simply referred to as an "image."

[0016] The filter array 100 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittance. The filter array 100 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 100 is referred to as "encoding" in this specification.

[0017] The image sensor 60 is a monochrome photodetector having a plurality of photodetection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 60 may be, for example, a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0018] The processing circuit 200 generates a plurality of separated images 220W1, 220W2, . . . 220W, each containing information of a plurality of wavelength bands, based on the image 120 acquired by the image sensor 60. N Generate data.

[0019] 1B and 1C are diagrams showing other configuration examples of the imaging device. In the example of FIG. 1B, a filter array 100 is disposed between the object 70 and the optical system 40. In the example of FIG. 1C, the imaging device includes two optical systems 40A and 40B, and the filter array 100 is disposed between them. As in these examples, an optical system may be disposed between the filter array 100 and the image sensor 60.

[0020] FIG. 2A is a diagram schematically illustrating an example of a filter array 100. The filter array 100 has a plurality of regions arranged two-dimensionally. In this specification, these regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0021] In the example shown in FIG. 2A, the filter array 100 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in the image sensor 60, for example. The number of filters included in the filter array 100 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0022] FIG. 2B shows multiple wavelength bands W1, W2, . . . , W included in the target wavelength range. N 2B is a diagram showing an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in FIG. 2B, the difference in shading in each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0023] 2C and 2D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 100 shown in FIG. 2A, respectively. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 100 varies depending on the region. However, the spectral transmittance of all regions does not necessarily have to be different. In the filter array 100, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 100 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns of the multiple regions included in the filter array 100 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 100 may be designed so that the spectral transmittances of more than half of the regions are different.

[0024] 3A and 3B show a target wavelength range W and multiple wavelength bands W1, W2, . . . , W N This figure explains the relationship between the wavelengths of the light and the wavelengths of the light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a radio wave range such as mid-infrared, far-infrared, terahertz waves, or millimeter waves. Thus, the wavelength range used is not limited to the visible light range. In this specification, for convenience, "light" refers not only to visible light but also to non-visible light such as near-ultraviolet, near-infrared, and radio waves.

[0025] In the example shown in FIG. 3A, N is an arbitrary integer equal to or greater than 4, and the target wavelength range W is divided into N equal wavelength ranges, each of which is designated as wavelength bands W1, W2, . . . , W N However, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the multiple wavelength bands may be determined arbitrarily as long as they are different from each other.

[0026] 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 100. In the example shown in FIG. 4A, the spectral transmittance is expressed as a function of wavelengths within the target wavelength range W. In the example shown in FIG. 4A, the optical transmittance in the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 4A, the optical transmittance in the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. N-1 In this way, the spectral transmittance of each region is expressed as a multiple of wavelength bands W1 to W2. NAmong them, there are maxima in a plurality of wavelength ranges. In the example of FIG. 4A, the maxima P1, P3, P4, and P5 are 0.5 or more.

[0027] As described above, the light transmittance of each region varies with wavelength. Therefore, the filter array 100 transmits a large amount of components in a certain wavelength range of the incident light and does not transmit components in other wavelength ranges so much. For example, for the light in k of the N wavelength bands, the transmittance may be greater than 0.5, and for the light in the remaining N - k wavelength ranges, the transmittance may be less than 0.5. k is an integer satisfying 2 ≤ k < N. If the incident light is white light that equally contains all visible light wavelength components, the filter array 100 modulates the incident light into light having a plurality of intensity peaks discrete with respect to wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0028] FIG. 4B shows, as an example, the result of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength ranges W1, W2, ···, W N This is a diagram showing the result of averaging the spectral transmittance T(λ) for each wavelength band by integrating it over each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is taken as the transmittance in that wavelength band. In this example, the transmittance is prominently high in three wavelength ranges taking the maxima P1, P3, and P5. In particular, in the two wavelength ranges taking the maxima P3 and P5, the transmittance exceeds 0.8.

[0029] In the examples shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessarily necessary to have a grayscale transmittance distribution. For example, a binary scale transmittance distribution in which the transmittance of each region can take either a value of approximately 0 or approximately 1 may be adopted. In the binary scale transmittance distribution, each region transmits most of the light in at least two of the plurality of wavelength ranges included in the target wavelength range and does not transmit most of the light in the remaining wavelength ranges. Here, "most" means generally 80% or more.

[0030] A portion of the cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength ranges W1 to W2 included in the wavelength range of interest. N The filter array 100 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 100, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0031] Such data indicating the spatial distribution of the spectral transmittance of the filter array 100 is obtained in advance using design data or actual measurement calibration, and is stored in a storage medium provided in the processing circuit 200. This data is used in the calculation processing described below.

[0032] The filter array 100 may be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing a metal. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing a metal layer may be used. In this case, the multilayer film is formed so that at least one of the thickness, material, and stacking order of each multilayer film differs for each cell. This makes it possible to realize different spectral characteristics for each cell. By using a multilayer film, it is possible to realize sharp rises and falls in the spectral transmittance. A configuration using an organic material is realized by containing different pigments or dyes for each cell, or by stacking different materials. The configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. When a microstructure containing metal is used, it can be fabricated by utilizing the plasmon effect for light separation.

[0033] Next, an example of signal processing by the processing circuit 200 will be described. The processing circuit 200 reconstructs a multi-wavelength separated image 220 based on the image 120 output from the image sensor 60 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 100. Here, "multiple wavelengths" means a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a typical color camera. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the number of bands. Depending on the application, the number of bands may exceed 100.

[0034] The data to be obtained is the separated image 220, and this data is designated as f. If the number of bands is designated as N, f is the image data of each band, f1, f2, . . . , f N Here, as shown in Figure 1A, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is n, and the number of pixels in the y direction is m, then the image data f1, f2, ..., f N Each of the data f is a collection of two-dimensional data of n×m pixels. Therefore, data f is three-dimensional data with n×m×N elements. On the other hand, data g of image 120 obtained by encoding and multiplexing by filter array 100 has n×m elements. Data g can be expressed by the following equation (1).

number

[0035] where f1, f2, . . . , f N is data with n × m elements. Therefore, the vector on the right side is strictly a one-dimensional vector with n × m × N rows and 1 column. The vector g is converted into a one-dimensional vector with n × m rows and 1 column, and then calculated. The matrix H is the components f1, f2, ..., f of the vector f. N represents a transformation in which each wavelength band is encoded with different encoding information, intensity-modulated, and then added together. Therefore, H is a matrix with n × m rows and n × m × N columns. In this specification, the matrix H may be referred to as the "system matrix."

[0036] Given the vector g and matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the processing circuit 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

number

[0037] Here, f' represents the estimated data for f. The first term in the parentheses in the above equation represents the amount of deviation between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The processing circuit 200 can converge the solution through recursive iterative calculations and calculate the final solution f'.

[0038] The first term in the parentheses in equation (2) represents the sum of squares of the difference between the acquired data g and Hf, which is the result of transforming the estimation process f by the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT) of f, the wavelet transform, the Fourier transform, or the total variation (TV) of f. For example, using the total variation transform can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the regularization term space can be selected. Alternatively, multiple regularization terms can be included in the calculation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0039] In the configurations shown in FIGS. 1A to 1C , the image encoded by the filter array 100 is acquired in a blurred state on the imaging plane of the image sensor 60. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned system matrix H, the separated image 220 can be reconstructed. Here, the blur information is expressed by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to a pixel on an image spreads to a k×k pixel region around the pixel due to blurring, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the influence on the luminance of each pixel within the region. The separated image 220 can be reconstructed by reflecting the influence of blurring of the encoding pattern due to the PSF in the system matrix H. The filter array 100 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 100 to be lost due to excessive diffusion can be selected.

[0040] In the above configuration, the processing circuit 200 generates a separation image 220 for each wavelength band for an area that is the same as or smaller than the area captured by the image sensor 60. To obtain an accurate separation image 220, it is necessary to accurately determine data indicating the spatial distribution of the spectral transmittance of the filter array 100 in advance (e.g., information about the above-mentioned system matrix H). The data indicating the spatial distribution of the spectral transmittance of the filter array 100 can be determined, for example, during design or manufacturing, by an experiment in which a light source that emits light of wavelengths belonging to each wavelength band is used to irradiate the filter array 100 from behind and measure the intensity of the transmitted light. The determined data indicating the spatial distribution of the spectral transmittance is pre-recorded in a storage medium of the processing circuit 200.

[0041] Thus, in order to generate multiple images for each wavelength band, it is important to accurately grasp the spatial distribution of the spectral transmittance of the filter array 100. If the above calculation is performed with an incorrect spatial distribution of the spectral transmittance of the filter array 100, an incorrect separated image will be generated. In other words, an incorrect spatial distribution of the spectral transmittance will result in a significant reduction in wavelength resolution.

[0042] This problem is particularly pronounced in a configuration in which the filter array 100 can be attached and detached from the imaging device. If the type of object 70 or the wavelength band of interest changes, the characteristics of the optimal filter array 100 also change. If the filter array 100 is replaceable, it is possible to select and attach an optimal filter array 100 depending on the object 70 or application. In this case, data indicating the spatial distribution of spectral transmittance for each filter array 100 is determined in advance and stored in a storage medium.

[0043] However, if the filter array 100 is detachable, the relative position between the filter array 100 and the image sensor 60 may deviate from the preset relative position. If such a deviation occurs, even if the same filter array 100 is used, the spatial distribution of the spectral transmittance of the filter array 100 as seen by the image sensor 60 may change. In particular, if the filters in the filter array 100 are arranged randomly or in a near-random manner, a deviation in the relative position may result in a completely different spatial distribution of the spectral transmittance. For example, if the relative position between the filter array 100 and the image sensor 60 changes by one pixel in the image sensor 60, the spatial distribution of the spectral transmittance of the filter array 100 as seen by the image sensor 60 will change by one pixel. If the filter array 100 is arranged randomly or in a near-random manner, a filter that is shifted by one pixel may have a completely different spectral transmittance from the original filter. This may result in an error in the results of calculations using the pre-defined system matrix H, potentially resulting in an inaccurate image. In other words, the wavelength resolution may be significantly reduced.

[0044] As described above, in a hyperspectral camera that acquires multi-wavelength images, changes in the relative position between the filter array and the image sensor can degrade the image quality of the multi-wavelength image. In particular, as in the example described above, when spatial distribution information of the spectral transmittance of the filter array 100 is used in the calculation to construct the multi-wavelength separated image 220, the impact of changes in relative position on image quality becomes significant. Therefore, in one embodiment of the present disclosure, one or more features of the filter array 100 are detected from an image acquired by the image sensor, and the relative positional deviation between the filter array and the image sensor can be detected based on the information of the features. Furthermore, the above problem can be solved by introducing an operation to compensate for the deviation. For example, calculation errors due to deviations in the relative position can be reduced by physically correcting the relative position between the filter array and the image sensor, or by correcting the acquired image data or data indicating the spatial distribution of the spectral transmittance of the filter array. According to an embodiment of the present disclosure, it is possible to prevent a decrease in wavelength resolution due to a change in the relative position between the filter array and the image sensor from a preset relative position.

[0045] The present inventors have also devised a calculation method that reduces errors that accompany the generation of images of multiple wavelength bands, regardless of whether there is a misalignment between the filter array and the image sensor relative to each other.

[0046] An outline of an embodiment of the present disclosure will be described below.

[0047] An imaging device according to one aspect of the present disclosure includes an image sensor, a filter array disposed on an optical path from an object to the image sensor and including a plurality of optical filters arranged two-dimensionally, the plurality of optical filters including a plurality of types of optical filters with different spectral transmittances, and a processing circuit configured to generate four or more spectral image data based on an image acquired by the image sensor. Each of the four or more spectral image data represents an image corresponding to one of four or more wavelength bands. The filter array includes one or more features. The processing circuit detects a relative position between the filter array and the image sensor based on the one or more features in the image acquired by the image sensor, and, when detecting a deviation between the relative position and a predetermined relative position, compensates for the deviation.

[0048] According to the above configuration, the filter array has one or more characteristic portions, for example, on its surface or outer periphery. The processing circuit can detect the relative position between the filter array and the image sensor based on information such as the position, shape, or size of the one or more characteristic portions included in the acquired image. When the relative position has changed, for example, from a preset initial relative position, the processing circuit performs an operation to compensate for the deviation of the relative position. For example, the processing circuit can physically correct the relative position between the filter array and the image sensor, or correct the relative position by signal processing. Actions can be taken to reduce the effects of the misalignment, thereby reducing errors in the spectroscopic image data due to the misalignment of the relative positions.

[0049] The imaging device may further include a first drive unit that moves the filter array, and in that case, the processing circuit may control the first drive unit to correct the relative position, thereby compensating for the misalignment.

[0050] According to the above configuration, when a deviation in the relative position is detected, the first drive device reduces the deviation in the relative position by moving the filter array. This reduces errors in the spectroscopic image data caused by the deviation in the relative position. The first drive device may also compensate for the deviation in the relative position by rotating the filter array.

[0051] The imaging device may further include a second drive unit that moves the image sensor, and in that case, the processing circuit may control the second drive unit to correct the relative position, thereby compensating for the misalignment.

[0052] According to the above configuration, when a deviation in the relative position is detected, the second drive device reduces the deviation in the relative position by moving the image sensor. This reduces errors in the spectroscopic image data caused by the deviation in the relative position. The second drive device may also compensate for the deviation in the relative position by rotating the image sensor.

[0053] The imaging device may further include an optical system arranged on an optical path connecting the filter array and the image sensor, the optical system forming an image on an imaging surface of the image sensor by light passing through the filter array, and a third driving device changing a position of the image formed by the optical system. In this case, the processing circuit may compensate for the deviation by controlling the third driving device to correct the position of the image.

[0054] According to the above configuration, when a deviation in the relative position is detected, the third driving device changes the position of the image formed by the optical system by, for example, moving or rotating the optical system, thereby reducing errors in the spectroscopic image data caused by the deviation in the relative position.

[0055] The processing circuit may generate the four or more spectral image data based on the image acquired by the image sensor and data indicating the spatial distribution of the spectral transmittance of the plurality of optical filters, and in this case, the processing circuit may compensate for the deviation by correcting the data indicating the spatial distribution of the spectral transmittance.

[0056] According to the above configuration, when the processing circuit detects a shift in the relative positions, it corrects the data representing the spatial distribution of the spectral transmittance of the plurality of optical filters. The data representing the spatial distribution of the spectral transmittance of the plurality of optical filters corresponds to, for example, the aforementioned matrix H. By correcting the data, it is possible to reduce errors in the spectral image data caused by the shift in the relative positions.

[0057] The processing circuitry may compensate for the misalignment by correcting the coordinates of the image captured by the image sensor.

[0058] According to the above configuration, when the processing circuit detects a deviation in the relative position, it corrects the coordinates of the image acquired by the image sensor. Here, correcting the coordinates of the image means correcting the correspondence between the coordinates and pixel values ​​in the data represented by the image. Image correction corresponds to, for example, an operation of performing a coordinate transformation such as translation or rotation on the aforementioned vector g. By correcting the acquired image, errors in the spectroscopic image data caused by a deviation in the relative position can be reduced. It can be reduced.

[0059] The filter array may include one or more alignment marks, and the one or more features may be the one or more alignment marks.

[0060] One or more alignment marks may be formed on the surface of the filter array, for example, in a manner that allows them to be distinguished from other portions on the image. For example, the alignment marks may be formed by a metal film having a transmittance lower than that of the surrounding filters.

[0061] Each of the one or more alignment marks may include a first portion extending in a first direction and a second portion extending in a second direction intersecting the first direction, and the processing circuit may further perform an operation of detecting tilt of the filter array and reducing an effect of the tilt based on a ratio between a length of the first portion and a length of the second portion in the image acquired by the image sensor.

[0062] The one or more alignment marks may include a plurality of alignment marks, and in that case, the processing circuit may further perform an operation of detecting tilt of the filter array and reducing an effect of the tilt based on a positional relationship of the plurality of alignment marks in the image acquired by the image sensor.

[0063] The tilt of the filter array refers to a state in which the orientation of the filter array is tilted from a predetermined proper orientation. Actions to reduce the influence of the tilt may include, for example, actions such as rotating the filter array or rotating the image sensor.

[0064] The processing circuitry can detect tilt of the filter array based on, for example, the ratio of the distance between two alignment marks separated in a first direction to the distance between two alignment marks separated in a second direction.

[0065] According to the above configuration, errors in the spectral image data caused by tilt of the filter array can be reduced.

[0066] The image sensor may include a plurality of photodetector elements, each outputting a signal according to the intensity of light received. The processing circuit may generate the four or more spectral image data based on a signal output from a photodetector element that receives light that has passed through a region of the filter array. The one or more alignment marks may be located inside or outside the partial region.

[0067] If the one or more alignment marks are outside of the region, the alignment marks are detected from image data in a region not used to generate the spectroscopic image data, thereby enabling the alignment marks to be detected without affecting the generated spectroscopic image data.

[0068] The one or more alignment marks may be located within a range that can be captured by the image sensor when the relative position is a preset relative position, in which case the alignment marks can be detected from an image captured by the image sensor, and the deviation in the relative position between the filter array and the image sensor can be easily detected based on information such as the positions of the alignment marks.

[0069] The filter array is configured to filter the image when the relative position is a preset relative position. The alignment mark may be larger than the range that can be captured by the image sensor. The one or more alignment marks may be outside the range. In this case, the alignment mark may be located near the boundary between the inside and outside of the imageable range. If the alignment mark is detected in the image captured by the image sensor, the processing circuit can determine that a relative positional deviation has occurred.

[0070] An imaging device according to another aspect of the present disclosure includes an image sensor, a filter array disposed on an optical path from an object to the image sensor and including a plurality of optical filters arranged two-dimensionally, the plurality of optical filters including a plurality of types of optical filters with different spectral transmittances, and a processing circuit that generates four or more sets of spectral image data based on an image acquired by the image sensor and data indicating the spatial distribution of the spectral transmittances of the plurality of optical filters. Each of the four or more sets of spectral image data represents an image corresponding to one of four or more wavelength bands. The processing circuit (a) a first operation of generating a plurality of sets of the four or more spectral image data by repeating the process of generating the four or more spectral image data a plurality of times while changing the relative position of the filter array and the image sensor; or (b) a second operation of generating a plurality of sets of the four or more spectral image data by repeating the process of generating the four or more spectral image data a plurality of times while changing the coordinates of the image acquired by the image sensor; and synthesizing the plurality of sets of the four or more spectral image data to generate output data.

[0071] According to the above configuration, by performing the first operation or the second operation and synthesizing the four or more sets of spectroscopic image data, it is possible to reduce errors contained in the spectroscopic image data.

[0072] The first operation may include correcting the data indicating the spatial distribution of the spectral transmittance in accordance with a change in the relative position.

[0073] The second operation may include correcting the data indicating the spatial distribution of the spectral transmittance in accordance with the change in the coordinate.

[0074] More specific embodiments of the present disclosure will be described below. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims. In the following description, identical or similar components are designated by the same reference numerals. The x, y, and z coordinates shown in the drawings will be used in the following description.

[0075] (Embodiment 1) 5 is a diagram schematically illustrating the configuration of an imaging device according to exemplary embodiment 1 of the present disclosure. The imaging device in this embodiment includes a filter array 100, optical systems 40A and 40B, an image sensor 60, a processing circuit 200, and a movable stage 80.

[0076] The filter array 100 is disposed on the optical path from the object 70 to the image sensor 60 and at a position separated from the image sensor 60. As described above, the filter array 100 includes a plurality of optical filters arranged two-dimensionally. The plurality of optical filters have spectral transmittances of It includes multiple different types of optical filters, each of which is designed to exhibit maximum light transmittance in two or more of N wavelength bands (N is an integer of 4 or greater).

[0077] Optical system 40A is disposed between object 70 and filter array 100. Optical system 40B is disposed between filter array 100 and image sensor 60. Each of optical systems 40A and 40B includes one or more lenses. Optical systems 40A and 40B form an image on the imaging plane of image sensor 60 using light from object 70 that has passed through filter array 100.

[0078] As described above, the image sensor 60 includes multiple photodetectors arranged two-dimensionally on the imaging surface. Each photodetector includes a photoelectric conversion element, such as a photodiode, and outputs an electrical signal corresponding to the amount of light received. The image sensor 60 may be, for example, a CCD or CMOS sensor, an infrared array sensor, a terahertz array sensor, or a millimeter-wave array sensor. The image sensor 60 does not necessarily have to be a monochrome sensor. For example, a color sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may also be used. Using a color sensor increases the amount of wavelength-related information and improves the accuracy of reconstructing the separated image 220. However, when a color sensor is used, the amount of information in the spatial directions (x and y directions) decreases, so there is a trade-off between the amount of wavelength-related information and resolution. The wavelength range to be acquired may be determined arbitrarily and may include not only the visible wavelength range but also ultraviolet, near-infrared, mid-infrared, far-infrared, microwave, and radio wave wavelength ranges.

[0079] Light coming from object 70 passes through optical system 40A, filter array 100, and optical system 40B in that order, and forms an image on the imaging surface of image sensor 60. The direction parallel to the optical axes of optical systems 40A and 40B and from object 70 toward image sensor 60 is defined as the z direction. Filter array 100 is disposed approximately parallel to the xy plane, which includes x and y axes that are orthogonal to the z axis. Image sensor 60 acquires an image using light from object 70 that has passed through filter array 100.

[0080] The processing circuit 200 generates image data corresponding to each of the N wavelength bands, i.e., separated images 220, based on the image data output from the image sensor 60 and data indicating the spatial distribution of the spectral transmittance of the filter array 100. The data indicating the spatial distribution of the spectral transmittance of the filter array 100 is stored in advance in a memory provided in the processing circuit 200. The processing circuit 200 generates the images 220 corresponding to each wavelength band by performing calculations based on the above-mentioned equation (2).

[0081] The processing circuit 200 can be realized by a combination of a digital signal processor (DSP), a programmable logic device (PLD) such as a field programmable gate array (FPGA), or a central processing unit (CPU), a graphics processing unit (GPU), and a computer program. Such a computer program is stored in a recording medium such as a memory, and the CPU executes the program to perform the calculation process for generating the separated image 220.

[0082] The processing circuit 200 in this embodiment further detects one or more feature portions of the filter array 100 from the image acquired by the image sensor 60. Then, based on information such as the position, shape, or size of the feature portions, the processing circuit 200 detects the relative position between the filter array 100 and the image sensor 60. When the relative position is deviated from a preset relative position, the processing circuit 200 executes an operation to compensate for the deviation. In this embodiment, the operation to compensate for the deviation of the relative position is performed by controlling the movable stage 80 to move the filter array 100 relative to the image sensor 60. This includes an operation to correct the relative position between the filter array 100 and the image sensor 60.

[0083] The movable stage 80 is a driving device that changes at least one of the position and orientation of the filter array 100. The filter array 100 is fixed to the movable stage 80. The movable stage 80 can move or rotate the filter array 100 independently of the object 70, the optical systems 40A and 40B, and the image sensor 60. The movable stage 80 can move the filter array 100 along the x-axis, y-axis, or z-axis, rotate it within the xy-plane, yz-plane, or xz-plane, or perform a combination of these movements. The movable stage 80 may only have the function of moving along the x-axis, y-axis, or z-axis. The movement of the movable stage 80 is controlled by the processing circuit 200. In this embodiment, the movable stage 80 is an electrically driven driving device, but the movable stage 80 may also be manually driven.

[0084] FIG. 6A is a diagram for explaining the filter array 100. The diagram on the left side of FIG. 6A shows a schematic diagram of the configuration of the filter array 100. The diagram on the right side of FIG. 6A shows an example of the spectral transmittance of two filters in the filter array 100. FIG. 6B shows the spectral transmittance of N wavelength bands W1, W2, ..., W included in the target wavelength range. N 6A and 6B show an example of the spatial distribution of the transmittance of the filter array 100 for . While FIGS. 6A and 6B show regions in which ten square filters are arranged in a grid pattern in each of the vertical and horizontal directions of the filter array 100, this is merely an example. The shape of each filter in the filter array 100 is not limited to a square, but may be any shape that can be formed into a plane tessellation. Furthermore, the number of filters arranged in each of the vertical and horizontal directions is not limited to ten, and may be any number. The size of each filter in the filter array 100 and the overall size of the filter array 100 may take any value.

[0085] FIG. 6A shows an example of the range 102 that can be imaged by the image sensor 60 and the range 104 in which the separated image 220 is generated when there is no misalignment of the relative positions of the filter array 100. As in this example, the range 104 in which the separated image 220 is generated may be smaller than the imageable range 102. In this case, the processing circuit 200 extracts a portion corresponding to the range 104 in which the separated image 220 is generated from the image data generated by the image sensor 60 to generate the separated image 220. In other words, the processing circuit 200 generates image data for each of the N wavelength bands based on signals output from photodetection elements that receive light that has passed through a portion of the filter array 100 (i.e., the range 104 in which the separated image 220 is generated) among the multiple photodetection elements in the image sensor 60. In the example of FIG. 6A, only a portion of the filter array 100 is included in the range 102 that can be imaged by the image sensor 60.

[0086] 6A includes a plurality of alignment marks 106 inside a range 102 that can be imaged by the image sensor 60. In this embodiment, four alignment marks 106 are arranged at the four corners inside the range 102 that can be imaged. Each alignment mark 106 has a size that, when imaged on the imaging surface of the image sensor 60, forms an image that is larger than one pixel of the image sensor 60. The number, shape, and size of the alignment marks 106 are not limited to the example shown in the figure and may be selected arbitrarily.

[0087] The size of the range 104 in which the separated image 220 is generated is the same as or smaller than the size of the range 102 that can be captured by the image sensor 60. When the range 104 in which the separated image 220 is generated is smaller than the range 102 that can be captured, the alignment mark 106 may be located either outside or inside the range 104 in which the separated image 220 is generated, as shown in FIG.

[0088] 6A shows a cross-shaped alignment mark 106 as an example, but the shape of the alignment mark 106 may be determined arbitrarily as long as its position can be identified. The shape of the alignment mark 106 may be, for example, any polygon, all or part of a circle, or a combination thereof.

[0089] 6C, alignment mark 106 may be outside range 102 that can be captured by image sensor 60 when the relative positions of filter array 100 and image sensor 60 are set to a predetermined relative position. In this case, alignment mark 106 is not detected in the captured image when there is no relative positional deviation, but alignment mark 106 is detected when there is a relative positional deviation. Based on the position of alignment mark 106 thus detected, the relative positional deviation can be detected.

[0090] 6D, the area 102 that can be imaged by the image sensor 60 may include an area outside the filter array 100. In this case, the boundary 107 between the inside and outside of the filter array 100 serves the same function as the alignment mark 106. In such a configuration, the filter array 100 does not include any explicit marks, and the outer periphery that separates the inside and outside of the filter array 100 serves as the alignment feature.

[0091] FIG. 7 is an image showing an example of an alignment mark 106 actually formed on the filter array 100. In this example, a cross-shaped chromium (Cr) film is formed on the surface of the filter array 100 as the alignment mark 106. The small square blocks in FIG. 7 represent individual filters. In this example, the length of one side of each filter is approximately 8 micrometers (μm). The cross-shaped alignment mark 106 has a shape in which two linear portions intersect. When irradiated with white light, the chromium film exhibits lower transmittance than the surrounding filter region. Therefore, as shown in FIG. 7, it is observed as a dark area. Note that the alignment mark 106 is not limited to a chromium film and can be made of any material as long as it can be distinguished from other parts.

[0092] The processing circuit 200 can detect the relative position of the filter array 100 as viewed from the image sensor 60 by detecting the center coordinates of the cross-shaped alignment mark 106 exemplified in FIG.

[0093] FIG. 8A shows an example of the rate of change (unit: %) in the transmittance of the filter array 100 when the relative position of the filter array 100 as viewed from the image sensor 60 is shifted by one pixel in the x and y directions. Here, the rate of change (unit: %) in the average transmittance in a 100×100 grid area is shown, based on the transmittance when the relative positions of the filter array 100 and the image sensor 60 are at their initial positions. As shown in FIG. 8A, a change in the transmittance occurs simply by shifting the relative position by the size of one pixel of the image sensor 60. As a result, the processing circuit 200 may generate an image that is significantly different from the correct image. FIG. 8B shows an example of the error in average luminance between the image generated by the processing circuit 200 and the correct image. As shown in the figure, a shift of one pixel in the x or y direction in the relative position of the filter array 100 and the image sensor 60 significantly increases the error in average luminance.

[0094] Thus, the change in the relative position between the filter array 100 and the image sensor 60 has a strong influence on the accuracy of image generation, and therefore, it is necessary to reduce the influence caused by the change in relative position.

[0095] In this embodiment, the filter array 100 having alignment marks 106 allows the relative position of the image sensor 60 and the filter array 100 to be detected. Depending on the change in the detected relative position, the hardware of the imaging device or the captured image may be adjusted. By correcting the processing, it is possible to prevent a decrease in wavelength resolution.

[0096] 5 controls the movable stage 80 in accordance with the detected change in relative position to correct the position of the filter array 100. This makes it possible to suppress degradation of image quality caused by the change in relative position.

[0097] FIG. 9A is a diagram illustrating an example of an image for each wavelength band generated by the processing circuit 200 in this embodiment. FIG. 9B is a diagram illustrating an example of an image for each wavelength band generated without correction in this embodiment. In these examples, the position of the filter array 100 has shifted by approximately 1.5 pixels in the vertical direction of the image from the time when the spatial distribution data of the spectral transmittance was acquired. Three types of filters that pass light of 510 nm, 520 nm, and 530 nm, respectively, were used as the object 70, and images were captured with white light irradiated from behind. As shown in FIG. 9B, without correction, the three wavelength bands are not properly separated, and accurate images cannot be generated for each wavelength band. In other words, the wavelength resolution is reduced. In contrast, in the example shown in FIG. 9A, the position of the filter array 100 has been corrected, so spots corresponding to each of the three types of filters are clearly displayed. In other words, the reduction in wavelength resolution is suppressed.

[0098] As described above, by using the imaging device shown in FIG. 5, it is possible to adjust the relative position between the filter array 100 and the image sensor 60, and it is possible to prevent a decrease in wavelength resolution due to a change in the relative position.

[0099] FIG. 10 is a flowchart showing an outline of the processing executed by the processing circuit 200 in this embodiment. First, in step S101, the processing circuit 200 acquires from the image sensor 60 a wavelength-multiplexed image in which multiple wavelength components of light passing through the filter array 100 are superimposed. In the following step S102, the processing circuit 200 detects the relative position between the filter array 100 and the image sensor 60 by detecting the alignment mark 106 from the acquired image. Here, the relative position refers to, for example, the amount of displacement in the x and y directions of the coordinates of the center of the filter array 100 relative to the coordinates of the center of the imaging surface of the image sensor 60. In step S103, the processing circuit 200 determines whether the relative position between the filter array 100 and the image sensor 60 has changed from a preset initial relative position. Here, the initial relative position refers to the relative position when data indicating the spatial distribution of the spectral transmittance of the filter array 100 was acquired. If a change in the relative position is detected in step S103, the processing circuit 200 proceeds to step S104. In step S104, the processing circuit 200 instructs the movable stage 80 to return the relative position between the filter array 100 and the image sensor 60 to the initial position. In response to this instruction, the movable stage 80 corrects the position of the filter array 100. If no change in the relative position is recognized in step S103, step S104 is not executed and the process proceeds to step S105. In step S105, the processing circuit 200 generates multiple image data for each wavelength band based on the image acquired by the image sensor 60 and data indicating the spatial distribution of the spectral transmittance of the filter array 100.

[0100] Through the above operations, deviations from the appropriate values ​​of the relative positions of the filter array 100 and the image sensor 60 can be detected, and the relative positions can be corrected to compensate for the deviations, thereby suppressing a decrease in wavelength resolution due to changes in the relative positions.

[0101] FIG. 11 is a diagram showing a schematic configuration of an imaging device according to a modification of this embodiment. In this modification, the imaging device includes a movable stage 82, which is a second driving device that changes the position of the image sensor 60 instead of the filter array 100. The image sensor 60 is disposed on the movable stage 82. The processing circuit 200 controls the movable stage 82 to 5, the position of the image sensor 60 is changed along one or more of the x-axis, y-axis, and z-axis, thereby correcting the relative position between the filter array 100 and the image sensor 60. As in the example shown in FIG. 5, the image sensor 60 may be made rotatable within one or more of the xy-plane, yz-plane, and xz-plane.

[0102] FIG. 12 is a diagram schematically illustrating the configuration of an imaging device according to another modification. In this modification, a movable stage for moving the filter array 100 or the image sensor 60 is not provided. The imaging device according to this modification includes an optical system 50 disposed between the optical system 40B and the image sensor 60, and a movable stage 84, which is a third driving device for changing the position of the optical system 50. The optical system 50 includes at least one lens. Instead of or in addition to the lens, an optical element having a reflective function, such as a mirror, may be provided.

[0103] Optical systems 40A, 40B, and 50 are disposed on an optical path connecting filter array 100 and image sensor 60, and form an image of light passing through filter array 100 on the imaging surface of image sensor 60. Movable stage 84 changes the position of the formed image by moving or rotating at least one lens included in optical system 50. Processing circuit 200 can change the imaging position and compensate for any deviation in the relative positions of filter array 100 and image sensor 60 by controlling movable stage 84.

[0104] (Embodiment 2) 13 is a schematic diagram showing the configuration of an imaging device according to a second exemplary embodiment of the present disclosure. In this embodiment, rather than physically adjusting the relative positions of the filter array 100 and the image sensor 60, the processing circuit 200 compensates for the deviation in the relative positions by signal processing. More specifically, the processing circuit 200 corrects data indicating the spatial distribution of the spectral transmittance of the filter array 100 or the image acquired by the image sensor 60, thereby reducing the effect of changes in the relative positions of the filter array 100 and the image sensor 60.

[0105] The data representing the spatial distribution of the spectral transmittance of the filter array 100 may be, for example, data representing the matrix H in the above-described equation (2). Correcting the data representing the spatial distribution of the spectral transmittance means, for example, correcting each component of the matrix H so as to cancel or reduce the detected change in relative position. Correcting the image acquired by the image sensor 60 means, for example, performing one of the following operations on the vector g in equation (2): translation, rotation, or scaling. As an example, consider a case where the relative position of the filter array 100 as seen from the image sensor 60 has changed by n pixels of the image sensor 60 in the y direction shown in FIG. 13. In this case, the processing circuit 200 translates each component of the matrix H or the vector g in equation (2) by n in the y direction.

[0106] 14A to 14C are diagrams for explaining the effects of this embodiment. FIG. 14A shows an example of the restoration result of the images of the three wavelength bands when the relative position between the filter array 100 and the image sensor 60 has not changed from the initial relative position (corresponding to the vector g). FIG. 14B shows an example of the restoration result of the images of the three wavelength bands when the relative position between the filter array 100 and the image sensor 60 has shifted from the initial relative position (corresponding to the vector changing from g to g') and the matrix H has not been corrected. FIG. 14C shows an example of the restoration result of the images of the three wavelength bands when the relative position between the filter array 100 and the image sensor 60 has shifted from the initial relative position (corresponding to the vector g') and the matrix has been appropriately corrected (corresponding to the vector being corrected to H'). In these examples, as in the examples of FIGS. 9A and 9B, three filters that transmit light of 510 nm, 520 nm, and 530 nm, respectively, are arranged as the object 70. As shown in FIG. 14A, the state in which each of the three filters is displayed as a spot is considered to be a correct restoration. represents the original image. In the example of FIG. 14B, an incorrect image is generated. On the other hand, as shown in FIG. 14C, when the matrix H is appropriately corrected, it was confirmed that the generated image is closer to the image shown in FIG. 14A. In this way, by correcting the matrix H, it is possible to prevent a decrease in wavelength resolution. Note that the same effect can be obtained even if the vector g indicating the image to be acquired is corrected instead of correcting the matrix H.

[0107] As described above, in this embodiment, the operation of compensating for the deviation in relative position includes an operation of correcting data indicating the spatial distribution of the spectral transmittance of the multiple optical filters in the filter array 100 or the coordinates of the image acquired by the image sensor 60. For example, the processing circuit 200 corrects the matrix data indicating the spatial distribution of the spectral transmittance of the filter array 100 or the data of the image acquired by the image sensor 60. This makes it possible to suppress a decrease in wavelength resolution caused by a change in the relative position between the filter array 100 and the image sensor 60.

[0108] (Embodiment 3) FIG. 15A is a diagram schematically illustrating an example of a filter array 100 according to a third exemplary embodiment of the present disclosure. The configuration of the imaging device according to this embodiment is similar to that illustrated in FIG. 5. In this embodiment, the processing circuit 200 detects tilt of the filter array 100 based on the positional relationship between multiple alignment marks 106, rather than the position of a single alignment mark 106. The tilt of the filter array 100 refers to the tilt of the filter array 100 from a plane perpendicular to the optical axes of the optical systems 40A and 40B. In the example illustrated in FIG. 15A, the processing circuit 200 detects tilt by measuring the spacing L1 between the alignment marks 106 in the y direction and the spacing L2 between the alignment marks 106 in the x direction based on the acquired image. When tilt occurs in the filter array 100 in the φ direction shown in FIG. 5, L1 is observed to be shorter than its original length. When tilt occurs in the η direction, L2 is observed to be shorter than its original length. When tilting occurs in the φ direction by an angle φ1 as shown in Fig. 5, the length of L1 becomes cos(φ1) times the original length. Similarly, when tilting occurs in the η direction by an angle η1 as shown in Fig. 5, the length of L2 becomes cos(η1) times the original length.

[0109] FIG. 15B is a schematic diagram illustrating a filter array 100 according to a modified example of this embodiment. Each alignment mark 106 has a shape including a first portion extending in the y-direction and a second portion extending in the x-direction. By measuring the length L3 of the first portion and the length L4 of the second portion, tilt of the filter array 100 can be detected. In this example, the first and second portions intersect at right angles, but they may also intersect at an angle other than 90 degrees. The processing circuit 200 detects tilt of the filter array 100 based on the ratio of the lengths of the first and second portions of one or more alignment marks 106 in an image acquired by the image sensor 60, and performs operations to reduce the effects of tilt.

[0110] The operation of reducing the influence of tilt can be performed, for example, by controlling the movable stage 80 shown in Fig. 5. Instead of the configuration shown in Fig. 5, for example, the configuration shown in Fig. 11 may be adopted. In that case, a movable stage 82 capable of changing the orientation of the image sensor 60 may be used. Alternatively, instead of correcting the relative position by mechanical driving, the influence of tilt may be reduced by software processing, as in the second embodiment.

[0111] (Embodiment 4) In the above embodiment, the processing circuit 200 detects the relative position between the filter array 100 and the image sensor 60 based on one or more feature portions (e.g., portions of the alignment marks 106) of the filter array 100 in the image acquired by the image sensor 60. Then, when the relative position is deviated from the preset relative position, the processing circuit 200 performs an operation to compensate for the deviation of the relative position.

[0112] In contrast, in the fourth embodiment, the processing circuit 200 executes one of the following operations (a) and (b). (a) The process of generating image data for each of the N wavelength bands is repeated multiple times while changing the relative position between the filter array 100 and the image sensor 60 and correcting the data indicating the spatial distribution of spectral transmittance in accordance with the change in relative position, thereby generating multiple sets of image data for each wavelength band. (b) The process of generating image data for each of the N wavelength bands is repeated multiple times by changing the coordinates of the image acquired by the image sensor 60 and correcting the data indicating the spatial distribution of the spectral transmittance in accordance with the change in coordinates, thereby generating multiple sets of image data for each wavelength band.

[0113] The processing circuit 200 generates output data by combining multiple sets of image data for each wavelength band through processing such as averaging, etc. This operation can reduce noise contained in the generated image.

[0114] FIG. 16 is a diagram illustrating signal processing in this embodiment. In this embodiment, a single filter array 100 is used to capture images multiple times while intentionally changing the relative position between the filter array 100 and the image sensor 60. As described with reference to FIG. 8A, when the relative position between the filter array 100 and the image sensor 60 changes, the spatial distribution of the spectral transmittance of the filter array 100 changes. That is, a single filter array 100 functions as another filter array 100 with a different spectral transmittance distribution. Therefore, this embodiment achieves the same effect as generating an image using n types of filter arrays 100 (n is an integer greater than or equal to 2) with different spectral transmittance distributions. When the n generated images are averaged, the noise added to the generated image is statistically reduced by 1 / √n times.

[0115] FIG. 17 is a flowchart illustrating an example of signal processing in this embodiment. In this example, the configuration shown in FIG. 5 is used, as in the first embodiment. The processing circuit 200 generates multiple images for each wavelength band with the filter array 100 in its initial position by executing steps S101 to S105 shown in FIG. 10 . In this example, a relative position correction process based on alignment marks is also performed, but this correction process may be omitted. In this case, steps S102, S103, and S104 are omitted, and the filter array 100 does not need to have a feature such as an alignment mark. In the following step S106, the processing circuit 200 determines whether the relative position between the filter array 100 and the image sensor 60 has been changed a predetermined number of times (n times). If the determination is No, the processing circuit 200 sets a new initial position different from the original initial position (step S107). For example, a position slightly shifted from the original initial position in a predetermined direction, such as the x direction or y direction, may be set as the new initial position. The "microscopic size" may be, for example, a size equivalent to one to several pixels of the image sensor 60. Next, the processing circuit 200 controls the movable stage 80 to move the filter array 100 to the set new initial position (step S108). Accordingly, the data indicating the spatial distribution of spectral transmittance used to generate the spectral image data is also corrected. Thereafter, the operations of steps S101 to S108 are repeated until the determination in step S106 is Yes. In this embodiment, the movement of the filter array 100 in step S108 is performed by physical movement. Instead of moving the filter array 100, the image sensor 60 may be moved. Alternatively, instead of physically changing the relative position, the coordinates of the image acquired by the image sensor 60 may be changed. Regarding the correction of the relative position in step S104, either physical movement or software matrix correction may be used. After repeating the processes of steps S101 to S108 a predetermined number of times, the processing circuit 200 averages the generated n images to obtain the synthesized output data. It is generated and output (step S109).

[0116] Through the above operations, noise removal processing can be performed by imaging using one filter array 100. This reduces errors in the generated hyperspectral image and improves wavelength resolution.

[0117] The features of each of the above-described first to fourth embodiments may be combined with the features of other embodiments as long as there is no contradiction. For example, the mechanical correction of relative position in the first embodiment may be combined with the software-based correction of relative position in the second embodiment. Furthermore, the processing of the fourth embodiment may be applied to any of the configurations shown in FIGS. 11 to 13. [Industrial Applicability]

[0118] The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength images. The technology disclosed herein can also be applied to, for example, biomedical, medical, and cosmetic sensing, food foreign matter and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems. [Explanation of symbols]

[0119] 40, 50 optical system 60 Image Sensor 70 Objects 80, 82, 84 Movable stage 100 filter array 102 Imageable range 104 Range in which separated images are generated 106 Alignment Mark 200 Processing Circuit 220 Separated Images

Claims

1. acquiring first information indicating a relative position between an encoding element including a plurality of regions with different spectral transmittances and an image sensor that detects light that has passed through the encoding element; correcting data including second information relating to the spatial distribution of the spectral transmittance based on the first information; the data is used in a process of generating four or more spectral images based on an image acquired by detecting light that has passed through the encoding element with the image sensor; each of the four or more spectral images represents an image corresponding to one wavelength band among four or more wavelength bands; method.

2. determining a deviation of a relative position between the encoding element and the image sensor from an optimum value based on the first information; correcting the data includes correcting the data in accordance with the deviation. The method of claim 1.

3. the image acquired by detecting the light that has passed through the encoding element with the image sensor has components of the four or more wavelength bands superimposed thereon; The method of claim 1.

4. the encoding element is a filter array including a plurality of optical filters arranged in a two-dimensional plane; each of the plurality of regions corresponds to each of the plurality of optical filters; The method according to claim 1 or 2.

5. A program that causes a processing circuit to perform the method of claim 1.

6. processing circuitry; The processing circuitry acquiring first information indicating a relative position between an encoding element including a plurality of regions having different spectral transmittances and an image sensor that detects light that has passed through the encoding element; correcting data including second information relating to the spatial distribution of the spectral transmittance based on the first information; the data is used in a process of generating four or more spectral images based on an image acquired by causing the image sensor to detect light that has passed through the encoding element; each of the four or more spectral images represents an image corresponding to one wavelength band among four or more wavelength bands; Processing equipment.

Images