Encoding imaging device

Abstract

To obtain an encoding imaging device that can suppress the drop or rise in the brightness level of an entire image due to reduction in dynamic range which is caused by noise generated in a masked pixel region when a high-resolution image is reconstructed from a low-resolution encoded image, thereby acquiring a high-resolution image.SOLUTION: An encoding imaging device includes a light modulator 12 that displays a predetermined encoding pattern to generate an encoded pattern of an image of a subject 10, a lens 11 that converges the encoded pattern of the image of the subject 10 so that the encoded pattern of the image of the subject 10 is equal to or smaller than the pixel size of an imaging device 14, the imaging unit 14 that captures the encoded pattern of the image of the subject 10, and a reconstruction processing unit 15 including a low-noise processing mechanism 15A that reduces noise generated in a pixel region masked for pattern formation when imaging the encoded pattern, and a reconstruction calculation mechanism 15B that reconstructs a low-resolution encoded image acquired by the imaging unit 14 into a high-resolution image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an encoded imaging device that acquires high-resolution images using super-resolution technology. [Background technology]

[0002] Super-resolution technology is known as a technique for creating high-resolution images from low-resolution images. One known technique for super-resolution imaging involves, for example, as shown in Figure 12, irradiating light (luminance information) (a) from a subject onto an optical modulator displaying multiple patterns to optically modulate it, capturing an image of the subject (b) carrying various encoded patterns from this optical modulator with a camera, and then reconstructing the multiple captured low-resolution encoded images (c) into a high-resolution image (d) using a computer.

[0003] The process described above, which involves reconstructing a low-resolution encoded image (c) into a high-resolution image (d) using a computer, is performed by solving the inverse problem Y=AX. Specifically, let Y be the low-resolution encoded image (c), and let A be the super-resolution factor representing the encoding and resampling information in the optical system. Solve the equation Y=AX, which is specific to the optical system, to obtain the high-resolution image (d), X.

[0004] Furthermore, while generally more conditional equations than the number of unknowns are needed to find the unknowns in a system of equations, if the unknowns are sparse, the unknowns can be estimated using fewer conditional equations than the number of unknowns. This method is known as compressed sensing technology, and by using this technology, it is possible to reduce the number of images to be captured.

[0005] In images, by using a basis that can compress some amount of information, such as the Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT), or Discrete Fourier Transform (DFT), and then converting to frequency components, it is possible to introduce sparsity to the unknowns. In this way, high-resolution images can be reconstructed by applying inverse DCT, inverse DWT, or inverse DFT processing to sparse solutions obtained using compressed sensing technology in the reconstruction process.

[0006] Incidentally, in actual optical systems, there is noise from the image sensor and stray light in the optical system, and if this noise becomes excessive, the quality of the reconstructed high-resolution image will be significantly reduced. The main cause of this noise lies in the discrepancy between the super-resolution factor used in the reconstruction process and the actual optical system and imaging system. To reduce this noise, two methods are known: one that uses signal processing to reduce noise, and another that focuses on the hardware of the optical system and imaging system.

[0007] For example, the technology described in Non-Patent Document 1 below uses a digital micromirror device (hereinafter referred to as DMD), which is an optical modulator in which a large number of tiny mirrors are arranged in a grid and the reflection can be switched ON or OFF for each mirror, to encode light from an object. This makes it possible to adjust hyperparameters using compressed sensing technology when performing super-resolution reconstruction calculations and to remove noise that follows a Poisson distribution appearing in the frequency domain.

[0008] Furthermore, when setting up the optical system, noise caused by misalignment can be reduced by precisely aligning the DMD, which is used as an optical modulator, and the camera at the pixel level. For example, the technology described in Patent Document 1 below is a method for aligning the DMD and the camera at the sub-pixel level using moiré fringes. By using this method, noise caused by resampling of simultaneous equations can be reduced.

[0009] Furthermore, the technology described in Non-Patent Document 2 below is a method that aims to reduce noise caused by positional misalignment between pixels by using a resampling factor that compensates for the misalignment between the DMD used as an optical modulator and the pixels of the camera during the reconstruction process. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Patent No. 3937024 [Non-patent literature]

[0011] [Non-Patent Document 1] Marco F. Duarte, et al., “Single-pixel imaging via compressive sampling: Building simpler, smaller, and less-expensive digital cameras”, IEEE Signal Processing Magazine, 25(2), pp. 83-91, 2008 [Non-Patent Document 2] Xudong Zhang, et al., “MEMS-based super-resolution remote sensing system using compressive sensing”, Optics Communications, 426, 410-417, 2018 [Overview of the project]

Problems to be Solved by the Invention

[0012] As factors that significantly affect the resolution of the reproduced encoded image described above, in addition to the above-mentioned noises, there is noise generated in the area masked by the pattern during encoding. Such noise also causes a deterioration in image quality. FIG. 13 shows the reconstruction results obtained by adjusting the hyperparameters. (A) is data showing a state where the luminance level is generally raised, (B) is data showing a state where the luminance level is generally lowered, and (C) is data showing the true value (Ground truth).

[0013] That is, as shown in FIG. 14, the reason for the above-mentioned deterioration in image quality is that noise occurs in the masked pixel area due to dark current, pixel unevenness, readout noise, optical shot noise, etc. For example, the luminance level of the low-resolution encoded image is generally raised (in the example of FIG. 14, a state where 20 units of noise are superimposed on each part is shown), and a deviation from an ideal situation as shown in the flow of FIG. 12 occurs.

[0014] Therefore, as in the technique shown in the above Non-Patent Document 1, when adjusting the hyperparameters during reconstruction, the noise generated in the masked pixel area cannot be suppressed, and the dynamic range of the high-resolution image decreases, resulting in an increase or decrease in the luminance level of the entire image. In each of the above-mentioned prior arts, no description or suggestion is made regarding a solution to such a problem.

[0015] Therefore, in the present invention, when reconstructing a high-resolution image from a low-resolution encoded image, it is an object to provide an encoding imaging device that can suppress a decrease or increase in the luminance level of the entire image associated with a decrease in the dynamic range caused by noise generated in the masked pixel area, and obtain a high-quality high-resolution image.

Means for Solving the Problems

[0016] The encoding imaging device of the present invention is An optical modulator having the function of displaying a predetermined coding pattern and generating a coding pattern for the image of a subject, A lens that converges the encoding pattern of the image of the subject so that the encoding pattern of the image of the subject becomes less than or equal to the pixel size of the image sensor, An imaging unit equipped with an image sensor that captures the encoded pattern of the image of the subject, A reconstruction processing unit includes a low-noise processing mechanism that reduces noise generated in the pixel area masked for pattern formation when capturing the encoded pattern, and a reconstruction calculation mechanism that reconstructs the low-resolution encoded image acquired by the imaging unit into a high-resolution image. Equipped with 、 The reconstruction processing unit is configured to perform the process of reconstructing the low-resolution encoded image into the high-resolution image using the reconstruction calculation mechanism, and then to perform the process of reducing the noise using the low-noise processing mechanism. A pattern with all pixels turned ON is displayed on the optical modulator, and this pattern is captured by the imaging unit, thereby capturing a low-resolution image Y of the optical modulator's pixel area when all pixels are lit. ON The optical modulator is configured to capture images, and the pixel area of ​​the optical modulator is divided into blocks of a predetermined number of pixels, and a predetermined number of pixels within each block are turned ON in sequence, displaying these patterns on the optical modulator, thereby capturing low-resolution encoded images for each pattern. It is characterized by the following:

[0019] Also, above Note In an encoding imaging device, light carrying the image information of the subject may be irradiated onto the optical modulator which displays the predetermined encoding pattern, and the light emitted from the optical modulator, carrying the encoding pattern information of the image of the subject, may be imaged by the imaging unit. Alternatively, the device may be configured such that light carrying predetermined coding pattern information from the optical modulator displaying the predetermined coding pattern is irradiated onto the subject, and light carrying the coding pattern information of the image of the subject, emitted from the subject, is captured by the imaging unit. [Effects of the Invention]

[0020] The encoding imaging apparatus of the present invention includes a low-noise processing mechanism that reduces noise generated in the pixel area masked for pattern formation when imaging an encoding pattern, and a reconstruction calculation mechanism that reconstructs a low-resolution encoding image captured by the imaging unit into a high-resolution image.

[0021] Therefore, by performing either the low-noise processing, which reduces noise generated in the pixel area masked for pattern formation, or the reconstruction calculation processing, which reconstructs the low-resolution encoded image captured by the imaging unit into a high-resolution image, before the other processing, it becomes possible to suppress the decrease or increase in the overall brightness level of the image, which occurs due to a decrease in dynamic range caused by noise generated in the area masked by the pattern during encoding, when reconstructing a high-resolution image from a low-resolution encoded image in super-resolution imaging using an encoded pattern. [Brief explanation of the drawing]

[0022] [Figure 1] This is a conceptual diagram illustrating the configuration for generating a high-resolution image from a low-resolution encoded image in the encoding imaging device according to the present invention. [Figure 2] This is a conceptual diagram illustrating a reflective imaging optical system using a DMD in an encoding imaging device according to an embodiment of the present invention. [Figure 3] This is a conceptual diagram illustrating an example of an encoded pattern to be displayed on a DMD in an imaging device according to an embodiment of the present invention. [Figure 4] This is a conceptual diagram illustrating the reconstruction calculation method (Method 1) according to Embodiment 1 of the present invention. [Figure 5] This is a conceptual diagram illustrating the reconstruction calculation method (method 2) according to Embodiment 1 of the present invention. [Figure 6] This is a conceptual diagram illustrating the low-noise processing according to Embodiment 1 of the present invention. [Figure 7] This figure shows (A) the reconstructed image result and (B) the ground truth when low-noise processing according to Embodiment 1 of the present invention is applied. [Figure 8] This is a conceptual diagram illustrating the low-noise processing (generation of a low-resolution noise map E) according to Embodiment 2 of the present invention. [Figure 9] This is a conceptual diagram illustrating the low-noise processing (method for determining the low-resolution noise pattern F) according to Embodiment 2 of the present invention. [Figure 10]This figure shows (A) the reconstructed image result and (B) the ground truth when low-noise processing according to Embodiment 2 of the present invention is applied. [Figure 11] This figure shows (A) the reconstructed image result and (B) the ground truth when low-noise processing according to Embodiment 3 of the present invention is applied. [Figure 12] This is a conceptual diagram showing the super-resolution flow using general coding. [Figure 13] This figure shows examples of reconstructed images when noise occurs in the mask region that forms the pattern during encoding ((A) when the brightness level is elevated, (B) when the brightness level is decreased) and (C) the true value (ground truth). [Figure 14] This is a conceptual diagram illustrating the super-resolution flow by encoding when noise occurs in the mask region that forms the pattern during encoding. [Modes for carrying out the invention]

[0023] <Concept of Invention> The coding imaging apparatus according to an embodiment of the present invention will be described below with reference to the drawings, but before that, the concept of the coding imaging apparatus according to the present invention will be briefly explained. As shown in Figure 1, the encoding imaging device 1 according to the present invention includes an optical modulator 12 that illuminates light from a subject 10 through a lens 11 and modulates this illuminated light with a spatial encoding pattern, a lens 13 that adjusts the size of the encoding pattern image so that the encoding pattern is less than or equal to the pixel size of the camera, an imaging unit 14 that captures this encoding pattern image, and a reconstruction processing unit 15 that reduces noise that appears in the area masked by the pattern during encoding and reconstructs the low-resolution encoding image acquired by the imaging unit 14 into a high-resolution image.

[0024] Furthermore, the reconstruction processing unit 15 includes a low-noise processing mechanism 15A that reduces noise appearing in areas masked by a pattern during encoding, and a reconstruction calculation mechanism 15B that reconstructs the low-resolution encoded image acquired by the imaging unit 14 into a high-resolution image. The encoding imaging device 1 according to the present invention can employ two methods depending on which of the processing by the low-noise processing mechanism 15A and the processing by the reconstruction calculation mechanism 15B is performed first and which is performed last.

[0025] Specifically, the first method is configured to correct a low-resolution encoded image with superimposed noise using a low-noise processing mechanism 15A to suppress the effects of noise, and then reconstruct a high-resolution image using a reconstruction calculation mechanism 15B. The second method is configured to reconstruct a high-resolution image from a low-resolution encoded image with superimposed noise using a reconstruction calculation mechanism 15B, and then correct the high-resolution image using a low-noise processing mechanism 15A to suppress the effects of noise. Both of these methods constitute an encoded imaging device according to the present invention.

[0026] In the following, Embodiments 1 to 3, which are methods for reducing noise generated in the mask region, will be described in order. Embodiments 1 and 2 specifically illustrate the first method described above, and Embodiment 3 specifically illustrates the second method described above.

[0027] <Embodiment 1> The encoding imaging device according to Embodiment 1 will be described below with reference to Figure 2. (Imaging optical system) First, the imaging optical system of the encoding imaging device 1A according to Embodiment 1 uses a DMD12A as the optical modulator 12, as specifically shown in Figure 2, thereby constructing a reflective optical system.

[0028] First, the light from subject 10A is imaged onto DMD 12A by lens 1 (11A), and the reflected light from DMD 12A is adjusted to be imaged onto the imaging plane of camera 14A by lens 2 (13A). Next, a pattern in which all pixels of the DMD12A are turned OFF (hereinafter referred to as "all off") is displayed. By capturing this with camera 14A, a low-resolution image Y of the DMD12A when all pixels are off is obtained. OFF Capture an image. This low-resolution image Y OFF This is then transferred to the reconstruction processing unit 15 described above.

[0029] Next, as shown in Figure 3, the pixel area of ​​the DMD12A is divided into blocks of (i,j) pixels, and one pixel in each block is turned ON to sequentially display the coding pattern (p types) on the DMD12A. A total of p low-resolution coded images for each coding pattern are captured by the camera 14A. The captured p low-resolution coded images are then transferred to the reconstruction processing unit 15.

[0030] In addition to the DMD12A, other optical modulators such as transmissive or reflective liquid crystal displays (LCDs) may also be used as optical modulators. Alternatively, encoding may be performed by setting a fixed mask on a stage or similar surface and moving the fixed mask and the stage relatively. Furthermore, the encoding pattern displayed on the DMD12A is not limited to the one shown in this embodiment, in which one pixel in each block is set to the ON state. It is also possible to use a pattern in which multiple pixels in each block are set to the ON state, or any other pattern including random patterns and orthogonal bases using Hadamard patterns. In addition, the pixels set to the ON state in each block do not necessarily have to be corresponding pixels between blocks.

[0031] Furthermore, the positional relationship between the DMD12A and the camera 14A may also be such that, in addition to the positional relationship of this embodiment described above, a positional shift exists between one block of the DMD12A and one pixel of the camera, by considering correction with the resampling factor. Furthermore, in this embodiment, the image of the subject 10A is formed on the DMD 12A and encoded, but the encoding imaging device according to the present invention is not limited to this, and may also be equipped with an imaging optical system that illuminates the subject 10A with an encoded pattern displayed on the DMD 12A, and captures an image of the subject 10A carrying the pattern information displayed on the DMD 12A with a camera 14A, thereby performing encoding on the illumination light.

[0032] (Reconstruction processing unit: Reconstruction computing mechanism) As mentioned above, the reconstruction processing unit 15 includes a reconstruction calculation mechanism 15B, and the following reconstruction calculation is performed in the reconstruction calculation mechanism 15B. In other words, if we let Y be the low-resolution encoded image, R be the resampling by camera 14A, D be the diagonal matrix of the encoding pattern, and O be the super-resolution high-resolution image, then applying the optical system to the formula gives the following equation (1). Y = RDO …(1) Note that the diagonal matrix D of the coding pattern uses a binary coding pattern.

[0033] Furthermore, if we let B be the DCT basis and X be the DCT component of the super-resolution image, then equation (1) above is transformed by DCT into equation (2) below. Y = RDBX …(2) Here, X and Y are one-dimensional signals and can be represented as a product of linear matrices, as shown in Figure 4.

[0034] Here, we construct a system of p equations based on the p low-resolution encoded images obtained. If the size of the low-resolution encoded image is (m,n), the size of the super-resolution high-resolution image is (M,N), and the super-resolution factor for the RDB as a whole is A, then A and Y(={Y1,Y2,…,Y p Since}) is known, from these two known elements X(={X1,…,X M×N The inverse problem for finding} is Y=AX (see Figure 5).

[0035] Generally, in a system of M×N linear equations (i.e., a system of linear equations with M×N unknowns), at least M×N conditions are required. However, by using compressed sensing techniques, it is possible to find solutions even if the number of conditions is less than the number of unknowns. In this embodiment, focusing on this point, the image is converted into a sparse solution by using the DCT transformation method. Furthermore, by using compressed sensing technology, super-resolution can be performed with a small number of images without the need for AI or other learning processes. Instead of using the DCT transformation method, it is also possible to use transformation methods such as DWT or DFT.

[0036] When solving the above system of equations, we use the ADMM (Alternating Direction Method of Multipliers) method to solve the optimization problem shown in equation (3) below, which is called LASSO (Least absolute shrinkage and selection operator) regression. Instead of the ADMM method, other methods such as Newton's method, quasi-Newton's method, coordinate descent, ISTA (Iterative shrinkage-thresholding algorithm), and FISTA (Fast iterative shrinkage-thresholding algorithm) may be used as the solution algorithm for the optimization problem below.

[0037]

number

[0038] (Reconstruction processing unit: Low-noise processing mechanism) As mentioned above, the reconstruction processing unit 15 includes a low-noise processing mechanism 15A, and the following low-noise processing is performed in the low-noise processing mechanism 15A. In other words, the low-resolution image Y obtained beforehand when all DMD12A lights are turned off. OFF This represents the brightness level when light is not incident on the entire surface of camera 14A, and is an image that displays only the noise component that raises the black level. Each obtained low-resolution encoded image Y k To perform noise correction on Y, k Y' = Y k - Y OFF is calculated, and Y' = {Y1', Y2', …, Y p '} is obtained (see FIG. 6).

[0039] (Reconstruction processing unit: configuration and function) Using Y' obtained by the low-noise processing mechanism 15A, the inverse problem Y' = AX is solved to reconstruct the frequency components X^ of the high-resolution image. X^ is converted from the frequency domain to the spatial domain to obtain the super-resolved high-resolution image O^. After performing low-noise processing by the low-noise processing mechanism 15A and then performing reconstruction processing, it is possible to reduce the noise appearing in the region masked by the pattern during encoding and suppress the decrease in the dynamic range of the super-resolved high-resolution image.

[0040] In this embodiment, with (i, j) = (2, 2), p = 4 low-resolution encoded images were captured. The low-resolution image Y OFF previously captured from the 4 images was subtracted to obtain Y'. The inverse equation Y' = AX was solved by ADMM to reconstruct the DCT components X^ of the high-resolution image. Further, an inverse DCT was performed to obtain a high-resolution image 0^ that was super-resolved by a factor of 2 in both the vertical and horizontal directions. As shown in FIG. 7(A), the result is that the PSNR is 22.5 dB, clearly indicating that the decrease in the dynamic range can be suppressed. Note that FIG. 7(B) shows the ground truth.

[0041] In the above-described Embodiment 1, super-resolution reconstruction processing is performed using the number of low-resolution encoded images for which the number of conditional expressions is equal to the number of unknowns. However, it is also possible to perform reconstruction processing by reducing the number of low-resolution encoded images to be captured compared to the number of unknowns.

[0042] <Embodiment 2> The following describes the encoding imaging device according to Embodiment 2. However, since the basic configuration is similar to that of Embodiment 1, to avoid unnecessary detail, descriptions that overlap with those of Embodiment 1 will be omitted, and only the differences in configuration will be described. Furthermore, corresponding components will be described using the reference numerals in Figure 2. In the above-described embodiment 1, by using the low-noise processing mechanism 15A, the decrease in the dynamic range of the super-resolved reconstructed image can be suppressed, but the low-resolution image Y OFF This is the brightness level when light is not incident on the entire surface of the image sensor of camera 14A, therefore, from each low-resolution encoded image, the low-resolution image Y OFF Subtracting this value reduces the brightness level of the reflected light component from the unmasked subject image region in the corrected low-resolution image Y', resulting in a decrease in the overall brightness level of the reconstructed high-resolution image. Therefore, by using a binary inverted encoding pattern to correct noise with high precision, it becomes possible to reconstruct high-resolution images with a wide dynamic range and brightness levels close to the true value (ground truth).

[0043] In the following, the sections on (imaging optical system) and (reconstruction processing unit: reconstruction calculation mechanism) in Embodiment 2 are the same as those in Embodiment 1 and will therefore be omitted. Only the sections on (reconstruction processing unit: low-noise processing mechanism) and (reconstruction processing unit: configuration and function), which differ from Embodiment 1, will be explained.

[0044] (Reconstruction processing unit: Low-noise processing mechanism) The reconstruction processing unit 15 includes a low-noise processing mechanism 15A, similar to Embodiment 1, and the following low-noise processing is performed in the low-noise processing mechanism 15A. In other words, the low-noise processing is performed in the manner shown in Figure 8. First, the known coding pattern shown in (a) is made one-dimensional as shown in (b). After this, the binary values ​​of the coding pattern are inverted to obtain the inverted pattern C shown in (c). INV To obtain. Next, inversion pattern C INVThen, multiply by R, which is the resampling by camera 14A (RC INV (Calculates) and then performs a low-resolution processing. This results in the low-resolution noise map E=RC shown in (d). INV To obtain.

[0045] Next, we have the low-resolution noise map E and the previously acquired low-resolution image Y. OFF The Hadamard product is calculated to find the low-resolution noise pattern F.

number

[0046] (Reconfiguration Processing Unit: Configuration and Function) Using Y' obtained by the low-noise processing mechanism 15A, the inverse problem Y'=AX is solved to reconstruct the frequency component X^ of the high-resolution image. X^ is converted from the frequency domain to the spatial domain to obtain the super-resolved high-resolution image O^. By performing low-noise processing using the low-noise processing mechanism 15A and then performing reconstruction processing, noise appearing in areas masked by patterns during encoding can be reduced, thereby suppressing a decrease in the dynamic range of the super-resolution high-resolution image.

[0047] In the following, the sections on (imaging optical system) and (reconstruction processing unit: reconstruction calculation mechanism) in Embodiment 2 are the same as those in Embodiment 1 and will be omitted. Only the sections on (reconstruction processing unit: low-noise processing mechanism) and (reconstruction processing unit: configuration and function), which differ from Embodiment 1, will be described below.

[0048] In this embodiment, (i, j) = (2, 2), and p = 4 low-resolution encoded images were captured. The previously captured low-resolution image Y OFF The low-resolution noise pattern F was obtained using the four encoded patterns. The low-resolution noise pattern F was subtracted from the four obtained low-resolution encoded images Y to obtain Y'. Next, the inverse equation Y'=AX was solved using ADMM to reconstruct the DCT component X^ of the high-resolution image. Furthermore, inverse DCT was performed to obtain a high-resolution image 0^ that was super-resolved twice in both the horizontal and vertical directions.

[0049] As shown in Figure 10(A), the PSNR was 36.7 dB, clearly demonstrating that the decrease in dynamic range could be suppressed. The overall brightness level of the image was even higher than that of Embodiment 1, confirming that a result close to the true value could be obtained. Figure 10(B) shows the true value (ground truth). In the embodiments described above, super-resolution reconstruction was performed using a number of low-resolution encoded images equal to the number of conditional expressions and unknowns. However, the reconstruction process may also be performed by reducing the number of low-resolution encoded images captured.

[0050] <Embodiment 3> The following describes the encoding imaging device according to Embodiment 3. However, since the basic configuration is similar to that of Embodiment 1, to avoid unnecessary detail, descriptions that overlap with those of Embodiment 1 will be omitted, and only the differences in configuration will be described. Furthermore, corresponding components will be described using the reference numerals in Figure 2.

[0051] In the following, the section on (Reconstruction Processing Unit: Reconstruction Calculation Mechanism) in Embodiment 3 is omitted as it is the same as in Embodiment 1. Only the sections on (Imaging Optical System), (Reconstruction Processing Unit: Low-Noise Processing Mechanism), and (Reconstruction Processing Unit: Configuration and Function), which differ from Embodiment 1, will be explained.

[0052] (Imaging optical system) As specifically shown in Figure 2, the imaging optical system of the encoding imaging device according to Embodiment 3 uses a DMD12A as the optical modulator 12, thereby constructing a reflective optical system. First, the light from subject 10A is imaged onto DMD 12A by lens 1 (11A), and the reflected light from DMD 12A is adjusted to be imaged onto the imaging plane of camera 14A by lens 2 (13A). Next, the DMD12A and camera 14A are aligned so that each block of pixels (i,j) in the DMD12A is imaged onto one pixel of the camera 14A.

[0053] Next, a pattern in which all pixels are turned ON (hereinafter referred to as "all-on") is displayed on the DMD12A. By capturing this with the camera 14A, a low-resolution image Y of the DMD12A when all pixels are turned ON is obtained. ON The image is captured. Furthermore, as shown in Figure 3, the pixel area of ​​the DMD12A is divided into blocks of (i, j) pixels, and one pixel in each block is turned ON to display a coding pattern (p types) sequentially on the DMD12A, and a total of p low-resolution coded images for each coding pattern are captured by the camera 14A. The p+1 captured images obtained in this way are then transferred to the reconstruction processing unit 15 described above.

[0054] In addition to the DMD12A, other optical modulators such as transmissive or reflective liquid crystal displays (LCDs) may also be used as optical modulators. Alternatively, encoding may be performed by setting a fixed mask on a stage or similar surface and moving the fixed mask and the stage relatively. Furthermore, the encoding pattern displayed on the DMD12A is not limited to the one shown in this embodiment, in which one pixel in each block is set to the ON state. It is also possible to use a pattern in which multiple pixels in each block are set to the ON state, or any other pattern including random patterns and orthogonal bases using Hadamard patterns. In addition, the pixels set to the ON state in each block do not necessarily have to be corresponding pixels between blocks.

[0055] Furthermore, the positional relationship between the DMD12A and the camera 14A may also be such that, in addition to the positional relationship of this embodiment described above, a positional shift exists between one block of the DMD12A and one pixel of the camera, by considering correction with the resampling factor. Furthermore, in this embodiment, the image of the subject 10A is formed on the DMD 12A and encoded, but the encoding imaging device according to the present invention is not limited to this, and may also be equipped with an imaging optical system that illuminates the subject 10A with an encoded pattern displayed on the DMD 12A, and captures an image of the subject 10A carrying the pattern information displayed on the DMD 12A with a camera 14A, thereby performing encoding on the illumination light.

[0056] (Reconfiguration Processing Unit: Configuration and Function) Using Y obtained by the low-noise processing mechanism 15A, the inverse problem Y=AX is solved to reconstruct the frequency component X^ of the high-resolution image. X^ is converted from the frequency domain to the spatial domain to obtain the super-resolved high-resolution image O^. The other configurations are the same as those of Embodiment 1 described above.

[0057] (Reconstruction processing unit: Low-noise processing mechanism) Noise in the low-resolution encoded image affects the overall brightness of the reconstructed high-resolution image. To improve this, the bias component of the entire image should be adjusted, so the low-resolution image Y ON This is used to correct the zeroth-order component in the frequency domain. For the DCT component X^ of the reconstructed high-resolution image, the low-resolution image Y obtained as described above is used. ON Then, the 0th-order component of the DCT component is corrected. ON We find the zeroth-order component of the DCT component. We multiply this by MN / mn, replace it with the zeroth-order component of X^, and perform inverse DCT. Here, mn represents the size of the low-resolution encoded image, and MN represents the size of the super-resolution high-resolution image. In this way, noise appearing in areas masked by the pattern during encoding can be reduced, and the decrease in the dynamic range of the super-resolution high-resolution image can be suppressed.

[0058] In this embodiment, (i, j) = (2, 2), and one low-resolution image Y ON Four low-resolution encoded images were acquired, and the inverse equation Y=AX was solved using ADMM with the four acquired low-resolution encoded images Y to reconstruct the DCT component X^ of the high-resolution image. ON The zeroth-order component of the DCT component was obtained. This was multiplied by MN / mn (=4), substituted with the zeroth-order component of X^, and the inverse DCT was performed.

[0059] As shown in Figure 11(A), the PSNR was 37.1 dB, clearly demonstrating that the decrease in dynamic range could be suppressed. The overall brightness level of the image was even higher than that of Embodiment 1, confirming that a result close to the true value could be obtained. Figure 11(B) shows the true value (ground truth). In the above embodiment, super-resolution reconstruction was performed using a number of low-resolution encoded images equal to the number of conditional equations and unknowns. However, the reconstruction process may also be performed by reducing the number of low-resolution encoded images captured.

[0060] <Changes> The encoding imaging device of the present invention is not limited to those of the embodiments described above, and various modifications are possible. For example, it can also be applied to super-resolution technology using a single-pixel sensor instead of a two-dimensional sensor in a camera. Furthermore, the encoded imaging device of the present invention is widely effective for super-resolution using encoded patterns, and is also effective for resolution methods other than Lasso, such as CGI (Computer Ghost Imaging). [Explanation of Symbols]

[0061] 1. 1A Encoding Imaging Device 10, 10A Subject 11, 13 lenses 11A Lens 1 12 Optical modulators 12A DMD 13A Lens 2 14 Imaging Unit 14A Camera 15 Reconstruction Processing Unit 15A Low-noise processing mechanism 15B Reconstruction calculation mechanism

Claims

1. An optical modulator having the function of displaying a predetermined coding pattern and generating a coding pattern for the image of a subject, A lens that converges the encoding pattern of the image of the subject so that the encoding pattern of the image of the subject becomes less than or equal to the pixel size of the image sensor, An imaging unit equipped with an image sensor that captures the encoded pattern of the image of the subject, A reconstruction processing unit includes a low-noise processing mechanism that reduces noise generated in the pixel area masked for pattern formation when capturing the encoded pattern, and a reconstruction calculation mechanism that reconstructs the low-resolution encoded image acquired by the imaging unit into a high-resolution image. Equipped with, The reconstruction processing unit is configured to perform the process of reconstructing the low-resolution encoded image into the high-resolution image using the reconstruction calculation mechanism, and then to perform the process of reducing the noise using the low-noise processing mechanism. The encoding and imaging device is configured to capture a low-resolution image Y ON when all pixels in the optical modulator are lit, by displaying a pattern with all pixels turned ON on the optical modulator and capturing this pattern with the imaging unit, and to divide the pixel area of ​​the optical modulator into blocks of a predetermined number of pixels, and sequentially display a pattern with a predetermined number of pixels turned ON in each block on the optical modulator, thereby capturing a low-resolution encoded image for each pattern.

2. The encoding imaging apparatus according to claim 1, characterized in that light carrying image information of the subject is irradiated onto the optical modulator which displays the predetermined encoding pattern, and the light emitted from the optical modulator, carrying the encoding pattern information of the image of the subject, is imaged by the imaging unit.

3. The encoding imaging device according to claim 1, characterized in that light carrying predetermined encoding pattern information from the optical modulator displaying the predetermined encoding pattern is irradiated onto the subject, and light carrying the encoding pattern information of the image of the subject, emitted from the subject, is captured by the imaging unit.

Images